Voice Mood AR

Arabic voice-to-mood labeling pipeline for building expressive TTS datasets.

voice-mood-ar turns Arabic audio files into ASR transcripts plus structured emotion, prosody, audio-quality, and TTS-usability labels.

Pipeline:

audio_data/*.wav, *.mp3, *.flac, ...
  -> Whisper Arabic ASR
  -> Arabic transcript + normalized transcript
  -> Gemini audio+transcript emotion/prosody labeling
  -> outputs/emotional_tts_dataset.jsonl

Models

ASR uses dev-ahmedhany/whisper-large-v3-turbo-arabic-ft by default. The model card positions this v1 checkpoint as best for Egyptian and Gulf Arabic, especially mixed-genre or phone-quality dialectal audio.

For MSA, consider --preset msa-base, which uses openai/whisper-large-v3-turbo.

Emotion/prosody labeling uses Gemini:

gemini-3-flash-preview

Setup

Create the environment:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Copy the environment example and add your Gemini key:

cp .env.example .env
nano .env

Expected .env format:

GEMINI_API_KEY=your-gemini-api-key-here

Model and pip caches are kept inside this project:

.cache/

Input Audio

Put audio files in:

audio_data/

Supported extensions:

.wav .flac .mp3 .m4a .ogg .opus .aac .wma

The default pipeline processes every supported audio file in audio_data/.

Run

Linux/macOS:

bash scripts/run_pipeline.sh

Windows PowerShell:

.\scripts\run_pipeline.ps1

Both scripts write:

outputs/asr_manifest.jsonl
outputs/emotional_tts_dataset.jsonl

Dry run without Gemini:

SKIP_GEMINI=1 bash scripts/run_pipeline.sh

PowerShell dry run:

.\scripts\run_pipeline.ps1 -SkipGemini

Direct Python

Run the complete folder pipeline:

python -m emotions_tts.complete_pipeline

Run ASR only:

python -m emotions_tts.asr_pipeline

Label an existing ASR manifest only:

python -m emotions_tts.gemini_labeler \
  --input outputs/asr_manifest.jsonl \
  --output outputs/emotional_tts_dataset.jsonl

Useful Options

Use the MSA Whisper baseline:

PRESET=msa-base bash scripts/run_pipeline.sh

Override dtype if needed:

DTYPE=bfloat16 bash scripts/run_pipeline.sh

Output Shape

Each JSONL row contains:

{
  "audio_path": "audio_data/sample.wav",
  "text_ar": "النص العربي المستخرج",
  "normalized_text_ar": "النص العربي بعد التطبيع",
  "asr": {
    "model_id": "dev-ahmedhany/whisper-large-v3-turbo-arabic-ft",
    "device": "cuda",
    "torch_dtype": "bfloat16"
  },
  "emotion_label": {
    "emotion_primary": "calm",
    "emotion_intensity": 0.5,
    "emotion_confidence": 0.95,
    "valence": "neutral",
    "arousal": "medium",
    "speaking_style": "formal",
    "prosody_tags": {
      "pitch": "medium",
      "energy": "medium",
      "pace": "medium",
      "pauses": "short",
      "stress": "medium",
      "clarity": "excellent"
    },
    "tts_training_quality": {
      "usable_for_tts": true,
      "recommended_split": "train"
    }
  }
}