TTS Voice Direction Benchmark

🏆 Leaderboard | 🛠️ Evaluation Suite

TTS Voice Direction is a benchmark of 700 reference-conditioned speech generation tasks. It evaluates whether a text-to-speech model can preserve a reference speaker while following a natural-language direction that controls how a new transcript is performed.

The benchmark emphasizes practical voice acting beyond basic emotion control. It covers accent, acoustic delivery, vocal events, emotion, physiological state, communicative intent, role performance, multi-attribute composition, and temporal variation.

Dataset At A Glance

• 700 English voice-direction tasks
• 25 synthetic reference voices
• 9 fine-grained direction categories
• 3 macro categories
• One reference audio, reference transcript, direction, and target transcript per task
• Four-digit output IDs from 0001 through 0700

Taxonomy

Foundational cases target directly controllable speech properties. Situational cases describe a high-level state, purpose, or role and require the model to realize an appropriate performance. Complex cases combine multiple controls or request a perceptible change over time.

Reference Voices

All 25 reference voices are synthetic and sourced from the BreezeBlue Voice Library. Reference files are stored in reference_audio/ and named voice-01.wav through voice-25.wav.

The benchmark rotates directions across the reference set so that each model is tested on both instruction following and speaker preservation. Reference voice identity is evaluated separately from direction following.

Data Format

Each line in voice_direction_dataset.jsonl is one task:

{
  "description_id": "0001",
  "language": "en",
  "macro_category": "foundational_speech_control",
  "category": "accent",
  "voice_id": "voice-15",
  "ref_audio_path": "reference_audio/voice-15.wav",
  "ref_audio_text": "Reference transcript...",
  "description": "Use a moderate, consistent General American English accent throughout the line.",
  "transcript": "Target transcript..."
}

Running The Benchmark

For every record:

1. Condition the model on ref_audio_path and, when required by the model, ref_audio_text.
2. Use description as the voice-direction instruction.
3. Synthesize the exact transcript.
4. Save one audio file named after description_id, for example 0001.wav.

Do not use macro_category or category as additional model inputs. They are provided only for evaluation and analysis.

Evaluation

The benchmark reports two complementary metrics:

• Voice Direction Score (VDS) is a holistic 1-5 judge score for audible fulfillment of the requested direction, requested degree or timing, semantic preservation, and coherent execution.
• Speaker Similarity (SIM) is cosine similarity between WavLM-Large ECAPA embeddings of the generated and reference audio.

Both metrics are aggregated in the same hierarchy: reference voices are macro-averaged within each fine-grained category, fine-grained categories are equally averaged within each macro category, and the three macro-category scores are equally averaged into the overall score.

The self-contained judge prompts, evaluation scripts, speaker-embedding code, and metric documentation are available in the evaluation suite.