TTS Voice Design Benchmark

🏆 Leaderboard | 🛠️ Evaluation Suite

TTS Voice Design is a high-quality benchmark of 1,000 character voice-design tasks spanning a broad range of media genres and real-world creative use cases. It evaluates whether a text-to-speech model can turn an open-ended character profile into a distinctive, appropriate, and usable voice.

Unlike benchmarks built around a fixed set of speakers or isolated acoustic attributes, this dataset covers complete character concepts: vocal identity, age and gender presentation, accent, timbre, pitch, speaking behavior, personality, and performance style.

Dataset At A Glance

• 1,000 English voice-design tasks
• 7 broad media-coverage categories
• 29 media types
• 1 self-contained voice-design prompt per task
• 1 natural preview transcript per task
• Structured voice-profile metadata for analysis
• Anonymized character identities and source references

Data Curation

The source pool is CharacterCodex, a collection of approximately 15,000 popular characters from games, film and television, animation, literature, mythology, web media, and factual or historical sources.

We select 1,000 tasks using a diversity-aware curation process that:

1. Balances coverage across media genres and source types.
2. Maximizes variation in perceived age, gender presentation, accent, pitch, speaking rate, energy, timbre, personality, and vocal style.
3. Removes low-quality cases such as multi-speaker scenes, non-speech content, and scripts containing unreadable or unpronounceable text.
4. Rewrites each selected character into a self-contained voice-design prompt and a natural preview transcript.
5. Removes character names, franchise references, and other identifying details.

The anonymization step reduces IP-related leakage and makes the task about realizing the described role rather than recognizing or reproducing a known character.

Media Coverage

Voice-Profile Coverage

The benchmark contains a broad range of human, non-human, and synthetic voice concepts. The profile metadata is intended for analysis and stratified evaluation; the voice_design_prompt remains the complete model input specification.

Accent coverage includes North American, British and Irish, Continental European, Latin American and Caribbean, East Asian, Oceanian, African, South Asian, Middle Eastern, Southeast Asian, and fictional accents, as well as cases where no accent is specified.

Frequently represented timbres include clear, breathy, rough, resonant, smooth, warm, gravelly, raspy, bright, and airy voices. Style labels span calm and reflective, dramatic and theatrical, authoritative, warm and nurturing, energetic, intimate, melancholic, scholarly, comedic, heroic, formal, eccentric, and sinister performances.

Data Format

Each line in the JSONL file is one task:

{
  "id": "tts-design-0001",
  "language": "en",
  "voice_design_prompt": "Male, middle-aged, European Portuguese. Warm, resonant journalist. Rhythmic, melodic cadence, moderate pace, precise articulation. Inquisitive, respectful, conveys curiosity and community empathy.",
  "transcript": "Tell me, when the music starts and the square fills with families, what do you feel this celebration says about your neighborhood, and about the pride people carry here every day?",
  "coverage_category": "Factual, News and Research Sources",
  "media_type": "Newspapers",
  "voice_profile": {
    "gender": "masculine",
    "age": "adult",
    "accent_region": "continental_european",
    "accent_raw": "European Portuguese",
    "speech_rate": "medium",
    "pitch": "mid",
    "energy": "mid",
    "timbre_tags": ["warm", "resonant", "clear"],
    "style_tags": ["formal_professional", "calm_reflective"],
    "humanity": "human"
  }
}

Field Reference

Running The Benchmark

For every record:

1. Provide voice_design_prompt as the voice-design instruction.
2. Synthesize the exact transcript.
3. Save one audio file named after the task ID, for example tts-design-0001.wav.

Do not use coverage_category, media_type, or voice_profile as additional model inputs. They are provided for analysis and visualization.

Evaluation

The benchmark reports three complementary metrics:

• Role Fit measures how convincingly the generated voice and performance realize the complete role described by the prompt.
• Transcript Pass Rate measures whether the intended preview text is spoken reliably.
• Voice Diversity measures the breadth of generated speaker identities using WavLM embeddings and average-linkage agglomerative clustering.

The self-contained judge prompt, evaluation script, WavLM embedding code, and metric documentation are available in the evaluation suite.