EmergentTTS-Eval

Code accompanying the paper, "EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge"

Introduction

Text-to-Speech (TTS) benchmarks often fail to capture how well models handle nuanced and semantically complex text. We introduce EmergentTTS-Eval, a comprehensive benchmark covering six challenging TTS scenarios: emotions, paralinguistics, foreign words, syntactic complexity, complex pronunciation (e.g. URLs, formulas), and questions. Crucially, our framework automates both test-case generation and evaluation, making the benchmark easily extensible. Starting from a small set of human-written seed prompts, we iteratively extend them using LLMs to target specific structural, phonetic and prosodic challenges, resulting in 1,645 diverse test cases. Moreover, we employ a model-as-a-judge approach, using a Large Audio Language Model (LALM) to assess the speech across multiple dimensions such as expressed emotion, prosodic, intonational, and pronunciation accuracy. We evaluate state-of-the-art open-source and proprietary TTS systems, such as 11Labs, Deepgram, and OpenAI's 4o-mini-TTS, on EmergentTTS-Eval, demonstrating its ability to reveal fine-grained performance differences. Results show that the model-as-a-judge approach offers robust TTS assessment and a high correlation with human preferences.

Updates

• [8-10-25] EmergentTTS-Eval has been accepted for publication in the datasets and benchmark track at NeurIPS'25 🎉.
• [8-10-25] We have updated our leaderboard with the latest gemini-2.5-pro model, check note under the leaderboard section for more details.

Leaderboard

WER ↓ and Win-rate ↑ over all categories with gpt-4o-mini-tts-alloy as baseline and gemini-2.5-pro as judger; ⭐ indicates that the model was strongly prompted(see paper for definition). For some expensive to use models, we only do strong prompting.

[!NOTE] The generated audios for all of the models we evaluated, along with the predictions made by the gemini-2.5-pro judger are available in this Google Drive link.

[!NOTE] The original leaderboard was created using gemini-2.5-pro-preview-05-06 as judge, since the model has been depreciated, we have moved the leaderboard to LEADERBOARD_gemini-2.5-pro-05-06.md. The current leaderboard is with the non-preview model pointed at gemini-2.5-pro, thinking budget=256. Both leaderboards have a spearmann correlation of 0.98 at p-value <0.05 indicating strong positive ranking correlation.

General Instructions

• The accompanying code provides the pipeline for reproducing the evaluation results for the models evaluated in the paper, and evaluating your own model on our dataset.
• To evaluate your own model, implement the generate_audio_out and prepare_emergent_tts_sample methods. If you want to load your model locally on GPU, implement it in model_clients.py file otherwise if it's an API call, put it in api_clients.py. In either case, expose your client in the evaluation_runner.py file following the structure of other clients.
• We use accelerate for local inference, so if you implement in model_clients.py, ensure to place the model on accelerator.device.

Setup Instructions

First, clone the repo, create a virtual environment, and install the requirements.

git clone https://github.com/boson-ai/EmergentTTS-Eval
cd EmergentTTS-Eval
python3 -m venv .test_env
source .test_env/bin/activate
pip3 install -r requirements.txt

Then, download all of the relevant data using download_data.py, this processes the baseline audio files, creates a jsonl structure for the data, and downloads the wv_mos checkpoint for MOS calculation.

python3 download_data.py

[!NOTE] If you are evaluating your own model or a new model from another provider, install any missing requirements for that model along with already installed requirements.txt, as the current requirements only cover the ones necessary for the benchmark and judger sdk's - google-generativeai and openai.

Evaluation Instructions

Inference for API-served models

For models in api_clients.py, we call the api with multiple threads for faster inference, the MOS model and WhisperV3 for transcription still needs to be placed in the GPU for this.

export <>_API_KEY=<key for model to evaluate, for openai models it's OPENAI_API_KEY, for deepgram, it's DEEPGRAM_API_KEY, for custom added api_client, follow the variable name you implement in api_clients.py> 
export JUDGER_API_KEY=<api key for the judger model, either gemini or openai model>
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python3 -u evaluation_runner.py \
--model_name_or_path "gpt-4o-mini-tts" \
--output_dir "<path_to_output_dir, absolute path is recommended>" \
--seed 42 \
--judge_model_provider "gemini-2.5-pro" \
--api_num_threads 14 \
--temperature 1.0 \
--text_speech_strong_prompting \
--tts_judger_evaluate_function "win_rate" \
--baseline_audios_path <path/to/EmergentTTS-Eval/data/baseline_audios, these will be stored when you run download_data.py>

Local Inference with Accelerate

If the model is in model_clients.py, evaluation will be run with accelerate. For example, to reproduce Orpheus-TTS results, install it's specific requirements(snac is the only missing requirement for orpheus) and use the command:

MODEL_NAME="orpheus-tts-0.1-finetune-prod"
export HF_TOKEN=<your_hf_token>
export JUDGER_API_KEY=<api key for the judger model, either gemini or openai model>
accelerate launch --config_file default_accelerate_config.yaml evaluation_runner.py \
--model_name_or_path "canopylabs/orpheus-tts-0.1-finetune-prod" \
--output_dir "<path_to_output_dir, absolute path is recommended>" \
--seed 42 \
--judge_model_provider "gemini-2.5-pro" \
--temperature 1.0 \
--tts_judger_evaluate_function "win_rate" \
--baseline_audios_path <path/to/EmergentTTS-Eval/data/baseline_audios, these will be stored when you run download_data.py>

Custom Judger Model

To use your own Judger Model, we recommend using vLLM to create an OpenAI compatible server. Next, you just have to pass the flag --judger_base_url pointing to the IP and port of the custom server.

Flags and Considerations

• For single GPU setup, edit the default_accelerate_config.yaml file and use gpu_ids: 0, and num_processes: 1.
• Use the --num_samples parameter to evaluate on only a subset of samples.
• Carefully tweak --api_num_threads and CUDA_VISIBLE_DEVICES for api client based inference in your setup. We place --api_num_threads instances of the MOS model and WhisperV3 uniformly in all visible gpus, so too high --api_num_threads can result in Cuda OOM.
• Omit passing the --tts_judger_evaluate_function if you do not want to use LALM judger to judge the output, and only calculate other metrics like WER, MOS, etc.
• Pass --text_speech_strong_prompting if the client supports strong_prompting, as described in the paper.
• To evaluate only on specific depths or categories, use the --depths_to_evaluate and --categories_to_evaluate flags.
• To use a specific voice of the model, if applicable, use the --voice_to_use paramter.
• --baseline_audios_path needs to be passed if win_rate is calculated.
• To disable audio generation and fetch generated audios from a path, use the --fetch_audios_from_path paramter, this can be useful for testing different judger and not changing the generated audios.

If our work proves useful to you, cite it as:

@misc{manku2025emergentttseval,
    title={EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge},
    author={Ruskin Raj Manku and Yuzhi Tang and Xingjian Shi and Mu Li and Alex Smola},
    year={2025},
    eprint={2505.23009},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}