Raon-OpenTTS-Eval

Technical Report

A robustness-oriented evaluation benchmark for zero-shot text-to-speech, covering 4 acoustic regimes (Clean, Noisy, Wild, Expressive) across 12 datasets with 6,000 prompt–text pairs.

Existing zero-shot TTS benchmarks typically evaluate models using prompts drawn from a single read-speech dataset, providing an incomplete view of robustness under realistic and challenging recording scenarios. Raon-OpenTTS-Eval addresses this by sampling prompts from diverse real-world conditions, enabling systematic analysis of TTS robustness across controlled, noisy, conversational, and expressive speech.

Dataset Structure

Raon-OpenTTS-Eval/
├── clean/
│   ├── metadata.csv      # 2,500 pairs
│   └── audio/            # reference (prompt) WAV files
├── noisy/
│   ├── metadata.csv      # 1,000 pairs
│   └── audio/
├── wild/
│   ├── metadata.csv      # 1,000 pairs
│   └── audio/
└── expressive/
    ├── metadata.csv      # 1,500 pairs
    └── audio/

Each metadata.csv has the following columns:

Construction

For each source dataset, 500 utterances are selected as speech prompts via stratified sampling by speaker metadata (emotion, dialect, speaking style) to ensure representative coverage. Each prompt is paired with a target text drawn from a disjoint utterance in the same dataset, resulting in cross-sentence pairs.

For AMI-SDM, a substantial number of segments contain noisy or misaligned transcriptions due to distant microphone recording conditions. To ensure reliable evaluation, only segments with zero WER (as estimated by Whisper) are retained before sampling, filtering out samples with severe transcription mismatches.

Quick Start: Evaluation

1. Install dependencies

pip install faster-whisper whisper-normalizer jiwer torchaudio soundfile torch

2. Download the WavLM speaker verification checkpoint

The SIM metric uses an ECAPA-TDNN model with WavLM-large features, finetuned for speaker verification. Download the checkpoint from UniSpeech:

# Direct download
wget https://github.com/microsoft/UniSpeech/releases/download/v1.0.0/wavlm_large_finetune.pth

ecapa_tdnn.py (included in this repository) must be in the same directory as eval_raon_tts.py when running evaluation.

3. Generate audio

For each row in metadata.csv, synthesize gen_text conditioned on the prompt audio at ref_audio. Save the output as {gen_id}.wav in a flat directory.

for row in metadata:
    wav = your_tts_model.synthesize(
        text=row["gen_text"],
        prompt_audio=f"{split_dir}/{row['ref_audio']}",
        prompt_text=row["ref_text"],
    )
    save_wav(wav, f"{output_dir}/{row['gen_id']}.wav")

4. Run evaluation

python eval_raon_tts.py \
    --gen_dir /path/to/generated_wavs \
    --dataset_dir /path/to/Raon-OpenTTS-Eval \
    --wavlm_ckpt /path/to/wavlm_large_finetune.pth

--gen_dir accepts two layouts:

Split names recognized: clean / raon-clean, noisy / raon-noisy, wild / raon-wild, expressive / raon-emo.

5. Output

RESULTS SUMMARY
==================================================
  clean        WER=0.0199  SIM=0.6793
  noisy        WER=0.0341  SIM=0.6969
  wild         WER=0.0641  SIM=0.6017
  expressive   WER=0.0117  SIM=0.6020
  overall      WER=0.0300  SIM=0.6505
==================================================
Results saved to: /path/to/generated_wavs/raon_eval_results.json

Metrics:

• WER — Word Error Rate computed by transcribing generated audio with Whisper-large-v3 and normalizing via EnglishTextNormalizer (avoids penalizing surface-form variants such as numeric expressions or hyphenated compounds)
• SIM — Cosine speaker similarity between generated and prompt audio using WavLM-large finetuned for speaker verification

Baseline Results

Zero-shot TTS models evaluated under the protocol above. WER (%) via Whisper-large-v3 (normalized); SIM via WavLM-large. Overall is computed over all evaluation samples across the four categories. Bold marks the best result and the Raon-OpenTTS rows. Numbers are from the technical report.

Splits

CLEAN (2,500 pairs)

Controlled read speech from studio and clean recording conditions.

NOISY (1,000 pairs)

Read and prompted speech in the presence of background noise or reverberation.

WILD (1,000 pairs)

Unscripted conversational speech from real-world meetings captured under natural conditions. AMI-SDM samples are filtered to WER=0 to ensure transcription reliability.

EXPRESSIVE (1,500 pairs)

Expressive speech covering a wide range of emotions and prosodic styles.

Licenses

This dataset is a compilation of audio excerpts from multiple sources, each retaining its original license. The overall dataset is released under CC BY-NC-ND 4.0 (the most restrictive license among the included sources). See the per-source table above for individual licenses.

Citation

@article{kim2026raonopentts,
  title     = {Raon-OpenTTS: Open Models and Data for Robust Text-to-Speech},
  author    = {Kim, Semin and Chung, Seungjun and Moon, Taehong and Lee, Sangheon and Ahn, Minyoung and Lee, Keon and Kim, Nam Soo and Cho, Jaewoong and Schmidt, Ludwig and Lee, Kangwook and Park, Dongmin},
  journal   = {arXiv preprint arXiv:2605.20830},
  year      = {2026},
  url       = {https://arxiv.org/abs/2605.20830}
}

© 2026 KRAFTON