Freya-TR-Eval — A General-Purpose Turkish TTS Evaluation Benchmark

Freya-TR-Eval is a compact, high-quality, reproducible benchmark of 495 natural conversational Turkish sentences for evaluating text-to-speech (TTS) systems on the everyday speech that conversational voice products actually synthesize. It is deliberately domain-neutral: it contains no bank-, call-center-, or product-specific content and no customer terms, so scores reflect general Turkish speech quality rather than familiarity with a narrow vertical.

Why another benchmark?

Existing Turkish TTS evaluation reuses ASR test splits (FLEURS, Common Voice, CoVoST2). Those skew toward long, encyclopedic sentences (news / wiki register) that over-weight long-horizon behavior and under-represent everyday conversational speech; and the standard Whisper-WER protocol confounds true intelligibility with Whisper's own Turkish error rate, a narrowband penalty against telephony-band systems, and digit-normalization artifacts. Freya-TR-Eval curates a length- and register-balanced everyday set and pairs it with a semantic metric that neutralizes those confounds.

Composition

• 495 sentences, everyday conversational register (statements, mI-questions, exclamations), 3–13 words (mean ~8).
• Built from two sources, merged and de-duplicated: (a) native Turkish sentence texts sampled from Common Voice 17-tr and CoVoST2-tr (short, natural); (b) everyday-conversational lines generated by gemini-3.1-pro-preview under fixed everyday-topic and sentence-type quotas (22 topics), then filtered for length, full Turkish grapheme coverage (ç ğ ı i ö ş ü), and duplication.
• No numbers-in-isolation, no brands, no bank/customer terms (verified by an automated leak scan).

Fields

{"id": "0007", "text": "Yarın öğleden sonra sağanak bekleniyormuş, şemsiyeni unutma.",
 "register": "everyday", "type": "decl", "topic": "weather_seasons", "n_words": 8}

register is everyday (generated conversational) or short-native (sampled read text).

Recommended scoring protocol

Apply identically to every system:

1. Semantic WER (sWER) — primary. Re-transcribe each synthesis with an ASR model, then score transcript-vs-reference with an LLM judge that counts only errors that change meaning (tolerant of Turkish morphology and number surface forms). Report sWER and correctness = 1 − min(sWER, 1). This measures what a cascaded voice agent depends on: whether the next-stage language model understands the utterance.
2. Band-matched WER — secondary. Downsample all systems to 8 kHz before ASR (removes the narrowband penalty), then standard Whisper WER/CER.
3. Naturalness. UTMOS or a human MOS (UTMOS is an English-trained proxy that penalizes narrowband audio; read cross-system gaps cautiously, and band-match before scoring for fairness).

The intended use is a like-for-like comparison within a weight class — e.g. open sub-1B Turkish TTS systems.

Reproducibility

The exact 495-sentence list is fixed and published here, together with the seeded build scripts (build_general_tr.py for the native split; generate.py / expand_eval.py + raw outputs for the generated split; verify.py for QA). Released under CC-BY-4.0. The native split derives from the texts of public CC-licensed corpora (Common Voice 17, CoVoST2) — text only, no source audio redistributed. The generated split is gemini-3.1-pro-preview output.

Citation

Freya-TR-Eval is introduced in the FreyaTTS technical report, arXiv:2607.09530. If you use this benchmark, please cite:

@misc{pamuk2026freyattstechnicalreport,
      title={FreyaTTS Technical Report}, 
      author={Ahmet Erdem Pamuk and Ömer Yentür and Ahmet Tunga Bayrak and Yavuz Alp Sencer Öztürk and Mustafa Yavuz},
      year={2026},
      eprint={2607.09530},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.09530}, 
}