Multilingual TTS Benchmarking Harness

This repository contains the benchmarking harness, evaluation scripts, and empirical results for English, Arabic, and Hindi text-to-speech pipelines using open-source engines (XTTS-v2, F5-TTS, MMS-TTS, Indic-Parler-TTS, Kokoro, Chatterbox, OpenVoice).

A complete landscape survey and deep-dive analysis are available in the Detailed Technical Report.

---

Results Summary

All benchmarks were run on an NVIDIA Tesla T4 GPU (15 GB VRAM). Runtime Environment: PyTorch 2.6.0 | CUDA 12.4 | Python 3.10.15

Comparative Model Evaluation Results

All benchmarks were run on an NVIDIA Tesla T4 GPU (15 GB VRAM). Values meeting target thresholds are highlighted in bold.

Note on MOS: UTMOS is an automated neural MOS model trained on English. While UTMOS scores for Arabic and Hindi drop below 4.00 due to English-only training bias, human listening checks confirm that XTTS-v2 sounds natural (>= 4.40) in both languages.

---

Technical Approach & Winner Rationale

We evaluated three distinct architectural approaches for each language:

1. Autoregressive Voice Cloning (GPT + Vocoder): XTTS-v2, Bark, Chatterbox.
2. Non-Autoregressive Flow-Matching / Diffusion: F5-TTS.
3. Feed-Forward / VITS or Preset Models: Kokoro, MMS-TTS, OpenVoice.

Winner: Coqui XTTS-v2

• Why it won: It is the only open-source model that successfully performs cross-lingual voice cloning (cloning the English speaker prompt into Arabic and Hindi) while keeping pronunciation accurate (WER < 10%).
• Latency Mitigation: While batch generation is slow (often > 2.5s), its streaming implementation reduces Time-to-First-Audio (TTFA) to 359 ms - 460 ms, satisfying real-time application constraints.
• Fallback Options: If zero-shot cloning is not required, Kokoro (English and Hindi) and MMS-TTS (Arabic) are much faster alternatives (RTF < 0.05).

---

Setup & Execution

Installation

git clone https://github.com/Shushant-k1/voice.git
cd voice
pip install -r requirements.txt

Running the Pipelines

Generates speech waveforms and outputs basic timing metrics:

python src/pipeline_english.py
python src/pipeline_arabic.py
python src/pipeline_hindi.py

Running Streaming Benchmark

Measures the Time-to-First-Audio (TTFA) for XTTS-v2 streaming:

python src/benchmark_streaming.py

Running Evaluation Scripts

Calculates WER (via Whisper large-v3) and speaker similarity (via Resemblyzer d-vector distance):

python src/evaluate.py
python src/mos_evaluation.py

Compiling Results

python src/compile_results.py

---

Repository Structure

• src/: Core Python pipeline and evaluation scripts.
  • pipeline_english.py, pipeline_arabic.py, pipeline_hindi.py: Main batch pipeline entries (each runs 3+ distinct models).
  • benchmark_streaming.py: XTTS-v2 streaming benchmark.
  • evaluate.py: ASR/WER and Similarity evaluation.
  • utils.py: PyTorch 2.6 patching and GPU cleanup helpers.
• results/: Consolidated CSV and Markdown tables.
• audio/: Reference voice files and all generated samples.