Infinia Voice Benchmark

Production-quality multilingual TTS benchmark for English, Arabic, and Hindi. Evaluates open-source TTS models and measures latency, RTF, WER, and speaker similarity.

---

Models

---

Installation

Requires Python 3.10. Create a venv first:

py -3.10 -m venv C:\venvs\voicebench310
C:\venvs\voicebench310\Scripts\Activate.ps1

Install dependencies in order:

pip install torch==2.2.2 torchaudio==2.2.2
pip install coqui-tts==0.24.3
pip install "librosa==0.10.2" "numba==0.60.0" "llvmlite==0.43.0"
pip install "click==8.1.7" "typer==0.12.3"
pip install faster-whisper==1.0.3 ctranslate2==4.4.0
pip install speechbrain==1.1.0 jiwer==3.0.4 soundfile==0.12.1
pip install "transformers==4.43.4" "tokenizers==0.19.1" "pandas==1.5.3"

---

Usage

Run full benchmark (all three models, all three languages)

python -m voicebench.cli benchmark `
    --models xtts --models mms_arabic --models mms_hindi `
    --languages en --languages ar --languages hi

With voice cloning reference

python -m voicebench.cli benchmark `
    --models xtts --models mms_arabic --models mms_hindi `
    --languages en --languages ar --languages hi `
    --speaker-wav assets/reference.wav

English only (XTTS-v2)

python -m voicebench.cli benchmark --models xtts --languages en

Skip WER (faster run, no Whisper download needed)

python -m voicebench.cli benchmark --models xtts --languages en --skip-wer

Regenerate comparison tables from existing results

python -m voicebench.cli compare

---

Metrics

All values are real measurements. Nothing is estimated or mocked.

---

Outputs

outputs/
├── audio/                           # Generated WAV files
├── metrics/
│   └── results.csv                  # Per-trial benchmark data
└── tables/
    ├── model_comparison_summary.csv # Averaged by model + language
    ├── en_ranked_by_wer.csv
    ├── en_ranked_by_latency.csv
    ├── en_ranked_by_rtf.csv
    ├── ar_ranked_by_wer.csv
    ├── ar_ranked_by_latency.csv
    ├── ar_ranked_by_rtf.csv
    ├── hi_ranked_by_wer.csv
    ├── hi_ranked_by_latency.csv
    └── hi_ranked_by_rtf.csv

---

Project Structure

infinia-voice-benchmark/
├── voicebench/
│   ├── __init__.py    — package version
│   ├── cli.py         — Typer CLI (benchmark / compare / report)
│   ├── config.py      — test texts (en/ar/hi), output paths
│   ├── pipeline.py    — benchmark orchestration, CSV/table export
│   ├── benchmark.py   — single trial runner, timing, WER, speaker similarity
│   ├── metrics.py     — RTF, WER, cosine similarity calculations
│   └── models.py      — TTSModel protocol, XTTS-v2, MMS-TTS Arabic, MMS-TTS Hindi
├── outputs/           — benchmark results (git-ignored)
├── assets/            — reference speaker WAV for voice cloning
├── requirements.txt
├── pyproject.toml
└── README.md

---

Known Limitations

• Latency on CPU: XTTS-v2 runs at 20–50s on CPU. On GPU (RTX 3060+) it runs at 1–3s. All benchmark numbers in this repo are CPU-only.
• WER evaluation for Arabic/Hindi: Faster-Whisper small is not well-calibrated for these languages. Reported WER overestimates actual intelligibility failure. Use whisper-large-v3 or IndicWhisper for accurate Arabic/Hindi WER.
• MMS-TTS has no voice cloning: MMS models always use their built-in voice regardless of --speaker-wav.
• Hindi production upgrade: AI4Bharat Indic-TTS is the recommended production model for Hindi but was not benchmarked here due to environment setup complexity.