Multilingual-TTS

Open-source multilingual TTS with Voice Cloning support for 150+ languages, built on Neucodec as the speech tokenizer at 50 TPS.

Models

Evaluation

TTS Evaluation

CER and MOS across 76 languages, compared against: Dia TTS, Orpheus, Chatterbox (23 languages), Fish Audio S2 Pro, Qwen3 TTS (11 languages).

Chatterbox covers 23 languages only; Qwen3 TTS covers 11 languages only. Their averages are not directly comparable to 76-language models.

VC Evaluation

Speaker similarity and CER across 76 languages, compared against: Dia TTS, Orpheus, Chatterbox (23 languages), Fish Audio S2 Pro.

Chatterbox covers 23 languages only; its averages are not directly comparable to 76-language models.

Dataset

Base

Sources

1. https://huggingface.co/datasets/malaysia-ai/Multilingual-TTS
2. https://huggingface.co/datasets/Scicom-intl/Emilia-YODAS-Voice-Conversion
3. https://huggingface.co/datasets/Scicom-intl/Malaysian-Emilia

Size

1. Multi-speaker multilingual Voice Cloning — up to 35.88B tokens
2. Multi-speaker multilingual TTS, 150+ languages — up to 25.35B tokens

Preparation: preparation

Expressive TTS

Natural-language descriptions of speech audio generated by computing acoustic statistics and categories, then prompting an LLM to summarise them.

Sources

1. https://huggingface.co/datasets/Scicom-intl/ExpressiveSpeech

Size

1. Multi-speaker multilingual Expressive TTS — up to 1.15B tokens

Pipeline

audio + transcript
      │
      ▼
speech_categories.py   ← categories (emotion, gender, fluency, accent, quality)
      │                   speech_stats_func  ← pitch, SNR, speaking rate, SQUIM
      ▼
calculate_bins.py      ← bin continuous stats, merge into a single parquet
      │
      ▼
synthetic.py           ← LLM generates a natural-language description per row

Uses Qwen/Qwen2.5-72B-Instruct via DeepInfra by default. Any OpenAI-compatible endpoint works — set API_KEY, BASE_URL, and MODEL_NAME.

Preparation: synthetic-description

Non-verbal Tags

Inline non-verbal event tags (laughter, cough, sigh, ...) mined from the Emilia-style corpora with PANNs SED + CLAP verification + whisper word-timestamp placement, in two renderings per row: Higgs-TTS style (<|sfx:laughter|>Haha) and Emilia-NV style ([Laughter]).

Sources → outputs

1. Malaysian-Emilia → Malaysian-Emilia-Nonverbal-Tags — 8,702 rows / 8,985 events
2. Malaysian-Tamil-Emilia → Malaysian-Tamil-Emilia-Nonverbal-Tags — 2,383 rows / 2,539 events
3. Malaysian-Chinese-Emilia → Malaysian-Chinese-Emilia-Nonverbal-Tags — 1,655 rows / 1,694 events

Preparation: nonverbal-tagging

Ablation

One Epoch

1. Global token size: 10240 × 256 × 8 GPUs ≈ 20,971,520 tokens
2. Warmup: 100 steps
3. FP32-BF16 mixed precision
4. Compared AdamW with WSD LR vs Muon + AdamW with WSD LR (decay = 10% of dataset)
5. Run on Qwen3 1.7B Base only
6. AdamW performed better

Hyperparameter Search

One-epoch results used conservative learning rates, so we ran a focused search:

1. Global token size: 10240 × 256 × 8 GPUs ≈ 20,971,520 tokens
2. 100 steps, warmup 50 steps
3. FP32-BF16 mixed precision
4. Run on Qwen3 1.7B Base only
5. Grid search over AdamW LR, Muon LR, and decay rate
6. Aggressive LRs (AdamW 1e-3, Muon 1e-2, decay 0.01) turned out best
7. AdamW-only with the same aggressive LR — 1.7B-adamw-aggresive.sh
8. Adding Muon performed better

Optimizer search

The original grid scripts only covered Muon+AdamW, so the harness was rebuilt around a pluggable-optimizer trainer: hyperparameter_search.py drives qwen3_optimizer_search.py and sweeps each optimizer over its own LR grid under the identical 100-step setup above.

Hybrid optimizers use the same 2D-hidden-weight/AdamW split as Muon ("Muon is Scalable for LLM Training", arXiv:2502.16982), so the comparison is apples-to-apples. Runs resume by name (search_state/<run>.json) and the harness ranks finished runs by mean train loss over the last 10 steps into search_state/summary.json.

pip install pytorch_optimizer   # needed for shampoo / soap / lion / ademamix

# everything, or a subset:
python hyperparameter_search.py --train-file <multipacking dir>
python hyperparameter_search.py --train-file <multipacking dir> --optimizers muon shampoo soap
python hyperparameter_search.py --train-file <multipacking dir> --dry-run   # print commands only

The train file must be a ChiniDataset multipacking directory (see preparation); custom grids go in --grid-json.

Training

Base

# 0.6B
bash 0.6B.sh

# 1.7B
bash 1.7B.sh

Expressive TTS

# 0.6B
bash 0.6B-expressive.sh

# 1.7B
bash 1.7B-expressive.sh

WandB

• Base experiments: https://wandb.ai/aies-scicom-scicom-ai/Multilingual-TTS
• Post TTS experiments: https://wandb.ai/aies-scicom-scicom-ai/Multilingual-TTS-Expressive

Optimize Inference

For better concurrency, you can use https://github.com/Scicom-AI-Enterprise-Organization/TTS-API-Neucodec

Acknowledgements

1. Malaysia-AI for the large-scale TTS dataset: https://huggingface.co/datasets/malaysia-ai/Multilingual-TTS
2. Scitix for H100 node access