Multilingual-TTS-language

malaysia-ai/Multilingual-TTS with two extra columns:

All original columns and the file/folder layout are preserved: 1552 subsets / 1561 parquet files, 121,842,371 rows (August 2026). Every subset of the parent dataset is exposed here as its own config:

from datasets import load_dataset

ds = load_dataset('malaysia-ai/Multilingual-TTS-language', 'assamese-tts-train', split='train')
ds[0]
# {'audio_filename': 'assamese-tts-train_audio/...-1ec6e7c0c9041b77_0.mp3',
#  'text': "মই বজাৰৰ অৱস্থাক লৈ চিন্তাত পৰিছোঁ। পৰিস্থিতি অধিক বেয়া হ'লে কি হ'ব?",
#  'speaker': 'assamese-tts-train_audio_0',
#  'language': 'asm_Beng',
#  'post-normalized': "মই বজাৰৰ অৱস্থাক লৈ চিন্তাত পৰিছোঁ. পৰিস্থিতি অধিক বেয়া হ'লে কি হ'ব?"}

language

• Detector: GlotLID v3 (fastText, 2102 language+script labels), chosen over lid.176 for its low-resource coverage (Yoruba, Hausa, Igbo, Bambara, Kabyle, dialectal Arabic, all Indic scripts, ...).
• Label format: ISO 639-3 + script, e.g. yor_Latn, zsm_Latn, arb_Arab, npi_Deva. Rows whose text has no letters (digits/punctuation only) are labeled und.
• Before detection, transcripts were cleaned (alignment markers like <um>/[noise] stripped); the stored text column is the original, unmodified.

Caveats

• Language identification on very short utterances (1–3 words) is unreliable for any model.
• Closely related pairs (Malay zsm vs Indonesian ind, bos/hrv/srp, Arabic dialects) can be confused at the row level; aggregate per-subset before drawing conclusions.

post-normalized

Produced by postnormalizer.py — a pure-Python, stdlib-only rule engine driven by Unicode script detection (no per-language config, no model, no network). It reproduces the deterministic half of an LLM punctuation + capitalization restoration pass:

• strips alignment markers (%, $) while keeping genuine 50% / $100;
• drops disallowed symbols (emojis, quotes, brackets, tone bars, word-attached Quranic annotation signs) and empty annotation groups (<?>, [?]);
• folds script-specific punctuation onto the canonical set (Arabic ،؛؟, Devanagari danda ।, Ethiopic, Armenian, Myanmar → , ; ? .), resolves …/... by context;
• normalizes whitespace, joins Han/Kana runs (Hangul keeps its spaces), un-glues word,word;
• capitalizes sentence starts for cased scripts (Latin/Cyrillic/Greek/Armenian) with guards for initials and abbreviation chains (P.U., p.m., m.in.);
• appends a sentence-final mark (。 for Han/Kana, . otherwise) when one is missing.

Caveats

• It is deliberately not semantic: it never inserts mid-sentence commas and never infers that an utterance is a question. Rules cannot do that reliably across 100+ languages, so those edits are left to the raw text → LLM path.
• Rows with no detectable script (phone numbers, IDs) and short unpunctuated fragments in Devanagari / Cyrillic / Arabic are left unterminated, matching the LLM reference behaviour.
• Rules were derived from and validated against Scicom-intl/Normalized-Multilingual-TTS (text → postprocessed_text, Qwen2.5-72B).