oron-cleaner

Builds the strict Mongolian speech corpus that oron-tts finetunes F5TTS_v1_Base on.

Sources are merged into one corpus rather than published separately, because the consumer is a single training run and the per-speaker cap and the speaker-disjoint split only make sense across the whole thing.

Changing anything here? Read AGENTS.md first. It lists the four failure modes that do not raise, and the order-of-operations constraints that look arbitrary until they break something downstream.

FLEURS' schema is id, num_samples, path, audio, transcription, raw_transcription, gender, lang_id, language, lang_group_id — there is no speaker field, and id indexes the sentence. Its clips therefore go wholly to training: a split cannot be shown disjoint from a voice it cannot name. Its gender is a ClassLabel, so a row yields 0/1/2 and needs decoding before it means anything.

All commercially usable.

disco-eth/WorldSpeech mn_mn is by far the largest Mongolian corpus — ~221 h, 24 kHz native, the only source with real full-band content — but it is CC-BY-NC-4.0. It is excluded from all and needs an explicit double opt-in, because including it makes the trained model non-commercial:

python clean_pipeline.py --datasets cv,ws --allow-non-commercial

Run it

pip install -e ../oron-tts                     # shared text normalisation
pip install -r requirements.lock               # pinned; -e . for the loose ranges
pip install -e . --no-deps

python clean_pipeline.py                       # all sources, then upload
python clean_pipeline.py --datasets cv --no-upload
python clean_pipeline.py --finalize-only       # re-split without refiltering

API_KEY (Mozilla Data Collective) and HF_TOKEN load from .env.

The Common Voice dataset id is release-specific and goes stale. Each language of each release is its own dataset, and the API has no list or search endpoint, so when _DATASET_ID stops resolving take the current one from the dataset page URL. Downloads also require the account to have accepted that dataset's terms in the web UI, and are capped at 30/day per organisation.

Calibrate before the full run

Every threshold below was set from published figures, small samples, or reasoning about what a strict corpus needs — none against this corpus's own distribution. A threshold 10% too strict silently discards hours of usable audio while looking like it worked, and the full pass is 24–48 h.

python clean_pipeline.py --datasets cv --calibrate --limit 500 --no-upload

Calibration mode scores every gate instead of stopping at the first failure. That matters: in a normal run a clip rejected for SNR is never scored for DNSMOS, so the DNSMOS rate is only the rate among clips that already passed SNR, and the gates cannot be compared. The report gives independent per-gate rejection rates, the full distribution of each metric, what the current threshold keeps, and the threshold that would hit a target yield.

Then edit pipeline/constants.py and re-run without --calibrate — the policy hash invalidates cached work automatically.

Output

output/oron_mn_strict/
  wavs/<clip_id>.wav        24 kHz mono, edge-trimmed, peak-normalised
  metadata.csv              audio_file|text — F5-TTS prepare_csv_wavs.py contract
  metadata_validation.csv   likewise
  metadata_test.csv         likewise — speaker-disjoint, for reference prompts
  metadata_withheld.csv     clips whose sentence is held out; trained on by nothing
  eval_sentences.txt        400 sentences no training clip contains — the CER text
  manifest.parquet          per-clip metrics and speaker metadata
  manifest.jsonl            the same rows, plus split and gender_resolved
  corpus_summary.txt        hours, speakers, acceptance criteria, text
                            diversity, and CER by source
  provenance.json           pinned revisions, package versions, content hash

Gates

Thresholds live in pipeline/constants.py and are hashed into FILTER_POLICY_VERSION, so changing one invalidates cached work instead of silently mixing policies.

That hash covers the thresholds and nothing else — not the ASR, aligner, VAD or DNSMOS revisions, not the source dataset revisions, and not oron_tts.text, whose every change rewrites every published transcript. provenance.json records all of those plus a content hash of the corpus, and the model and dataset revisions are pinned to commit SHAs in pipeline/provenance.py so a rebuild is a rebuild rather than a re-derivation.

Why forced alignment rather than ASR

Free-running ASR is the wrong instrument in Mongolian. The best available model has a CER floor of 0.123 on clean, correctly-transcribed speech (whisper-large-v3 is 0.311), so an absolute CER threshold sits near the recogniser's own error — it rejects good clips and admits bad ones.

Forced alignment is constrained to the transcript it is given, so a low score is evidence the audio does not contain those words. Measured on real Mongolian audio, scoring each clip against its own transcript and against another's:

Clean separation on both. The threshold sits at 0.65 and is biased toward rejecting: a mismatched clip teaches a wrong text-to-audio mapping, a rejected good clip only costs data.

Two holdouts, not one

A speaker-disjoint split is not a text-disjoint one, and conflating them makes both useless. Common Voice mn repeats each of its 6,062 sentences 4.76 times, so a sentence read by an evaluation speaker was almost certainly read by a training speaker too — measured under an earlier split, 99.6% of test clips had their text in train.

They are held out separately because they answer different questions:

Requiring one clip to satisfy both intersects the two, which on the measured corpus shape leaves 47 clips in test — not enough for one reference prompt per gender. Kept apart, test keeps ~2 h across ~100 speakers and there are 400 genuinely unseen sentences, at a cost of ~1.5% of training hours.

Gender

F5-TTS takes voice identity from a reference clip, not a token, so the corpus needs enough clean speech of each gender and a way to find the best candidate.

• Common Voice ≥v17 emits male_masculine / female_feminine.
• 39.7% of clips carry no label, but there are only 520 speakers — so a label on any one of a speaker's clips settles all of them.
• Where a speaker declared nothing anywhere, gender is inferred from median F0, calibrated on 80 self-declared clips: male 70–150 Hz, female 173–306 Hz, zero overlap. The 155–170 Hz dead band is left unknown rather than guessed.

Values describing identity rather than vocal tract (non_binary, intersex, transgender, do_not_wish_to_say) map to unknown and are never inferred.

Architecture

pipeline/dsp.py and pipeline/speakers.py hold pure functions over arrays and record dicts — no models, no IO — so the logic that decides what enters the corpus is testable without a GPU or a 3 GB download. pipeline/audio_filter.py is the only module that loads models.

Clips are written to disk the moment they pass. They used to accumulate in a list with their decoded audio and then be copied again, which is roughly 14 GB at Common Voice scale. Resume is keyed by clip id, so a restart re-does no work regardless of dataset ordering.

Tests

pytest                       # no models required
RUN_SLOW_TESTS=1 pytest      # adds the alignment test (downloads 1.18 GB)