Tone-aware Vietnamese ASR Starter Kit

This repository contains the code scaffold for a Vietnamese noisy-ASR project: controlled VIVOS + MUSAN benchmark construction, Whisper/PhoWhisper inference, WER/CER scoring, Vietnamese diagnostic metrics, and a PhoWhisper LoRA + tone-aware multi-task learning training path.

The exact paper-v2 protocol, distributed runner assignments, hash handoff, resume commands, and stop conditions are documented in Guide.md.

The repository retains an archived midterm comparison of three systems on the same clean and noisy VIVOS manifests:

1. openai/whisper-base zero-shot
2. vinai/PhoWhisper-base zero-shot
3. vinai/PhoWhisper-base tone-aware LoRA trained on VIVOS with MUSAN noise augmentation

Those archived results use six metrics: WER, CER, TER, DER, FCER, and SWDR. They were derived from the official VIVOS test partition and must not be used as paper-v2 selection or final evidence.

What is included

configs/                 Training/evaluation config files
docs/                    Midterm runbook and experiment notes
latex/                   Paper skeleton
notebooks/               Result analysis notebook
scripts/                 Manifest, noise, inference, scoring, pipeline scripts
src/vitonesr/            Dataset, model, tone, noise, and metric utilities
evaluate.py              Evaluation entrypoint scaffold
train.py                 LoRA + tone-aware MTL training entrypoint
requirements.txt         Python dependencies

Large generated files are intentionally excluded from Git by default. The only exceptions are small, reviewed protocol manifests/audits under data/manifests/paper_v2/ and outputs/paper_v2/, plus the explicitly allow-listed final benchmark under Git LFS:

data/                    Downloaded VIVOS/MUSAN and generated audio (ignored by default)
data/derived/paper_v2/final_benchmark/
                         Sole audio allow-list: 2,300 final WAVs in Git LFS
outputs/                 Generated CSV/report/slide artifacts
experiments/             Fine-tuned checkpoints and training outputs

Keep raw data and checkpoint weights out of normal Git commits. Reviewed result tables, prediction CSVs, protocol locks, and portable provenance may be staged selectively; transfer checkpoints with the hash-verified handoff procedure in Guide.md (or a separately approved Git-LFS workflow).

The final benchmark is built once by the designated builder and consumed read-only by every inference machine. Its metadata is one JSONL file with exactly 2,300 rows: outputs/paper_v2/benchmark/final_benchmark_manifest.jsonl. The JSONL is not an audio container; the corresponding 2,300 WAV files remain under the LFS directory above. Do not rebuild the benchmark after it is published. Install Git LFS and run git lfs pull after cloning or pulling. See the exact publish, pull, and SHA verification commands in Guide.md, and review dataset licensing and attribution before making the LFS objects public.

Dataset links

Download datasets through the provided scripts or the official pages below. Do not commit these files to GitHub.

Expected local dataset layout after download:

data/raw/vivos/
data/raw/musan/musan/noise/
data/manifests/noise/musan_noise.jsonl  # legacy/unlocked, not formal Gate-2 input
data/manifests/paper_v2/vivos_train.jsonl
data/manifests/paper_v2/vivos_dev.jsonl
data/manifests/paper_v2/vivos_test_legacy_exposed.jsonl
data/manifests/paper_v2/vivos_test_locked.jsonl
outputs/paper_v2/protocol/split_lock.json
outputs/paper_v2/protocol/split_audit.csv
outputs/paper_v2/protocol/legacy_test_exposure.csv
data/manifests/fleurs/paper_v2/test.jsonl
data/manifests/fleurs/paper_v2/audio/test/*.wav
outputs/paper_v2/protocol/fleurs_test_lock.json

Environment setup

Recommended for training: Colab GPU or another CUDA machine. Local CPU is acceptable for manifest generation and small inference smoke tests.

python -m pip install -r requirements.txt
python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'cpu')"

On Colab, clone the repository, enable GPU runtime, then run the same commands:

git clone [email protected]:KietIT/SLP.git
cd SLP
python -m pip install -r requirements.txt

If SSH is not configured in Colab, use the HTTPS clone URL instead.

Paper-v2 VIVOS split protocol

The paper protocol splits the official VIVOS train partition by speaker. Of the 760 official-test utterances, 300 already appeared in the historical benchmark used for lambda comparison and are therefore recorded as vivos_test_legacy_exposed.jsonl. Only their 460-utterance complement is sealed as the unseen vivos_test_locked.jsonl; neither partition is a valid input for checkpoint or lambda selection. Paper-v2 training verifies the split-lock, manifest SHA-256, and every locked VIVOS train/dev audio SHA-256 before model/output creation. Setting protocol.final_test_unlocked: true is necessary but never sufficient to open the final test: inference also requires the reviewed decision-v3 lock and the matching final-benchmark lock. The decision lock names each permitted configuration and binds its method_id, train_type, lambda, seed, immutable backbone revision, checkpoint fingerprint, resolved-config hash, and training-contract hash. Anonymous checkpoint allow-lists are not accepted.

bash scripts/download_vivos.sh

python scripts/make_vivos_manifest.py \
  --vivos-root data/raw/vivos \
  --out-dir data/manifests/paper_v2 \
  --protocol-dir outputs/paper_v2/protocol \
  --legacy-benchmark-manifest outputs/benchmark/benchmark_manifest.csv \
  --expected-legacy-exposed 300 \
  --seed 42 \
  --dev-speaker-fraction 0.20

Running the command again verifies byte-identical outputs. A changed source, seed, or split configuration is refused unless --overwrite is explicit. The trainer requires train and dev manifests respectively, evaluates dev after each epoch, and saves the lowest-dev-ASR-loss checkpoint under best/. Checkpoint/lambda selection is dev-only; after the method, lambda, and exact checkpoint identities are locked, the evaluator may open only the 460 unseen final-test utterances through that reviewed authorization. The 300 exposed utterances remain historical/regression evidence and can never be relabeled as unseen final evidence.

Gate 1.2 alone does not authorize definitive training. The formal pipeline is fail-closed while the Gate-2 locked noise registry and derived noisy-dev benchmark lock are absent, and definitive runs begin only after Gates 1-3 are complete. The clean dev manifest may be used only for diagnostics, with outputs isolated under a smoke/diagnostic path; it is not the low-SNR selection screen.

Historical midterm artifacts

The statistics and full pipeline below describe the earlier coursework run. That run selected models on the official VIVOS test partition and therefore is not valid evidence for the paper-v2 protocol.

Dataset statistics from the historical midterm run:

Full midterm pipeline (archival only)

Do not run this command for paper-v2. It rebuilds an official-test-derived coursework pipeline and can only reproduce archival artifacts after the final-test boundary has already been crossed.

Archived invocation:

conda activate slp
python scripts/run_full_midterm_pipeline.py

The pipeline performs these steps:

1. Create VIVOS clean manifests.
2. Create the MUSAN noise manifest.
3. Create a fixed noisy VIVOS test set with SNR 20/10/5/0.
4. Run Whisper-base and PhoWhisper-base zero-shot inference on clean/noisy sets.
5. Train PhoWhisper tone-aware LoRA with configs/phowhisper_base_lora.yaml.
6. Evaluate the LoRA checkpoint on the same clean/noisy sets.
7. Score all predictions with six metrics.
8. Build the final comparison table.

Main outputs:

outputs/metrics_whisper_clean.csv
outputs/metrics_whisper_noisy_by_snr.csv
outputs/metrics_phowhisper_clean.csv
outputs/metrics_phowhisper_noisy_by_snr.csv
outputs/metrics_phowhisper_lora_clean.csv
outputs/metrics_phowhisper_lora_noisy_by_snr.csv
outputs/model_comparison_6metrics.csv
notebooks/midterm_results_analysis.ipynb

Historical robust benchmark and zero-shot baseline (archival only)

Do not build, run, aggregate, or select from this benchmark for paper-v2. It is derived from data/manifests/vivos/test.jsonl, the official VIVOS test partition. The commands below are retained only to reproduce coursework artifacts after test access; they are not a substitute for the Gate-2 noise-disjoint dev screen or locked final benchmark.

The archived larger benchmark pipeline keeps the midterm outputs untouched and writes artifacts under outputs/benchmark/, outputs/zero_shot/, and data/noisy_eval/.

Archived typed-MUSAN invocation:

python scripts/make_musan_noise_manifest_typed.py \
  --musan_root data/raw/musan/musan \
  --out data/manifests/noise/musan_noise_typed.jsonl \
  --seed 42

Archived official-test benchmark invocation:

python scripts/build_robust_benchmark.py \
  --vivos_manifest data/manifests/vivos/test.jsonl \
  --noise_manifest data/manifests/noise/musan_noise_typed.jsonl \
  --out_manifest outputs/benchmark/benchmark_manifest.csv \
  --pool_manifest outputs/benchmark/benchmark_pool_manifest.csv \
  --report_out outputs/benchmark/benchmark_report.md \
  --out_noisy_dir data/noisy_eval \
  --pool_size 500 \
  --eval_size 300 \
  --snrs 20 10 5 0 \
  --seed 42 \
  --sample_rate 16000

Archived one-model invocation:

python scripts/infer_zero_shot.py \
  --benchmark_manifest outputs/benchmark/benchmark_manifest.csv \
  --model_name_or_path openai/whisper-tiny \
  --model whisper \
  --model_size tiny \
  --out outputs/zero_shot/pred_whisper_tiny.csv \
  --batch_size 4 \
  --device auto \
  --resume

Archived full zero-shot invocation:

python scripts/run_zero_shot_pipeline.py \
  --vivos_root data/raw/vivos \
  --musan_root data/raw/musan/musan \
  --seed 42 \
  --pool_size 500 \
  --eval_size 300 \
  --batch_size 4 \
  --device auto

Archived smoke invocation (still official-test-derived):

python scripts/run_zero_shot_pipeline.py \
  --vivos_root data/raw/vivos \
  --musan_root data/raw/musan/musan \
  --seed 42 \
  --pool_size 10 \
  --eval_size 3 \
  --snrs 20 0 \
  --models whisper_tiny \
  --batch_size 1 \
  --device auto \
  --smoke_test

Archived aggregate/validation invocation:

python scripts/aggregate_zero_shot.py \
  --pred_dir outputs/zero_shot \
  --out_by_snr outputs/zero_shot/zero_shot_results_by_snr.csv \
  --out_by_noise_type outputs/zero_shot/zero_shot_results_by_noise_type.csv

python scripts/validate_robust_benchmark.py \
  --benchmark_manifest outputs/benchmark/benchmark_manifest.csv \
  --pool_manifest outputs/benchmark/benchmark_pool_manifest.csv \
  --pred_dir outputs/zero_shot \
  --expected_eval_size 300 \
  --expected_pool_size 500 \
  --snrs 20 10 5 0

The benchmark manifest has 1500 rows for the full run: 300 clean utterances and 300 noisy utterances at each SNR level. Prediction CSV files use the schema utt_id,dataset,model,model_size,snr,noise_type,ref,hyp.

Metrics

All metrics are error rates, so lower is better.

TER, DER, FCER, and SWDR are diagnostic metrics for this midterm project. They should be reported as simple/prototype Vietnamese ASR error indicators, not as external benchmark-standard metrics.

Historical midterm results (archival only)

These official-test-derived results are from the archived small midterm subset. They must not be interpreted as paper-v2 development, selection, or final-test results:

Noisy WER by SNR:

Interpretation:

• PhoWhisper-base strongly outperforms Whisper-base for Vietnamese ASR.
• Tone-aware LoRA slightly improves noisy overall WER compared with PhoWhisper zero-shot.
• The clearest LoRA gain appears at the hardest 0 dB noisy condition.
• LoRA does not improve every condition; clean WER is slightly worse than the PhoWhisper zero-shot baseline.

Fine-tuning configuration

Phat: five-lambda LoRA ablation pipeline

The paper-v2 ordinary/tone-aware PhoWhisper-base workflow uses the five configs under configs/phat/. It now requires speaker-disjoint train/dev manifests, evaluates dev after every epoch, saves best/ by dev_asr_loss, and binds every lambda-ablation result to the full dev manifest, selected-row hash, checkpoint fingerprint, and metric_version=aligned_v1. select_best_lambda.py rejects missing, mixed, partial, legacy, or test provenance. It also requires candidate TER and DER denominator coverage to be at least 0.98 of ordinary LoRA at the configured low-SNR conditions.

The locked training-noise registry and derived noisy-dev benchmark are now present, so the paper-v2 configs enable formal training. This flag alone grants no authority: training still fails closed unless the reviewed environment and method locks bind the exact source revision, split/noise locks, backbone revision, and training contract. The noisy-dev lock binds the source-dev hash, the disjoint noise split, and every derived audio file; merely replacing evaluation.manifest is not sufficient.

Use the existing slp conda environment and its exact interpreter for this workspace. Limited smoke outputs must stay separate and must not be reported as full ablation results.

The active paper-v2 configs are:

configs/phat/lambda_0.yaml
configs/phat/lambda_005.yaml
configs/phat/lambda_01.yaml
configs/phat/lambda_03.yaml
configs/phat/lambda_05.yaml

The older configuration example below is retained only for the historical midterm entry point; it is not the paper-v2 selection protocol.

Legacy commands below are archival and prohibited for paper-v2. They use the old manifests/checkpoint layout, including official-test inputs. Do not execute them before the reviewed decision-v3 lock opens final test.

Legacy fields:

model:
  name_or_path: vinai/PhoWhisper-base
  language: vi
  task: transcribe
  use_lora: true
  lora:
    r: 8
    alpha: 16
    dropout: 0.05
    target_modules: [q_proj, v_proj]

training:
  output_dir: experiments/phowhisper_base_lora_mtl
  num_train_epochs: 3
  per_device_train_batch_size: 1
  gradient_accumulation_steps: 16
  learning_rate: 1.0e-4
  fp16: true
  gradient_checkpointing: true
  lambda_tone: 0.10

data:
  train_manifest: data/manifests/vivos/train.jsonl
  valid_manifest: data/manifests/vivos/dev.jsonl
  test_manifest: data/manifests/vivos/test.jsonl
  test_noisy_manifest: data/manifests/vivos/test_noisy.jsonl

noise:
  enable_train_noise: true
  noise_manifest: data/manifests/noise/musan_noise.jsonl
  snr_choices: [20, 10, 5, 0]
  prob: 0.70

Archived legacy fine-tuning invocation:

python train.py --config configs/phowhisper_base_lora.yaml

Archived legacy official-test evaluation invocation:

python evaluate.py \
  --config configs/phowhisper_base_lora.yaml \
  --checkpoint experiments/phowhisper_base_lora_mtl \
  --split test_noisy \
  --out outputs/phowhisper_lora_noisy.csv

Archived legacy scoring invocation:

python scripts/score_predictions.py \
  --pred outputs/phowhisper_lora_noisy.csv \
  --out outputs/metrics_phowhisper_lora_noisy_by_snr.csv \
  --group_by snr

FLEURS Vietnamese external evaluation

Historical pre-paper-v2 result: the three adapters below were trained before the exact tone-label alignment fix and before the speaker-disjoint dev/test protocol. The measurements remain useful for debugging the evaluation code, but they are not valid evidence for the paper. Re-run FLEURS only after the paper-v2 checkpoints have been retrained and the dev-only decision is locked.

The external evaluation uses all 857 utterances in the FLEURS Vietnamese test split and the same aligned_v1 metric implementation as the internal benchmark. It compares ordinary LoRA, tone-aware LoRA with lambda 0.05, and tone-aware LoRA with lambda 0.1 on exactly the same utterance IDs and references. Audio longer than 30 seconds is split into balanced, non-overlapping chunks to avoid sub-second tails, and each chunk has a 440-token decode limit within Whisper's 448-position target window. If the fast tokenizer encounters an invalid byte-BPE sequence, the runner validates the slow tokenizer has the same vocabulary and decodes only that sequence with invalid bytes ignored.

Archived-run provenance: FLEURS manifest SHA-256 4BEF10B833B7AE8B39D0202F2849564ED4299562B8C546C8828DB7900DC1EA22, PhoWhisper-base revision 7ebdb9e88f5cc5271fb88f4d642c82ff9388650e, and adapter SHA-256 values 821B91821DBE30029C044DD106692CA85B92307B9589799A2115D708A22A79F6 (ordinary), A62F81FFB31BF2C72F01B405ED322CDEBE30C5D2ECD54F426FE43131D641DDB3 (lambda 0.05), and 28BFBE3D3C1BEDF63A4CF92DAA5446BD6F536F220EA02293BD5071D420606C3B (lambda 0.1).

The following command block is archival only; do not run it against the old adapters or publish its outputs as paper-v2 evidence:

python scripts/download_fleurs.py
python scripts/run_external_fleurs.py --device cuda
python scripts/error_analysis.py --pred-glob "outputs/external/fleurs/predictions/pred_*.csv" --out-dir outputs/external/fleurs/error_analysis
python scripts/build_error_breakdowns.py --events outputs/external/fleurs/error_analysis/error_events.csv --out-dir outputs/external/fleurs/error_analysis
python scripts/bootstrap_ci.py --ordinary outputs/external/fleurs/predictions/pred_lora_ordinary_lambda0.csv --lambda-005 outputs/external/fleurs/predictions/pred_tone_lora_lambda_005.csv --lambda-01 outputs/external/fleurs/predictions/pred_tone_lora_lambda_01.csv --output outputs/external/fleurs/bootstrap_ci_results.csv

Use scripts/build_error_artifacts.py with --overall-only and one --focus-run for each of ordinary_lora:0, tone_aware_lora:0.05, and tone_aware_lora:0.1 to build the tone, final-coda, and short-word artifacts. orthographic_breakdown.csv additionally reports the requested missing-mark, wrong-tone-mark, and wrong-vowel-mark diagnostics. These three feature labels are explicitly nonexclusive; the primary TER and DER tables remain additive.

Historical full-test results (lower is better):

Paired percentile bootstrap uses 1,000 shared utterance-level resamples, seed 42, and reports delta as configuration B minus configuration A. The 95% CI excludes zero only for WER and CER in ordinary LoRA versus lambda 0.05, and for WER in ordinary LoRA versus lambda 0.1. All three significant deltas are positive, so they favor ordinary LoRA on this FLEURS test set. The complete 12 intervals are in outputs/external/fleurs/bootstrap_ci_results.csv.

Historical completed experiment order

1. Reproduce the openai/whisper-base zero-shot baseline.
2. Run vinai/PhoWhisper-base zero-shot on the same clean/noisy subset.
3. Fine-tune PhoWhisper tone-aware LoRA using VIVOS train and MUSAN augmentation.
4. Evaluate all three systems on the same clean and noisy manifests.
5. Compare WER, CER, TER, DER, FCER, and SWDR across clean/noisy/SNR splits.
6. Build the final table at outputs/model_comparison_6metrics.csv.

Historical midterm claim

For the archived midterm run only, the correct claim was:

The project has a reproducible Vietnamese noisy-ASR pipeline, real VIVOS + MUSAN evaluation data, Whisper/PhoWhisper zero-shot baselines, a trained PhoWhisper tone-aware LoRA checkpoint, and six-metric comparison results.

The strongest safe conclusion is that PhoWhisper is much better than Whisper-base for Vietnamese ASR, while tone-aware LoRA gives a small noisy-set gain and clearer improvement at severe 0 dB noise, but does not improve every condition.