Court Proceedings ASR Pipeline

A reproducible pipeline for building a speech‑to‑text dataset from Indian Supreme Court proceedings and fine‑tuning Whisper models. It captures what we did in order: data ingestion → alignment → cleaning → dataset export → fine‑tuning → evaluation.

Quick Start

Prereqs: pip install -r requirements.txt

Repository Map

• download.py: Download PDFs/audio from data.csv (Dropbox mp3s)
• audio_files/, transcripts/, meta/: Downloaded data
• aligner.py + config.py: Extract PDF text, run NeMo Forced Aligner (NFA), chunk to 10–30s with punctuation/gap constraints paired .wav/.txt files.
• clean.py: Duration and word count gates, writes filter_report.csv.
• wer_filtering.py: Builds curation_table.csv with WER-based tags (clean, fair, hard, reject) for optional filtering during fine-tuning.
• upload_data_to_hf.py: Builds train/test JSON and pushes dataset to HF (features include audio, sentence, metadata) and curation_table.csv for optional filtering.
• train.py: Minimal fine‑tuning for Whisper (small/medium), optional curation by curation_table.csv tags.
• eval.py: Batched inference with WER/CER + normalized metrics; OOM‑safe batch reductions.
• baseline_wer_v2.py, format_results.py, duration.py: Baselines, result formatting, and duration stats.

Data Ingestion

Input spreadsheet: data.csv

• Command: - python download.py --csv data.csv
• Outputs: - Audio: audio_files/audio_<idx>.wav - PDFs: transcripts/transcript_<idx>.pdf - Metadata: meta/metadata_<idx>.json

Alignment (NeMo Forced Aligner)

• Requirements: NEMO_ROOT points to a NeMo checkout that contains tools/nemo_forced_aligner/align.py.
• Extracts clean text from PDFs (drops headers/footers/speaker tags/IST lines). - Inserts sentence separator (|) to encourage denser NFA segments. - Aligns text to audio with NFA; chunks to 10–30s, respecting hard gaps; falls back to word‑CTM to guarantee max length.
• Command:
  • python aligner.py --nemo_root "$NEMO_ROOT" --audio_dir audio_files --transcript_dir transcripts --output_dir aligned_output_30s_final
• Outputs: Paired chunks: aligned_output_30s_final/audio_<idx>_<chunk>.wav/.txt

Cleaning

• Gates: min_duration=2.0s, min_words=3, min_alnum_chars=6 (no VAD).
• Command:
  • python clean.py --in_dir aligned_output_30s_final --out_dir aligned_output_30s_final_clean --reject_dir final_data_rejected --report_csv filter_report.csv
• Outputs: Kept: aligned_output_30s_final_clean/ and Rejected: final_data_rejected/
• Report: filter_report.csv with drop = 0|1 for each sample.

WER Filtering + Curation Table

• Loads predictions from multiple ASR models (Parakeet RNNT 1.1B, Conformer CTC Large, Whisper Large v3).
• Merges predictions with filter_report.csv (keeps only samples where drop=0).
• Computes per-sample WER for each model, then calculates cross-model statistics: wer_min, wer_med, wer_mean, wer_std.
• Assigns dynamic quality tags based on ensemble behavior:
  • clean: wer_min ≤ 0.25 (at least one model performs well)
  • fair: median WER in 45th–85th percentile range AND wer_min ≤ 0.60
  • ambiguous: high disagreement (wer_std > 0.35)
  • drop: wer_min > 0.55 AND median WER above 95th percentile of hard samples
  • no_pred: no model produced predictions
• Coalesces reference text across models (picks first non-empty).
• Command:
  • python wer_filtering.py
• Outputs:
  • curation_table.csv with columns: basename, path, duration, wer_<model> (per model), ref, no_pred, wer_min, wer_med, wer_mean, wer_std, tag
  • missing_no_predictions.csv (samples with no predictions from any model)

Dataset Export + HF Hub

• Script builds train.json/test.json from aligned pairs and splits test by fixed sr_no set.
• Command (push to hub):
  • python upload_data_to_hf.py --push-to-hub --hf-username <your_hf_user> --hf-token $HF_TOKEN
• Outputs:
  • Local: sc_stt_export/train.json, sc_stt_export/test.json
  • Hub dataset: e.g., karynaur/court_proceedings_stt_dataset (contains train/test splits and curation_table.csv) https://huggingface.co/datasets/karynaur/court_proceedings_stt_dataset

Fine‑Tuning (Whisper)

• Uses HF datasets (local or Hub), simple collator, and computes WER/CER + normalized metrics.
• Common run:
  • python train.py --model_id openai/whisper-small --output_dir whisper_small_ft --fp16
• Filtering by tags: ensure curation_table.csv has tag and basename|path columns, then set --include_tags clean,fair as needed.
• Outputs:
  • Checkpoint + processor in whisper_small_ft/ (or whisper_medium_ft/).

Evaluation

• Evaluate any HF id or local folder with batched decoding and OOM‑aware batch shrinking.
• Baseline example:
  • python eval.py --model_path openai/whisper-small --dataset karynaur/court_proceedings_stt_dataset --split test --batch_size 16 --fp16 --out_file eval/openai_whisper-small_eval.txt
• Fine‑tuned example:
  • python eval.py --model_path whisper_small_ft --dataset karynaur/court_proceedings_stt_dataset --split test --batch_size 16 --fp16 --out_file eval/resport_whisper_small_ft.txt
• Outputs:
  • Text report with WER/CER + normalized metrics and REF/HYP rows under eval/.

Results (Highlights)

• Baselines (from eval/ files):
  • openai/whisper-small: WER 44.19, CER 27.50, NORM WER 32.82, NORM CER 24.55
  • openai/whisper-medium: WER 40.87, CER 25.86, NORM WER 29.58, NORM CER 22.99
  • openai/whisper-large-v3: WER 40.77, CER 23.50, NORM WER 26.18, NORM CER 19.97
  • nvidia/parakeet-rnnt-1.1b: WER 47.37, CER 24.90, NORM WER 25.98, NORM CER 20.14
  • nvidia/stt_en_conformer_ctc_large: WER 50.74, CER 27.79, NORM WER 32.57, NORM CER 23.39
• Fine‑tuned:
  • whisper-small-ft: WER 33.08, CER 19.32, NORM WER 23.78, NORM CER 16.90 https://huggingface.co/karynaur/whisper-small-ft-court-proceedings
  • whisper-medium-ft: WER 31.27, CER 18.99, NORM WER 22.53, NORM CER 16.61 https://huggingface.co/karynaur/whisper-medium-ft-court-proceedings

Reproduce In Order

1. python download.py --csv data.csv
2. python aligner.py --nemo_root "$NEMO_ROOT" --audio_dir audio_files --transcript_dir transcripts --output_dir aligned_output_30s_final
3. python clean.py --in_dir aligned_output_30s_final --out_dir aligned_output_30s_final_clean --reject_dir final_data_rejected --report_csv filter_report.csv
4. python wer_filtering.py
5. python upload_data_to_hf.py --push-to-hub --hf-username <you> --hf-token $HF_TOKEN
6. python train.py --model_id openai/whisper-small --output_dir whisper_small_ft --fp16
7. python eval.py --model_path whisper_small_ft --out_file eval/resport_whisper_small_ft.txt