whisper-arabic-dialects

Production-aware multi-dialect Arabic ASR via parameter-efficient fine-tuning of Whisper variants. Code, paper, and reproducible benchmarking pipeline accompanying the paper Production-Aware Fine-Tuning of Whisper Variants for Multi-Dialect Arabic ASR: A Cross-Platform CPU Inference Study.

The paper draft is in paper/paper.md. Deployment runbooks for GCP training and CPU benchmarking on GCP + Hetzner are in deploy/.

Released artifacts

v3 large-v3 fine-tune (paper headline, Apr 2026):

• 🤗 LoRA adapter — every save during training (30+ Git revisions for trajectory analysis)
• 🤗 Merged HF model (~3 GB safetensors, transformers-ready)
• 🤗 CTranslate2 int8 (~1.6 GB, production-deployable on CPU)

v2 turbo fine-tune (smaller/faster, paper §6.5):

• 🤗 LoRA adapter, Merged, CT2 int8

Other:

• 🤗 Interactive demo — Gradio Space, side-by-side vs zero-shot
• 🌐 Project case study

Headline results — v3 large-v3 fine-tune (clean held-out test, May 2026)

Note: Earlier headline numbers in this README were inflated by a test-set contamination (96 of 728 utterances overlapped with the training pool — 50 MGB-3-train rows reused as Egyptian test, 46 MASC-train rows as Levantine test). Those numbers have been removed. The table below uses only Casablanca test split rows (built by scripts/build_v3_test_clean.py, which asserts no overlap against the training JSONL before writing). Postmortem: deploy/07_contamination_postmortem.md.

Clean held-out 4-dialect test (n=100/dialect, all conversational Casablanca-test for dialects, FLEURS-test for MSA). Same decoding config across rows: CT2 int8, beam=2, 8 threads, c3-standard-8 (Sapphire Rapids).

FT-v3 beats zero-shot Whisper-large-v3 by 5.39 pp average on the contamination-free test. Double-digit gain on Egyptian (−10.38), strong on Levantine (−7.53) and Gulf (−6.13). MSA loses 2.49 pp — known dialect-vs-MSA tradeoff that v4 (planned) addresses with an MSA-rebalanced 74 h+ mix.

Cross-family comparison (same clean test, May 2026)

FT-v3-ckpt-4750 is the lowest-WER row on every dialect except MSA. Reproduction in deploy/06_v3_data_and_eval.md and deploy/07_contamination_postmortem.md.

The v2-turbo numbers (paper §6.5) used the same now-deprecated mixed test set and need re-evaluation on the clean splits — those rows have been removed pending re-test.

Dialects scoped out

• Maghrebi (Moroccan/Algerian/Tunisian) — excluded from training and reporting. Whisper has insufficient Maghrebi Arabic in pretraining (84.7% zero-shot WER at large-v3 int8); QLoRA cannot bring it within range of other dialects in this training budget. Paper §3.7.

Quickstart

# Local sanity (no GPU, no datasets needed):
pip install -r requirements-bench.txt -e .
pytest tests/

# Tiny zero-shot smoke run (downloads ~50 FLEURS samples):
python -m scripts.run_zero_shot_baseline --tiny

Repo map

Reproducing the paper results

Follow the runbooks in order. Every command used to produce the paper's results is captured in one of these — no improvised steps.

1. deploy/00_gcp_bootstrap.md — install gcloud, authenticate, link billing, enable APIs, confirm L4 quota, set up budget alerts.
2. deploy/01_dataset_acquisition.md — pull the seven Arabic datasets from HuggingFace Hub and assemble the dialect-balanced splits.
3. deploy/02_gcp_training.md — provision the L4 training instance and run QLoRA on turbo (then large-v3).
4. deploy/03_gcp_benchmark.md — provision the c3-standard-8 CPU instance and run the benchmark matrix.
5. deploy/04_hetzner_benchmark.md — provision a Hetzner CX53 and replay the benchmark for cross-platform validation.
6. deploy/05_artifacts_publishing.md — push models to HF Hub, make the W&B project public, push code.

Approximate compute spend: ~$120 on GCP (4 fine-tuning runs across small / medium / turbo / large-v3 + benchmarking sweep) plus ~$2 on Hetzner (cross-platform validation).

Citing

If you use this code, the released models, or results from the accompanying paper, please cite:

@misc{hany2026whisperarabic,
  title        = {Production-Aware Fine-Tuning of Whisper Variants for Multi-Dialect
                  Arabic ASR: A Cross-Platform CPU Inference Study},
  author       = {Hany, Ahmed},
  year         = {2026},
  howpublished = {Preprint, arXiv},
}

A CITATION.cff is also provided for the GitHub "Cite this repository" button.

License

Code: Apache-2.0 (LICENSE). Paper text under paper/: CC-BY-4.0.