South Azerbaijani ASR Benchmark

Official research repository for the INTERSPEECH 2026 paper “Preserving the Iranian Turkic Language: Community-Driven ASR Datasets and Benchmarking for South Azerbaijani.” It provides reproducible dataset loading, South Azerbaijani text normalization, Whisper and MMS fine-tuning, Whisper/MMS/CTC inference, and WER/CER/DIR benchmark evaluation without storing large audio or model files in Git.

Contributions

• Community-driven South Azerbaijani speech resources in Perso-Arabic script.
• Evaluation across Community, External, and held-out GoldSet data.
• Whisper Tiny/Base/Small, cross-lingual Whisper, and MMS-1B-All experiments.
• A single explicit evaluation protocol for WER, CER, and deletion/insertion ratio (DIR).

Datasets

All source rows are converted internally to audio, text, and optional speaker_id/duration. GoldSet is programmatically rejected by training commands. See dataset roles and known schema TODOs.

Models

The unified Whisper trainer accepts openai/whisper-tiny, openai/whisper-base, openai/whisper-small, or any compatible Hugging Face Whisper checkpoint. This is also how the Farsi, North Azerbaijani, Turkish, and Arabic cross-lingual initializations are supplied. The MMS trainer activates the azb adapter of facebook/mms-1b-all and trains the adapter and CTC head by default. Public paper checkpoints are collected at Kartal-Ol/ASR-AZB.

Hugging Face model downloads

The links point directly to each checkpoint directory so model weights, processor/tokenizer files, and configuration files can be downloaded together.

Google Colab inference workshop

The workshop notebook loads any published Whisper checkpoint directly from Hugging Face, accepts an uploaded audio sample, and provides one-sample WER, CER, DIR, and auxiliary SER analysis with a word alignment. SER is a tutorial diagnostic and is not substituted for DIR in the paper benchmark.

Benchmark results

WER and CER are percentages. DIR is the raw word-level deletion/insertion ratio, deletions / insertions. Lower WER and CER are better. These values are transcribed unchanged from the repository’s paper README; they have not been recomputed during cleanup.

Community-only cross-lingual comparison (WER / CER / DIR):

Installation

Python 3.10 or newer is required. Audio decoding for MP3/OGG may additionally require an FFmpeg installation supported by the local audio backend.

git clone https://github.com/Kartalol/Kartalol-azb-asr.git
cd Kartalol-azb-asr
python -m venv .venv
source .venv/bin/activate
pip install -e .

Inference

Whisper, MMS, and compatible Wav2Vec2/CTC checkpoints are detected from their Transformers configuration. Input is converted to mono and resampled to the processor’s required rate.

python scripts/transcribe.py \
  --model openai/whisper-tiny \
  --audio example.ogg

Use --device cpu, --device cuda, or --device cuda:1 to override automatic selection. Batch inference is available with --folder AUDIO_DIRECTORY; add --recursive if needed and --output-json outputs/transcriptions.json for machine-readable output.

Training

The checked-in configurations preserve parameters recoverable from the old scripts. Every inferred reproducibility default is marked TODO in YAML.

python scripts/train_whisper.py --config configs/whisper_tiny_community.yaml
python scripts/train_whisper.py --config configs/whisper_base_full.yaml
python scripts/train_mms.py --config configs/mms_community.yaml --fp16

For a cross-lingual Whisper experiment, override the compatible checkpoint and output directory without editing source:

python scripts/train_whisper.py \
  --config configs/whisper_small_community.yaml \
  --model-id YOUR_COMPATIBLE_WHISPER_CHECKPOINT \
  --output-dir outputs/whisper-small-cross-lingual

Available paper configurations:

• configs/whisper_tiny_community.yaml
• configs/whisper_tiny_full.yaml
• configs/whisper_base_community.yaml
• configs/whisper_base_full.yaml
• configs/whisper_small_community.yaml
• configs/mms_community.yaml

Evaluation

Official scores use raw references and predictions; normalization is never enabled silently.

python scripts/evaluate.py \
  --model YOUR_MODEL_OR_HF_ID \
  --dataset goldset \
  --split test \
  --normalization none \
  --output-dir outputs/goldset

This writes metrics.json and predictions.csv. Use --dataset community or --dataset external for the other benchmark sets. An additional normalized analysis can use --normalization published; see normalization documentation.

Reproducing the benchmark

Copy the example manifest, replace placeholder checkpoints, and run:

cp configs/benchmark.example.yaml configs/benchmark.local.yaml
python scripts/reproduce_benchmark.py \
  --manifest configs/benchmark.local.yaml \
  --output-dir outputs/benchmark

The command evaluates each supplied checkpoint on External, Community, and GoldSet and writes CSV, JSON, and Markdown tables in the paper format. Exact reproduction still requires the final checkpoint IDs, exact Hugging Face External split/config, package/environment lock, and confirmation of the original random seeds and generation settings; these were not recoverable from the checked-in experiment scripts.

Repository structure

configs/                 Paper experiment and benchmark manifests
docs/                    Dataset and normalization protocol
legacy/                  Sanitized historical experiment notes
scripts/                 Training, inference, evaluation, reproduction CLIs
src/kartalol_azb_asr/    Reusable data, audio, normalization, metrics, inference
tests/                   Lightweight unit tests (no model downloads)

Citation

@inproceedings{farsi2026preserving,
  title     = {Preserving the Iranian Turkic Language: Community-Driven ASR
               Datasets and Benchmarking for South Azerbaijani},
  author    = {Farsi, Farhan and Bali, Shayan and Nourmohammadi Khiarak, Jalil
               and Aref, Mohammad Hossein and Akbari Saeed, Taher},
  booktitle = {Proceedings of INTERSPEECH 2026},
  year      = {2026}
}

Machine-readable citation metadata is in CITATION.cff.

License and acknowledgements

Code is released under the existing MIT License. Datasets and models retain the licenses shown on their respective hosting pages.

We thank the Kartal Ol Foundation and every community contributor whose speech, transcription, review, and organizational work made these resources possible.