Open Universal Arabic ASR Leaderboard

This repository contains the evaluation code for the Open Universal Arabic ASR Leaderboard, a continuous benchmark project for open-source multi-dialectal Arabic ASR models across various multi-dialectal datasets. The leaderboard is hosted at elmresearchcenter/open_universal_arabic_asr_leaderboard. For more detailed analysis such as models' robustness, speaker adaption, model efficiency and memory usage, please check our paper.

Updates

• [2026/04/21]: New model included: google/gemma-4-E4B-it
• [2026/04/21]: New model included: google/gemma-4-E2B-it
• [2026/04/16]: New model included: cohere-transcribe-03-2026
• [2026/01/30]: New model included: Qwen3-ASR-0.6B
• [2026/01/30]: New model included: Qwen3-ASR-1.7B
• [2026/01/30]: New model included: VibeVoice-ASR-7B
• [2025/11/13]: New model included: 8 * omnilingual-asr CTC&LLM models
• [2025/09/30]: New model included: Qwen/Qwen3-Omni-30B-A3B-Instruct
• [2025/09/22]: New model included: mistralai/Voxtral-Small-24B-2507
• [2025/09/22]: New model included: mistralai/Voxtral-Mini-3B-2507
• [2025/01/11]: New model included: Nvidia Parakeet-CTC-XXL-1.1B-Universal
• [2025/01/11]: New model included: Nvidia Parakeet-CTC-XXL-1.1B-Concat
• [2025/01/11]: New dataset included: Casablanca

Datasets

Please first download the following test sets

Requirements

We collected models from different toolkits, such as HuggingFace, SpeechBrain, Nvidia-NeMo, etc. Requirements for each library can be installed to evaluate a desired model. To install all the dependencies, run:

pip install -r requirements.txt

Evaluate a model

We provide easy-to-use inference functions, to run an ASR model:

1. Go under models/, run the corresponding model inference function to generate an output manifest file containing ground-truths and predictions.
2. Run calculate_wer function in eval.py on the output manifest file.
3. Details can be found in the methods' docstrings.

Add a new model

Please run the above evaluation for all the test sets under datasets/, calculate the average WER/CER, then launch an issue or PR letting us know about your model, training data, and its performance.

We welcome models that:

1. with a model architecture that is not present in the leaderboard.
2. avoid using training sets in the same dataset as the test sets to avoid the in-domain issue.

Citation

@article{wang2024open,
  title={Open Universal Arabic ASR Leaderboard},
  author={Wang, Yingzhi and Alhmoud, Anas and Alqurishi, Muhammad},
  journal={arXiv preprint arXiv:2412.13788},
  year={2024}
}