Arabic ASR & AST Benchmarking Framework

This project provides a research-grade benchmarking suite for evaluating Arabic Automatic Speech Recognition (ASR) and Arabic Speech-to-Translation (AST). It was developed to compare state-of-the-art models like Whisper Large V3 against optimized smaller models and custom adapter-based systems.

📊 Final Results

The comprehensive final report with all metrics, visualizations, and key takeaways can be found here: 👉 Final Benchmarking Report

🚀 Key Features

• Deterministic Evaluation: Ensures every model is tested on the exact same audio samples.
• Arabic Normalization: Systematic text preprocessing for fair WER/CER evaluation.
• Robust Pipeline: Chunked processing to handle large-scale datasets without memory issues.
• Multi-Metric Assessment: Evaluates Word Error Rate (WER), Character Error Rate (CER), Semantic Similarity, BLEU score, and Latency.

🧪 Sample Pipeline Results

ASR Sample (Whisper Large)

{
  "sample_id": "common_voice_ar_19060089.mp3",
  "model_name": "whisper-large",
  "reference": "ماتت عمتي منذ سنتين.",
  "prediction": "ماتت عمتي منذ سنتين",
  "wer": 0.0,
  "cer": 0.0,
  "semantic_similarity": 0.9977
}

AST Sample (E2E Whisper)

{
  "sample_id": "common_voice_ar_19060089.mp3",
  "mode": "e2e",
  "reference_en": "My aunt was dead two years ago.",
  "prediction": "My aunt died two years ago",
  "bleu": 27.22,
  "semantic_similarity": 0.9569
}

🏗️ Project Structure

• asr/: ASR model wrappers (Whisper, Adapter) and evaluation logic.
• speech_translation/: AST evaluation pipeline (Oracle, E2E, Adapter-Cascaded).
• shared/: Common utilities for metrics, data loading, and preprocessing.
• final_delivery/: Final benchmark summary and the comprehensive Markdown report.
• reports/: Detailed logs and plots from various benchmark runs.

🛠️ Setup & Usage

1. Environment Setup

bash setup_environment.sh

2. Running Benchmarks

You can run the full benchmark or a smaller validation set:

# Run validation on 50 samples
python benchmark_runner.py --limit 50 --run_name validation_test

# Run full benchmark
bash run_full_benchmark.sh

📜 Acknowledgments

• Datasets: Common Voice 25.0 (Arabic) and CoVoST 2 (Ar-En).
• Models: OpenAI Whisper, Helsinki-NLP Opus-MT, and custom SparQLe-based adapters.

---

Developed for benchmarking Arabic speech technologies.