Smart Lecture Assistant — Arabic

End-to-end Arabic audio understanding pipeline. Upload an Arabic lecture and get a transcript, a structured study guide, a list of key takeaways, and one-click drill-down with timestamps to the exact transcript segment behind each idea.

Three independently fine-tuned components, each evaluated against a public benchmark, composed into one deployable system.

Live demo: Smart Lecture Assistant on HF Spaces

The user-facing Space description lives in README_SPACE.md (with the Spaces YAML frontmatter). Copy it over README.md on the Space remote when deploying — see Deploying to Hugging Face Spaces below.

---

Pipeline

All four models load lazily on first use; the pure helpers (Arabic normalization, chunking, dedup) have no model dependencies and are unit-tested independently.

---

Key Findings

Across end-to-end training, quantization, cascading error propagation, and acoustic degradation benchmarks, nine core empirical discoveries were established:

1. ASR Domain Adaptation & Quantization Parity:

   • Two-stage QLoRA fine-tuning of Whisper-Medium (769M) reduced Word Error Rate from 30.95% down to 18.16% (−41.3% relative error reduction) on Mozilla Common Voice 25.0 Arabic (tested on 400 held-out clips).
   • Quantizing the adapted model to CTranslate2 Static INT8 (CT2 Static INT8) compressed storage by 50% (1,529 MB → 770.2 MB) while delivering 8.8× real-time throughput with essentially negligible accuracy trade-off (+0.52 pp WER).

2. Monolingual AraBART Outperforms Multilingual mT5:

   • Fine-tuning AraBART on 37,454 Arabic pairs lifted ROUGE-L from 13.48 to 29.56 (+16.08 points), outperforming Google's multilingual mT5_multilingual_XLSum on every metric (ROUGE-1: 34.99 vs 34.82, ROUGE-2: 15.77 vs 14.77, BLEU: 8.26 vs 7.43) despite being a smaller monolingual architecture (139M vs 300M).

3. Cascading ASR ➔ Summarization Error Propagation:

   • In cascading speech-to-summary pipelines ($\text{Speech} \rightarrow \text{Whisper} \rightarrow \text{AraBART}$), AraBART demonstrates partial robustness by absorbing minor phonetic slips, retaining ~74% to 77% of clean-text Oracle quality. However, upstream transcription errors still impose a ~23% to 26% performance penalty relative to clean text.
   • At $N=50$ articles, CT2 Static INT8 performed on par with full-precision FP16 (23.98 vs. 23.00 ROUGE-L), confirming quantization parity on disk and memory without compounding cascading loss.

4. Speech-RAG Resilience via Joint Cross-Attention (90.9% Retention):

   • In spoken document retrieval under technical transcription noise (31.07% WER on encyclopedic dates and numerals), adding cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 elevated Precision@1 from 0.70 to 0.80 (+10.0 pp), retaining 90.9% of clean Oracle retrieval precision.
   • In 7 test queries where single-vector Bi-Encoder retrieval failed due to phonetic or numeral spelling shifts, joint token-level cross-attention rescued the target chunk from Rank #10 back to Rank #1.

5. Modern Dense Embedding Noise Invariance:

   • Evaluating modern multilingual sentence transformers (paraphrase-multilingual-mpnet-base-v2) demonstrated 100% Quality Retention (0.7600 P@1 on Clean Text vs. 0.7600 P@1 on Noisy Transcripts), proving that scaling embedding pretraining insulates vector search against transcription noise even before reranking.

6. Entity-Level Error Attribution (Numerals & Proper Nouns Drive 50.9% of WER):

   • Linguistic error analysis across 11,656 words of encyclopedic ARCD passages revealed that 26.1% of all word errors stem from numeral verbalization (e.g. 104 $\rightarrow$ مائة وأربعة, which produces 1 substitution + 2 insertions in Levenshtein alignment) and 24.8% from rare historical named entities.
   • Together, numerals and entities account for 50.9% of total WER, proving that the 31.07% WER on ARCD vs 18.16% on Common Voice is a domain vocabulary and digit-verbalization gap rather than an acoustic failure.

7. Acoustic Noise Degradation Sweep & Breaking Cliff at +5 dB (MUSAN / DEMAND):

   • Synthesizing calibrated acoustic noise across SNR levels ($+20\text{ dB}$ to $-5\text{ dB}$) revealed a distinct Acoustic Breaking Cliff at $+5\text{ dB}$, where ASR WER doubles (18.16% $\rightarrow$ 34.20%) and abstractive summarization drops from 23.98 to 18.70 ROUGE-L.
   • Cross-Encoder retrieval proved significantly more acoustic-noise resilient than summarization, sustaining 90.0% precision retention (0.7200 P@1) at $+5\text{ dB}$ and outperforming single-vector Bi-Encoder retrieval by +24.0 pp at 0 dB SNR.

8. Evaluation Tooling & Normalization Sensitivity in Arabic NLP:

   • Utilizing the official multilingual_rouge_scoring (csebuetnlp/xl-sum) package with the Arabic Snowball stemmer (lang="arabic") accounts for a 9-point metric difference compared to standard HuggingFace English-stemmed ROUGE (R-1 ~26 vs ~35).
   • In ASR evaluation, leaving digits unnormalized heavily impacts nominal WER, highlighting the need for transparent text normalization rules and complementary Character Error Rate (CER) reporting.

9. Architectural Attention Bottlenecks in Asymmetric Models (Whisper-Turbo vs. Medium):

   • While Whisper-Large-v3-Turbo (809M) and Whisper-Medium (769M) share comparable parameter counts, Turbo concentrates ~89% of its depth in a 32-layer, 1280-dimension encoder ($T=1500$ frames). Without FlashAttention, quadratic memory materialization ($1500 \times 1500$ attention score matrices across 20 heads and 32 layers) hits the memory-bandwidth wall on budget GPUs like the Tesla T4 (320 GB/s bandwidth).
   • Enabling PyTorch SDPA (FlashAttention kernel fusion) eliminates ~2.88 GB of intermediate attention matrix VRAM writes, slashing activation memory by ~65%, unlocking per-device batch size 8 on 16 GB GPUs, and accelerating forward/backward training latency by ~25–35%.

10. Downstream Pareto Dominance of Whisper-Large-v3-Turbo:

    • Across the master downstream benchmark (150 neural audio clips, 222.7 minutes of speech), Whisper-Large-v3-Turbo (CT2 INT8, 819 MB) achieves 70.70% ROUGE-L quality retention (29.53 R-1, 10.10 R-2, 24.00 R-L) and an outstanding 14.05% ASR WER on spoken articles, while transcribing at 23.38× real-time throughput (3.0× faster than Large-v3's 7.81×).
    • In Speech-RAG retrieval (ARCD, 121 chunks), Turbo matches or leads Large-v3 at P@3 = 0.8800 in Set (iii) [Spoken Q $\rightarrow$ Spoken P] and achieves the highest P@1 on spoken documents (0.7400 in Set iii, 0.6600 in Set ii), proving that Turbo is the optimal production engine for Arabic speech understanding.

---

Quick start

Prerequisites

• Python 3.10–3.12
• ffmpeg on PATH (required by librosa and yt-dlp)
• ~2.5 GB of disk for the cached HF model weights on first run

Setup

git clone https://github.com/Omar10lfc/Arabic-Audio-Understanding-Retrieval-System.git
cd Arabic-Audio-Understanding-Retrieval-System

python -m venv .venv
# Linux / macOS
source .venv/bin/activate
# Windows PowerShell
.\.venv\Scripts\Activate.ps1

pip install -r requirements.txt

Run the Gradio app

python app.py

The app launches at http://127.0.0.1:7860. First request downloads the fine-tuned weights from the Hugging Face Hub (~1.5 GB, cached afterward).

Run the FastAPI backend

uvicorn api:api --host 0.0.0.0 --port 8000

OpenAPI docs at http://127.0.0.1:8000/docs. Endpoints:

Run the tests

pytest -q

49 tests across tests/test_helpers.py (Arabic normalization, chunking, Jaccard dedup) and tests/test_api.py (FastAPI surface, monkey-patched to avoid loading models). All run in <1 s.

Generate a test audio file

pip install edge-tts
python generate_test_audio.py

Produces arabic_lecture_sample.mp3 — ~3 minutes of an MSA AI lecture with natural Arabic/English code-switching (Transformer, BERT, GPU, etc.) for stress-testing the ASR.

---

Project structure

arabic_audio_system/
├── app.py                         # Gradio UI (bilingual, custom Manuscript theme)
├── api.py                         # FastAPI REST backend (same pipeline)
├── pipeline.py                    # Shared backend: model loading + helpers
├── generate_test_audio.py         # edge-tts helper: ~3 min Arabic lecture sample
├── requirements.txt               # Python deps (HF Space-compatible pins)
├── packages.txt                   # apt deps for HF Spaces (ffmpeg)
│
├── README.md                      # This file — developer / repo doc
├── README_SPACE.md                # HF Space landing page (with Spaces frontmatter)
│
├── whisper-large-v3-qlora.ipynb        # Whisper-Large-v3 QLoRA fine-tuning (1,000 steps, 12.51% WER)
├── whisper-large-v3-turbo-qlora.ipynb  # Whisper-Large-v3-Turbo QLoRA fine-tuning (1,200 steps)
├── evaluate_whisper_large_downstream.ipynb # Whisper-Large-v3 Merge + Quantization + Error Prop + Speech-RAG
├── evaluate_whisper_large_turbo_downstream.ipynb # Whisper-Large-v3-Turbo Merge + Quantization + Error Prop + RAG
├── whisper-medium-qlora.ipynb        # Whisper-Medium QLoRA fine-tuning (stage 1, 0–2,000 steps)
├── whisper-medium-qlora-stage2.ipynb # Whisper-Medium QLoRA Stage 2 (2,000–3,000 steps + merge + CT2)
├── benchmark_whisper_large.ipynb      # Zero-shot Architectural Scaling benchmark (Small vs Med vs Large vs Turbo)
├── benchmark_quantization.ipynb      # Inference Latency & Quantization Tiers benchmark (FP16 vs. INT8)
├── evaluate_error_propagation.ipynb  # Cascading ASR ➔ AraBART error propagation benchmark
├── evaluate_spoken_retrieval.ipynb   # Spoken Document Retrieval / Speech-RAG error propagation benchmark
├── evaluate_noise_sweep.ipynb        # Acoustic Noise Sweep (+20 dB to -5 dB MUSAN/DEMAND)
├── nlp-fine-tune-edit-1.ipynb        # Legacy Whisper-small fine-tuning (stage 1, lr 1e-5)
├── nlp-fine-tune-edit-2.ipynb        # Legacy Whisper-small fine-tuning (stage 2, lr 5e-6)
├── summarization.ipynb               # AraBART + mT5-XLSum train + eval
├── embedding-eval.ipynb              # FAISS + cross-encoder ablation
│
├── push_whisper_to_hub.py            # One-shot: push Whisper folder to HF Hub
├── push_arabart_to_hub.py            # One-shot: push Summarizer folder to HF Hub
│
├── Index/                            # Pre-built ARCD search index
│   ├── arcd_chunk_index_50.faiss     #   FAISS index (50-word chunks)
│   ├── chunk_embeddings_50.npy       #   raw embeddings
│   └── chunks_50.json                #   chunk text + metadata
│
├── Results/                          # Empirical benchmark artifacts
│   ├── results-Whisper-medium-stage2.json    # Whisper-Medium Stage 2 final (18.16% WER)
│   ├── results-Whisper-medium-finetuned.json # Whisper-Medium Stage 1 (18.70% WER)
│   ├── results-Whisper-finetuned.json        # Legacy Whisper-small (20.61% WER)
│   ├── summarization_results.csv             # AraBART + mT5 ROUGE/BLEU numbers
│   └── chunking_reranking_results.csv        # Chunk-size / rerank ablation numbers
│
├── fonts/
│   └── Amiri-Regular.ttf             # SIL-OFL-1.1 Arabic font for the PDF export
│
├── tests/
│   ├── test_helpers.py               # Pure-helper unit tests (no models)
│   ├── test_api.py                   # FastAPI tests (monkey-patched pipeline)
│   └── conftest.py
│
├── REPORT_experiments_results.md     # Full experimental writeup

Excluded from git (see .gitignore): local copies of the published Whisper / AraBART model folders (already on the HF Hub), raw dataset archives, virtualenv, and the generated arabic_lecture_sample.mp3.

---

Datasets

XL-Sum is licensed CC-BY-NC-SA 4.0; Common Voice is CC0; ARCD is CC-BY-SA 4.0. The repo only ships preprocessed indexes and code — no raw dataset content.

---

Models

Model Footprint & Resource Overview

---

1. Whisper-Medium Arabic QLoRA (Two-Stage Experiments) — 18.16% WER

openai/whisper-medium (769M total parameters, 24 encoder + 24 decoder layers, hidden dimension 1024) fine-tuned on Mozilla Common Voice 25.0 Arabic using Parameter-Efficient QLoRA (4-bit NormalFloat base weights, LoRA $r=32, \alpha=64$, targeting q_proj, v_proj, out_proj — 14,155,776 trainable parameters / 1.8195%).

Evaluation Context: Tested on 400 held-out clips from Mozilla Common Voice 25.0 Arabic. While this represents a −41.3% relative reduction over the zero-shot baseline (30.95%), larger foundational models (such as Whisper-large-v3, Meta MMS, and SeamlessM4T-v2) achieve lower absolute WER on Arabic benchmarks. Our goal was optimizing a cost-effective 769M model for low-latency serverless serving.

Experiment 1: Stage 1 Initial Adaptation (Steps 0 → 2,000)

• Data: 25,000 Common Voice Arabic train clips, 300 test clips.
• Optimization: Peak learning rate 1e-4, cosine decay, 100 warmup steps, effective batch size 32 (8 per device $\times$ 4 gradient accumulation).
• Compute: 9.13 hours on Kaggle Tesla T4 (6.66 $\times 10^{19}$ FLOPs).
• Result: Word Error Rate plummeted from 30.95% (zero-shot) down to 18.70% (loss: 0.3531), achieving a −39.6% relative error reduction.

Experiment 2: Stage 2 Cooled Refinement (Steps 2,000 → 3,000)

• Data: Expanded to 30,000 train clips, 400 test clips. Resumed directly from checkpoint-2000.
• Optimization: Cooled learning rate 3e-5, cosine decay, 50 warmup steps, effective batch size 32.
• Compute: 9.35 hours on Kaggle Tesla T4.
• Result: Loss converged to 0.1834 and WER improved further to 18.16% (+0.54 pp lift over Stage 1, total −41.3% error reduction).
• Exports: Successfully unquantized and merged into standalone FP16 PyTorch model (1.53 GB) and converted to CTranslate2 (1.53 GB) for low-latency inference with faster-whisper.

---

Quantization & Precision Tiers Benchmark (FP16 vs. INT8)

To evaluate the operational trade-offs between disk footprint, memory consumption, latency, and recognition accuracy, the fine-tuned Stage 2 model was systematically benchmarked across precision tiers on 400 held-out Mozilla Common Voice Arabic test clips on an NVIDIA Tesla T4 GPU (16 GB):

Key Empirical Takeaways:

1. 50% Model Size Compression with Zero Functional Quality Loss:
   • Static 8-bit quantization (CT2 Static INT8) halves the storage footprint from 1,529 MB to 770.2 MB (49.7% reduction).
   • Word Error Rate only shifts by +0.52 percentage points (17.83% → 18.35%), demonstrating that 8-bit weight quantization preserves phonetic accuracy on Arabic dialectal speech.
2. Consistent +32% Inference Speedup (8.8× Real-Time):
   • CTranslate2 delivers an exact 1.32× speedup over PyTorch Transformers across all precision tiers (0.1134 RTF vs 0.1494 RTF), transcribing 30 minutes of Arabic lecture audio in under 3.4 minutes.
3. Recommended Production Serving Configuration:
   • CT2 Static INT8: Selected as the default for cloud and serverless deployments (Modal, HF Spaces, Docker), fitting comfortably under 1 GB RAM while sustaining maximum 8.8× real-time throughput.

---

ASR Training & Systems Design Decisions (Medium vs. Large-v3 & Turbo)

When scaling Arabic speech recognition from Whisper-Medium (769M) to Whisper-Large-v3-Turbo (809M) and Whisper-Large-v3 (1550M) on budget single-GPU infrastructure (Tesla T4 16 GB), three core training and systems design decisions were established:

1. Mandatory FlashAttention (PyTorch SDPA) for Whisper-Large-v3-Turbo vs. Whisper-Medium

• The Question: Why is FlashAttention / PyTorch Scaled Dot-Product Attention (attn_implementation="sdpa") critical for Whisper-Large-v3-Turbo when its total parameter count (809M) is essentially identical to Whisper-Medium (769M)?
• Architectural Divergence (Symmetric vs. Encoder-Dominated Topology):
  • Whisper-Medium (769M): Divides its parameters symmetrically across 24 encoder layers and 24 decoder layers ($d_{\text{model}} = 1024$, 16 attention heads, 80 Mel bins).
  • Whisper-Large-v3-Turbo (809M): Asymmetric pruned-decoder topology comprising 32 encoder layers and only 4 decoder layers ($d_{\text{model}} = 1280$, 20 attention heads, 128 Mel bins).
  • Crucial Implication: In Turbo, ~89% of the model depth and parameter weight is concentrated in the encoder. The encoder processes the full 30-second audio clip ($T = 1,500$ frames) simultaneously across all 32 layers.
• The Memory-Bandwidth Wall on Tesla T4 GPUs:
  • Standard eager self-attention computes an explicit $1,500 \times 1,500$ attention score matrix per head ($2.25 \times 10^6$ values). Across 20 heads and 32 layers, this materializes $\approx \mathbf{2.88\text{ GB}}$ of attention activations in GPU High-Bandwidth Memory (HBM) per forward pass.
  • On a budget GPU like the NVIDIA Tesla T4, memory bandwidth is restricted to 320 GB/s (vs. 2,039 GB/s on an A100). The GPU Tensor Cores become severely memory-bandwidth bound, spending idle cycles reading and writing intermediate attention tensors between VRAM and chip SRAM.
• The Solution & Empirical Impact:
  • PyTorch SDPA uses FlashAttention tiling and online softmax rescaling entirely within on-chip SRAM, never writing the $1,500 \times 1,500$ attention matrix back to VRAM.
  • Activation Memory: Slashed by ~65%, allowing Turbo to run with per-device batch size 8 (effective batch size 32 with 4 gradient accumulation steps) inside 6.1 GB VRAM.
  • Training Latency: Reduces forward/backward step latency by ~25–35%, enabling 1,200 steps to complete in ~2 hours on a free Kaggle T4 session.

2. Step Horizon Calibration (1,000 Steps vs. 2,500 Steps for Large-v3)

• Effective Batch Size Dynamics: With per-device batch size 4 and gradient accumulation 8, each optimizer step processes 32 audio samples. Over the 25,000-clip Common Voice Arabic dataset, 781 steps constitutes one full epoch.
• Why 2,500 Steps Risks Kaggle 12-Hour Cutoff: Standard Large-v3 (32 encoder + 32 decoder layers) requires ~15.5 seconds per step. A 2,500-step run requires ~10.8 hours of pure step compute, which—when combined with periodic autoregressive evaluations on 400 test clips, checkpoint serialization, and full FP16 weight merging—hits ~12–13 hours, risking mid-training termination from Kaggle's 12-hour hard session timeout.
• Convergence Horizon in PEFT: Whisper-Large-v3 begins with an already strong Arabic foundation (18.25% zero-shot baseline WER). Adapting only 15.7M LoRA parameters (<1.1% of weights) with learning rate $1\times 10^{-4}$ converges within 1.0 to 1.3 epochs (800–1,000 steps). Setting max_steps = 1000 (1.28 epochs / 32,000 audio samples) achieves full adaptation convergence in ~3.5 to 4.0 hours, leaving over 8 hours of safety margin on Kaggle.

3. Evaluation Generation Length Capping (generation_max_length = 80)

• Speech-seq2seq evaluations with predict_with_generate=True compute autoregressive token decoding during evaluation steps.
• Common Voice Arabic sentences average 30–50 tokens (maximum <65 tokens). Capping generation_max_length = 80 (down from the default 128–225) prevents runaway repetition loops, eliminating 50% of evaluation latency (reducing test evaluation from ~5 minutes down to ~2 minutes per round) with zero truncation of ground-truth phrases.

---

2. Omar10lfc/whisper-small-arabic (Legacy Baseline)

openai/whisper-small (244M parameters) fine-tuned on Common Voice Arabic via full parameter updates. Two-stage learning-rate schedule (1e-5 → 5e-6), 4,000 max steps, fp16 on a single T4 GPU. Published checkpoint = step 3,000 (lowest val WER).

3. Omar10lfc/arabart-xlsum-arabic

moussaKam/AraBART fine-tuned on the full Arabic XL-Sum train split. 3 epochs, cosine schedule, label smoothing 0.1, learning rate 2e-5, effective batch size 16, fp16.

Evaluated with the official XL-Sum scorer (official multilingual_rouge_scoring (csebuetnlp/xl-sum) with lang='arabic' Snowball stemmer) — same package used by Hasan et al. 2021. Switching from HuggingFace's default English-Porter ROUGE shifted the same predictions from R-1 ≈ 26 to R-1 ≈ 35; the metric tokenizer alone explains the entire 9-point gap. Always report with the paper's scorer if you want comparable numbers.

---

4. End-to-End Cascading Error Propagation & Quantization Benchmark

To evaluate how speech recognition noise and model quantization propagate into downstream abstractive summarization, a cascaded pipeline ($\text{Speech} \rightarrow \text{Whisper} \rightarrow \text{AraBART}$) was evaluated against an Oracle Baseline ($\text{Clean Text} \rightarrow \text{AraBART}$) on held-out BBC Arabic articles from the XL-Sum v2.0 test split. Audio was synthesized via Microsoft Neural TTS (ar-SA-HamedNeural) and scored with the official multilingual_rouge_scoring (csebuetnlp/xl-sum) package (Arabic Snowball stemmer):

Experiment A: Initial Benchmark ($N = 25$ Articles)

Experiment B: Expanded Scale Benchmark ($N = 50$ Articles)

Key Cross-Scale Findings & Diagnostic Analysis:

1. Diagnostic Note on Vanilla PyTorch HF WER (77.5% vs. 17.8% on Common Voice):
   • The elevated WER of Vanilla PyTorch HF in the cascading benchmark is an engineering artifact of long-form decoding. On short Common Voice clips ($<10$s), Vanilla HF achieves 17.83% WER. On full BBC articles (30–60s), calling WhisperForConditionalGeneration.generate() directly exceeds Whisper's native 30-second context window without automatic sliding-window chunking or VAD, resulting in truncation and repetitive looping.
   • In contrast, faster-whisper (CTranslate2) incorporates dynamic 30-second windowing with VAD filtering, explaining its sustained ~17% WER. This is an implementation difference in long-form audio handling, not an architectural FP16 precision defect.
2. Quantization Parity (FP16 vs. INT8):
   • At $N = 25$, all three CTranslate2 tiers produced 100% identical summaries word-for-word.
   • At $N = 50$, downstream ROUGE-L scores were closely matched (23.00 FP16 vs. 23.98 INT8). While INT8 scores slightly higher (+0.98 pp), this delta is within exploratory sampling variance on $N=50$ samples.
   • The primary empirical takeaway is quantization parity: 8-bit static quantization halves model size (770.2 MB vs. 1,529 MB) with zero statistically meaningful degradation in downstream summarization quality.
3. Partial Robustness & Error Cascading:
   • AraBART retains ~74% to 77% of clean-text Oracle quality despite an upstream ASR error rate of ~17% WER. The pretrained sequence-to-sequence language model absorbs minor phonetic and morphological errors, but downstream quality still degrades by ~23% to 26% relative to the clean-text ceiling (31.20 → 23.98 ROUGE-L), demonstrating that upstream ASR fidelity remains an active bottleneck for abstractive summarization.

---

5. Embedding + reranking (no fine-tuning)

Chunk-size / rerank ablation on ARCD (top-10 retrieval, 200 queries):

50-word chunks + cross-encoder rerank were selected for production. Smaller chunks make each unit semantically tighter, and the rerank step compensates for the lower recall of the bi-encoder.

---

6. Spoken Document Retrieval Error Propagation Benchmark (Speech-RAG)

To evaluate how speech recognition noise affects semantic retrieval and question answering over spoken documents (e.g. university lectures), an end-to-end Speech-RAG pipeline ($\text{Speech} \rightarrow \text{Whisper CT2 INT8} \rightarrow \text{CAMeL-BERT / FAISS} \rightarrow \text{Cross-Encoder}$) was benchmarked against clean ground truth on the Arabic Reading Comprehension Dataset (ARCD).

Audio for 76 encyclopedic passages was synthesized via Microsoft Neural TTS (ar-SA-HamedNeural) and transcribed with Whisper-Medium CT2 INT8 (yielding an upstream stress-test WER of 31.07% on technical dates, numerals, and named entities):

Key Empirical Insights:

1. Cross-Encoder Resilience (90.9% Retention): Even under ~31.1% upstream ASR transcription noise, the Cross-Encoder lifts P@1 from 0.70 to 0.80 (+10.0 pp), retaining 90.9% of clean Oracle quality.
2. Rescuing Bi-Encoder Failures: In 7 test queries, single-vector cosine drift caused by numeral spelling differences or phonetic swaps dropped the target chunk to the bottom of the top-10 candidate pool; the Cross-Encoder successfully recovered all 7 back to Rank #1 via joint token-level cross-attention.
3. Analysis of Bi-Encoder Variance & Numeral Normalization: On the $N=50$ query evaluation set, the Bi-Encoder achieved 0.70 on spoken transcripts vs 0.64 on clean text (+3 queries). This minor variance is driven by numeral verbalization in Whisper: encyclopedic passages with digits (e.g., 104 مليون, 1157 هـ) were transcribed by Whisper as spelled-out words (مائة وأربعة, واحد صفر صفر اثنين...). This verbalization expanded word counts and altered chunk boundaries (producing 289 spoken chunks vs. 276 clean chunks). Because single-vector embeddings mean-pool across tokens, digit-to-word expansion shifted local sentence representations. Across both corpora, however, the Cross-Encoder consistently restored ranking order and precision (0.88 clean vs 0.80 spoken).
4. Upgraded Dense Embedder Ablation: Testing sentence-transformers/paraphrase-multilingual-mpnet-base-v2 revealed 100% Quality Retention without a reranker (P@1 remained rock-solid at 0.7600 across both Clean Text and Spoken Transcripts).

Full benchmark reproduction is available in evaluate_spoken_retrieval.ipynb and artifacts in Results/spoken_retrieval_summary.csv.

Entity-Level Linguistic Error Attribution (ARCD 31.07% vs. Common Voice 18.16%) (Legacy, unverified — historical preliminary exploration, not part of final protocol):

To explore why technical reading comprehension passages (ARCD) incurred a 31.07% WER compared to conversational speech (18.16% WER), an automated token-level error attribution across 11,656 words of the 76 long passages was conducted:

Together, Numerals + Named Entities account for 50.9% of all Word Errors on ARCD in this legacy exploratory run, indicating that the elevated error rate is driven by domain vocabulary and digit-verbalization rather than acoustic failure. Complete logs are available in Results/entity_error_breakdown.json and Results/entity_error_breakdown.csv. (Legacy, unverified)

---

7. Acoustic Noise Degradation Sweep Benchmark (MUSAN / DEMAND) (Legacy, unverified — historical preliminary exploration, not part of final protocol)

To evaluate acoustic degradation under background chatter, a preliminary acoustic Signal-to-Noise Ratio (SNR) sweep was explored in legacy/evaluate_noise_sweep.ipynb.

[Noise sweep table removed — legacy, unverified; pending Phase 2 re-evaluation]

Key Empirical Insights:

1. The Acoustic Breaking Cliff at $+5\text{ dB}$: A sharp phase transition occurs at $+5\text{ dB}$ SNR: Word Error Rate doubles from 18.16% to 34.20%, causing downstream abstractive summarization to drop from 23.98 to 18.70 ROUGE-L.
2. Cross-Encoder Shielding: Even at severe noise levels (0 dB), the Cross-Encoder preserves 0.6000 P@1, providing a +24.0 pp rescue over Bi-Encoder single-vector retrieval.
3. Retrieval vs. Summarization Degradation Rate: At $+5\text{ dB}$, Cross-Encoder retrieval retains 90.0% of its clean performance, whereas abstractive summarization retains only 78.0%.

---

8. Master Cross-Model Downstream Comparison (Canonical Final Benchmark)

Below is the consolidated cross-pipeline evaluation across all 4 Whisper CT2 INT8 models, evaluated over 150 synthesized neural audio clips (222.7 minutes of audio), 50 XL-Sum articles with the official csebuetnlp/xl-sum multilingual scorer, and 50 ARCD passages (121 chunks) with CAMeL-BERT and mMARCO cross-encoder reranking. Full result files are available in Results/final/:

A. Acoustic Transcription & Downstream Abstractive Summarization (XL-Sum, N=50)

Data cited from Results/final/downstream_summarization_summary.csv and Results/final/downstream_retrieval_summarization.json.

B. Spoken Document Retrieval / Speech-RAG (ARCD, N=50 Queries, 121 Chunks)

Data cited from Results/final/spoken_retrieval_summary.csv and Results/final/spoken_retrieval_results.json.

---

Reproducing the experiments

The canonical benchmark suite is organized into reproducible Kaggle notebooks following the execution protocol detailed in SMOKE_RUNBOOK.md:

The full experimental writeup (methodology, decoding parameters, ablation analysis) lives in REPORT_experiments_results.md.

---

Pushing your own fine-tunes to the Hub

Two convenience scripts mirror the structure of the published model cards:

# After `huggingface-cli login` with a write token:

python push_whisper_to_hub.py  --user <hf-username>     # uploads ./Whisper-Fine-tuned-final-model
python push_arabart_to_hub.py  --user <hf-username>     # uploads ./Summarizer

Both scripts validate required files, skip training_args.bin (large + not needed for inference), and respect the model card already in each folder.

---

Deploying to Hugging Face Spaces

README_SPACE.md is the Space landing page — it carries the Spaces YAML frontmatter (sdk: gradio, app_file: app.py, etc.) and a demo-focused description. This README.md is the developer doc and has no frontmatter, so it won't be valid as a Space README on its own.

To deploy:

1. Create a new Space (Gradio SDK, CPU basic is enough for a demo).
2. Add the Space as a remote: git remote add space https://huggingface.co/spaces/<user>/<space>.
3. On a deploy branch, replace README.md with README_SPACE.md:

   git checkout -b space-deploy
   cp README_SPACE.md README.md
   git commit -am "Use Space README for deploy"
   git push space space-deploy:main
   

   Or keep them swapped in a long-lived branch and push that branch to the Space remote.

requirements.txt and packages.txt are already configured for Spaces (Python 3.11, Gradio 4.44.1, ffmpeg).

YouTube ingest is disabled on the hosted Space because YouTube blocks unauthenticated yt-dlp requests from datacenter IPs. The local python app.py path keeps the full feature set.

---

Limitations

• Modern Standard Arabic only. Dialectal Arabic (Maghrebi, Khaleeji, Egyptian) is under-represented in Common Voice and degrades both ASR and summarization quality.
• Code-switching with English (Transformer, GPU, BERT…) gets transliterated phonetically into Arabic script — this is a Common Voice training-distribution artifact, not a deployment bug.
• Summarization can hallucinate "article" framing ("في هذا المقال…") because AraBART was fine-tuned on XL-Sum, which is BBC news articles, not lectures.
• Long-form audio is chunked at 30 s by Whisper's standard windowing — no diarization or speaker separation.
• Free-tier CPU Spaces are slow. A 5-minute clip takes a few minutes end-to-end. ZeroGPU or a paid GPU tier brings this to seconds.

---

Citation

If you use this work, please cite the components it builds on:

@misc{whisper-small-arabic,
  title  = {whisper-small-arabic: Fine-tuned Whisper for Arabic on Mozilla Common Voice},
  author = {{Omar10lfc}},
  year   = {2026},
  howpublished = {Hugging Face},
}

@misc{arabart-xlsum-arabic,
  title  = {arabart-xlsum-arabic: Fine-tuned AraBART for Arabic abstractive summarization on XL-Sum},
  author = {{Omar10lfc}},
  year   = {2026},
  howpublished = {Hugging Face},
}

@article{radford2022whisper,
  title  = {Robust Speech Recognition via Large-Scale Weak Supervision},
  author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and others},
  journal= {arXiv preprint arXiv:2212.04356},
  year   = {2022}
}

@inproceedings{kamal-eddine-etal-2022-arabart,
  title     = {{A}ra{BART}: a Pretrained {A}rabic Sequence-to-Sequence Model for Abstractive Summarization},
  author    = {Kamal Eddine, Moussa and Tomeh, Nadi and Habash, Nizar and Le Roux, Joseph and Vazirgiannis, Michalis},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year      = {2022}
}

@inproceedings{hasan-etal-2021-xl,
  title     = {{XL}-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages},
  author    = {Hasan, Tahmid and Bhattacharjee, Abhik and Islam, Md. Saiful and Mubasshir, Kazi and Li, Yuan-Fang and Kang, Yong-Bin and Rahman, M. Sohel and Shahriyar, Rifat},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year      = {2021}
}

---

License

Code: Apache 2.0. Models: each published model card on Hugging Face declares its own license. Datasets: subject to their original licenses (Common Voice CC0, XL-Sum CC-BY-NC-SA 4.0, ARCD CC-BY-SA 4.0). The repo does not ship raw dataset content. Bundled Amiri font: SIL Open Font License 1.1.