Golden Transcription Selection System (GTSS)

Automated Multi-Signal Arabic and Urdu Transcription Quality Pipeline

A production-grade system for automatically selecting the most accurate transcription from multiple candidate transcriptions of the same audio, designed specifically for Arabic and Urdu speech.

---

🎯 Problem Statement

When multiple transcriptions exist for the same audio sample (from different annotators or ASR systems), identifying the most accurate one requires manual expert review. GTSS automates this process using a multi-signal ensemble approach that evaluates:

• Audio-text alignment via Whisper ASR rescoring
• Cross-candidate consensus via ROVER-inspired voting
• Linguistic quality via language model perplexity
• Structural completeness via heuristic analysis

🏗️ Architecture

Audio + Transcription Candidates
        ↓
[Stage 1] Audio Preprocessing + Arabic/Urdu Text Normalization
        ↓
[Stage 2] Multi-Signal Scoring (4 independent signals)
        ↓
[Stage 3] Ensemble Fusion + Golden Selection
        ↓
[Stage 4] WER Computation + Export
        ↓
predictions.csv (Golden labels + WER metrics)

The 4 Signals

---

🚀 Quick Start

Installation

# Clone the repository
git clone https://github.com/your-team/gtss.git
cd gtss

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Download models (first run only)
python scripts/download_models.py

Basic Usage

# Process a dataset
python main.py \
    --input data/transcriptions.csv \
    --audio_dir data/audio/ \
    --output output/ \
    --language auto

# With specific language
python main.py \
    --input data/transcriptions.csv \
    --audio_dir data/audio/ \
    --output output/ \
    --language ar  # or 'ur' for Urdu

Expected Input Format

CSV with columns:

audio_id, audio_filename, option_1, option_2, option_3, option_4, option_5

Or flexible column naming:

audio_id, filename, transcription_1, transcription_2, ...

Output Format

predictions.csv:

audio_id, language, audio_filename, option_1, ..., option_5, golden_ref, 
wer_option1, ..., wer_option5, asr_score, consensus_score, lm_score, 
structural_score, ensemble_score, selection_margin

---

📊 Key Features

✅ Arabic/Urdu Specific

• Orthographic normalization: Alef unification, diacritic stripping, Ya/alef-maqsura handling
• Language-specific LMs: AraGPT2 for Arabic, mGPT for Urdu and code-switching
• Clitic-aware length tolerance: Accounts for Arabic attachment variability
• Verbatim-aware filler detection: Treats disfluencies (يعني, مطلب) as positive signals

✅ Production-Ready

• Graceful degradation: Audio unavailable? Weights auto-redistribute
• Hardware adaptive: Auto-selects Whisper base vs. large-v3 based on speed
• Zero manual intervention: Fully automated from CSV to predictions
• Comprehensive diagnostics: Per-sample JSON output for error analysis

✅ Scientifically Rigorous

• Ablation studies: Compare 4 model variants to justify design
• Arabic-aware WER: Normalization applied before error computation
• Calibrated confidence: Selection margin scores for borderline cases

---

📁 Repository Structure

gtss/
├── main.py                     # CLI entry point
├── pipeline/
│   ├── __init__.py
│   ├── arabic_normalize.py    # Arabic/Urdu text normalization (zero dependencies)
│   ├── asr_scoring.py         # Whisper-based audio-text alignment
│   ├── consensus_scoring.py   # ROVER-inspired cross-candidate voting
│   ├── lm_scoring.py          # AraGPT2/mGPT perplexity computation
│   ├── structural.py          # Heuristic quality signals
│   ├── ensemble.py            # Weighted fusion + golden selection
│   └── wer_compute.py         # Arabic-aware WER calculation
├── tests/
│   ├── test_arabic_normalize.py
│   ├── test_asr_scoring.py
│   ├── test_consensus.py
│   └── test_integration.py
├── scripts/
│   ├── download_models.py     # Pre-download Whisper, AraGPT2, mGPT
│   └── run_ablation.py        # Automated ablation study
├── data/
│   ├── train/                 # Training data (if applicable)
│   ├── test/                  # Test CSV goes here
│   └── audio/                 # Audio files directory
├── output/                    # predictions.csv + diagnostics.json
├── docs/
│   ├── PROPOSAL.pdf           # Full technical proposal
│   ├── TECHNICAL_REPORT.pdf   # Results and analysis
│   └── API.md                 # Module API documentation
├── requirements.txt
├── .gitignore
└── README.md                  # This file

---

🧪 Running Tests

# Run all tests
pytest tests/ -v

# Test specific modules
pytest tests/test_arabic_normalize.py -v
pytest tests/test_integration.py -v

# With coverage
pytest tests/ --cov=pipeline --cov-report=html

---

📈 Performance Targets

---

🔧 Configuration

Ensemble Weights (tunable via CLI)

python main.py \
    --input data/test.csv \
    --audio_dir data/audio/ \
    --output output/ \
    --weight_asr 0.35 \
    --weight_consensus 0.30 \
    --weight_lm 0.20 \
    --weight_structural 0.15

Whisper Model Selection

# Force Whisper base (faster)
python main.py --input data/test.csv --whisper_model base

# Force Whisper large-v3 (more accurate)
python main.py --input data/test.csv --whisper_model large-v3

# Auto-select based on hardware (default)
python main.py --input data/test.csv --whisper_model auto

---

📚 Technical Details

Arabic Normalization Rules

Implemented in pipeline/arabic_normalize.py with zero external dependencies:

1. Alef unification: أ إ آ ٱ → ا
2. Ya normalization: ى → ي
3. Diacritic stripping: Remove U+064B–U+065F (harakat, tanwin, shadda, sukun)
4. Tatweel removal: Remove ـ (elongation character)
5. Punctuation normalization: Remove ، ؟ ۔

Language Detection

Uses langdetect to route each sample to:

• Whisper language flag (ar or ur)
• LM model (AraGPT2 for Arabic, mGPT for Urdu/mixed)
• Filler word list for Signal 4

Ablation Models

Four variants for comparison:

1. ASR-only: Whisper overlap + argmax
2. Consensus-only: ROVER voting alone
3. LM-only: AraGPT2/mGPT perplexity ranking
4. Full fusion: All 4 signals weighted (proposed system)

Run ablation:

python scripts/run_ablation.py --input data/test.csv --audio_dir data/audio/

---

🚧 Known Limitations

1. Language scope: Designed for Arabic and Urdu. Other languages require updating normalization rules and LM models.

2. Dialectal Arabic: Covers MSA orthographic variants. Dialectal transliterations may introduce residual WER inflation.

3. Whisper errors: Signal 1 uses Whisper as reference. Whisper mistakes on proper nouns/domain vocabulary can penalize correct candidates. Partially offset by Signals 2-4.

4. Consensus assumptions: Assumes independent annotator errors. Correlated errors (same ASR pre-fill tool) can reward incorrect majority.

5. Hardware requirements:

   • Whisper large-v3: ~10GB GPU VRAM or ~1× real-time on CPU
   • AraGPT2: ~500MB storage
   • mGPT: ~700MB storage

---

👥 Team

BITS Pilani, Goa Campus

• Mayank Joshi - Team Leader, Arabic/Urdu NLP, Pipeline Architecture
• Saurabh Kumar - Documentation, Testing, CLI Design
• Krutik Rana - Deep Learning, Whisper/LM Integration
• Kushagra Singh - Audio Processing, Feature Extraction

---

📄 License

MIT License - see LICENSE for details.

---

📖 Citation

If you use this system in your research, please cite:

@misc{gtss2026,
  title={Golden Transcription Selection System: Multi-Signal Ensemble for Arabic and Urdu Transcription Quality Evaluation},
  author={Joshi, Mayank and Kumar, Saurabh and Rana, Krutik and Singh, Kushagra},
  year={2026},
  institution={BITS Pilani, Goa Campus}
}

---

🔗 References

• Radford et al. (2023) - Whisper: Robust Speech Recognition via Large-Scale Weak Supervision
• Fiscus (1997) - ROVER: Recognizer Output Voting Error Reduction
• Antoun et al. (2020) - AraGPT2: Pre-Trained Transformer for Arabic Language Generation
• Shliazhko et al. (2022) - mGPT: Few-Shot Learners Go Multilingual

---

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

---

Hackathon 2026 | Speech & Language Processing Track