ASR System Assessment

An end-to-end Automatic Speech Recognition system for transcribing Bangla and English audio with a web-based interface.

---

i. Objective

Develop an end-to-end ASR system capable of transcribing audio files in both Bangla and English with:

• Low-latency inference
• High accuracy using a hybrid model approach
• User-friendly web interface for audio upload and transcription
• Real-time performance metrics display

---

ii. Dataset Sources and Explanation

Mozilla Common Voice (English)

• Samples: ~195,000 audio clips
• Location: data/common_voice/cv-valid-train/
• Metadata: data/common_voice/cv-valid-train.csv
• Description: Crowdsourced English speech recordings with transcriptions

BUET DL Sprint 2022 (Bangla)

• Samples: ~207,000 audio clips
• Location: data/buet/train_files/
• Metadata: data/buet/train.csv
• Description: Bangla speech dataset from BUET Deep Learning competition

Training Data: Due to the time-bounded nature of the assessment, fine-tuning uses a representative subset of 5,000 samples from each dataset (10,000 total). See Training Note for rationale.

---

iii. Dataset Preprocessing Steps

The prepare_data.py script performs the following:

1. Load CSV files from both dataset sources
2. Normalize audio paths to absolute paths
3. Add language labels (en for English, bn for Bangla)
4. Merge datasets into unified format with columns: audio_path, text, language
5. Split data into train/validation/test sets (70:15:15 ratio)
6. Save processed files to data/processed/

The datasets were originally released for training purposes, which is why they use "train" naming conventions. In this project, they are treated as raw corpora. A custom preparation step standardizes the data and creates train/validation/test splits solely for evaluation and reproducibility. No model training or fine-tuning is performed.

python prepare_data.py

---

iv. Model Architecture and Training Details

Hybrid Model Approach

I use a hybrid approach combining two pretrained models for optimal accuracy:

BanglaASR (for Bangla)

• Source: bangla-speech-processing/BanglaASR
• Base: Fine-tuned Whisper model
• Training Data: Mozilla Common Voice Bangla (~400 hours)
• Architecture: Transformer encoder-decoder

Whisper Base (for English)

• Source: OpenAI Whisper
• Training Data: 680,000 hours of multilingual audio
• Architecture: Transformer encoder-decoder

Language Detection Flow

1. Whisper analyzes audio mel spectrogram
2. If Bangla detected (or confidence > 15%) → Route to BanglaASR
3. If English detected → Route to Whisper
4. Fallback: If output contains Bengali script characters → Re-route to BanglaASR

Training Note

This project includes a fine-tuning pipeline for training a unified Whisper model on bilingual data.

Initial Attempt (Stopped)

The original plan was to fine-tune on 100,000 samples (50k English + 50k Bangla) with 5,000 training steps. However, this approach proved impractical:

• ~400k utterances across both datasets
• RTX 3060 (8GB VRAM) with batch size 2-4
• Evaluation every 500 steps alone cost ~4 hours
• Estimated total training time: 60+ hours

Time-Efficient Approach (Current)

To demonstrate the complete pipeline within a reasonable timeframe, the following adjustments were made:

Expected runtime: ~60-90 minutes on RTX 3060

What This Demonstrates

• ✅ Multilingual/domain mixing (English + Bangla)
• ✅ Loss convergence and stable optimization
• ✅ Correct evaluation wiring
• ✅ TensorBoard logging and checkpoint saving

Training Results

The fine-tuning completed in ~8 minutes with excellent loss convergence:

What Would Be Done With More Time

• Scale to 50,000+ samples per dataset
• Train for 5,000+ steps with proper learning rate scheduling
• Enable periodic evaluation every 1,000 steps
• Grid search over learning rates and batch sizes

"I trained on a representative subset to validate the pipeline and learning dynamics under time constraints."

Fine-tuning Details

• Base Model: openai/whisper-small
• Training Data: 5,000 English (Common Voice) + 5,000 Bangla (BUET) samples
• Output: Fine-tuned model saved to models/finetuned-whisper-assessment/

The app automatically detects and loads the fine-tuned model if available, otherwise falls back to pretrained models.

---

v. Optimization Techniques

• Audio Preprocessing: Resampled to 16kHz mono for consistency
• FP16 Inference: Half-precision on GPU when available for faster inference
• Model Caching: Both models loaded once at startup, reused for all requests
• Fallback Detection: Bengali script detection catches misclassified Bangla audio
• Confidence Threshold: Low Bangla confidence (>15%) triggers BanglaASR
• Temporary File Cleanup: Uploaded files deleted after processing

---

vi. Performance Evaluation Results

Run evaluation on test set:

python evaluate.py --test_csv data/processed/test.csv --max_samples 100

---

vii. Model Checkpoint Links

Both models are automatically downloaded on first run (~300MB total).

---

viii. How to Run

1. Install Dependencies

pip install -r requirements.txt

2. Prepare Data (Optional - for evaluation)

python prepare_data.py

3. Fine-tuning (Optional)

Prepare training data (5k samples per language for time-efficient training):

python prepare_finetune_data.py

For the full dataset (not recommended due to time constraints):

python prepare_finetune_data.py --cv_samples 50000 --bn_samples 50000

Start fine-tuning (~90 minutes on RTX 3060):

python finetune_whisper.py --fp16

Training outputs:

• Checkpoints saved to models/finetuned-whisper-assessment/checkpoint-500/
• TensorBoard logs for loss curve visualization
• View progress: tensorboard --logdir models/finetuned-whisper-assessment

Manual evaluation after training:

python evaluate.py --test_csv data/finetune/val.csv --model models/finetuned-whisper-assessment

4. Start FastAPI Backend

python app.py
# or with hot-reload:
uvicorn app:app --reload --port 8000

API will be available at http://localhost:8000

5. Start Streamlit Frontend

streamlit run streamlit_app.py

UI will be available at http://localhost:8501

---

Docker & MLflow

Note: Docker containers were not executed locally due to virtualization being disabled on the development machine (Docker Desktop requires hardware virtualization support). The Dockerfiles and docker-compose configuration are provided and verified for correctness, and the setup is expected to run on any standard Docker-enabled environment.

Running with Docker Compose

Start all services (backend, frontend, mlflow):

docker-compose up --build

Services will be available at:

• Backend API: http://localhost:8000
• Frontend UI: http://localhost:8501
• MLflow UI: http://localhost:5000

Stop all services:

docker-compose down

MLflow Experiment Tracking

Note: MLflow is used for evaluation tracking and reproducibility of inference experiments, not for full model training.

Run evaluation with MLflow logging enabled:

python evaluate.py --max_samples 50 --mlflow

With custom MLflow server:

python evaluate.py --max_samples 50 --mlflow --mlflow_uri http://localhost:5000

What gets logged to MLflow:

• Parameters: model name, samples evaluated
• Metrics: wer, cer, wacc, rtfx
• Artifacts: model info json
• Model Registry: auto-registers models with wacc >= 80%

View experiments at http://localhost:5000 after running evaluation.

---

ix. Example API Requests and Responses

POST /transcribe

Transcribe an audio file.

Request:

curl -X POST "http://localhost:8000/transcribe" \
  -F "[email protected]"

Response:

{
  "transcription": "Everything in the universe evolved, he said.",
  "language": "en",
  "latency_ms": 524.57,
  "duration_seconds": 3.5
}

GET /metrics

Get model performance metrics.

Request:

curl "http://localhost:8000/metrics"

Response:

{
  "wer": 8.54,
  "cer": 3.95,
  "wacc": 91.46,
  "rtfx": 4.15,
  "model": "banglaasr + whisper-base",
  "samples_evaluated": 400,
  "breakdown": {
    "bangla": {"model": "bangla-speech-processing/BanglaASR", "wer": 4.58, "wacc": 95.42},
    "english": {"model": "whisper-base", "wer": 12.5, "wacc": 87.5}
  }
}

GET /health

Check API health status.

Response:

{
  "status": "healthy"
}

---

x. Screenshots of Streamlit UI

English Transcription

Bangla Transcription

---

xi. Limitations and Future Improvements

Current Limitations

Bangla:

• Very short utterances may be misclassified as other languages
• Heavy background noise reduces accuracy

English:

• Accuracy decreases with heavy accents or fast speech
• Background noise affects word recognition

General:

• Long audio files (>5 min) have higher latency
• First run downloads ~300MB of model files
• Requires internet connection for initial model download

Future Improvements

• Real-time streaming transcription
• Support for additional South Asian languages
• Batch processing for multiple files
• Automatic punctuation and formatting

---

Project Structure

.
├── app.py               # FastAPI backend with hybrid model inference
├── streamlit_app.py     # Streamlit web frontend
├── inference.py         # Standalone CLI inference script
├── evaluate.py          # Model evaluation with MLflow logging
├── prepare_data.py      # Dataset preprocessing for evaluation
├── prepare_finetune_data.py  # Prepare fine-tuning data (5k + 5k samples)
├── finetune_whisper.py  # Whisper fine-tuning script (time-efficient config)
├── requirements.txt     # Python dependencies
├── Dockerfile.backend   # Docker image for backend
├── Dockerfile.frontend  # Docker image for frontend
├── docker-compose.yml   # Multi-container orchestration
├── images/              # Screenshots for documentation
├── mlruns/              # MLflow experiment data (auto-created)
├── mlartifacts/         # MLflow artifacts (auto-created)
└── data/
    ├── common_voice/    # English audio dataset
    ├── buet/            # Bangla audio dataset
    ├── processed/       # Preprocessed train/val/test splits
    └── finetune/        # Fine-tuning data (train.csv, val.csv)

---

Built for Technohaven Company Ltd. by Ahnaf Samin