Efficient Low-Resource Transliteration for Urdu and Roman-Urdu

Optimizing CPU-GPU Training Pipelines with MinatoLoader

Authors: Fatima Zahra & Ramalah Amir — Data Science, FAST-NUCES Islamabad

---

Overview

Low-resource languages like Urdu remain heavily underexplored in NLP. This project fine-tunes M2M-100, a multilingual machine translation model, for Roman-Urdu ↔ Urdu transliteration — while tackling a critical but often overlooked bottleneck: the CPU-GPU communication gap during training.

The core contribution is integrating MinatoLoader, an asynchronous data loading framework that eliminates GPU idle time through dynamic batch scheduling, reducing training time from 35+ hours to ~30 hours on 6.3M sentence pairs.

---

Problem Statement

Fine-tuning large multilingual models like M2M-100 on low-resource language pairs is computationally expensive. A key bottleneck is head-of-line blocking (HOLB) — when variable-length sequences stall preprocessing, the GPU sits idle waiting for data. Most existing work assumes preprocessing is fully pre-done, which is not always feasible at scale.

---

Methodology

The system is designed as an asynchronous CPU-GPU pipeline:

• CPU handles tokenization, preprocessing, and dynamic batch construction
• GPU performs forward/backward passes and parameter updates on M2M-100
• MinatoLoader overlaps CPU preprocessing with GPU training using fast/slow queue routing to prevent slow sequences from blocking fast ones

Key Optimizations

---

Experiments & Results

All experiments conducted on Kaggle free-tier (dual NVIDIA T4 GPUs, 13GB RAM).

Training Time Comparison

MinatoLoader achieves ~38% speedup on 100k samples and reduces full-scale training from 385h to 55h.

Translation Quality (Roman-Urdu → Urdu)

Translation quality is identical across all pipeline configurations — the optimizations affect training efficiency only, not model performance.

---

Key Finding

MinatoLoader's benefit scales with dataset size. At 35k samples it introduces overhead (queue initialization costs), but at 100k+ samples it significantly outperforms standard PyTorch DataLoader — confirming it is most valuable for production-scale training.

---

Tech Stack

• Model: facebook/m2m100_418M (HuggingFace)
• Framework: PyTorch, HuggingFace Transformers
• Tokenizer: SentencePiece (M2M-100)
• Dataset: Roman Urdu Parl (6.3M sentence pairs)
• Environment: Kaggle free-tier (NVIDIA T4)
• Data Loader: MinatoLoader

---

Repository Structure

├── Minato/               # MinatoLoader integration and training scripts
├── notebooks/            # Experiment notebooks
├── i232616_i232644_paper.pdf  # Full research paper
└── README.md

---

References

• Butt et al. — Low-resource transliteration for Roman-Urdu and Urdu using transformer-based models
• Isenko et al. — Where is my training bottleneck? (SIGMOD '22)
• Nouaji et al. — MinatoLoader: Accelerating ML training through efficient data preprocessing (arXiv 2509.10712)
• Fan et al. — Beyond English-centric multilingual machine translation (M2M-100)