English to Tulu Translation using IndicTrans2

A neural machine translation system for translating English text to Tulu language using IndicTrans2 with LoRA (Low-Rank Adaptation) fine-tuning.

Overview

This project implements a production-ready translation system for English-to-Tulu translation, leveraging the power of IndicTrans2 (200M parameter model) fine-tuned on a custom Tulu dataset. The system includes both training infrastructure and a web application for real-time translation with confidence scoring and alternative predictions.

Key Features

• LoRA Fine-tuning: Efficient adaptation of IndicTrans2 using Parameter-Efficient Fine-Tuning (PEFT)
• Flash Attention 2: Optimized training and inference on modern GPUs
• Web Interface: Flask-based web application with real-time translation
• Database Integration: Track translation history and usage statistics
• Confidence Scoring: Quality estimates for each translation
• Alternative Predictions: Multiple translation candidates using beam search

Project Structure

LangTranslation/
├── data/
│   └── experimental_full.csv          # Training dataset (~10k sentence pairs)
├── indictrans2-200m-en-tulu/          # Fine-tuned LoRA adapters
│   ├── adapter_model.safetensors      # LoRA weights
│   ├── adapter_config.json            # LoRA configuration
│   └── checkpoint-*/                   # Training checkpoints
├── templates/
│   └── index_shadcn.html              # Web UI
├── exp_train.py                       # Initial training script
├── exp_continue_train.py              # Continue training from checkpoint
├── exp_flask_app.py                   # Flask web application
├── database.py                        # Database utilities
└── requirements.txt                   # Python dependencies

Approach

Model Architecture

Base Model: IndicTrans2 (ai4bharat/indictrans2-en-indic-dist-200M)

• 200M parameter Seq2Seq transformer model
• Pre-trained on 22 Indic languages
• Distilled version optimized for efficiency

Fine-tuning Strategy: LoRA (Low-Rank Adaptation)

• Rank (r): 64
• Alpha: 128
• Dropout: 0.1
• Target modules: All attention and FFN projections
• Trainable parameters: ~1.5% of base model (highly efficient)

Language Mapping:

• Source: English (eng_Latn)
• Target: Tulu using Kannada script (kan_Knda)

Training Configuration

Training Parameters:
- Batch size: 8 per device
- Gradient accumulation: 4 steps (effective batch size: 32)
- Learning rate: 3e-4 with linear decay
- Epochs: 10-30 (configurable)
- Precision: FP16 (half-precision)
- Optimization: Flash Attention 2 enabled

Dataset

• Size: ~10,197 English-Tulu sentence pairs
• Format: CSV with columns (Kannada, English, Tulu)
• Preprocessing: IndicProcessor handles normalization and script conversion
• Split: Training only (validation available in separate files)

Translation Pipeline

1. Input Preprocessing: IndicProcessor normalizes and adds language tags
2. Tokenization: IndicTrans2 tokenizer with 128 token limit
3. Generation: Beam search (1-5 beams) for best translations
4. Postprocessing: IndicProcessor handles script conversion and cleanup
5. Confidence Scoring: Heuristic-based quality estimation

Installation

Prerequisites

• Python 3.8+
• CUDA-capable GPU (recommended: RTX 3050+, 6GB+ VRAM)
• 16GB+ RAM
• Ubuntu/Linux (Windows WSL2 supported)

Step 1: Clone Repository

git clone <repository-url>
cd LangTranslation

Step 2: Create Environment

Using Conda (recommended):

conda env create -f environment.yml
conda activate translation-env

Or using venv:

python -m venv venv
source venv/bin/activate  # Linux/Mac
# venv\Scripts\activate   # Windows

Step 3: Install PyTorch

For CUDA 12.1+ (check with nvidia-smi):

conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

For CUDA 11.8:

conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

For CPU only (not recommended):

conda install pytorch torchvision torchaudio cpuonly -c pytorch

Step 4: Install Dependencies

pip install transformers==4.42.3
pip install datasets sentencepiece sacrebleu
pip install peft  # For LoRA support
pip install flash-attn --no-build-isolation  # Optional but recommended
pip install IndicTransToolkit  # IndicTrans2 preprocessing
pip install flask python-dotenv pandas tqdm

Or install all at once:

pip install -r requirements.txt
pip install peft flash-attn IndicTransToolkit

Step 5: Hugging Face Authentication

IndicTrans2 requires authentication:

# Login to Hugging Face
huggingface-cli login

# Or set token as environment variable
export HF_TOKEN="your_huggingface_token_here"

Get your token from: https://huggingface.co/settings/tokens

Access the model: https://huggingface.co/ai4bharat/indictrans2-en-indic-dist-200M

Usage

Training the Model

Initial Training (10 epochs)

python exp_train.py

This will:

• Load the IndicTrans2 base model
• Apply LoRA adapters
• Train on data/experimental_full.csv
• Save checkpoints to indictrans2-200m-en-tulu/
• Save final adapter to indictrans2-200m-en-tulu/

Training time: ~2-4 hours on RTX 4050 (6GB VRAM)

Continue Training (additional epochs)

python exp_continue_train.py

This resumes from the last checkpoint and trains for 10-20 more epochs.

Running the Web Application

python exp_flask_app.py

The Flask app will:

1. Load the base IndicTrans2 model
2. Load your trained LoRA adapter
3. Initialize the database
4. Start the server at http://localhost:5000

Web Interface Features:

• Real-time translation
• Confidence scores
• Alternative translations (beam search variants)
• Translation history
• Search functionality
• Usage statistics

API Endpoints

POST /translate

{
  "text": "Hello, how are you?",
  "include_details": true
}

GET /history?limit=50

GET /search?q=hello

GET /statistics

Training Configuration

Hyperparameters

Edit exp_train.py or exp_continue_train.py:

# Data
DATA_FILE = "data/experimental_full.csv"
COL_ENGLISH = "English"
COL_TULU = "Tulu"

# Model
MODEL_NAME = "ai4bharat/indictrans2-en-indic-dist-200M"
OUTPUT_DIR = "indictrans2-200m-en-tulu"

# Languages
SRC_LANG = "eng_Latn"  # English
TGT_LANG = "kan_Knda"  # Tulu (Kannada script)

# LoRA Config
r = 64                 # Rank
lora_alpha = 128       # Scaling factor
lora_dropout = 0.1     # Dropout rate

# Training
batch_size = 8         # Per device
gradient_accumulation = 4
learning_rate = 3e-4
num_epochs = 10

Memory Optimization

For 4GB VRAM GPUs:

per_device_train_batch_size = 2
gradient_accumulation_steps = 16

For 8GB+ VRAM GPUs:

per_device_train_batch_size = 16
gradient_accumulation_steps = 2

Model Performance

Training Progress

Loss steadily decreases indicating successful learning of English-Tulu patterns.

Inference Speed

• GPU (RTX 4050): ~0.5-1s per sentence
• CPU: ~5-10s per sentence

Troubleshooting

CUDA Out of Memory

Reduce batch size:

per_device_train_batch_size = 2
gradient_accumulation_steps = 16

Flash Attention Installation Issues

Skip Flash Attention (slower but works):

# In training script, remove:
attn_implementation="flash_attention_2"

Tokenizer Error: "multiple values for keyword argument"

Ensure you're loading tokenizer from base model, not adapter:

tokenizer = AutoTokenizer.from_pretrained(
    BASE_MODEL,  # Not ADAPTER_PATH
    trust_remote_code=True
)

Model Access Denied

1. Accept terms: https://huggingface.co/ai4bharat/indictrans2-en-indic-dist-200M
2. Login: huggingface-cli login
3. Set token: export HF_TOKEN="your_token"

Requirements Summary

Hardware

• GPU: 6GB+ VRAM (RTX 3050, 4050, or better)
• RAM: 16GB+ system memory
• Storage: 10GB for model + checkpoints

Software

• Python 3.8+
• PyTorch 2.0+ with CUDA
• Transformers 4.42+
• PEFT (for LoRA)
• IndicTransToolkit
• Flask (for web app)

Data

• Training data: CSV with English and Tulu columns
• ~10k sentence pairs minimum for decent results

Citation

If you use this work, please cite:

IndicTrans2:

@article{gala2023indictrans,
  title={IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages},
  author={Gala, Jay and others},
  journal={arXiv preprint arXiv:2305.16307},
  year={2023}
}

LoRA:

@article{hu2021lora,
  title={LoRA: Low-Rank Adaptation of Large Language Models},
  author={Hu, Edward J and others},
  journal={arXiv preprint arXiv:2106.09685},
  year={2021}
}

License

This project builds upon IndicTrans2 which is released under MIT License. Please refer to the original model's license terms.

Acknowledgments

• AI4Bharat for IndicTrans2 model
• Hugging Face for Transformers library
• Microsoft for PEFT/LoRA implementation

Contact

For questions or issues, please open an issue on GitHub or contact the maintainers.

---

Status: ✅ Production Ready | 🔄 Active Development | 📊 10k+ Training Samples