🌐 Multilingual News Classification using XLM-RoBERTa

---

📌 Table of Contents

• Problem Statement
• Dataset
• Data Preprocessing
• Models Implemented
• Results & Model Comparison
• Key Observations
• Challenges & Decisions
• Project Structure
• Quick Start
• Deployment
• Sample Inputs
• Features
• Future Work
• Team
• References

---

🎯 Problem Statement

Billions of news articles are published daily in Indian regional languages. Most existing classifiers either:

• Work only for English, or
• Require expensive and lossy translation as a preprocessing step

This project solves that. We built a single unified model that reads Telugu, Malayalam, Marathi, Tamil, and Gujarati news headlines natively and classifies them into 10 predefined categories — with no translation required.

Input:  "హైదరాబాద్‌లో క్రికెట్ టోర్నమెంట్ ప్రారంభమైంది"   (Telugu)
Output: 🏏 Sports  →  Confidence: 91.3%

Input:  "मुंबई शेअर बाजारात आज मोठी तेजी"               (Marathi)  
Output: 📈 Business  →  Confidence: 88.7%

---

📦 Dataset

HuggingFace Subset Names

🏷️ Target Categories (10 Classes)

---

🧹 Data Preprocessing

All text is processed through a Unicode-safe pipeline before being passed to any model.

def clean_text(text: str) -> str:
    text = re.sub(r"https?://\S+|www\.\S+", " ", text)   # Remove URLs
    text = re.sub(r"<[^>]+>", " ", text)                  # Remove HTML tags
    text = re.sub(r"[\u200b\u200c\u200d\ufeff\u00ad]", "", text)  # Zero-width chars
    text = re.sub(
        r"[^\w\s"
        r"\u0900-\u097F"   # Devanagari  (Hindi, Marathi)
        r"\u0C00-\u0C7F"   # Telugu
        r"\u0D00-\u0D7F"   # Malayalam
        r"\u0B80-\u0BFF"   # Tamil
        r"\u0A80-\u0AFF"   # Gujarati
        r"]", " ", text
    )
    return re.sub(r"\s+", " ", text).strip()

What the pipeline does:

⚠️ No stemming or lemmatization — XLM-RoBERTa's SentencePiece tokenizer handles morphology natively, making these steps unnecessary and potentially harmful.

---

🧠 Models Implemented

Three progressively powerful approaches are compared — each one motivated by the limitations of the previous.

---

1️⃣ TF-IDF + Logistic Regression (Baseline)

The baseline establishes the minimum achievable performance without any deep learning.

Why TF-IDF with character n-grams?
Word-level TF-IDF fails on Indic languages because the same root word produces dozens of inflected forms due to agglutinative morphology. Character n-grams (1–3) capture shared sub-word patterns across these forms, making the approach far more robust across all five scripts.

TfidfVectorizer(
    analyzer    = 'char_wb',    # Word-boundary-aware character n-grams
    ngram_range = (1, 3),       # Unigrams, bigrams, trigrams
    max_features= 80_000,       # Top 80k most discriminative n-grams
    sublinear_tf= True,         # log(1 + tf) — prevents common chars dominating
    min_df      = 2,            # Ignore n-grams in < 2 documents
)

LogisticRegression(
    C            = 1.0,
    max_iter     = 1000,
    class_weight = 'balanced',  # Corrects for class imbalance
    solver       = 'lbfgs',
    multi_class  = 'multinomial',
)

---

2️⃣ Bidirectional LSTM

The LSTM introduces sequential modelling — reading text word-by-word and remembering context.

Why Bidirectional?
In Indic languages, the verb often appears at the end of a sentence (subject-object-verb order). A standard left-to-right LSTM misses this. A Bidirectional LSTM reads the sentence in both directions simultaneously and combines both representations.

Architecture:
  Embedding(60000+1, 128, mask_zero=True)
  → Bidirectional LSTM(128, return_sequences=True)
  → Dropout(0.3)
  → Bidirectional LSTM(64, return_sequences=True)
  → GlobalMaxPooling1D()         ← captures strongest signal across all timesteps
  → Dense(128, activation='relu')
  → Dropout(0.3)
  → Dense(10, activation='softmax')

Hyperparameters:
  Vocab size   : 60,000   (increased for 5-language vocabulary)
  Max length   : 150      (longer to accommodate Indic articles)
  Embedding dim: 128
  Batch size   : 32
  Epochs       : 15 (with EarlyStopping, patience=3)
  Optimizer    : Adam (lr=2e-4)

---

3️⃣ XLM-RoBERTa — Final Model 🚀

XLM-RoBERTa is a 125M-parameter transformer pre-trained by Meta AI on 2.5TB of text across 100 languages — including all five of our target languages.

Why XLM-RoBERTa over mBERT or IndicBERT?
XLM-R was trained on significantly more data with a larger vocabulary (250K SentencePiece tokens) and consistently outperforms mBERT on multilingual benchmarks (XNLI, XQuAD). Its SentencePiece tokenizer handles Indic scripts natively without any special character pre-processing.

Fine-tuning Strategy — Partial Layer Unfreezing:

xlm-roberta-base (12 transformer layers)
  ├── Layers 0–9  : FROZEN   ← preserve pre-trained multilingual knowledge
  ├── Layers 10–11: TRAINABLE ← adapt to our news domain
  └── Classifier head (linear, 10 outputs): TRAINABLE

Why freeze most layers?
  - Training all 125M params on ~26k samples causes catastrophic forgetting
  - Frozen layers = fast training (only ~3M params updated)
  - Last 2 layers + head = sufficient for task-specific adaptation

🔗 Link to the Model and its files: huggingface.co/Jaykumardas/Multilingual_News_Model

Training Configuration:

TrainingArguments(
    num_train_epochs              = 5,
    per_device_train_batch_size   = 16,
    learning_rate                 = 2e-5,
    weight_decay                  = 0.01,
    warmup_ratio                  = 0.1,    # Gradual LR warmup — prevents early instability
    evaluation_strategy           = "epoch",
    load_best_model_at_end        = True,
    metric_for_best_model         = "f1_macro",
    fp16                          = True,   # Mixed precision — 2× faster on GPU
)

---

📊 Results & Model Comparison

Summary Table

XLM-RoBERTa outperforms the baseline by +2.28% accuracy and +0.90% F1 Macro.
LSTM underperforms the baseline — see Challenges for explanation.

---

       Test Accuracy (%)          F1 Macro (%)
       ┌─────────────────┐        ┌─────────────────┐

TF-IDF+LR ████████████ 83.84 ████████████ 77.85 BiLSTM ███████████ 79.36 ████████ 67.16 XLM-R ⭐ █████████████ 86.12 ████████████ 78.75 └─────────────────┘ └─────────────────┘

---

---

🔍 Key Observations

1. Transformer > Traditional ML > LSTM for multilingual classification. XLM-R's pre-training on 100 languages gives it a head start that neither TF-IDF features nor an LSTM trained from scratch can match.

2. Character n-grams save the baseline. Word-level TF-IDF performed ~9% worse than character n-grams on Indic text due to morphological richness. This alone closed much of the gap to the LSTM.

3. LSTM underperformed the baseline — not a bug, but an expected outcome for multilingual data with a relatively small training set. LSTMs need far more data to learn cross-lingual representations from scratch. XLM-R has this baked in.

4. Topic-based categories are easiest. Sports, entertainment, and tamil-cinema have rich domain-specific vocabularies that all three models learn well. Sentiment classes are hardest because sentiment is expressed differently across languages and contexts.

5. 5% warmup ratio was critical. Without gradual learning rate warmup, XLM-R training showed instability in the first epoch and final accuracy dropped by ~3%. Warmup protects pre-trained weights during the initial batches.

6. Layer freezing cut training time by ~60% with less than 1% accuracy loss vs. full fine-tuning — validating partial layer freezing as the right strategy for this dataset size.

---

⚠️ Challenges & Decisions — Honest Engineering Notes

These are real problems encountered during development, not textbook descriptions.

---

1. Wrong Label Mapping → Biased Predictions

What happened: Early predictions were almost always "state" regardless of input. Accuracy was stuck below 20%.

Root cause: We manually wrote a {0: "entertainment", 1: "business", ...} dictionary based on guesswork. The actual integer-to-label mapping in the HuggingFace dataset was completely different.

Fix: Extract the label map directly from the dataset's ClassLabel feature:

label_names = dataset['train'].features['label'].names
id2label    = {i: name for i, name in enumerate(label_names)}

Lesson: Never hardcode label mappings. Always read them from the data source.

---

2. Should We Reduce to 3 Classes?

The temptation: Merge all 10 categories into 3 broader groups (Topic / Sentiment / Other) to make the problem easier.

Why we didn't:

• The dataset's 10-class structure is meaningful and well-curated — collapsing it loses real information
• XLM-RoBERTa handles 10-class classification without any special tricks
• The assignment objective explicitly required demonstrating multilingual multi-class capability
• Reducing classes would have hidden the real challenge: distinguishing semantically similar categories across scripts

Decision: Keep all 10 classes. Let the model learn the harder problem.

---

3. Class Imbalance in Sentiment Categories

What happened: F1 scores for positive, negative, and neutral were 10–15 points lower than topic categories. The model learned to avoid predicting these classes under uncertainty.

Mitigations applied:

• class_weight='balanced' in Logistic Regression — weights each class inversely proportional to its frequency
• Relied on XLM-R's pre-trained representations for the final model — transformers are more robust to imbalance than shallow models
• Did not oversample (SMOTE etc.) because text oversampling on multilingual data adds more noise than signal

Honest assessment: Class imbalance in sentiment classes remains the biggest gap in this system. Addressed in Future Work.

---

4. Model Files Too Large for Standard Git

What happened: model.safetensors (~1.1GB) exceeded GitHub's 100MB file limit and HuggingFace Space's direct upload limit.

Solution — three-repo separation (industry standard pattern):

GitHub (this repo) → Source code, app.py, requirements.txt HuggingFace Model Hub → model.safetensors, config.json, tokenizer files HuggingFace Spaces → Gradio app (loads model from Hub at startup)

Why this pattern:

• GitHub stays lightweight and fast to clone
• Model versioning is handled by HuggingFace's Git LFS
• Space can be updated without touching model weights

---

5. label_map.json Worked Locally, Failed in HuggingFace Space

What happened: App ran perfectly in Kaggle. After deploying to HuggingFace Spaces, it crashed on startup with a FileNotFoundError.

Root cause: Local code used os.path.exists("./xlmr/label_map.json"). In HuggingFace Spaces, files are fetched from a remote Git repo — os.path.exists returns False for remote paths.

Fix: Use hf_hub_download() to explicitly pull the file from the Hub:

from huggingface_hub import hf_hub_download
path = hf_hub_download(repo_id="YourUsername/your-model", filename="label_map.json")
with open(path) as f:
    id2label = {int(k): v for k, v in json.load(f)["id2label"].items()}

Lesson: Local file path assumptions always break in containerised deployments. Use the Hub SDK.

---

6. LSTM Underperformed the Baseline

What happened: BiLSTM scored 79.36% vs TF-IDF's 83.84% — a deep learning model lost to a 50-year-old algorithm.

Why:

• 26k samples is genuinely small for training multilingual embeddings from scratch
• The Keras tokenizer treats each Indic script's tokens independently — it has no concept of cross-lingual similarity
• The LSTM's 60k vocabulary is dominated by high-frequency tokens from majority languages, leaving minority-language tokens under-represented

Why XLM-R doesn't have this problem: It starts with 2.5TB of pre-trained cross-lingual knowledge. Fine-tuning on 26k samples only needs to teach it domain adaptation, not language understanding from scratch.

Lesson: Raw dataset size is not the bottleneck for transformers. It is for LSTMs.

---

🎯 Final Takeaways

"Correct data handling beats model complexity. Deployment issues are as real as training issues. And for multilingual NLP in 2025, XLM-RoBERTa is the right starting point."

---

📁 Project Structure

multilingual-news-classification/
│
├── 📄 app.py                    # Gradio UI — main entry point for HuggingFace Spaces
├── 📄 requirements.txt          # All Python dependencies
├── 📄 README.md                 # This file
│
├── 📂 src/
│   ├── preprocess.py            # Text cleaning pipeline (Unicode-safe)
│   ├── baseline_model.py        # TF-IDF + Logistic Regression
│   ├── lstm_model.py            # Bidirectional LSTM (TensorFlow/Keras)
│   ├── transformer_model.py     # XLM-RoBERTa fine-tuning (HuggingFace Trainer)
│   ├── evaluate.py              # Metrics, confusion matrix, comparison charts
│   └── main.py                  # CLI orchestrator — runs full pipeline
│
├── 📂 notebooks/
│   └── multilingual_news.ipynb  # Complete Kaggle notebook (all phases in one)
│
├── 📂 outputs/
│   ├── model_comparison.png     # Bar chart: accuracy + F1 across all models
│   ├── confusion_matrix_*.png   # Per-model confusion matrices
│   └── training_curves_lstm.png # BiLSTM loss/accuracy curves
│
└── 📂 models/                   # ⚠️ NOT in this repo — hosted on HuggingFace Hub
    ├── xlmr/                    # → HuggingFace Model: YourUsername/indic-news-xlmr
    │   ├── config.json
    │   ├── model.safetensors
    │   ├── tokenizer.json
    │   └── label_map.json       # id → label name mapping
    └── baseline/
    │    ├── tfidf_vectorizer.pkl
    │    └── logistic_regression.pkl
    └── lstm/
        ├── lstm_model.keras
        └── tokenizer.pkl



    

---

⚡ Quick Start

Option A — Run the Gradio App Locally

# 1. Clone the repository
git clone https://github.com/YourUsername/multilingual-news-classification.git
cd multilingual-news-classification

# 2. Install dependencies
pip install -r requirements.txt

# 3. Set your HuggingFace model repo in app.py
#    Change: HF_MODEL_REPO = "YourUsername/indic-news-xlmr"

# 4. Launch the app
python app.py
# → Open http://localhost:7860

Option B — Run Training Pipeline (Kaggle / GPU machine)

# Run the full pipeline (preprocessing → all 3 models → evaluation)
python main.py --data data/news_dataset.csv --mode all --out_dir outputs/

# Run only the transformer
python main.py --mode xlmr

# Run only the baseline
python main.py --mode baseline

Option C — Use the Hosted Demo

Visit the live HuggingFace Space — no installation required:
🔗 https://huggingface.co/spaces/YourUsername/indic-news-classifier

---

Requirements

# requirements.txt
torch>=2.0.0
transformers>=4.35.0
datasets>=2.14.0
sentencepiece>=0.1.99
gradio>=4.0.0
scikit-learn>=1.3.0
tensorflow>=2.13.0
numpy>=1.24.0
pandas>=2.0.0
matplotlib>=3.7.0
seaborn>=0.12.0
huggingface_hub>=0.19.0
joblib>=1.3.0

---

🚀 Deployment

The project follows a clean three-component deployment pattern:

┌─────────────────────────────────────────────────────────────────┐
│                     DEPLOYMENT ARCHITECTURE                     │
├──────────────────┬──────────────────┬───────────────────────────┤
│   GitHub Repo    │  HF Model Hub    │    HuggingFace Space      │
│  (Source Code)   │  (Model Weights) │    (Live Gradio UI)       │
├──────────────────┼──────────────────┼───────────────────────────┤
│ app.py           │ config.json      │ Loads model from Hub      │
│ requirements.txt │ model.safetensors│ Auto-scales on CPU/GPU    │
│ src/ modules     │ tokenizer.json   │ Public URL, zero setup    │
│ README.md        │ label_map.json   │ Free tier supported       │
└──────────────────┴──────────────────┴───────────────────────────┘
         ↕ git push           ↕ hf_hub_download()      ↕ iframe embed

Gradio UI Features

---

🧪 Sample Inputs

Try these in the live demo:

---

✨ Features

• 🌐 True multilingual support — 5 Indic scripts, no translation needed
• ⚡ Real-time inference — ~50–200ms per headline (CPU), ~20ms (GPU)
• 📊 Confidence visualisation — horizontal bar chart per prediction
• 📋 Batch predictions — classify up to 50 headlines simultaneously with summary pie chart
• 🏷️ 10-class classification — topic and sentiment categories
• 🧹 Unicode-safe preprocessing — handles all 5 Indic scripts without stripping valid characters
• 🎨 Clean dark-themed UI — built with Gradio, no installation needed for end users
• 🔌 Modular codebase — each pipeline phase is a standalone, testable module

---

🔮 Future Work

---

👥 Team

🎓 B.Tech AI & ML · Chaitanya Bharathi Institute of Technology, Hyderabad
📧 Guided by Mr. Panigrahi Srikanth, Assistant Professor, Dept. of AIML

---

📚 References

| 1 | IndicGLUE Dataset | HuggingFace | | 2 | Abid et al. (2019) — Gradio: Hassle-Free Sharing and Testing of ML Models | arXiv |

---