Data Augmentation for Low-Resource African NLP

Comparing LLM-based Augmentation vs Back-Translation for Hausa and Fongbe

This project investigates whether synthetic data augmentation improves NLP model performance for low-resource African languages. We compare two augmentation strategies across three tasks: Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and Sentiment Analysis.

---

Table of Contents

• Research Questions
• Languages & Hypothesis
• Tasks & Datasets
• Project Structure
• Installation
• Quick Start Guide
• Running Experiments
• Expected Results
• Troubleshooting
• Citation

---

Research Questions

1. RQ1: Does data augmentation improve NER/POS/Sentiment performance for Hausa and Fongbe?
2. RQ2: Which method works better: LLM-generated synthetic data or back-translation?
3. RQ3: Does the effectiveness vary by language and task?

---

Languages & Hypothesis

*Based on prior SIGIR paper findings on LLM translation quality for African languages.

Key Hypothesis: High translation quality → augmentation helps; Low quality → may introduce noise.

---

Tasks & Datasets

NER Labels

O, B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, B-DATE, I-DATE

POS Labels (Universal Dependencies)

NOUN, VERB, ADJ, ADV, PROPN, PRON, DET, ADP, NUM, CONJ, PUNCT, ...

Sentiment Labels

positive, negative, neutral

---

Experimental Conditions

---

Project Structure

augmentation_tasks_fon_ha/
│
├── configs/
│   └── experiment_config.yaml     # Model settings, API config
│
├── data/
│   ├── raw/                       # Downloaded benchmark datasets
│   │   ├── masakhaner2_hau/       # Hausa NER (train/val/test.jsonl)
│   │   ├── masakhaner2_fon/       # Fongbe NER
│   │   ├── masakhane_pos_hau/     # Hausa POS
│   │   └── masakhane_pos_fon/     # Fongbe POS
│   │
│   ├── original/
│   │   └── sentiment/hausa/       # AfriSenti sentiment data
│   │
│   └── synthetic/                 # Generated augmented data
│       ├── llm/                   # LLM-generated (hausa_ner.jsonl, etc.)
│       └── backtrans/             # Back-translated examples
│
├── scripts/
│   ├── download_datasets.py       # Download NER + POS data
│   └── download_snt_dataset.py    # Download sentiment data
│
├── src/
│   ├── utils.py                   # Gemini setup, data loading, helpers
│   ├── augment_llm.py             # LLM-based augmentation (Gemini API)
│   ├── augment_backtrans.py       # Back-translation (NLLB-200)
│   ├── train_ner.py               # NER model training
│   ├── train_pos.py               # POS model training
│   ├── train_sentiment.py         # Sentiment model training
│   ├── evaluate.py                # Aggregate results & statistics
│   └── analyze_comparative.py     # Generate tables/plots
│
├── experiments/
│   ├── run_all.sh                 # Full experiment pipeline
│   ├── run_augmentation.sh        # Generate all synthetic data
│   ├── run_ner.sh                 # NER experiments (16 runs + cleanup)
│   ├── run_pos.sh                 # POS experiments (16 runs + cleanup)
│   └── run_senti.sh               # Sentiment experiments (8 runs + cleanup)
│
├── results/                       # Experiment outputs
│   ├── ner/{lang}_{condition}_seed{N}/
│   ├── pos/
│   ├── sentiment/
│   └── analysis/                  # Aggregated tables & plots
│
├── tests/
│   └── test_pipeline.py           # Validation tests (16 tests)
│
├── requirements.txt               # Python dependencies
├── .env                           # API keys (GEMINI_API_KEY, HF_TOKEN)
└── README.md

---

Installation

System Requirements

• Python: 3.9+
• GPU: Recommended (NVIDIA with CUDA). CPU works but is slower.
• Disk Space: ~50GB per task (checkpoints are auto-cleaned)
• RAM: 16GB+ recommended

Step 1: Clone and Setup Environment

# Navigate to project
cd augmentation_tasks_fon_ha

# Create virtual environment
python -m venv venv
source venv/bin/activate  # Linux/Mac
# venv\Scripts\activate   # Windows

# Install dependencies
pip install -r requirements.txt

Step 2: Configure API Keys

Create a .env file with your API keys:

# Get a free Gemini API key at: https://aistudio.google.com/
GEMINI_API_KEY=your_gemini_api_key_here

# Optional: HuggingFace token for faster downloads
HF_TOKEN=your_hf_token_here

Step 3: Download Datasets

# Download NER and POS data (MasakhaNER + MasakhaPOS)
python scripts/download_datasets.py

# Download sentiment data (AfriSenti - Hausa only)
python scripts/download_snt_dataset.py

Step 4: Verify Setup

# Run tests (should show 16/16 passed)
python tests/test_pipeline.py

---

Quick Start Guide

Generate Augmented Data

# LLM-based augmentation (uses Gemini API)
python src/augment_llm.py -l hau -t ner -n 500        # Hausa NER
python src/augment_llm.py -l fon -t ner -n 500        # Fongbe NER
python src/augment_llm.py -l hau -t pos -n 500        # Hausa POS
python src/augment_llm.py -l fon -t pos -n 500        # Fongbe POS
python src/augment_llm.py -l hau -t sentiment -n 500  # Hausa Sentiment

# Back-translation (uses NLLB-200, slower ~1 sec/example)
python src/augment_backtrans.py -l hau -t ner -n 500
python src/augment_backtrans.py -l fon -t ner -n 500

Train a Single Model

# Baseline (no augmentation)
python src/train_ner.py --language hau --condition baseline --seed 42

# With LLM augmentation
python src/train_ner.py --language hau --condition llm --seed 42

# With back-translation
python src/train_ner.py --language hau --condition backtrans --seed 42

Analyze Results

python src/evaluate.py
# Results saved to: results/analysis/

---

Running Experiments

Full Experiment Matrix

Using Experiment Scripts

Each script runs all experiments for a task and automatically cleans up checkpoints to save disk space:

# Run NER experiments (16 runs, ~5 hours on GPU)
bash experiments/run_ner.sh

# Run POS experiments (16 runs, ~5 hours on GPU)
bash experiments/run_pos.sh

# Run Sentiment experiments (8 runs, ~3 hours on GPU)
bash experiments/run_senti.sh

# Generate final analysis
python src/evaluate.py

Phased Execution (Recommended for Limited Disk Space)

If disk space is limited (<100GB free), run experiments in phases:

# Phase 1: NER (~51GB, auto-cleanup after)
bash experiments/run_ner.sh

# Phase 2: POS (~51GB, auto-cleanup after)
bash experiments/run_pos.sh

# Phase 3: Sentiment (~26GB, auto-cleanup after)
bash experiments/run_senti.sh

# Final: Generate analysis
python src/evaluate.py

Time & Storage Estimates

Note: Scripts automatically clean up model checkpoints after completion, keeping only test_results.json files.

---

How It Works

1. Data Augmentation

LLM Augmentation (augment_llm.py)

Original Examples → Few-shot Prompt → Gemini API → NEW Synthetic Examples

• Uses Gemini 2.5 Flash to generate completely new training examples
• Provides 5 real examples as context in the prompt
• Generates token-level annotations (NER/POS tags) or sentiment labels

Back-Translation (augment_backtrans.py)

Original Sentence → NLLB (→English) → NLLB (→Hausa/Fongbe) → Paraphrased Sentence

• Uses Facebook's NLLB-200-distilled-600M multilingual translation model
• Creates paraphrased versions that preserve meaning
• Labels are projected using positional alignment

2. Model Training

All tasks use AfroXLMR-base (Davlan/afro-xlmr-base):

• XLM-RoBERTa pretrained on African languages
• Better baseline than generic multilingual models
• Fine-tuned with HuggingFace Transformers

Training flow:

1. Load original data from data/raw/ or data/original/
2. If augmentation: append synthetic data from data/synthetic/
3. Tokenize with subword alignment for token classification
4. Fine-tune for N epochs
5. Evaluate on test set, save results JSON

3. Evaluation

evaluate.py collects all test_results.json files and:

• Computes mean ± std across seeds
• Runs paired t-tests for significance
• Generates LaTeX tables and bar plots

---

Training Parameters

---

Expected Results

NER Task (F1 Score)

Key Finding

"LLM-based augmentation significantly improves NER performance for Hausa (+3% F1), but shows no benefit for Fongbe, likely due to lower translation quality for this language."

---

Output Files

After running experiments:

results/
├── ner/
│   ├── hausa_baseline_seed42/test_results.json
│   ├── hausa_llm_seed42/test_results.json
│   └── ...
├── pos/
│   └── ...
├── sentiment/
│   └── ...
└── analysis/
    ├── summary_table.csv      # Copy-paste to Excel/Sheets
    ├── summary_table.tex      # LaTeX for paper
    ├── significance_tests.csv # p-values for comparisons
    └── plots/
        ├── ner_comparison.pdf
        └── pos_comparison.pdf

---

Troubleshooting

"GEMINI_API_KEY not found"

# Option 1: Export directly
export GEMINI_API_KEY="your-key-here"

# Option 2: Create .env file
echo 'GEMINI_API_KEY=your-key' > .env

"ModuleNotFoundError: No module named 'google.genai'"

pip install google-genai

"'NoneType' object has no attribute 'replace'" (NLLB tokenizer)

This is fixed in the latest version. If you encounter it:

pip install --upgrade transformers

"CUDA out of memory"

# Reduce batch size
python src/train_ner.py --batch_size 8 ...

"Dataset scripts are no longer supported"

The sentiment download script has been updated to use direct file downloads. Pull the latest version.

Back-translation is slow

• NLLB runs at ~1 second per example on CPU
• Use --num 100 for quick testing
• Consider using LLM augmentation instead (faster with API)

Tests failing

# Upgrade dependencies
pip install --upgrade huggingface_hub transformers datasets google-genai

---

Citation

@inproceedings{yourname2026augmentation,
  title={When Does Synthetic Data Help? Comparing Augmentation Methods
         for Low-Resource African NLP},
  author={Your Name},
  booktitle={Proceedings of KDD},
  year={2026}
}

Dataset Citations

@inproceedings{adelani2022masakhaner,
  title={MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition},
  author={Adelani, David Ifeoluwa and others},
  booktitle={EMNLP},
  year={2022}
}

@inproceedings{dione2023masakhapos,
  title={MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languages},
  author={Dione, Cheikh M Bamba and others},
  booktitle={ACL},
  year={2023}
}

@inproceedings{muhammad2023afrisenti,
  title={AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages},
  author={Muhammad, Shamsuddeen Hassan and others},
  booktitle={EMNLP},
  year={2023}
}

---

License

MIT License