TopXGen: Topic-Diverse Parallel Data for Low-Resource MT

Dataset Summary

This dataset is a synthetic parallel dataset for 10 low-resource languages, created by applying the TopXGen pipeline with recent multilingual LLMs. It is designed for machine translation (MT) fine-tuning and few-shot experiments (as a selection pool).
The pipeline works as follows:

1. Topic-diverse paragraph generation in the target low-resource language using an LLM (generator), with diversity controlled via topic selection and temperature settings.
2. Sentence splitting followed by translation/back-translation with a MT model (back-translator).
3. Redundancy removal similar to the self-instruct approach.

Models trained on this TopXGen dataset achieve translation performance close to that of the generator and back-translator. For more details, see our paper.

Supported Languages

• Basque (eus)
• Hausa (hau)
• Igbo (ibo)
• Kinyarwanda (kin)
• Nepali (nep)
• Somali (som)
• Sundanese (sun)
• Swahili (swh)
• Urdu (urd)
• Xhosa (xho)

Data Generation

• Generator: gemma-3-27b-it
• Back-translator: nllb-200-3.3B

Example Usage

from datasets import load_dataset

dataset = load_dataset("almanach/topxgen-gemma-3-27b-and-nllb-3.3b", split="Basque")
print(dataset)

Output

Dataset({
    features: ['source', 'target', 'source_language', 'target_language'],
    num_rows: 120031
})

Licensing

This dataset is derived from outputs of Google’s Gemma-3 and Meta’s NLLB. Users must comply with the licenses and usage guidelines of both models.