MILU: A Multi-task Indic Language Understanding Benchmark

Overview

MILU (Multi-task Indic Language Understanding Benchmark) is a comprehensive evaluation dataset designed to assess the performance of Large Language Models (LLMs) across 11 Indic languages. It spans 8 domains and 41 subjects, reflecting both general and culturally specific knowledge from India.

This repository contains code for evaluating language models on the MILU benchmark using the lm-eval-harness framework.

Usage

Prerequisites

• Python 3.7+
• lm-eval-harness library
• HuggingFace Transformers
• vLLM (optional, for faster inference)

1. Clone this repository:

git clone --depth 1 https://github.com/AI4Bharat/MILU.git
cd MILU
pip install -e .

2. Request access to the HuggingFace 🤗 dataset here.

3. Set up your environment variables:

export HF_HOME=/path/to/HF_CACHE/if/needed
export HF_TOKEN=YOUR_HUGGINGFACE_TOKEN

The following languages are supported for MILU:

• Bengali
• English
• Gujarati
• Hindi
• Kannada
• Malayalam
• Marathi
• Odia
• Punjabi
• Tamil
• Telugu

HuggingFace Evaluation

For HuggingFace models, you may use the following sample command:

lm_eval --model hf \
    --model_args 'pretrained=google/gemma-2-27b-it,temperature=0.0,top_p=1.0,parallelize=True' \
    --tasks milu \
    --batch_size auto:40 \  
    --log_samples \
    --output_path $EVAL_OUTPUT_PATH \
    --max_batch_size 64 \
    --num_fewshot 5 \
    --apply_chat_template

vLLM Evaluation

For vLLM-compatible models, you may use the following sample command:

lm_eval --model vllm \
    --model_args 'pretrained=meta-llama/Llama-3.2-3B,tensor_parallel_size=$N_GPUS' \
    --gen_kwargs 'temperature=0.0,top_p=1.0' \
    --tasks milu \
    --batch_size auto \
    --log_samples \
    --output_path $EVAL_OUTPUT_PATH

Single Language Evaluation

To evaluate your Model on a specific language, modify the --tasks parameter:

--tasks milu_English

Replace English with the available language (e.g., Odia, Hindi, etc.).

Evaluation Tips & Observations

1. Make sure to use --apply_chat_template for Instruction-fine-tuned models, to format the prompt correctly.
2. vLLM generally works better with Llama models, while Gemma models work better with HuggingFace.
3. If vLLM encounters out-of-memory errors, try reducing max_gpu_utilization else switch to HuggingFace.
4. For HuggingFace, use --batch_size=auto:<n_batch_resize_tries> to re-select the batch size multiple times.
5. When using vLLM, pass generation kwargs in the --gen_kwargs flag. For HuggingFace, include them in model_args.

Key Features

• 11 Indian Languages: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu, and English
• Domains: 8 diverse domains including Arts & Humanities, Social Sciences, STEM, and more
• Subjects: 41 subjects covering a wide range of topics
• Questions: ~80,000 multiple-choice questions
• Cultural Relevance: Incorporates India-specific knowledge from regional and state-level examinations

Dataset Statistics

Dataset Structure

Test Set

The test set consists of the MILU (Multi-task Indic Language Understanding) benchmark, which contains approximately 85,000 multiple-choice questions across 11 Indic languages.

Validation Set

The dataset includes a separate validation set of 8,933 samples that can be used for few-shot examples during evaluation. This validation set was created by sampling questions from each of the 41 subjects.

Subjects spanning MILU

Citation

If you use MILU in your work, please cite us:

@inproceedings{verma-etal-2025-milu,
    title = "{MILU}: A Multi-task {I}ndic Language Understanding Benchmark",
    author = "Verma, Sshubam  and
      Khan, Mohammed Safi Ur Rahman  and
      Kumar, Vishwajeet  and
      Murthy, Rudra  and
      Sen, Jaydeep",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.naacl-long.507/",
    doi = "10.18653/v1/2025.naacl-long.507",
    pages = "10076--10132",
    ISBN = "979-8-89176-189-6",
    abstract = "Evaluating Large Language Models (LLMs) in low-resource and linguistically diverse languages remains a significant challenge in NLP, particularly for languages using non-Latin scripts like those spoken in India. Existing benchmarks predominantly focus on English, leaving substantial gaps in assessing LLM capabilities in these languages. We introduce MILU, a Multi-task Indic Language Understanding Benchmark, a comprehensive evaluation benchmark designed to address this gap. MILU spans 8 domains and 41 subjects across 11 Indic languages, reflecting general and culturally specific knowledge. With an India-centric design, incorporates material from regional and state-level examinations, covering topics such as local history, arts, festivals, and laws, alongside standard subjects like science and mathematics. We evaluate over 42 LLMs, and find that current LLMs struggle with MILU, with GPT-4o achieving the highest average accuracy at 74 percent. Open multilingual models outperform language-specific fine-tuned models, which perform only slightly better than random baselines. Models also perform better in high resource languages as compared to low resource ones. Domain-wise analysis indicates that models perform poorly in culturally relevant areas like Arts and Humanities, Law and Governance compared to general fields like STEM. To the best of our knowledge, MILU is the first of its kind benchmark focused on Indic languages, serving as a crucial step towards comprehensive cultural evaluation. All code, benchmarks, and artifacts are publicly available to foster open research."
}

License

This dataset is released under the CC BY 4.0.

Contact

For any questions or feedback, please contact:

• Sshubam Verma (sshubamverma@ai4bharat.org)
• Mohammed Safi Ur Rahman Khan (safikhan@ai4bharat.org)
• Rudra Murthy (rmurthyv@in.ibm.com)
• Vishwajeet Kumar (vishk024@in.ibm.com)

Links

• GitHub Repository 💻
• Paper 📄
• Hugging Face Dataset 🤗