IndicIFEval is an evaluation dataset to assess the instruction-following capability of LLMs in Indic languages with verifiable, rule-based constraints.
Currently it supports 14 Indic languages, in addition to English.
| Assamese (as) | Nepali (ne) |
| Bengali (bn) | Odia (or) |
| Gujarati (gu) | Punjabi (pa) |
| Hindi (hi) | Sanskrit (sa) |
| Kannada (kn) | Tamil (ta) |
| Malayalam (ml) | Telugu (te) |
| Marathi (mr) | Urdu (ur) |
This repository contains the complete codebase for the IndicIFEval benchmark from Data Creation to Benchmark Evaluation.
This includes the scripts for indicifeval-trans creation, the pipeline for indicifeval-ground creation, and the custom evaluation configurations required for evaluation with lm-evaluation-harness.
Clone the repository and install the required dependencies. Please refer to the individual README for detailed usage.
The indicifeval-trans directory contains scripts to translate the English IFEval dataset into 14 Indic languages. Navigate to this directory and execute the main translation script to generate the localized prompts.
The indicifeval-ground directory houses the pipeline for synthetically generating instructions from native Indic content.
We use the Language Model Evaluation Harness for benchmarking. The lm-evaluation-harness directory contains the custom configurations required for our tasks. You must run the evaluation script specifying the model and the specific task configuration.
indicifeval-ground vs. indicifeval-trans constraint categoriesindicifeval-ground and indicifeval-trans do not use identical evaluation setups, even where a constraint category shares its name or checker code. In particular:
indicifeval-trans tests num_paragraphs and nth_paragraph/first_word together as independent, randomly-sized constraints. In indicifeval-ground, this category was simplified to match how the underlying source content was collected: num_paragraphs is implicitly fixed at 1 and only the first-word constraint is evaluated. Released indicifeval-ground rows therefore omit num_paragraphs from kwargs for this instruction — this is expected, not missing data.keywords:frequency, length_constraints:number_sentences). indicifeval-ground prompts for these two constraint types are always phrased as an exact count (e.g. "the word X must appear exactly N times"), reflecting how the source text was mined (documents where a word/sentence count matched a target exactly). The evaluation harness now honors this: when relation is omitted in kwargs for these two ground checkers, it defaults to an exact-equality comparison rather than randomly choosing "less than"/"at least". indicifeval-trans, by contrast, always supplies an explicit relation and keeps the original less-than/at-least semantics from IFEval.Not every translated prompt released in indicifeval-trans was judged correct by human review. Each row carries a tags list that includes exactly one of correct or incorrect (translation-quality verdict), and optionally parallel for the subset of rows that are aligned 1:1 across all 14 languages (used for the paper's cross-lingual comparison). To reproduce paper-reported numbers, filter to rows tagged correct (add parallel as well if you need the cross-lingual-aligned subset). Use for gold prompt or silver prompt for the indicifeval-ground release.
If you use IndicIFEval in your work, please cite us:
@article{jayakumar2026indicifeval,
title={IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages},
author={Thanmay Jayakumar and Mohammed Safi Ur Rahman Khan and Raj Dabre and Ratish Puduppully and Anoop Kunchukuttan},
year={2026},
eprint={2602.22125},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2602.22125},
}
This dataset is released under the CC BY 4.0.
IndicIFEval is an evaluation dataset to assess the instruction-following capability of LLMs in Indic languages with verifiable, rule-based constraints.
Currently it supports 14 Indic languages, in addition to English.
| Assamese (as) | Nepali (ne) |
| Bengali (bn) | Odia (or) |
| Gujarati (gu) | Punjabi (pa) |
| Hindi (hi) | Sanskrit (sa) |
| Kannada (kn) | Tamil (ta) |
| Malayalam (ml) | Telugu (te) |
| Marathi (mr) | Urdu (ur) |
This repository contains the complete codebase for the IndicIFEval benchmark from Data Creation to Benchmark Evaluation.
This includes the scripts for indicifeval-trans creation, the pipeline for indicifeval-ground creation, and the custom evaluation configurations required for evaluation with lm-evaluation-harness.
Clone the repository and install the required dependencies. Please refer to the individual README for detailed usage.
The indicifeval-trans directory contains scripts to translate the English IFEval dataset into 14 Indic languages. Navigate to this directory and execute the main translation script to generate the localized prompts.
The indicifeval-ground directory houses the pipeline for synthetically generating instructions from native Indic content.
We use the Language Model Evaluation Harness for benchmarking. The lm-evaluation-harness directory contains the custom configurations required for our tasks. You must run the evaluation script specifying the model and the specific task configuration.
indicifeval-ground vs. indicifeval-trans constraint categoriesindicifeval-ground and indicifeval-trans do not use identical evaluation setups, even where a constraint category shares its name or checker code. In particular:
indicifeval-trans tests num_paragraphs and nth_paragraph/first_word together as independent, randomly-sized constraints. In indicifeval-ground, this category was simplified to match how the underlying source content was collected: num_paragraphs is implicitly fixed at 1 and only the first-word constraint is evaluated. Released indicifeval-ground rows therefore omit num_paragraphs from kwargs for this instruction — this is expected, not missing data.keywords:frequency, length_constraints:number_sentences). indicifeval-ground prompts for these two constraint types are always phrased as an exact count (e.g. "the word X must appear exactly N times"), reflecting how the source text was mined (documents where a word/sentence count matched a target exactly). The evaluation harness now honors this: when relation is omitted in kwargs for these two ground checkers, it defaults to an exact-equality comparison rather than randomly choosing "less than"/"at least". indicifeval-trans, by contrast, always supplies an explicit relation and keeps the original less-than/at-least semantics from IFEval.Not every translated prompt released in indicifeval-trans was judged correct by human review. Each row carries a tags list that includes exactly one of correct or incorrect (translation-quality verdict), and optionally parallel for the subset of rows that are aligned 1:1 across all 14 languages (used for the paper's cross-lingual comparison). To reproduce paper-reported numbers, filter to rows tagged correct (add parallel as well if you need the cross-lingual-aligned subset). Use for gold prompt or silver prompt for the indicifeval-ground release.
If you use IndicIFEval in your work, please cite us:
@article{jayakumar2026indicifeval,
title={IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages},
author={Thanmay Jayakumar and Mohammed Safi Ur Rahman Khan and Raj Dabre and Ratish Puduppully and Anoop Kunchukuttan},
year={2026},
eprint={2602.22125},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2602.22125},
}
This dataset is released under the CC BY 4.0.