Releases:

• Version 3.0 (May 2026): GMMLU Lite 3.0 release with 5 new languages: Czech, Hungarian, Italian (updated), Oriya, Slovak and Tajik
• Version 2.0 (Dec 2025): GMMLU Lite 2.0 release with 3 new languages: Albanian, Burmese and Welsh
• Version 1.0 (Dec 2024): GMMLU Lite initial release with 15 languages.

Dataset Summary

Global-MMLU-Lite is a multilingual evaluation set spanning 23 languages, including English. It is "lite" version of the original Global-MMLU dataset 🌍. It includes 200 Culturally Sensitive (CS) and 200 Culturally Agnostic (CA) samples per language. All samples in Global MMLU lite are fully human translated or post edited.

NOTE: Of the 23 languages presently included in Global-MMLU-Lite, 15 are taken from Global-MMLU dataset whereas Albanian, Burmese, Welsh have been contributed by external collaborators or members of Cohere Labs Open Science Community. The evaluation results reported in our paper are corresponding to the 15 languages originally released as part of Global-MMLU-Lite.

• Curated by: Professional annotators and contributors of Cohere Labs Community
• Language(s): 23 languages
• License: Apache 2.0

Global-MMLU Dataset Family:

Load with Datasets

To load this dataset with datasets, you'll first need to install it using pip install datasets and then use the following code:

from datasets import load_dataset

# load HF dataset
gmmlu_lite = load_dataset("CohereLabs/Global-MMLU-Lite", 'en')

# can also be used as pandas dataframe
gmmlu_lite.set_format("pandas")
gmmlu_lite_test = gmmlu_lite['test'][:]
gmmlu_lite_dev = gmmlu_lite['dev'][:]

import ast

# convert string values to list
gmmlu_lite_test['required_knowledge'] = gmmlu_lite_test['required_knowledge'].apply(lamda x: ast.literal_eval(x))

Data Fields

The data fields are the same among all splits. Brief description of each field is provided below.

Data Splits

The following are the splits of the data:

Data Instances

An example from test set looks as follows:

{'sample_id': 'astronomy/test/58',
 'subject': 'astronomy',
 'subject_category': 'STEM',
 'question': 'When traveling north from the United States into Canada you’ll see the North Star (Polaris) getting _________.',
 'option_a': 'Brighter',
 'option_b': 'Dimmer',
 'option_c': 'Higher in the sky',
 'option_d': 'Lower in the sky',
 'answer': 'C',
 'required_knowledge': "['regional', 'regional', 'regional', 'regional']",
 'time_sensitive': "['No', 'No', 'No', 'No']",
 'reference': "[{'end': 55, 'label': 'Geographic', 'score': None, 'start': 5}, {'end': 43, 'label': 'Geographic', 'score': None, 'start': 30}, {'end': 55, 'label': 'Geographic', 'score': None, 'start': 5}, {'end': 43, 'label': 'Geographic', 'score': None, 'start': 30}]",
 'culture': '[]',
 'region': "['North America', 'North America', 'North America', 'North America']",
 'country': "['United States of America (USA)', 'United States of America (USA)', 'United States of America (USA)', 'United States of America (USA)']",
 'cultural_sensitivity_label': 'CS',
 'is_annotated': True
}

Statistics

Annotation Types

The following is the breakdown of CS🗽, CA⚖️ and MA📝 samples in the final dataset.

Languages

The dataset covers 23 languages. The following is details about the languages included in the dataset.

Known Limitations

A brief overview of limitations of this dataset is provided below.

Additional Information

Provenance

• Methods Used: Professional annotations as well as crowd-sourced through volunteer annotations.
• Methodology Details: We collected cultural bias annotations as well as post-edits of translations for different mmlu questions.
  • Cultural Sensitivity Annotation Platform
  • Translation Quality Annotation Platform
  • Dates of Collection: May 2024 - Aug 2024

Dataset Version and Maintenance

• Maintenance Status: Actively Maintained
• Version Details:
  • Current version: 3.0
  • Last Update: 05/2026
  • First Release: 12/2024

Authorship

• Publishing Organization: Cohere Labs
• Industry Type: Not-for-profit - Tech

Licensing Information

This dataset can be used for any purpose, under the terms of the Apache 2.0 License.

Continuous Improvement:

If you want to contribute to improving the quality of translations in Global-MMLU-Lite then please contribute using our annotation UI. You can also help review and edit machine translations in additional languages using our annotation interface to help improve language coverage of Global-MMLU-Lite.

Additional Details

For any additional details, please check our paper, Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation.

Citation Information

@misc{singh2024globalmmluunderstandingaddressing,
      title={Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation}, 
      author={Shivalika Singh and Angelika Romanou and Clémentine Fourrier and David I. Adelani and Jian Gang Ngui and Daniel Vila-Suero and Peerat Limkonchotiwat and Kelly Marchisio and Wei Qi Leong and Yosephine Susanto and Raymond Ng and Shayne Longpre and Wei-Yin Ko and Madeline Smith and Antoine Bosselut and Alice Oh and Andre F. T. Martins and Leshem Choshen and Daphne Ippolito and Enzo Ferrante and Marzieh Fadaee and Beyza Ermis and Sara Hooker},
      year={2024},
      eprint={2412.03304},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.03304}, 
}