MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark

[📜 Paper] • [🤗 HF Dataset] • [🐱 GitHub]

📢 News and Updates

[2024.12.01] 🔥We have initialized the repository.
[2024.12.16] 🔥We have added the evaluation results of Phi-4-14B and Llama-3.3-70B-Instruct.
[2024.12.20] 🔥We have released the validation dataset of MMLU-CF.
[2025.01.09] 🔥OpenCompass now support the MMLU-CF. Feel free to give it a try!
[2025.05.18] 🔥Accepted by ACL'25 (main).

1. The Motivation of MMLU-CF

• The open-source nature of these benchmarks and the broad sources of training data for LLMs have inevitably led to benchmark contamination, resulting in unreliable evaluation results. To alleviate this issue, we propose MMLU-CF.
• (a) An instance of leakage in MMLU. When questions are used as prompt from the MMLU, certain LLMs, due to their memorization capabilities, directly provide choices identical to the original ones. (b) When questions are used as prompt from the MMLU-CF, LLMs only provide guessed choices. This indicates that the MMLU test set suffers from data contamination and memorization by some LLMs, while the proposed MMLU-CF avoids such leakage.

2. How to Evaluate Your Models on the MMLU-CF Validation/Test Set

(1) We perform automated testing only on Huggingface models. After following the steps outlined below and obtaining the validation set results from OpenCompass, the test set results can then be accessed via GitHub Issues.

Step 1. Validation set evaluation: Obtaining the validation results for your model using LLM evaluation tools, OpenCompass. The validation data will be automatically loaded from Hugging Face.

• For a 5-shot evaluation with Internlm 2.5:

opencompass --models hf_internlm2_5_1_8b_chat --datasets mmlu_cf_few_shot --summarizer mmlu_cf

• For a 0-shot evaluation with Internlm 2.5:

opencompass --models hf_internlm2_5_1_8b_chat --datasets mmlu_cf_zero_shot --summarizer mmlu_cf

Step 2. Test set evaluation: With the validation results, submit a GitHub issue on the MMLU-CF GitHub homepage to request the test set results. Please follow the format below:

Example 1,

Title: 
Test set evaluation Request - add HF model [microsoft/phi-4]  
Content: 
The result on validation set: 68.5%

Example 2,

Notably:

• Ensure you use the format with square brackets [ ] as shown. The model name microsoft/phi-4 corresponds to the name on HuggingFace.
• We will automatically submit your model. The time to receive the results depends on the number of models being evaluated, but it typically takes 1-2 weeks.

(2) For API models, if OpenCompass updates the model interface, you can obtain the test set results by sending a temporary key to Email after receiving the validation set results.

3. What is the Difference between MMLU-CF and MMLU

MMLU focuses on the breadth and reasoning without considering contamination prevention. We apply three decontamination rules to mitigate unintentional data leakage while collecting data from a broader domain. Meanwhile, our MMLU-CF benchmark maintains the test set closed-source to prevent malicious data leakage.

4. Leaderboard

5. Data Construction Pipeline

The pipeline involves (1) MCQ Collection to gather a diverse set of questions; (2) MCQ Cleaning to ensure quality; (3) Difficulty Sampling to ensure an appropriate difficulty distribution for questions; (4) LLMs checking: The LLMs, including GPT-4o, Gemini, and Claude, are reviewing the accuracy and safety of the data; and (5) Contamination-Free Processing to prevent data leakage and maintain dataset purity. Ultimately, this process results in the MMLU-CF, consisting of 10,000 questions for the closed-source test set and 10,000 for the open-source validation set.

6. Contact

For any inquiries or concerns, feel free to reach out to us via Email: Qihao Zhao and Yangyu Huang.

7. Citation

@misc{zhao2024mmlucfcontaminationfreemultitasklanguage,
      title={MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark}, 
      author={Qihao Zhao and Yangyu Huang and Tengchao Lv and Lei Cui and Qinzheng Sun and Shaoguang Mao and Xin Zhang and Ying Xin and Qiufeng Yin and Scarlett Li and Furu Wei},
      year={2024},
      eprint={2412.15194},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.15194}, 
}

8. License

This repository is licensed under the MIT License. The validation dataset of MMLU-CF is subject to the CDLA-2.0 License.