CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding Capabilities

📌 CodeMMLU

CodeMMLU is a comprehensive benchmark designed to evaluate the capabilities of large language models (LLMs) in coding and software knowledge. It builds upon the structure of multiple-choice question answering (MCQA) to cover a wide range of programming tasks and domains, including code generation, defect detection, software engineering principles, and much more.

📄 Dataset Description

• Leaderboard: CodeMMLU
• Repository: CodeMMLU-Evaluator
• Paper: https://arxiv.org/abs/2410.01999

List of CodeMMLU subset:

• Syntactic test:
  • programming_syntax
  • api_frameworks
• Semantic test:
  • software_principles
  • dbms_sql
  • others
• Real-world problems:
  • code_completion
  • fill_in_the_middle
  • code_repair
  • defect_detection

📌 Citation

If you find this repository useful, please consider citing our paper:

@article{nguyen2024codemmlu,
  title={CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding Capabilities},
  author={Nguyen, Dung Manh and Phan, Thang Chau and Le, Nam Hai and Doan, Thong T. and Nguyen, Nam V. and Pham, Quang and Bui, Nghi D. Q.},
  journal={arXiv preprint},
  year={2024}
}