OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI

OlympicArena is a comprehensive, highly-challenging, and rigorously curated benchmark featuring a detailed, fine-grained evaluation mechanism designed to assess advanced AI capabilities across a broad spectrum of Olympic-level challenges.

This benchmark encompasses seven disciplines: Mathematics, Physics, Chemistry, Biology, Geography, Astronomy, and Computer Science. Each discipline is divided into two splits: validation (val) and test. The validation split includes publicly available answers for small-scale testing and evaluation, while the test split does not disclose the answers, users could submit their results.

An Example to load the data

from datasets import load_dataset
dataset=load_dataset("GAIR/OlympicArena", "Math", split="val")

print(dataset[0])

More details on loading and using the data are at our github page.

If you do find our code helpful or use our benchmark dataset, please citing our paper.

@article{huang2024olympicarena,
      title={OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI},
      author={Zhen Huang and Zengzhi Wang and Shijie Xia and Xuefeng Li and Haoyang Zou and Ruijie Xu and Run-Ze Fan and Lyumanshan Ye and Ethan Chern and Yixin Ye and Yikai Zhang and Yuqing Yang and Ting Wu and Binjie Wang and Shichao Sun and Yang Xiao and Yiyuan Li and Fan Zhou and Steffi Chern and Yiwei Qin and Yan Ma and Jiadi Su and Yixiu Liu and Yuxiang Zheng and Shaoting Zhang and Dahua Lin and Yu Qiao and Pengfei Liu},
      year={2024},
      journal={arXiv preprint arXiv:2406.12753},
      url={https://arxiv.org/abs/2406.12753}
}