OlymMATH Evaluation Results

OlymMATH is a dataset we introduced in Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models by Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, Lei Fang, and Ji-Rong Wen. You can find more information on GitHub and HuggingFace 🤗.

We have made our evaluation results for the avg@{8, 64} and cons@{8, 64} metrics in this dataset publicly available for academic research purposes.

For DeepSeek R1, OpenAI o3 mini (high) and Gemini 2.5 Pro Exp 0325, we provided an additional column called reasoning_tokens, which was extracted from the API response body. This is because the API calls for the latter two models do not return the chain-of-thought content.

Live demonstrations are currently available on OlymMATH Demo, aiming to help the community better analyze the problem-solving patterns and characteristics of large language models. 🥳

Citation

If you find this helpful in your research, please give a 🌟 to our repo and consider citing

@misc{sun2025challengingboundariesreasoningolympiadlevel,
      title={Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models}, 
      author={Haoxiang Sun and Yingqian Min and Zhipeng Chen and Wayne Xin Zhao and Zheng Liu and Zhongyuan Wang and Lei Fang and Ji-Rong Wen},
      year={2025},
      eprint={2503.21380},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2503.21380}, 
}