Measuring Multimodal Mathematical Reasoning with the MATH-Vision Dataset

[💻 Github] [🌐 Homepage] [📊 Main Leaderboard ] [📊 Open Source Leaderboard ] [🌿 Wild Leaderboard ] [🔍 Visualization] [📖 Paper]

---

🌿 NEW: MATH-Vision-Wild

MATH-Vision-Wild is a photographic, real-world variant of MATH-Vision. The same testmini problems are physically captured on printed paper, iPads, laptops, and projectors under varying lighting and angles — the conditions VLMs actually face when a user holds up a phone to a math problem.

📦 Dataset: MathLLMs/MathVision-Wild · 🏆 Leaderboard: mathllm.github.io/mathvision/#wildleaderboard

Key finding — almost every model regresses in the wild:

Only o4-mini improves when problems are photographed; long-reasoning models degrade less than fast/non-thinking models. A full 25-model comparison with MATH-Vision-Screenshot and Δ% columns is on the Wild Leaderboard.

from datasets import load_dataset

wild = load_dataset("MathLLMs/MathVision-Wild", split="testmini_photo")
screenshot = load_dataset("MathLLMs/MathVision-Wild", split="testmini_screenshot")
photo_full = load_dataset("MathLLMs/MathVision-Wild", split="test_photo")  # 3,040 photos

---

🚀 Data Usage

from datasets import load_dataset

dataset = load_dataset("MathLLMs/MathVision")
print(dataset)

🙏 Acknowledgments

We would like to thank the following contributors for helping improve the dataset quality:

• @Zhiqi Huang for correcting answers for ID 21, ID 691, ID 2522 and ID 2524
• @Big-Brother-Pikachu for correcting answers for ID 338 and ID 1826

💥 News

• [2026.04.24] 🌿🔥🔥 MATH-Vision-Wild released! A photographic variant that re-captures MATH-Vision problems on printed paper, iPads, laptops, and projectors. o4-mini is the only model whose accuracy improves in the wild (57.2%, +2.33% vs baseline); every other VLM — including Gemini 2.5 Pro, GPT-4.1, and Qwen2.5-VL-72B — regresses, often by 20-50%. Dataset: MathLLMs/MathVision-Wild · Leaderboard: #wildleaderboard
• [2025.05.16] 💥 We now support the official open-source leaderboard! 🔥🔥🔥 Skywork-R1V2-38B is the best open-source model, scoring 49.7% on MATH-Vision. 🔥🔥🔥 MathCoder-VL-2B is the best small model on MATH-Vision, scoring 21.7%. See the full open-source leaderboard.
• [2025.05.16] 🤗 MathCoder-VL-2B, MathCoder-VL-8B and FigCodifier-8B is available now! 🔥🔥🔥
• [2025.05.16] Our MathCoder-VL is accepted to ACL 2025. 🔥🔥🔥
• [2025.05.13] 🔥🔥🔥 Seed1.5-VL achieves 68.7% on MATH-Vision! 🎉 Congratulations!
• [2025.04.11] 💥 Kimi-VL-A3B-Thinking achieves strong multimodal reasoning with just 2.8B LLM activated parameters! Congratulations! See the full leaderboard.
• [2025.04.10] 🔥 SenseNova V6 Reasoner achieves 55.39% on MATH-Vision! 🎉 Congratulations!
• [2025.04.05] 💥 Step R1-V-Mini 🥇 Sets New SOTA on MATH-V with 56.6%! See the full leaderboard.
• [2025.03.10] 💥 Kimi k1.6 Preview Sets New SOTA on MATH-V with 53.29%! See the full leaderboard.
• [2025.02.28] 💥 Doubao-1.5-pro Sets New SOTA on MATH-V with 48.62%! Read more on the Doubao blog.
• [2025.01.26] 🚀 Qwen2.5-VL-72B achieves 38.1%, establishing itself as the best-performing one in open-sourced models. 🎉 Congratulations!
• [2025.01.22] 💥 Kimi k1.5 achieves new SOTA on MATH-Vision with 38.6%! Learn more at the Kimi k1.5 report.
• [2024-09-27] MATH-V is accepted by NeurIPS DB Track, 2024! 🎉🎉🎉
• [2024-08-29] 🔥 Qwen2-VL-72B achieves new open-sourced SOTA on MATH-Vision with 25.9! 🎉 Congratulations! Learn more at the Qwen2-VL blog.
• [2024-07-19] open-compass/VLMEvalKit now supports MATH-V, utilizing LLMs for more accurate answer extraction!🔥
• [2024-05-19] OpenAI's GPT-4o scores 30.39% on MATH-V, considerable advancement in short time! 💥
• [2024-03-01] InternVL-Chat-V1-2-Plus achieves 16.97%, establishing itself as the new best-performing open-sourced model. 🎉 Congratulations!
• [2024-02-23] Our dataset is now accessible at huggingface.
• [2024-02-22] The top-performing model, GPT-4V only scores 23.98% on MATH-V, while human performance is around 70%.
• [2024-02-22] Our paper is now accessible at ArXiv Paper.

👀 Introduction

Recent advancements in Large Multimodal Models (LMMs) have shown promising results in mathematical reasoning within visual contexts, with models approaching human-level performance on existing benchmarks such as MathVista. However, we observe significant limitations in the diversity of questions and breadth of subjects covered by these benchmarks. To address this issue, we present the MATH-Vision (MATH-V) dataset, a meticulously curated collection of 3,040 high-quality mathematical problems with visual contexts sourced from real math competitions. Spanning 16 distinct mathematical disciplines and graded across 5 levels of difficulty, our dataset provides a comprehensive and diverse set of challenges for evaluating the mathematical reasoning abilities of LMMs.

The accuracies of four prominent Large Multimodal Models (LMMs), random chance, and human
performance are evaluated on our proposed MATH-Vision (MATH-V) across 16 subjects.

Through extensive experimentation, we unveil a notable performance gap between current LMMs and human performance on MATH-V, underscoring the imperative for further advancements in LMMs.

You can refer to the project homepage for more details.

🏆 Leaderboard

The leaderboard is available here.

We are commmitted to maintain this dataset and leaderboard in the long run to ensure its quality! 🔔 If you find any mistakes, please paste the question_id to the issue page, we will modify it accordingly.

📐 Dataset Examples

Some examples of MATH-V on three subjects: analytic geometry, topology, and graph theory.

Analytic geometry

Topology

Graph Geometry

📑 Citation

If you find this benchmark useful in your research, please consider citing this BibTex:

@inproceedings{
wang2024measuring,
title={Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset},
author={Ke Wang and Junting Pan and Weikang Shi and Zimu Lu and Houxing Ren and Aojun Zhou and Mingjie Zhan and Hongsheng Li},
booktitle={The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2024},
url={https://openreview.net/forum?id=QWTCcxMpPA}
}

@inproceedings{
wang2025mathcodervl,
title={MathCoder-{VL}: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning},
author={Ke Wang and Junting Pan and Linda Wei and Aojun Zhou and Weikang Shi and Zimu Lu and Han Xiao and Yunqiao Yang and Houxing Ren and Mingjie Zhan and Hongsheng Li},
booktitle={The 63rd Annual Meeting of the Association for Computational Linguistics},
year={2025},
url={https://openreview.net/forum?id=nuvtX1imAb}
}