μ-MATH (Meta U-MATH) is a meta-evaluation dataset derived from the U-MATH benchmark.

It is intended to assess the ability of LLMs to judge free-form mathematical solutions.
The dataset includes 1,084 labeled samples generated from 271 U-MATH tasks, covering problems of varying assessment complexity.

For fine-grained performance evaluation results, in-depth analyses and detailed discussions on behaviors and biases of LLM judges, check out our paper.

• 📊 U-MATH benchmark at Huggingface
• 🔎 μ-MATH benchmark at Huggingface
• 🗞️ Paper
• 👾 Evaluation Code at GitHub

Key Features

• Dataset Construction:
  • Subset of U-MATH problems (25%).
  • Includes solutions generated by four top-performing LLMs: Llama-3.1 70B, Qwen2.5 72B, GPT-4o, Gemini 1.5 Pro
  • Solutions labeled as correct or incorrect with the use of math experts and formal auto-verification.
  • Samples validated by math experts at Toloka AI, Gradarius
• Focus: Meta-evaluation of LLMs as evaluators, testing their accuracy in judging free-form solutions.
• Primary Metric: Macro F1-score.
• Secondary Metrics: True Positive Rate, True Negative Rate, Positive Predictive Value, Negative Predictive Value.

For original tasks on mathematical problem-solving, refer to the U-MATH dataset.

Use it

from datasets import load_dataset
ds = load_dataset('toloka/mu-math', split='test')

Dataset Fields

uuid: problem id
problem_statement: problem formulation, written in natural language
golden_answer: a correct solution for the problem to compare the generated solutions against, written in natural language
model: name of the instruction-finetuned LLM that generated the solution
model_output: the LLM's solution
label: boolean flag on whether the generated solution is correct or not

Evaluation Results

The prompt used for judgment:

You'll be provided with a math problem, a correct answer for it and a solution for evaluation.
You have to answer whether the solution is correct or not.
---
PROBLEM STATEMENT:
{problem_statement}

CORRECT ANSWER:
{golden_answer}

SOLUTION TO EVALUATE:
{model_output}
---
Now please compare the answer obtained in the solution with the provided correct answer to evaluate whether the solution is correct or not.

Think step-by-step, following these steps, don't skip any:
1. Extract the answer from the provided solution
2. Make any derivations or transformations that may be necessary to compare the provided correct answer with the extracted answer
3. Perform the comparison
4. Conclude with your final verdict — put either "Yes" or "No" on a separate line

Licensing Information

• The contents of the machine-generated model_output column are subject to the underlying LLMs' licensing terms.
• Contents of all the other fields are available under the MIT license.

Citation

If you use U-MATH or μ-MATH in your research, please cite the paper:

@inproceedings{umath2024,
title={U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs},
author={Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov and Sergei Tilga},
year={2024}
}

Contact

For inquiries, please contact kchernyshev@toloka.ai