Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning

Code: https://github.com/jincan333/MAS-TTS Project page: https://github.com/jincan333/MAS-TTS

The M500 dataset is a curated collection of 500 challenging, interdisciplinary problems designed to evaluate and improve multi-agent collaboration and reasoning in large language models (LLMs). Each sample includes a full trace of interactions among multiple specialized agents solving a complex task collaboratively.

Related Research

This dataset is referenced in the survey paper A Survey of Deep Learning for Geometry Problem Solving. Its GitHub repository provides a continuously updated list of papers on the topic: https://github.com/majianz/gps-survey

Sample Usage

from datasets import load_dataset

dataset = load_dataset("Can111/m500")
print(dataset)
# Output:
# DatasetDict({
#     train: Dataset({
#         features: ['question', 'solution', 'expert_recruiter_messages', 'problem_solver_messages', 'critic_messages', 'evaluator_messages'],
#         num_rows: 500
#     })
# })

print(dataset["train"][0])
# Output (example of content structure):
# {
#   "question": "...",
#   "solution": "...",
#   "expert_recruiter_messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}],
#   "problem_solver_messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}],
#   "critic_messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}],
#   "evaluator_messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}],
# }

Dataset Structure

Each entry in the dataset contains:

{
  "question": "...",
  "solution": "...",
  "expert_recruiter_messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}],
  "problem_solver_messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}],
  "critic_messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}],
  "evaluator_messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}],
}

Data Fields

• question (string): The problem statement.
• solution (string): The ground truth solution for the question.
• expert_recruiter_messages (list of dict): A sequence of messages representing the dialogue involving the Expert Recruiter agent. Each message has role (string) and content (string).
• problem_solver_messages (list of dict): A sequence of messages representing the dialogue involving the Problem Solver agent(s). Structure is the same as expert_recruiter_messages.
• critic_messages (list of dict): A sequence of messages representing the dialogue involving the Critic agent. Structure is the same as expert_recruiter_messages.
• evaluator_messages (list of dict): A sequence of messages representing the dialogue involving the Evaluator agent. Structure is the same as expert_recruiter_messages.

Data Splits

The dataset contains a single split: train, consisting of 500 examples.

Source Data

• Questions and Solutions: Sampled from SimpleScaling)
• Reasoning Traces: Generated using DeepSeek-R1) on AgentVerse

Citation

If you use this dataset, please cite the relevant papers:

@article{jin2025two,
  title={Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning},
  author={Jin, Can and Peng, Hongwu and Zhang, Qixin and Tang, Yujin and Metaxas, Dimitris N and Che, Tong},
  journal={arXiv preprint arXiv:2504.09772},
  year={2025}
}