ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning

📄 Paper  |  💻 Code  |  📊 Dataset

ReasonMed is the largest open-source medical reasoning dataset to date, containing 370 K high-quality question–answer examples with multi-step chain-of-thought (CoT) rationales and concise summaries. We distilled these from 1.75 M initial reasoning paths generated by three competitive large-language models (Qwen-2.5-72B, DeepSeek-R1-Distill-Llama-70B, and HuatuoGPT-o1-70B), using a rigorous multi-agent verification and refinement pipeline.

---

📚 Dataset Composition

We sourced 194,925 unique multiple-choice medical questions from six established benchmarks, then generated and validated CoT paths:

---

🔍 Data Generation & Curation Pipeline

1. Multi-Agent CoT Generation

   • Three LLMs each generate 3 CoT trajectories per question at temperatures {0.7, 0.9, 1.0}, yielding 1.75 M raw paths.

2. Verification (Qwen-2.5-72B)

   • Judge each CoT for correctness, logical coherence, and medical factuality.
   • Label as “Correct” or “Error” with error reasons.

3. Difficulty Tiers & Refinement

   • Easy (0–4 errors): select top 2 CoTs via Quality Ranker.
   • Medium (5–7 errors): refine top 2 CoTs via Error Refiner (GPT-4o-mini).
   • Difficult (8–9 errors): regenerate full CoT via GPT-o1 with a 6-step template.

4. Summarization (GPT-4o-mini)

   • Condense each CoT into a concise answer rationale.

5. Final Dataset

   • Each dataset contains 370k pieces of data, for a total of 1.1M pieces of data:
     • ReasonMed({CoT}{response})
     • CoTMed({CoT})
     • ResponseMed({response})

---

📊 Data Quality Evaluation

Medium Pipeline Validity Verification

To evaluate the Medium Pipeline, we sampled 1 000 questions + CoTs and used our Score Evaluator to score before and after GPT-4o-mini corrections. The average score improved by 0.8 points.

Comparison with Other Medical Reasoning Corpora

We compared ReasonMed against two open datasets, sampling 1 000 instances each, and also evaluated 3 000 ReasonMed samples:

---

🎯 Multiscale Supervised Fine-Tuning Results

We fine-tuned Qwen2.5-7B under three regimes—CoT, Response, and hybrid Reason—over three epochs and one epoch. Evaluation on MedQA, MedMCQA, PubMedQA, and six MMLU subdomains yields:

Note:

• CK = Clinical Knowledge
• C-Bio = College Biology
• C-Med = College Medicine
• Med-Gene = Medical Genetics
• P-Med = Professional Medicine

• One-epoch vs Three-epoch: Three-epoch models outperform one-epoch variants (e.g., ReasonMed-7B improves from 67.7% to 69.6%)
• Token Length: CoTMed and ReasonMed generate deeper reasoning (≈555–626 tokens) vs ResponseMed (≈225 tokens).

---

Citation

@misc{sun2025reasonmed370kmultiagentgenerated,
      title={ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning}, 
      author={Yu Sun and Xingyu Qian and Weiwen Xu and Hao Zhang and Chenghao Xiao and Long Li and Yu Rong and Wenbing Huang and Qifeng Bai and Tingyang Xu},
      year={2025},
      eprint={2506.09513},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2506.09513}, 
}
@misc{lasateam2025lingshugeneralistfoundationmodel,
      title={Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning}, 
      author={LASA Team and Weiwen Xu and Hou Pong Chan and Long Li and Mahani Aljunied and Ruifeng Yuan and Jianyu Wang and Chenghao Xiao and Guizhen Chen and Chaoqun Liu and Zhaodonghui Li and Yu Sun and Junao Shen and Chaojun Wang and Jie Tan and Deli Zhao and Tingyang Xu and Hao Zhang and Yu Rong},
      year={2025},
      eprint={2506.07044},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2506.07044}, 
}