MedXpertQA is a highly challenging and comprehensive benchmark designed to evaluate expert-level medical knowledge and advanced reasoning capabilities. It features both text-based and multimodal question-answering tasks, with the multimodal subset leveraging structured clinical information alongside images.
MedXpertQA comprises 4,460 questions spanning diverse medical specialties, tasks, body systems, and image types. It includes two subsets:
Key features:
For more details, please refer to our preprint, GitHub repository, and project page.
The following is an example from MedXpertQA MM:
id: Question ID (e.g., "MM-26").question: Question text with formatted answer choices.options: Answer choices (5 for MM, 10 for Text).label: Correct answer choice (single letter).images: List of image filenames (MM subset only). Images are provided in images.zip.medical_task: Main medical task (Diagnosis, Treatment, or Basic Medicine).body_system: Relevant body system.question_type: Question type (Reasoning or Understanding).{
"id": "MM-26",
"question": "A 70-year-old female patient seeks medical attention with complaints of dizziness and widespread rash that developed over the past week, following a viral respiratory infection. Physical examination reveals a generalized, macular, purplish rash that does not blanch with pressure. What is the most probable diagnosis?
Answer Choices: (A) Erythema infectiosum (B) Cutaneous larva migrans (C) Cold agglutinin disease (D) Cutis marmorata (E) Erythema ab igne",
"options": {
"A": "Erythema infectiosum",
"B": "Cutaneous larva migrans",
"C": "Cold agglutinin disease",
"D": "Cutis marmorata",
"E": "Erythema ab igne"
},
"label": "C",
"images": ["MM-26-a.jpeg"],
"medical_task": "Diagnosis",
"body_system": "Lymphatic",
"question_type": "Reasoning"
}
Each subset (Text and MM) contains dev.jsonl (development set) and test.jsonl (test set).
If you find our work helpful, please use the following citation.
@article{zuo2025medxpertqa,
title={MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding},
author={Zuo, Yuxin and Qu, Shang and Li, Yifei and Chen, Zhangren and Zhu, Xuekai and Hua, Ermo and Zhang, Kaiyan and Ding, Ning and Zhou, Bowen},
journal={arXiv preprint arXiv:2501.18362},
year={2025}
}
MedXpertQA is a highly challenging and comprehensive benchmark designed to evaluate expert-level medical knowledge and advanced reasoning capabilities. It features both text-based and multimodal question-answering tasks, with the multimodal subset leveraging structured clinical information alongside images.
MedXpertQA comprises 4,460 questions spanning diverse medical specialties, tasks, body systems, and image types. It includes two subsets:
Key features:
For more details, please refer to our preprint, GitHub repository, and project page.
The following is an example from MedXpertQA MM:
id: Question ID (e.g., "MM-26").question: Question text with formatted answer choices.options: Answer choices (5 for MM, 10 for Text).label: Correct answer choice (single letter).images: List of image filenames (MM subset only). Images are provided in images.zip.medical_task: Main medical task (Diagnosis, Treatment, or Basic Medicine).body_system: Relevant body system.question_type: Question type (Reasoning or Understanding).{
"id": "MM-26",
"question": "A 70-year-old female patient seeks medical attention with complaints of dizziness and widespread rash that developed over the past week, following a viral respiratory infection. Physical examination reveals a generalized, macular, purplish rash that does not blanch with pressure. What is the most probable diagnosis?
Answer Choices: (A) Erythema infectiosum (B) Cutaneous larva migrans (C) Cold agglutinin disease (D) Cutis marmorata (E) Erythema ab igne",
"options": {
"A": "Erythema infectiosum",
"B": "Cutaneous larva migrans",
"C": "Cold agglutinin disease",
"D": "Cutis marmorata",
"E": "Erythema ab igne"
},
"label": "C",
"images": ["MM-26-a.jpeg"],
"medical_task": "Diagnosis",
"body_system": "Lymphatic",
"question_type": "Reasoning"
}
Each subset (Text and MM) contains dev.jsonl (development set) and test.jsonl (test set).
If you find our work helpful, please use the following citation.
@article{zuo2025medxpertqa,
title={MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding},
author={Zuo, Yuxin and Qu, Shang and Li, Yifei and Chen, Zhangren and Zhu, Xuekai and Hua, Ermo and Zhang, Kaiyan and Ding, Ning and Zhou, Bowen},
journal={arXiv preprint arXiv:2501.18362},
year={2025}
}