🏥 Extended Multilingual Multimodal Medical Exam Dataset for Visual Question Answering in Healthcare

---

The Extended Multilingual Multimodal Medical Exam Dataset (Extended MMMED) is the new, larger release of MMMED for evaluating Vision-Language Models (VLMs) on medical multiple-choice question answering (MCQA) tasks.

Compared to the original benchmark, this extension substantially increases the number of questions and updates the benchmark with 28 tested VLMs (general-purpose, medical-specialized, and closed-source) across Spanish, English, and Italian.

The dataset includes challenging, real-world medical content from Médico Interno Residente (MIR - Spain) and Scuole di Specializzazione in Medicina (SSM - Italy) exam settings, with heterogeneous diagnostic images and clinically grounded questions.

🔓 How to Access the Dataset

You can access the dataset via Hugging Face. Follow these steps to download it:

⚠️ Disclaimer: This dataset contains medical images that may be sensitive for some users. Viewer discretion is advised, especially if the content may evoke strong emotional reactions or be distressing.

Login using e.g. huggingface-cli login to access this dataset

from datasets import load_dataset

# Login using e.g. `huggingface-cli login` to access this dataset
ds = load_dataset("praiselab-picuslab/MMMED", split="extended")

🌟 Key Features:

• Languages: 🇪🇸 Spanish, 🇬🇧 English, 🇮🇹 Italian
• Scale: 955 questions per language (2,865 total samples)
• Medical Content: Questions based on Spanish residency exam material
• Image Types: Diagnostic medical images (e.g., CT scans, X-rays)
• Categories: 26 medical categories per language
• Multimodal: Each question comes with a medical image 📸
• Benchmarking: 28 VLMs evaluated in multilingual settings

🔄 Dataset Workflow

Here is the general workflow for building the MMMED dataset for Vision-Language Model (VLM) evaluation:

📊 Dataset Overview

The Extended MMMED benchmark contains 955 questions for each language and is organized into 26 medical categories per language. The table below reports updated corpus statistics used in the new study.

* Token counts are computed with the preprocessing pipeline used in this repository (SpaCy-based analysis notebooks).

🖼️ Image Types

Categorization of Image Types in the Extended MMMED Dataset. This figure presents the four main categories of images included in the dataset and their respective distributions.

✨ Example MMCQA

Each multimodal multiple-choice question-answer (MMCQA) pair integrates three essential components with the following structure:

• Category: $C$
• Question: $Q$
• Image URL: $I$
• Answer Options: $O$
• Correct Answer: 💡

Here’s an illustrative example of multimodal QA in three languages:

🔍 VLMs Evaluated in the Extended Benchmark (28 Models)

The following table reports architecture details for all tested models.

📈 VLM Performance on MMMED

The following figure presents the overall multilingual performance trend.

For complete analysis outputs (tables and publication-quality figures), see:

• Analysis/analysis_output/tables/accuracy_table.csv
• Analysis/analysis_output/tables/summary_table.csv
• Analysis/analysis_output/figures/

🖋️ Original MMMED Citation

Please cite also the original work as follows:

@inproceedings{riccio2025multilingual,
  title={A Multilingual Multimodal Medical Examination Dataset for Visual Question Answering in Healthcare},
  author={Riccio, Giuseppe and Romano, Antonio and Barone, Mariano and Orlando, Gian Marco and Russo, Diego and Postiglione, Marco and La Gatta, Valerio and Moscato, Vincenzo},
  booktitle={2025 IEEE 38th International Symposium on Computer-Based Medical Systems (CBMS)},
  pages={435--440},
  year={2025},
  organization={IEEE Computer Society}
}

🌐 Notes

Dataset Usage: The dataset is intended for academic and research purposes only. It is not recommended for clinical decision-making or commercial use.

👨‍💻 This project was developed by Mariano Barone, Francesco Di Serio, Giuseppe Riccio, Antonio Romano, Vincenzo Moscato, and Marco Postiglione University of Naples, Federico II

📜 License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.