FreedomIntelligence/Med-MAT

Dataset

18

stars

11

commits

7

linked in READMEs

Nov 28, 2025

updated

README

Med-MAT: On the Compositional Generalization of Multimodal LLMs for Medical Imaging

✨ Latest News

⚡ Introduction

Welcome to the repository of Med-MAT, a VQA dataset consisting of 106 open-source medical datasets, which we hope will advance generalization experiments and aid in training powerful medical multimodal large language models (MLLMs).

Through this dataset, we have demonstrated that Compositional Generalization (CG) is one of the key mechanisms for MLLMs to understand unseen images, enabling them to handle unfamiliar images and achieve data-efficient training.

Here is a list of what has been released:

  1. QA Pairs for 106 Medical Datasets: Image-label pairs converted into VQA pairs for MLLM training.
  2. QA Pairs for 53 Aggregated Subsets: Datasets categorized by Modality, Anatomical Area, and Task (MAT), with identical entries merged into subsets.
  3. Image Download Links: Some datasets cannot be shared due to licensing. Users can download them to specified directories.

💭 QA Pairs Construction

To enable MLLMs to directly train and test on Med-MAT, the image-label pairs were converted into a Visual Question-Answering (VQA) format. The process involves the following steps:

  1. Task Definition: Each subset was manually assigned 6 instructions to guide the MLLM in answering the task related to the subset.
  2. Conversion to VQA Format: All image-label pairs were converted into single-choice questions with up to four answer options.
  3. Distractor Selection: Distractor options were randomly drawn from other labels within the subset to ensure variety.
  4. Final Dataset: The resulting dataset consisted of VQA pairs, where each image is paired with a question and four options, one of which is correct.

📚 Data

You can access the QA pairs of Med-MAT in this page.

The tables below record the download URLs for the images and QA pairs for each dataset and subset. If you only wish to use part of Med-MAT, you can selectively download the corresponding data.

Original_Medical_Datasets

Click to view the details of 106 Medical Datasets
No.Name with linkModalityAreaTaskQA
1Intel and MobileODT Cervical ScreeningCoCervixCervix Type in ScreeningHF
2CT Kindney DatasetCTKidneyNormal or Cyst or TumorHF
3SARS-COV-2 Ct-ScanCTLungCOVID19, Classification DatasetHF
4COVID CT COVID-CTCTLungCOVID19, Classification Dataset.HF
5Chest CT-ScanCTLungCancer, 3 Cancer Categories, Multiple Classification DatasetHF
6COVID-19-CT SCAN IMAGESCTLungCOVID19, ClassificationHF
7Head CTCTBrainHead HemorrhageHF
8CT of BrainCTBrainHead CancerHF
9MED-NODEDerSkinMelanoma or NaevusHF
10ISIC 2020DerSkinMelanoma, Benign or MalignantHF
11PED-UFES-20DerSkinSkin Multi ClassificationHF
12Web-scraped Skin ImageDerSkinSkin Desease Multi ClassificationHF
13ISBI 2016DerSkinSkin Lesion ClassificationHF
14ISIC 2019DerSkinSkin Desease Multi ClassificationHF
15Skin Cancer ISICDerSkinSkin Cancer Multi ClassificationHF
16Dental Condition DatasetDPTeethTeeth condition classificationHF
17Oral Cancer DatasetDPTeethOral cancer ClassificationHF
18The Nerthus DatasetEndIntestineCleanliness levelHF
19Endoscopic Bladder TissueEndBladderCanser Degree ClassificationHF
20KvasirEndIntestineMulti Disease ClassificationHF
21ACRIMAFPFundusGlaucomaHF
22Augemnted ocular diseases AODFPFundusMulti Classification of eye diseasesHF
23JSIECFPFundusMulti Classification of eye diseasesHF
24Multi-Label Retinal DiseasesFPFundusMulti Classification of eye diseasesHF
25RFMiD 2.0FPFundusMulti Classification of eye diseasesHF
26ToxoFundus(Data Processed Paper)FPFundusOcular toxoplasmosisHF
27ToxoFundus(Data Raw 6class All)FPFundusOcular toxoplasmosisHF
28Adam datasetFPFundusAge-related Macular DegenerationHF
29APTOS 2019 BlindnessFPFundusBlindness Level Identification 0~4HF
30DRIMBDFPFundusQuality Testing of Retinal ImagesHF
31Glaucoma DetectionFPFundusGlaucoma ClassificationHF
32AIROGSFPFundusGlaucoma ClassificationHF
33ICPR-HEp-2MicCellMulti ClassificationHF
34SICAPv2MicCellCancer Degree ClassificationHF
35Blood Cell ImagesMicCellBlood Cell Classificaion (Multi)HF
36BreakHisMicCellCell type and beginormagHF
37ChaoyangMicCellMulti Classification of pathologistsHF
38HuSHeMMicCellSperm Head Morphology ClassificaionHF
39Bone Marrow Cell ClassificationMicCellBone Marrow Cell ClassificationHF
40NCT-CRC-HE-100KMicCellMulti ClassificationHF
41Malignant Lymphoma ClassificationMicCellMulti ClassificationHF
42Histopathologic Cancer DetectionMicCellCancer ClassificationHF
43LC25000MicCellMulti Classification of Lung and ColonHF
44Brain Tumor 17 ClassesMRIBrainMulti ClassificationHF
45Tumor ClassificationMRIBrainPituitary or Glioma or Meningioma or NotumorHF
46Malignant Lymphoma ClassificationOCTRetinaMulti Classification of eye diseasesHF
47Retinal OCT-C8OCTRetinaMulti Classification of eye diseasesHF
48BUSIUSBreastBreast CancerHF
49Digital Knee X-Ray ImagesX-RayBonesDegree Classification of KneeHF
50Bone Fracture Multi-Region X-ray DataX-RayBonesFractured ClassificationHF
51Fracture detectionX-RayBonesFractured ClassificationHF
52The vertebrae X-ray imageX-RayBonesVertebraeHF
53Knee Osteoarthritis DatasetX-RayBonesKnee Osteoarthritis with severity gradingHF
54Shenzhen Chest X-Ray SetX-RayLungCOVID19, Classification Dataset.HF
55Chest X-ray PDX-RayLungCOVID and PneumoniaHF
56COVID-19 CHEST X-RAY DATABASEX-RayLungCOVID and PneumoniaHF
COVIDGRX-RayLungCOVID19, ClassificationHF
58MIASX-RayBreastMulti Classification of BreastHF
59Tuberculosis Chest X-Ray DatabaseX-RayLungTuberculosisHF
60Pediatric Pneumonia Chest X-RayX-RayLungPneumonia ClassificationHF
61Random Sample of NIH Chest X-Ray DatasetX-RayChestMulti Classificaiton of ChestHF
62CoronaHack-Chest X-RayX-RayLungPnemonia Classifcition with Virus typeHF
63Brain Tumor DatasetX-RayBrainTumor ClassificationHF
64Fitzpatrick 17k (Nine Labels)DerSkinMulti ClassificationHF
65BioMediTechMicCellMulti ClassificationHF
66Diabetic retinopathyFPFundusDiabetic Retinopathy LevelHF
67LeukemiaMicCellCancer ClassificationHF
68ODIR-5KFPFundusMultiple Labels ClassificationHF
69ArthrosisX-RayBonesBone Age ClassificationHF
70HSA-NRLMicCellMulti Classification of pathologistsHF
71ISIC 2018 (Task 3)DerSkinMulti ClassificationHF
72ISIC 2017 (Task 3)DerSkinMulti ClassificationHF
73ChestX-DetX-RayChestMulti ClassificationHF
74Monkeypox Skin Lesion DatasetDerSkinOnly MonkeypoxHF
75Cataract DatasetFPFundusMulti ClassificationHF
76ChestX-rays IndianaUniversityX-RayChestMulti-label ClassificationHF
77CheXpert v1.0 smallX-RayChestMulti-label ClassificationHF
78CBIS-DDSMX-RayBreastMulti ClassificationHF
79NLM-TBX-RayLungTuberculosisHF
80ChestXray-NIHCCX-RayChestMulti-label ClassificationHF
81COVIDx CXR-4X-RayLungCOVID19, ClassificationHF
82VinDr-MammoX-RayBreastMulti-label ClassificationHF
83PBC dataset normal DIBMicCellMulti ClassificationHF
84Human Protein AtlasMicCellMulti-label Classification (Only green)HF
85RSNA Pneumonia Detection Challenge 2018X-RayChestMulti-label ClassificationHF
86VinDr-SpineXRX-RayBonesMulti Classification of Bones DiseasesHF
87VinDr-PCXRX-RayChestMulti-label ClassificationHF
88PH2DerSkinMelanoma SegmentationTODO
89ISBI 2016 (Task3B)DerSkinMelanoma SegmentationTODO
90ISIC 2016 (Task 1)DerSkinMelanoma SegmentationTODO
91ISIC 2017DerSkinMelanoma SegmentationTODO
92CVC-ClinicDBEndIntestinePolyp SegmentationTODO
93Kvasir-SEGEndIntestinePolyp segmentationTODO
94m2caisegEndIntestineSurgical Instrument SegmentationTODO
95EDD 2020EndIntestineMultiple Diseases Segmentation in IntestineTODO
96SICAPv2MicCellCancer Cells SegmentationTODO
97BUSIUltrasoundBreastCancer SegmentationTODO
98TN3KUltrasoundThyroidThyroid Nodule SegmentationTODO
99NLM-TBX-RayLungLung Segmentation (With left or right)TODO
100VinDr-SpineXRX-RayBonesSpinal X-ray Anaomaly DetectionTODO
101VinDr-PCXRX-RayChestMultiple Diseases Segmentation in ChestTODO
102ChestX-DetX-RayChestMultiple Diseases Segmentation in ChestTODO
103UW-Madison Gl Tract Image SegmentationMRIIntestineSurgical Instrument SegmentationTODO
104Duke Liver Dataset MRI v1MRILiverLiver SegmentationTODO
105Duke Liver Dataset MRI v2MRILiverLiver SegmentationTODO
106SIIM-ACR Pneumothorax SegmentationX-RayLungPneumothorax SegmentationTODO
107FIVESFPFundusFundus Vascular SegmentationTODO
108RIM-ONE DLFPFundusOptic Disc and Cup SegmentationTODO
109PALM19FPFundusOptic Disc SegmentationTODO

Aggregated_Subsets

Click to view the details of 53 Subsets
No.ModalityAreaTaskQA
01CoCervixCervical Picture Quality EvaluationHF
02CTKidneyKidney Diseases ClassificationHF
03CTLungCOVID-19 ClassificationHF
04CTLungLung Cancer ClassificationHF
05CTBrainBrain Hemorrhage ClassificationHF
06CTBrainBrain Cancer ClassificationHF
07DerSkinMelanoma Type ClassificationHF
08DerSkinSkin Diseases ClassificationHF
09DPMouthTeeth Condition ClassificationHF
10DPMouthOral Cancer ClassificationHF
11EndIntestineIntestine Cleanliness LevelHF
12EndBladderCancer Degree ClassificationHF
13EndIntestineIntestine Diseases ClassificationHF
14FPFundusEye Diseases ClassificationHF
15FPFundusMultiple-labels Eye Diseases ClassificationHF
16FPFundusBlindness LevelHF
17FPFundusRetinal Images Quality EvaluationHF
18MicCellCell Type ClassificationHF
19MicCellProstate Cancer Degree ClassificationHF
20MicCellMultiple-labels Blood Cell ClassificationHF
21MicCellCancer ClassificationHF
22MRIBrainHead Diseases ClassificationHF
23OCTRetinaRetina Diseases ClassificationHF
24USBreastBreast Cancer ClassificationHF
25X-rayBonesDegree Classification of KneeHF
26X-rayBonesFractured ClassificationHF
27X-rayBonesVertebrae Diseases ClassificationHF
28X-rayLungCOVID-19 and Pneumonia ClassificationHF
29X-rayBreastBreast Diseases ClassificationHF
30X-rayLungTuberculosis ClassificationHF
31X-rayChestMultiple-labels Chest ClassificationHF
32X-rayBrainTumor ClassificationHF
33MicCellMulti-labels DiseasesHF
34FPFundusLevel IdentificationHF
35X-rayBonesLevel IdentificationHF
36X-rayBonesSpinal lesion ClassificationHF
37X-rayBreastMulti-labels DiseasesHF
38DerSkinLesion Det/SegTODO
39EndIntestinePolyP Det/SegTODO
40EndIntestineSurgical Procedures Det/SegTODO
41EndIntestineMulti-labels Det/SegTODO
42MicCellCancer Cell Det/SegTODO
43USChestCancer Det/SegTODO
44USThyroidThyroid Nodule Region Det/SegTODO
45MRIIntestineMulti-labels Det/SegTODO
46MRILiverLiver Det/SegTODO
47X-rayLungLung Det/SegTODO
48X-rayLungPneumothorax Det/SegTODO
49X-rayBonesSpinal Anomaly DetTODO
50X-rayChestMulti-labels DetTODO
51FPFundusVessel SegTODO
52FPFundusOptic Disc and Cup SegTODO
53FPFundusOptic Disc SegTODO

After downloading the images to the "med-mat" folder and placing the corresponding JSON files as shown, you can easily access Med-MAT.

┬─ med-mat
│   ├─ CT_Kindney_Dataset
│   └─ ... (unzipped datasets)
└─ Aggregated_Subsets
│   ├─ Subset--01-train.json
│   ├─ Subset--02-train.json
│   └─ ... (other subsets)
└─ Original_Medical_Datasets
    ├─ Ori--01-train.json
    ├─ Ori--02-train.json
    └─ ... (other medical datasets)

⚒️ Data Construction

Here’s a sample from Med-MAT:

  • caption: The original label from the collected medical datasets.
  • image: Path to the corresponding image.
  • Question and Answer: Caption-based QA pairs.
  • Question-choice and Answer-choice: Multiple-choice QA pairs.
  • data-no: Number of its original medical dataset.
{
    "id": 1,
    "caption": "Cyst",
    "image": "med-mat/CT_Kindney_Dataset/CT-KIDNEY-DATASET-Normal-Cyst-Tumor-Stone/CT-KIDNEY-DATASET-Normal-Cyst-Tumor-Stone/Cyst/Cyst- (561).jpg",
    "Question": "Review this kidney CT scan and determine the possible condition it represents.",
    "Answer": "Cyst",
    "Question-choice": "Review this kidney CT scan and determine the possible condition it represents.\nA: Stone\nB: Cyst\nC: Normal\nD: Tumor\nAnswer with the option's letter from the given choices directly.",
    "Answer-choice": "B",
    "data-no": "2"
}

Acknowledgement

We appreciate the previous efforts in open-sourcing the medical imaging datasets used in this project.

Please be sure to credit them when citing these datasets.

📖 Citation

@misc{cai2024compositionalgeneralizationmultimodalllms,
      title={On the Compositional Generalization of Multimodal LLMs for Medical Imaging}, 
      author={Zhenyang Cai and Junying Chen and Rongsheng Wang and Weihong Wang and Yonglin Deng and Dingjie Song and Yize Chen and Zixu Zhang and Benyou Wang},
      year={2024},
      eprint={2412.20070},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.20070}, 
}

Contributors

Eric3200

11 commits

FreedomIntelligence/Med-MAT

Dataset

18

stars

11

commits

7

linked in READMEs

Nov 28, 2025

updated

README

Med-MAT: On the Compositional Generalization of Multimodal LLMs for Medical Imaging

✨ Latest News

⚡ Introduction

Welcome to the repository of Med-MAT, a VQA dataset consisting of 106 open-source medical datasets, which we hope will advance generalization experiments and aid in training powerful medical multimodal large language models (MLLMs).

Through this dataset, we have demonstrated that Compositional Generalization (CG) is one of the key mechanisms for MLLMs to understand unseen images, enabling them to handle unfamiliar images and achieve data-efficient training.

Here is a list of what has been released:

  1. QA Pairs for 106 Medical Datasets: Image-label pairs converted into VQA pairs for MLLM training.
  2. QA Pairs for 53 Aggregated Subsets: Datasets categorized by Modality, Anatomical Area, and Task (MAT), with identical entries merged into subsets.
  3. Image Download Links: Some datasets cannot be shared due to licensing. Users can download them to specified directories.

💭 QA Pairs Construction

To enable MLLMs to directly train and test on Med-MAT, the image-label pairs were converted into a Visual Question-Answering (VQA) format. The process involves the following steps:

  1. Task Definition: Each subset was manually assigned 6 instructions to guide the MLLM in answering the task related to the subset.
  2. Conversion to VQA Format: All image-label pairs were converted into single-choice questions with up to four answer options.
  3. Distractor Selection: Distractor options were randomly drawn from other labels within the subset to ensure variety.
  4. Final Dataset: The resulting dataset consisted of VQA pairs, where each image is paired with a question and four options, one of which is correct.

📚 Data

You can access the QA pairs of Med-MAT in this page.

The tables below record the download URLs for the images and QA pairs for each dataset and subset. If you only wish to use part of Med-MAT, you can selectively download the corresponding data.

Original_Medical_Datasets

Click to view the details of 106 Medical Datasets
No.Name with linkModalityAreaTaskQA
1Intel and MobileODT Cervical ScreeningCoCervixCervix Type in ScreeningHF
2CT Kindney DatasetCTKidneyNormal or Cyst or TumorHF
3SARS-COV-2 Ct-ScanCTLungCOVID19, Classification DatasetHF
4COVID CT COVID-CTCTLungCOVID19, Classification Dataset.HF
5Chest CT-ScanCTLungCancer, 3 Cancer Categories, Multiple Classification DatasetHF
6COVID-19-CT SCAN IMAGESCTLungCOVID19, ClassificationHF
7Head CTCTBrainHead HemorrhageHF
8CT of BrainCTBrainHead CancerHF
9MED-NODEDerSkinMelanoma or NaevusHF
10ISIC 2020DerSkinMelanoma, Benign or MalignantHF
11PED-UFES-20DerSkinSkin Multi ClassificationHF
12Web-scraped Skin ImageDerSkinSkin Desease Multi ClassificationHF
13ISBI 2016DerSkinSkin Lesion ClassificationHF
14ISIC 2019DerSkinSkin Desease Multi ClassificationHF
15Skin Cancer ISICDerSkinSkin Cancer Multi ClassificationHF
16Dental Condition DatasetDPTeethTeeth condition classificationHF
17Oral Cancer DatasetDPTeethOral cancer ClassificationHF
18The Nerthus DatasetEndIntestineCleanliness levelHF
19Endoscopic Bladder TissueEndBladderCanser Degree ClassificationHF
20KvasirEndIntestineMulti Disease ClassificationHF
21ACRIMAFPFundusGlaucomaHF
22Augemnted ocular diseases AODFPFundusMulti Classification of eye diseasesHF
23JSIECFPFundusMulti Classification of eye diseasesHF
24Multi-Label Retinal DiseasesFPFundusMulti Classification of eye diseasesHF
25RFMiD 2.0FPFundusMulti Classification of eye diseasesHF
26ToxoFundus(Data Processed Paper)FPFundusOcular toxoplasmosisHF
27ToxoFundus(Data Raw 6class All)FPFundusOcular toxoplasmosisHF
28Adam datasetFPFundusAge-related Macular DegenerationHF
29APTOS 2019 BlindnessFPFundusBlindness Level Identification 0~4HF
30DRIMBDFPFundusQuality Testing of Retinal ImagesHF
31Glaucoma DetectionFPFundusGlaucoma ClassificationHF
32AIROGSFPFundusGlaucoma ClassificationHF
33ICPR-HEp-2MicCellMulti ClassificationHF
34SICAPv2MicCellCancer Degree ClassificationHF
35Blood Cell ImagesMicCellBlood Cell Classificaion (Multi)HF
36BreakHisMicCellCell type and beginormagHF
37ChaoyangMicCellMulti Classification of pathologistsHF
38HuSHeMMicCellSperm Head Morphology ClassificaionHF
39Bone Marrow Cell ClassificationMicCellBone Marrow Cell ClassificationHF
40NCT-CRC-HE-100KMicCellMulti ClassificationHF
41Malignant Lymphoma ClassificationMicCellMulti ClassificationHF
42Histopathologic Cancer DetectionMicCellCancer ClassificationHF
43LC25000MicCellMulti Classification of Lung and ColonHF
44Brain Tumor 17 ClassesMRIBrainMulti ClassificationHF
45Tumor ClassificationMRIBrainPituitary or Glioma or Meningioma or NotumorHF
46Malignant Lymphoma ClassificationOCTRetinaMulti Classification of eye diseasesHF
47Retinal OCT-C8OCTRetinaMulti Classification of eye diseasesHF
48BUSIUSBreastBreast CancerHF
49Digital Knee X-Ray ImagesX-RayBonesDegree Classification of KneeHF
50Bone Fracture Multi-Region X-ray DataX-RayBonesFractured ClassificationHF
51Fracture detectionX-RayBonesFractured ClassificationHF
52The vertebrae X-ray imageX-RayBonesVertebraeHF
53Knee Osteoarthritis DatasetX-RayBonesKnee Osteoarthritis with severity gradingHF
54Shenzhen Chest X-Ray SetX-RayLungCOVID19, Classification Dataset.HF
55Chest X-ray PDX-RayLungCOVID and PneumoniaHF
56COVID-19 CHEST X-RAY DATABASEX-RayLungCOVID and PneumoniaHF
COVIDGRX-RayLungCOVID19, ClassificationHF
58MIASX-RayBreastMulti Classification of BreastHF
59Tuberculosis Chest X-Ray DatabaseX-RayLungTuberculosisHF
60Pediatric Pneumonia Chest X-RayX-RayLungPneumonia ClassificationHF
61Random Sample of NIH Chest X-Ray DatasetX-RayChestMulti Classificaiton of ChestHF
62CoronaHack-Chest X-RayX-RayLungPnemonia Classifcition with Virus typeHF
63Brain Tumor DatasetX-RayBrainTumor ClassificationHF
64Fitzpatrick 17k (Nine Labels)DerSkinMulti ClassificationHF
65BioMediTechMicCellMulti ClassificationHF
66Diabetic retinopathyFPFundusDiabetic Retinopathy LevelHF
67LeukemiaMicCellCancer ClassificationHF
68ODIR-5KFPFundusMultiple Labels ClassificationHF
69ArthrosisX-RayBonesBone Age ClassificationHF
70HSA-NRLMicCellMulti Classification of pathologistsHF
71ISIC 2018 (Task 3)DerSkinMulti ClassificationHF
72ISIC 2017 (Task 3)DerSkinMulti ClassificationHF
73ChestX-DetX-RayChestMulti ClassificationHF
74Monkeypox Skin Lesion DatasetDerSkinOnly MonkeypoxHF
75Cataract DatasetFPFundusMulti ClassificationHF
76ChestX-rays IndianaUniversityX-RayChestMulti-label ClassificationHF
77CheXpert v1.0 smallX-RayChestMulti-label ClassificationHF
78CBIS-DDSMX-RayBreastMulti ClassificationHF
79NLM-TBX-RayLungTuberculosisHF
80ChestXray-NIHCCX-RayChestMulti-label ClassificationHF
81COVIDx CXR-4X-RayLungCOVID19, ClassificationHF
82VinDr-MammoX-RayBreastMulti-label ClassificationHF
83PBC dataset normal DIBMicCellMulti ClassificationHF
84Human Protein AtlasMicCellMulti-label Classification (Only green)HF
85RSNA Pneumonia Detection Challenge 2018X-RayChestMulti-label ClassificationHF
86VinDr-SpineXRX-RayBonesMulti Classification of Bones DiseasesHF
87VinDr-PCXRX-RayChestMulti-label ClassificationHF
88PH2DerSkinMelanoma SegmentationTODO
89ISBI 2016 (Task3B)DerSkinMelanoma SegmentationTODO
90ISIC 2016 (Task 1)DerSkinMelanoma SegmentationTODO
91ISIC 2017DerSkinMelanoma SegmentationTODO
92CVC-ClinicDBEndIntestinePolyp SegmentationTODO
93Kvasir-SEGEndIntestinePolyp segmentationTODO
94m2caisegEndIntestineSurgical Instrument SegmentationTODO
95EDD 2020EndIntestineMultiple Diseases Segmentation in IntestineTODO
96SICAPv2MicCellCancer Cells SegmentationTODO
97BUSIUltrasoundBreastCancer SegmentationTODO
98TN3KUltrasoundThyroidThyroid Nodule SegmentationTODO
99NLM-TBX-RayLungLung Segmentation (With left or right)TODO
100VinDr-SpineXRX-RayBonesSpinal X-ray Anaomaly DetectionTODO
101VinDr-PCXRX-RayChestMultiple Diseases Segmentation in ChestTODO
102ChestX-DetX-RayChestMultiple Diseases Segmentation in ChestTODO
103UW-Madison Gl Tract Image SegmentationMRIIntestineSurgical Instrument SegmentationTODO
104Duke Liver Dataset MRI v1MRILiverLiver SegmentationTODO
105Duke Liver Dataset MRI v2MRILiverLiver SegmentationTODO
106SIIM-ACR Pneumothorax SegmentationX-RayLungPneumothorax SegmentationTODO
107FIVESFPFundusFundus Vascular SegmentationTODO
108RIM-ONE DLFPFundusOptic Disc and Cup SegmentationTODO
109PALM19FPFundusOptic Disc SegmentationTODO

Aggregated_Subsets

Click to view the details of 53 Subsets
No.ModalityAreaTaskQA
01CoCervixCervical Picture Quality EvaluationHF
02CTKidneyKidney Diseases ClassificationHF
03CTLungCOVID-19 ClassificationHF
04CTLungLung Cancer ClassificationHF
05CTBrainBrain Hemorrhage ClassificationHF
06CTBrainBrain Cancer ClassificationHF
07DerSkinMelanoma Type ClassificationHF
08DerSkinSkin Diseases ClassificationHF
09DPMouthTeeth Condition ClassificationHF
10DPMouthOral Cancer ClassificationHF
11EndIntestineIntestine Cleanliness LevelHF
12EndBladderCancer Degree ClassificationHF
13EndIntestineIntestine Diseases ClassificationHF
14FPFundusEye Diseases ClassificationHF
15FPFundusMultiple-labels Eye Diseases ClassificationHF
16FPFundusBlindness LevelHF
17FPFundusRetinal Images Quality EvaluationHF
18MicCellCell Type ClassificationHF
19MicCellProstate Cancer Degree ClassificationHF
20MicCellMultiple-labels Blood Cell ClassificationHF
21MicCellCancer ClassificationHF
22MRIBrainHead Diseases ClassificationHF
23OCTRetinaRetina Diseases ClassificationHF
24USBreastBreast Cancer ClassificationHF
25X-rayBonesDegree Classification of KneeHF
26X-rayBonesFractured ClassificationHF
27X-rayBonesVertebrae Diseases ClassificationHF
28X-rayLungCOVID-19 and Pneumonia ClassificationHF
29X-rayBreastBreast Diseases ClassificationHF
30X-rayLungTuberculosis ClassificationHF
31X-rayChestMultiple-labels Chest ClassificationHF
32X-rayBrainTumor ClassificationHF
33MicCellMulti-labels DiseasesHF
34FPFundusLevel IdentificationHF
35X-rayBonesLevel IdentificationHF
36X-rayBonesSpinal lesion ClassificationHF
37X-rayBreastMulti-labels DiseasesHF
38DerSkinLesion Det/SegTODO
39EndIntestinePolyP Det/SegTODO
40EndIntestineSurgical Procedures Det/SegTODO
41EndIntestineMulti-labels Det/SegTODO
42MicCellCancer Cell Det/SegTODO
43USChestCancer Det/SegTODO
44USThyroidThyroid Nodule Region Det/SegTODO
45MRIIntestineMulti-labels Det/SegTODO
46MRILiverLiver Det/SegTODO
47X-rayLungLung Det/SegTODO
48X-rayLungPneumothorax Det/SegTODO
49X-rayBonesSpinal Anomaly DetTODO
50X-rayChestMulti-labels DetTODO
51FPFundusVessel SegTODO
52FPFundusOptic Disc and Cup SegTODO
53FPFundusOptic Disc SegTODO

After downloading the images to the "med-mat" folder and placing the corresponding JSON files as shown, you can easily access Med-MAT.

┬─ med-mat
│   ├─ CT_Kindney_Dataset
│   └─ ... (unzipped datasets)
└─ Aggregated_Subsets
│   ├─ Subset--01-train.json
│   ├─ Subset--02-train.json
│   └─ ... (other subsets)
└─ Original_Medical_Datasets
    ├─ Ori--01-train.json
    ├─ Ori--02-train.json
    └─ ... (other medical datasets)

⚒️ Data Construction

Here’s a sample from Med-MAT:

  • caption: The original label from the collected medical datasets.
  • image: Path to the corresponding image.
  • Question and Answer: Caption-based QA pairs.
  • Question-choice and Answer-choice: Multiple-choice QA pairs.
  • data-no: Number of its original medical dataset.
{
    "id": 1,
    "caption": "Cyst",
    "image": "med-mat/CT_Kindney_Dataset/CT-KIDNEY-DATASET-Normal-Cyst-Tumor-Stone/CT-KIDNEY-DATASET-Normal-Cyst-Tumor-Stone/Cyst/Cyst- (561).jpg",
    "Question": "Review this kidney CT scan and determine the possible condition it represents.",
    "Answer": "Cyst",
    "Question-choice": "Review this kidney CT scan and determine the possible condition it represents.\nA: Stone\nB: Cyst\nC: Normal\nD: Tumor\nAnswer with the option's letter from the given choices directly.",
    "Answer-choice": "B",
    "data-no": "2"
}

Acknowledgement

We appreciate the previous efforts in open-sourcing the medical imaging datasets used in this project.

Please be sure to credit them when citing these datasets.

📖 Citation

@misc{cai2024compositionalgeneralizationmultimodalllms,
      title={On the Compositional Generalization of Multimodal LLMs for Medical Imaging}, 
      author={Zhenyang Cai and Junying Chen and Rongsheng Wang and Weihong Wang and Yonglin Deng and Dingjie Song and Yize Chen and Zixu Zhang and Benyou Wang},
      year={2024},
      eprint={2412.20070},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.20070}, 
}

Contributors

Eric3200

11 commits