GMAI-VL-5.5M Dataset

GMAI-VL-5.5M is a comprehensive, large-scale medical General Medical AI Vision-Language (GMAI-VL) dataset built specifically for training multimodal foundation models in the medical domain. It contains an extraordinary scale of high-quality instructions encompassing over 5.5 million multimodal question-answering pairs, carefully constructed based on hundreds of medical classification, segmentation, and detection datasets.

This repository is a unified hub that hosts both the openly redistributable data and the annotation-only non-open-source split, organized under a single HuggingFace repo: General-Medical-AI/GMAI-VL-5.5M.

Dataset Structure

The repository is divided into two top-level directories, each containing its own annotations and (where permitted) image data:

The full 5.5M dataset is organized into five subsets by task type. The number in each filename (e.g., 1.7M) indicates the total count across both splits — the actual entry count per folder varies depending on whether the underlying source datasets allow redistribution.

GMAI-VL-5.5M-OpenSource

• zips/: To facilitate stable and reliable downloads, the massive high-resolution imagery has been losslessly compressed into manageable volume chunks (e.g., images_chunk_001.zip, images_chunk_002.zip). Once downloaded, extract them together into an images/ directory.
• annotations/: Contains the open-source portion of each multimodal subset (with redistributable images in zips/), plus the complete text-only subsets (which have no image copyright restrictions).

GMAI-VL-5.5M-NonOpenSource

• annotations/: Contains only the model-generated annotations for the non-open-source portion of each multimodal subset (no images). Users must obtain the original restricted images from the respective dataset authors (see list below).

The overall repository layout is:

General-Medical-AI/GMAI-VL-5.5M/                <- HuggingFace Repository
├── GMAI-VL-5.5M-NonOpenSource/
│   └── annotations/
│       ├── GMAI-MM-Caption-1.7M.jsonl           <- Caption (non-open-source portion)
│       ├── GMAI-MM-Instrunct-0.9M.jsonl         <- Instruction (non-open-source portion)
│       └── GMAI-MM-Percept-1.3M.jsonl           <- Perception (non-open-source portion)
└── GMAI-VL-5.5M-OpenSource/
    ├── annotations/
    │   ├── GMAI-MM-Caption-1.7M.jsonl           <- Caption (open-source portion)
    │   ├── GMAI-MM-Instrunct-0.9M.jsonl         <- Instruction (open-source portion)
    │   ├── GMAI-MM-Percept-1.3M.jsonl           <- Perception (open-source portion)
    │   ├── GMAI_Text_Multi_0.7M.jsonl           <- Text-only multi-turn QA (complete)
    │   └── GMAI_Text_Single_1M.jsonl            <- Text-only single-turn QA (complete)
    ├── zips/
    │   ├── images_chunk_001.zip                 <- Compressed image chunks
    │   ├── images_chunk_002.zip
    │   └── ...
    ├── download_and_prepare.py
    ├── README.md
    └── README_zh-CN.md

All datasets provide standard LLaVA-style dictionary objects for immediate training integration:

{
    "image": "images/2d/cls/fundus_photography/diabetic/images/1/36808_left.jpeg",
    "conversations": [
        {
            "from": "human",
            "value": "<image>\nExamine the given fundus photograph. Identify the specific vascular abnormalities present and explain their significance..."
        },
        {
            "from": "gpt",
            "value": "The image shows mild abnormalities such as microaneurysms, which are small bulges..."
        }
    ]
}

Special Note on Commercial & Secondary Redistribution Restrictions

A significant portion of the constructed 5.5M pairs relies on underlying medical datasets that explicitly prohibit secondary redistribution of their images or require distinct academic clearance, email authorization, or challenge participation. To respect the original authors' intellectual properties and terms of service, we strictly omit these restricted raw images from our public release.

However, the Model-Generated Annotations (Q&A JSON pairs) are considered derived works/intellectual property generated by our team and are 100% openly released. To make this clear, our repository is separated into two major segments:

1. GMAI-VL-5.5M-OpenSource/: Contains the open-source portion of each multimodal subset's annotations (with redistributable images in zips/), plus the complete text-only subsets.
2. GMAI-VL-5.5M-NonOpenSource/: Contains ONLY the non-open-source portion of each multimodal subset's annotations. Images are omitted.

Both directories reside within the same HuggingFace repository: General-Medical-AI/GMAI-VL-5.5M.

Instruction for researchers wanting the FULL 5.5M dataset:

• For the Open Source split, everything works out of the box — run download_and_prepare.py and images will be automatically downloaded and extracted.
• For the Non-Open Source split, please visit the URLs of the respective restricted datasets provided below. After obtaining access and downloading the raw sources according to the authors' guidelines, structure those images locally into your GMAI-VL-5.5M-NonOpenSource/images/ directory strictly matching the relative paths defined in our JSON conversations.

Below is a detailed list of the original datasets mapped in our system that are subject to such redistribution restrictions:

Datasets Restricted from Secondary Open Source

Please visit their respective homepages for access procedures if needed. Some datasets might require email requests or have restricted access. Notes have been attached for datasets whose links have naturally expired over time.

• 2018 Data Science Bowl (Microscopy Images, Segmentation) - Homepage - License: Non-commercial, No Redistribution
• 5K+ CT Images on Fractured Limbs (CT, Segmentation) - Homepage - License: Subject to Competition Rules
• AIDA-E_1 (Confocal Endoscopy in Celiac Imaging) (Endoscopy, Classification) - Homepage - License: Challenge/Competition Only
• AIDA-E_2 (Esophagus microendoscopy images in Barrett's surveillance) (Endoscopy, Classification) - Homepage - License: Challenge/Competition Only
• AIDA-E_3 (Endoscopy, Classification) - Homepage - License: Challenge/Competition Only
• AIROGS (Fundus Photography, Classification) - Homepage - License: CC BY-NC-ND 4.0
• APTOS 2019 Blindness Detection (Fundus Photography, Classification) - Homepage - License: Subject to Competition Rules
• Arteriovenous Nicking (Fundus Photography, Classification) - Homepage - License: 链接失效
• BCNB Task-1 (whole-slide images, Classification) - Homepage
• BCNB Task-2 (whole-slide images, Classification) - Homepage
• BCNB Task-4 (whole-slide images, Classification) - Homepage
• BCNB Task-5 (whole-slide images, Classification) - Homepage
• BCNB Task-6 (whole-slide images, Classification) - Homepage
• BraTS2023 (MR, Augmentation) - Homepage - License: Not allowed
• BRIGHT (Histopathology, Classification) - Homepage
• BUSI (Ultrasound, Segmentation) - Homepage
• Cervix93 Cytology Dataset (Microscopy Images, Classification) - Homepage
• ChinaSet - (Link expired or unavailable)
• Corneal Nerve Tortuosity (Microscopy Images, Classification) - Homepage
• Corneal Nerve (Microscopy Images, Classification) - Homepage
• COVIDx_CXR-4 - (Link expired or unavailable)
• CRAG (Histopathology, Segmentation) - Homepage
• CRASS (X_Ray, Segmentation) - Homepage
• Dataset for AO-SLO cone photoreceptor automatic segmentation and analysis (Ophthalmoscope, Classification) - Homepage
• DigestPath19 (whole-slide images, Detection) - Homepage - License: Challenge/Competition Only
• DigestPath19 (whole-slide images, Segmentation, Classification) - Homepage
• DRHAGIS (Fundus Photography, Segmentation) - Homepage - License: 链接失效
• DRISHTI-GS (Fundus Photography, Segmentation) - Homepage
• DRIVE (Fundus Photography, Segmentation) - Homepage - License: Challenge/Competition Only
• EndoVis 2017 - KBD (Endoscopy, Segmentation) - Homepage
• EndoVis 2017 - RIS (Endoscopy, Segmentation) - Homepage
• EndoVis 2018 - RSS (Endoscopy, Segmentation) - Homepage
• EndoVis15 (Endoscopy, Segmentation) - Homepage
• EndoVis2023-PitVis (Endoscopy, Classification) - Homepage - License: CC BY-NC-ND 4.0
• EndoVis2023-SIMS (Endoscopy, Segmentation) - Homepage - License: CC BY-NC-ND 4.0
• EndoVis2023-Syn-ISS (Endoscopy, Segmentation) - Homepage - License: CC BY-NC-ND
• Finding and Measuring Lungs in CT Data (CT, Segmentation) - Homepage - License: Not allowed
• FUND (Fundus Photography) - (Link expired or unavailable)
• GAMMA (Fundus Photography, Classification) - Homepage - License: Not allowed
• GAMMA (Fundus Photography, Segmentation) - Homepage - License: Not allowed
• HarvardGlaucoma (Fundus Photography) - (Available via original authors) - License: CC BY-NC-ND 4.0
• HErlev (Pathology, Classification) - Homepage
• hyper-kvasir-segmented-images - (Link expired or unavailable)
• iChallenge - GOALS (OCT, Classification) - Homepage
• iChallenge - GOALS (OCT, Segmentation) - Homepage - License: Challenge/Competition Only
• iChallenge-REFUGE2 (Fundus Photography, Classification) - Homepage - License: Only academic, no redistribution
• iChallenge-REFUGE2 (Fundus Photography, Segmentation) - Homepage - License: Only academic, no redistribution
• ICIAR 2018 (Microscopy Images, whole-slide images, Classification) - [Homepage](Home - Grand Challenge (grand-challenge.org)) - License: CC BY-NC-ND 3.0
• ICIAR 2018 (Microscopy Images, whole-slide images, Segmentation) - [Homepage](Home - Grand Challenge (grand-challenge.org)) - License: CC BY-NC-ND 3.0
• JSRT (X_Ray, Classification) - Homepage
• JSRT (X_Ray, Segmentation) - Homepage
• Kavsir (Endoscopy, Classification) - Homepage
• Kvasir-SEG (Endoscopy, Segmentation) - Homepage - License: Non-commercial, No Redistribution
• KvasirCapsule-SEG (Colposcopy, Segmentation) - Homepage
• LAG (Fundus Photography) - Homepage - License: Not allowed
• m2cai16-tool-locations (Endoscopy, Tracing) - Homepage
• MED-NODE (Dermoscopy, Classification) - Homepage
• MESSIDOR-2 (Fundus Photography, Classification) - Homepage - License: Only academic, no redistribution
• MontgomerySet - (Link expired or unavailable)
• MRL Eye Eye state cls (Infrared Reflectance Imaging, Classification) - Homepage
• MRL Eye Gender (Infrared Reflectance Imaging, Classification) - Homepage
• MRL Eye Glasses cls (Infrared Reflectance Imaging, Classification) - Homepage
• MRL Eye Image quality cls (Infrared Reflectance Imaging, Classification) - Homepage
• MRL Eye Reflections cls (Infrared Reflectance Imaging, Classification) - Homepage
• MRL Eye Sensor type cls (Infrared Reflectance Imaging, Classification) - Homepage
• NODE21 (X_Ray, Detection) - Homepage - License: CC BY-NC-ND 4.0
• OcularD (Fundus Photography, Classification) - (Available via original authors)
• ORVS (Fundus Photography, Segmentation) - Homepage
• Overlapping Cervical Cytology Image Segmentation Challenge (Microscopy Images, Segmentation) - Homepage
• PANDA (Histopathology, Classification) - Homepage - License: Subject to Competition Rules
• PH2 (Dermoscopy, Classification) - Homepage - License: No commercial use, no redistribution
• PH2 (Dermoscopy, Segmentation) - Homepage - License: No commercial use, no redistribution
• Pneumothorax Masks X-Ray (X_Ray, Segmentation) - Homepage
• QUBIQ2020 (CT, MR, Segmentation) - Homepage - License: Challenge/Competition Only
• Retina (Fundus Photography, Classification) - Homepage
• RetinaCheck-Scanning Laser Ophthalmoscopy-Microaneurysm (IOSTAR) (Fundus Photography, Segmentation) - Homepage
• RITE (Retinal Images vessel Tree Extraction) (Fundus Photography, Segmentation) - Homepage
• RSNA Bone Age (X_Ray, Estimation) - Homepage
• SIIM-ACR Pneumothorax Segmentation (X_Ray, Classification) - Homepage - License: Subject to Competition Rules
• SinaFarsiu-003-Fang_BOE_2012/ (OCT, Classification) - Homepage
• SinaFarsiu-009-Chiu_BOE_2013_dataset/ (OCT, Segmentation) - Homepage
• SinaFarsiu-010-Rabbani_IOVS_2014_dataset/ (OCT, Segmentation) - Homepage
• SinaFarsiu-013-Estrada_PAMI_2015_dataset/ (OCT, Classification) - Homepage
• SinaFarsiu-018-Yang_BOE_2021/ (OCT, Segmentation) - Homepage
• STARE (Fundus Photography, Segmentation) - Homepage
• SUN_SEG (Endoscopy, Segmentation) - Homepage - License: Non-commercial, No Redistribution
• TCB-Challenge (Bone Radiograph, Classification) - Homepage
• Ultrasound Nerve Segmentation (Ultrasound, Segmentation) - Homepage - License: Non-commercial, No Redistribution
• UW-Madison GI Tract Image Segmentation (MR, Segmentation) - Homepage - License: Subject to Competition Rules
• VinBigData Chest X-ray Abnormalities Detection (X_Ray, Detection) - Homepage - License: Subject to Competition Rules

Researchers desiring to train locally incorporating the non-opensource instances should obtain local access to those datasets adhering strictly to the authors' guidelines.

How to Download and Prepare the Dataset

To simplify the process of gathering the multi-part ZIP chunks and annotations, we provide a unified script download_and_prepare.py.

Prerequisites

• Python 3.8+
• huggingface_hub library:

  pip install huggingface_hub
  

Usage

1. Clone this repository or download download_and_prepare.py.
2. Run the script to download the full dataset and automatically extract OpenSource images:

   python download_and_prepare.py --local_dir ./GMAI-VL-5.5M
   

   Note: This will download both OpenSource and NonOpenSource splits. All images_chunk_*.zip files from GMAI-VL-5.5M-OpenSource/zips/ will be automatically extracted into an images/ folder.
3. To skip image extraction (download only):

   python download_and_prepare.py --local_dir ./GMAI-VL-5.5M --skip_unzip
   

Citation

If you find this dataset or our work helpful, please cite our AAAI 2026 paper:

@inproceedings{li2026gmai,
  title={Gmai-vl \& gmai-vl-5.5 m: A large vision-language model and a comprehensive multimodal dataset towards general medical ai},
  author={Li, Tianbin and Su, Yanzhou and Li, Wei and Fu, Bin and Chen, Zhe and Huang, Ziyan and Wang, Guoan and Ma, Chenglong and Chen, Ying and Hu, Ming and others},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={40},
  number={28},
  pages={23177--23185},
  year={2026}
}

Links & Resources

• Paper (AAAI 2026): Link
• Paper (arXiv Preprint): arXiv:2411.14522
• **Official GitHub Code **: uni-medical/GMAI-VL

🤝 免责声明

GMAI-VL-5.5M是基于多个公开的数据集构建，旨在取之于社区，回馈于社区，为研究人员和开发者提供一个用于学术和技术研究的资源。使用本数据集的任何个人或组织（以下统称为“使用者”）需遵守以下免责声明：

• 数据集来源：本数据集基于多个公开的数据集构建，这些数据集的来源已在论文中明确标明。使用者应当遵守原始数据集的相关许可和使用条款。
• 数据准确性：尽管我们已经努力确保数据集的准确性和完整性，但使用者应自行承担使用数据集可能带来的风险和责任。
• 责任限制：在任何情况下，数据集的提供者及相关贡献者均不对使用者的任何行为或结果承担责任。
• 使用约束：使用者在使用本数据集时，应遵守适用的法律法规和伦理规范。使用者不得将本数据集用于非法、侵犯隐私、诽谤、歧视或其他违法或不道德的目的。
• 知识产权：本数据集所有影像数据的知识产权归原始数据集的相关权利人所有，使用者不得以任何方式侵犯数据集的知识产权。

作为非盈利机构，团队倡导和谐友好的开源交流环境，若在开源数据集内发现有侵犯您合法权益的内容，请联系我们，我们将尽最大努力协助您处理。

通过下载、复制、访问或使用本数据集，即表示使用者已阅读、理解并同意遵守本免责声明中的所有条款和条件。如果使用者无法接受本免责声明的任何部分，请勿使用本数据集。

🤝 Disclaimer

GMAI-VL-5.5M is constructed based on multiple publicly available datasets and aims to provide a resource for academic and technical research to researchers and developers. Any individual or organization (hereinafter referred to as "User") using this dataset must comply with the following disclaimer:

• Dataset Source: GMAI-VL-5.5M is constructed based on multiple publicly available datasets, and the sources of these datasets have been clearly indicated in the paper. Users should adhere to the relevant licenses and terms of use of the original datasets.
• Data Accuracy: While efforts have been made to ensure the accuracy and completeness of the dataset, users assume all risks and liabilities associated with the use of the dataset.
• Limitation of Liability: Under no circumstances shall the dataset providers or contributors be held liable for any actions or outcomes of the Users.
• Usage Constraints: Users must comply with applicable laws, regulations, and ethical norms when using this dataset. The dataset must not be used for illegal, privacy-infringing, defamatory, discriminatory, or other unlawful or unethical purposes.
• Intellectual Property: The intellectual property rights of the image data in this dataset belong to the relevant rights holders of the original datasets. Users must not infringe upon the intellectual property rights of the dataset in any way.

As a non-profit organization, we advocate for a harmonious and friendly open-source communication environment. If any content in the open dataset is found to infringe upon your legitimate rights and interests, please contact us and we will make our best effort to assist you in addressing the issue.

By downloading, copying, accessing, or using this dataset, the User indicates that they have read, understood, and agreed to comply with all the terms and conditions of this disclaimer. If the User cannot accept any part of this disclaimer, please refrain from using this dataset.

🤝 Acknowledgement

We thank all medical workers and dataset owners for making public datasets available to the community. If you find that your dataset is included in our GMAI-VL-5.5M but you do not want us to do so, please contact us to remove it.