MedMax Dataset

Mixed-Modal Instruction Tuning for Training Biomedical Assistants

Authors: Hritik Bansal, Daniel Israel†, Siyan Zhao†, Shufan Li, Tung Nguyen, Aditya Grover
Institution: University of California, Los Angeles
† Equal Contribution

Paper | Code | Project Page

Overview

Large Language Models (LLMs) and Large Multimodal Models (LMMs) have demonstrated remarkable capabilities in multimodal information integration, opening transformative possibilities for biomedical AI in image analysis, diagnosis, and dataset creation. However, their potential in medical domains remains constrained by limited data availability, narrow domain coverage, and restricted origins (e.g., medical papers).

To address these gaps, we present MedMax, the first large-scale multimodal biomedical instruction-tuning dataset for mixed-modal foundation models. Comprising 1.47 million instances, MedMax encompasses diverse tasks including multimodal content generation (interleaved image-text data), biomedical image captioning and generation, visual chatting, and report understanding. These tasks span diverse medical domains such as radiology and histopathology.

Dataset Details

Data Structure

Each dataset instance includes:

• text: Instruction, context, and the expected response (can be purely textual or multimodal).
• tokens: Tokenized representations for text and images (credentialed entries have no pre-included tokens, users need to download images and tokenize them).
• image_path: References to corresponding image files.
• task: The type of biomedical task (e.g., VQA, Image Generation, Report Understanding).
• source: Data source origin.
• credential: Access level ('yes'/'no') indicating if special credentials are required.

Tasks & Sources

Tasks: Visual Chat, VQA (Visual Question Answering), Image Captioning, Multimodal Generation, Image Generation, Report Understanding

Sources: PubMedVision-IT, PMC-VQA, PMC_OA, QUILT_1M, QUILT_Instruct, PubMedVision-Alignment, MIMIC-CXR, OmniMedVQA, LLaVA-Med-IT, LLaVA-Med-PMC, PathVQA, SLAKE, VQA-RAD

Dataset Statistics

• Total instances: 1.47M
• Unique images: 725K
• Unique words: 947K
• Token distribution: 1.7B tokens (0.7B visual, 1B text)

Using the Dataset

First, download this dataset repo. Then, download images:

Non-Credentialed Data (credential='no')

1. Download images:

cat images.tar.gz.* | tar -xzf -

2. Access text and tokenized data directly from the dataset files

Note:

We observed that there was some error in saving the images for the slake dataset. We fixed and uploaded them in https://huggingface.co/datasets/mint-medmax/medmax_data/blob/main/slake.tar.gz. Please untar this data and copy them in the images/slake folder.

Credentialed Data (credential='yes')

1. Obtain original images from respective sources (e.g., MIMIC-CXR, Quilt)
2. Process images to generate tokens

Credentialed Images Access

1. MIMIC-CXR Dataset

Visit PhysioNet and complete the required steps for credentialing.

Images: https://physionet.org/content/mimic-cxr-jpg/2.1.0/

2. Quilt and Quilt-Instruct Dataset

Submit access request through their official form: Request Form can be found in Quilt-1m repo: https://github.com/wisdomikezogwo/quilt1m

After downloading these images, you can map the image paths back to our dataset's image paths and tokenize these images yourself using the same VQGAN encoder.

We provide the instructions to do so in github repo: https://github.com/Hritikbansal/medmax?tab=readme-ov-file#data-setup

Citation

@misc{bansal2024medmaxmixedmodalinstructiontuning,
      title={MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants}, 
      author={Hritik Bansal and Daniel Israel and Siyan Zhao and Shufan Li and Tung Nguyen and Aditya Grover},
      year={2024},
      eprint={2412.12661},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2412.12661}, 
}

License

Apache 2.0