MedPLIB

Fine-Grained Medical Vision-Language Models from Images to Pixels

MedPLIB: Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine
From Image to Pixels: Towards Fine-Grained Medical Vision-Language Models

If you like our project, please give us a star ⭐ on GitHub for the latest updates.

😮 Highlights

This repository contains the code for two related works, MedPLIB and From Image to Pixels. MedPLIB focuses on pixel-level understanding for biomedical images.

• ✨ MedPLIB is a biomedical MLLM that supports multiple imaging modalities. It handles image-level tasks such as VQA and extends visual question answering to pixel-level reasoning.

• ✨ We construct the MeCoVQA dataset. It comprises an array of 8 modalities with a total of 310k pairs for complex medical imaging question answering and image region understanding.

🔥 Updates

• 2026-04-16: From Image to Pixels was accepted by IEEE TPAMI.
• 2025-01-09: 🔥🔥🔥 We release the model weight and MeCoVQA dataset.
• 2024-12-19: We release the code.
• 2024-12-10: We release the paper.
• 2024-12-09: 🔥🔥🔥 This paper was accepted by AAAI2025.
• 2024-06-14: We release the code link.

Contents

• Demo
• Installation
• Dataset
• Train
• Test
• Model Zoo
• Acknowledgement
• License
• Model Use
• Citation

🍉Demo

Gradio Web UI

We recommend trying our web demo, which includes all the features currently supported by MedPLIB. To run our demo, you need to download or train MedPLIB to make the checkpoints locally. Please run the following commands one by one.

# launch the server controller
python -m model.serve.controller --host 0.0.0.0 --port 64000

# launch the web server
python -m model.serve.gradio_web_server --controller http://localhost:64000 --model-list-mode reload --add_region_feature --port 64001 

# launch the model worker
CUDA_VISIBLE_DEVICES=0 python -m model.serve.model_worker --host localhost --controller http://localhost:64000 --port 64002 --worker http://localhost:64002 --model-path /path/to/the/medplib_checkpoints --add_region_feature --device_map cuda --vision_pretrained /path/to/the/sam-med2d_b.pth

• Pixel grounding:

• Region VQA:

• VQA:

🛠️Installation

1. Clone this repository and navigate to MedPLIB folder

git clone https://github.com/ShawnHuang497/MedPLIB.git
cd MedPLIB

2. Install Package

conda create -n medplib python=3.10 -y
conda activate medplib
pip install --upgrade pip 
pip install -r requirements.txt

3. Install additional packages for training cases

pip install ninja==1.11.1.1
pip install flash-attn==2.5.2 --no-build-isolation

🗃️Dataset

1. Please download MeCoVQA dataset from Google Drive.
2. Download the images from SA-Med2D-20M at Huggingface.

📀Train

Stage I

We perform the pre-training stage I to get the projector checkpoints. Please obtain the llava_med_alignment_500k dataset according to LLaVA-Med, and then follow the usage tutorial of LLaVA-v1.5 to pretrain.

Stage II

sh scripts/train_stage2.sh

Stage III

sh scripts/train_stage3.sh

Stage IV

sh scripts/train_stage4.sh

MedPLIB-ICL

MedPLIB-ICL adds in-context segmentation to MedPLIB. Each query image can use 1-3 (image, mask) examples as visual context.

sh scripts/train_medplib_icl.sh

MedPLIB-ICL can be trained with two configurable example-mask input modes:

• overlay: overlay each example mask on its image.
• separate: encode each example image and mask separately.

To use separate mode:

ICL_MASK_MODE=separate sh scripts/train_medplib_icl.sh

The ICL script also includes two optional components:

• --icl_mask_encoder: encode example masks into 64 tokens.
• --mm_token_compress: compress each CLIP image from 576 tokens to 256 tokens.

For inference:

sh scripts/infer_medplib_icl.sh

To retrieve examples automatically with image-RAG:

sh scripts/build_medplib_icl_rag_index.sh
sh scripts/infer_medplib_icl_rag.sh

🥭 Model_Zoo

Please download MedPLIB model weight from Huggingface.

🥭 Test

Pixel Grounding

TRANSFORMERS_OFFLINE=1 deepspeed --include=localhost:1 --master_port=64995 model/eval/vqa_infer.py \
    --version="/path/to/the/medplib_checkpoints" \
    --vision_tower='/path/to/the/clip-vit-large-patch14-336' \
    --answer_type='open' \
    --val_data_path='/path/to/the/pixel_grounding_json_file' \
    --image_folder='/path/to/the/SAMed2D_v1' \
    --vision_pretrained="/path/to/the/sam-med2d_b.pth" \
    --eval_seg \
    --moe_enable \
    --region_fea_adapter \
    # --vis_mask \

Region_VQA & VQA

Infer to generate the prediction jsonl file.

sh model/eval/infer_parallel_medplib.sh

Calculate the metrics.

python model/eval/cal_metric.py \
    --pred="/path/to/the/jsonl_file" \

👍Acknowledgement

We thank the following works for giving us the inspiration and part of the code: LISA, MoE-LLaVA, LLaVA, SAM-Med2D, SAM and SEEM.

🪜Model Use

Intended Use

The data, code, and model checkpoints are intended to be used solely for (I) future research on visual-language processing and (II) reproducibility of the experimental results reported in the reference paper. The data, code, and model checkpoints are not intended to be used in clinical care or for any clinical decision making purposes.

Primary Intended Use

The primary intended use is to support AI researchers reproducing and building on top of this work. MedPLIB and its associated models should be helpful for exploring various biomedical pixel grounding and vision question answering (VQA) research questions.

Out-of-Scope Use

Any deployed use case of the model --- commercial or otherwise --- is out of scope. Although we evaluated the models using a broad set of publicly-available research benchmarks, the models and evaluations are intended for research use only and not intended for deployed use cases.

🔒License

• The majority of this project is released under the Apache 2.0 license as found in the LICENSE file.
• The service is a research preview intended for non-commercial use only, subject to the model License of LLaMA, Terms of Use of the data generated by OpenAI, and Terms of Use of SAM-Med2D-20M. Please contact us if you find any potential violation.

✏️Citation

If you find our paper and code useful in your research, please consider giving a star and citation.

@article{huang2024towards,
  title={Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine},
  author={Huang, Xiaoshuang and Shen, Lingdong and Liu, Jia and Shang, Fangxin and Li, Hongxiang and Huang, Haifeng and Yang, Yehui},
  journal={arXiv preprint arXiv:2412.09278},
  year={2024}
}

@article{shen2026image,
  title={From Image to Pixels: towards Fine-Grained Medical Vision-Language Models},
  author={Shen, Lingdong and Huang, Xiaoshuang and Shang, Fangxin and Zhang, Xudong and Yang, Yehui and Fan, Bin and Xiang, Shiming},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2026},
  publisher={IEEE}
}