Biomedical Few-shot Image Classification for Vision-Language Models

Overview

Recent advancements in vision-language models (VLMs), such as CLIP, have demonstrated substantial success in self-supervised representation learning for vision tasks. However, effectively adapting VLMs to downstream applications remains challenging, as their accuracy often depends on time-intensive and expertise-demanding prompt engineering, while full model fine-tuning is costly. This is particularly true for biomedical images, which, unlike natural images, typically suffer from limited annotated datasets, unintuitive image contrasts, and nuanced visual features. Recent prompt learning techniques, such as Context Optimization (CoOp) intend to tackle these issues, but still fall short in generalizability. Meanwhile, explorations in prompt learning for biomedical image analysis are still highly limited. In this work, we propose BiomedCoOp, a novel prompt learning framework that enables efficient adaptation of BiomedCLIP for accurate and highly generalizable few-shot biomedical image classification. Our approach achieves effective prompt context learning by leveraging semantic consistency with average prompt ensembles from Large Language Models (LLMs) and knowledge distillation with a statistics-based prompt selection strategy. We conducted comprehensive validation of our proposed framework on 11 medical datasets across 9 modalities and 10 organs against existing state-of-the-art methods, demonstrating significant improvements in both accuracy and generalizability.

Datasets Description

Download the datasets

All the datasets can be found here on HuggingFace. Download each dataset seperately:

• BTMRI [Drive | HuggingFace]
• BUSI [Drive | HuggingFace]
• CHMNIST [Drive | HuggingFace]
• COVID_19 [Drive | HuggingFace]
• CTKidney [Drive | HuggingFace]
• DermaMNIST [Drive | HuggingFace]
• KneeXray [Drive | HuggingFace]
• Kvasir [Drive | HuggingFace]
• LungColon [Drive | HuggingFace]
• OCTMNIST [Drive | HuggingFace]
• RETINA [Drive | HuggingFace]

Domain Generalization Datasets

• BTMRI-P HuggingFace
• BTMRI-S HuggingFace
• BRISC HuggingFace
• BUID HuggingFace
• BUSBRA HuggingFace
• UDIAT HuggingFace

After downloading each dataset, unzip and place each under its respective directory like the following

BTMRI/
|–– BTMRI/
|   |–– glioma_tumor/
|   |–– meningioma_tumor/
|   |–– normal_brain/
|   |–– pituitary_tumor/
|–– split_BTMRI.json

Citation

If you use our work, please consider citing:

@inproceedings{koleilat2025biomedcoop,
  title={Biomedcoop: Learning to prompt for biomedical vision-language models},
  author={Koleilat, Taha and Asgariandehkordi, Hojat and Rivaz, Hassan and Xiao, Yiming},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={14766--14776},
  year={2025}
}

@article{
koleilat2026clipsvd,
title={{CLIP}-{SVD}: Efficient and Interpretable Vision{\textendash}Language Adaptation via Singular Values},
author={Taha Koleilat and Hassan Rivaz and Yiming Xiao},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2026},
url={https://openreview.net/forum?id=XYy8pwqwMR}
}

@article{koleilat2026evi,
  title={Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning},
  author={Koleilat, Taha and Rivaz, Hassan and Xiao, Yiming},
  journal={arXiv preprint arXiv:2605.26292},
  year={2026}
}