Histo-VL: Evaluating Robustness and Clinical Reliability of Histopathology Vision-Language Foundation Models

Mohamed Bin Zayed University of Artificial Intelligence, Khalifa University and Shaukat Khanum Cancer Hospital

Histo-VL is a large-scale multimodal benchmark for stress-testing histopathology vision–language models (VLMs) across 26 organs, 32 cancer types, and 5.3M patches from 14 international cohorts. Organized into 7 clinically defined diagnostic tasks, it evaluates 9 VLMs under magnification shifts, stain variation, adversarial perturbations, and textual corruptions. Results reveal that top models (KEEP, CONCH) drop up to 34% in balanced accuracy under adversarial attack, are universally miscalibrated, and fall short of expert pathologists in cancer grading — exposing standard benchmark scores as an incomplete proxy for clinical readiness. Histo-VL advocates for systematic stress testing as a prerequisite before deploying histopathological VLMs in clinical settings. The paper was accepted in MIUA 2026. Link to the paper will be provided upon publishing. The following is a link to an earlier version preprint

Installation

1. Clone this repo and the CONCH github repository.
2. Create an environment using the provided .yml file

conda env create -f benchmark.yml
conda activate benchmark
cd CONCH
pip install --upgrade pip
pip install -e .
pip install ../timm_ctp.tar --no-deps

3. Install requirements in the updated_requirements file.
4. Add the relative paths in :
   • The paths to the data, caching and results folder in the dotenv file : "plip/reproducibility/config_example.env"
   • The paths to the CONCH and MI-Zero models in : "plip/reproducibility/factory.py". The CONCH models can be downloaded from here, and MI-Zero models from here
   • Path to MI-Zero configuration path in : "plip/src/models/factory.py"
5. The captions for both available in the BenchCaption folder as well as the captions and splits for other experiments as well.

Running Zero-shot

6. Run the following bash file in "plip/reproducibility" :

bash zero_shot.sh

7. To run experiments for the adversarial image or text corruptions or caption ensembling, you can specify that in the zero_shot.sh bash file or in plip/reproducibility/train.py
8. To create artifacts of a specific dataset, you can run the script at plip/reproducibility/generate_validation_datasets/artifactor.sh, which allows the creation of multiple types of artifacts, including blurring, rotation, addition of fat, squamous epithelial cells, and more.

Datasets

Current models included in the evaluation