Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammography

---

Shantanu Ghosh¹ , Clare B. Poynton² , Shyam Visweswaran³, Kayhan Batmanghelich^1
1BU ECE, ² BUMC, ³ Pitt DBMI

---

🔥🔥🔥 Major Update: Mammo-FM (Bigger and stronger Mammography FM) 👁️

We upgraded by components of Mammo-CLIP and train it with a much larger dataset (140k patients, 800K mammograms) across 4 US institutions. Check out the following links for details

• 📄 Paper: ArXiv
• 🌐 Checkpoints and weights: Hugging Face
• 💻 Code on GitHub: batmanlab/Mammo-FM

🔥🔥🔥 Major Update: LADDER & Mammo‑CLIP for Bias Discovery & Mitigation 👁️

Interested in finding biases for breast cancer prediction models?

We’re thrilled to announce the ACL2025 paper: LADDER: Language‑Driven Slice Discovery and Error Rectification in Vision Classifiers, now integrated with Mammo‑CLIP for detecting and fixing bias in breast cancer prediction models. Check out the following links for details

• 📄 Paper: ACL2025
• 🌐 Project Page: ACL‑2025‑LADDER project
• 💻 Code on GitHub: batmanlab/Ladder

🩻 Example Use Case: Mammogram Bias Detection

Suppose your mammography classifier underperforms on dense-breast cases or specific implant types. LADDER will:

• Retrieve radiology text describing correctly vs incorrectly labeled cases
• Generate hypotheses like “missing calcifications in high-density tissue”
• Use those to define an error slice and retrain with pseudo-labeled correction

All of this is achieved without group annotations.

---

⚠️ WARNING: Look for /restricted/projectnb/batmanlab/shawn24/PhD and replace it with your own path. E.g,

.src/codebase/breastclip/data/datasets/imagetext.py, change the json path

⚠️ WARNING: There is a plethora of pre-processing settings available for RSNA and VinDr Mammo datasets. We recommend using the pre-processing discussed in the following sections. We are not responsible for any discrepancies in the results due to different pre-processing settings. If you use the VinDr png dataset uploaded in kaggle, it is fully pre-processed. Else you can use the pre-processing scripts provided in the following sections.

⚠️ WARNING: If you find the punkt_tab error, run the following command in the python environment:

import nltk

nltk.download('punkt_tab')

FAQ

After going through the instruction, it is recommended to visit the following queries logged in the issues:

• Issue-2 for any further clarification on pretraining.
• Issue-10 for getting the embeddings.
• Issue-6 for further clarification on the downstream tasks and corresponding datasets.
• Issue-13 for setting up the baselines.
• Issue-9 and Issue-35 for problems related to BioClinincalBert from Hugging Face.

If we hear more queries, we may add a separate FAQs in the future.

Table of Contents

1. Environment Setup
2. Data Download
3. Pre-processing Images
4. Data Preparation for Pretraining
5. Data Preparation for Downstream Evaluation Tasks
6. Final Dataset Directory Structures
7. Mammo-CLIP checkpoints
8. Pretraining Mammo-CLIP
9. Creating classifiers and detectors for downstream evaluations
10. Evaluation
11. Tutorial Notebooks
12. Additional Scripts
13. Mammo-FActOR
14. Citation
15. License and Copyright
16. Contact
17. Acknowledgements
18. Contributing

Environment Setup

Use environment.yml to setup the environment.

git clone git@github.com:batmanlab/Mammo-CLIP.git
cd Mammo-CLIP
conda env create --name Mammo-CLIP -f environment.yml
conda activate Mammo-CLIP

Mammo-CLIP is implemented with following specification:

• Python version: 3.8.18
• PyTorch version: 2.2.2
• CUDA version: 11.8

Data Download

Download the original versions VinDr and RSNA from the links for downstream evaluations:

• RSNA
• VinDr

For the PNG images converted from the original Dicom images, as mentioned in the preprocessing steps in the paper, refer to the following links:

• VinDr

To preprocess the dicom images directly, follow the instructions in the next section. If you downloaded the PNG images, skip the preprocessing steps.

Pre-processing images

Convert to png: RSNA

python ./src/preprocessing/preprocess_image_to_png_kaggle.py \
  --phase="test" \
  --base_folder="/restricted/projectnb/batmanlab/shawn24/PhD/RSNA_Breast_Imaging/Dataset/RSNA_Cancer_Detection"

convert to png: VinDr

python ./src/preprocessing/preprocess_image_to_png_vindr.py \
  --phase="test" \
  --base_folder="/restricted/projectnb/batmanlab/shawn24/PhD/RSNA_Breast_Imaging/Dataset/External/Vindr/vindr-mammo-a-large-scale-benchmark-dataset-for-computer-aided-detection-and-diagnosis-in-full-field-digital-mammography-1.0.0"

Data preparation for pretraining

Image-text dataset

1. Our image-text dataset is an in-house dataset. You can have your own image+text dataset where images are 2D mammograms and texts are radiology reports. If you have access to such dataset, follow the setup here. Extract the IMPRESSION and FINDINGS sections from the report and create a csv. The sample csv: img_text_dicom_consolidated_final_folds_BIRADS_num_1_report.csv

2. Note the FINDINGS and IMPRESSION columns are used to generate the text for the image. The HISTORY, FINDINGS and IMPRESSION columns contains templated text due to privacy.

3. Next run the following command to augment the text with img_text_dicom_consolidated_final_folds_BIRADS_num_1_report.csv file:

# input: img_text_dicom_consolidated_final_folds_BIRADS_num_1_report.csv
# output: clip_pretrain_100.csv

python ./src/codebase/augment_text.py \
  --dataset-path="/restricted/projectnb/batmanlab/shawn24/PhD/Mammo-CLIP/src/codebase/data_csv" \
  --csv-path="img_text_dicom_consolidated_final_folds_BIRADS_num_1_report.csv" \
  --dataset="img_text" 

The augment_text.py script will do the preprocessing for the FINDINGS and IMPRESSION columns by converting them to lower case and removing punctuations. Look at the _split_report_into_segment_concat function in the script for more details.

4. The csv file of the final image-text dataset should have the following format:

5. The final sample csv file as the output of step3 is here: clip_pretrain_100.csv. clip_pretrain_100.csv is used for pretraining the image-text variant of Mammo-CLIP.

Image-label dataset

We use VinDr dataset as image-label dataset though it can be expanded to any such datasets. Make sure that every patient should have atleast one CC and MLO image per laterality. So if you are planning to use it in the pre-training setup, use the following notebook to preprocess the VinDr dataset:

./src/codebase/notebooks/preprocess-clip/VinDr.ipynb

When you download the VinDr dataset, you will get these two csv files: breast-level_annotations.csv and finding_annotations.csv . We preprocess the finding_annotations.csv file to get vindr_detection_v1_folds.csv . VinDr.ipynb notebook requires vindr_detection_v1_folds.csv file as input and generate clip_vindr_final.csv file.

Both clip_pretrain_100.csv and clip_vindr_final.csv files are used for pretraining the image-text and image-label variant of Mammo-CLIP.*

The csv file of the final image-label (VinDr) dataset should have the following format:

Explanation for CC_FINDING and MLO_FINDING Columns: In the above table, for the row, CC_FINDING can be expanded as:

[
  [+ve findings for CC_img if laterality of CC_img is R],
  [+ve findings for CC_img if laterality of CC_img is L],
  [-ve findings for CC_img if laterality of CC_img is R],
  [-ve findings for CC_img if laterality of CC_img is L],
]

As VinDr contains a single image per patient-laterality combo, we did n

Similarly, in the above table, for the row, MLO_FINDING can be expanded as:

[
  [+ve findings for MLO_img if laterality of MLO_img is R],
  [+ve findings for MLO_img if laterality of MLO_img is L],
  [-ve findings for MLO_img if laterality of MLO_img is R],
  [-ve findings for MLO_img if laterality of MLO_img is L],
]

Data preparation for downstream evaluation tasks

Use the following csv files as metadata for the downstream tasks (classification, detection, zero-shot):

For detection/localization tasks, we have included the coordinates of the resized bounding boxes of VinDr in the above csv file. Somebody interested in resizing the bounding boxes by themselves, run the following command with finding_annotations.csv file as input:

python ./src/preprocessing/preprocess_VinDr_detector.py

Final dataset directory structures

Image+Text pretraining dataset

.
├── list_tree_files.sh
├── img_text_dicom_consolidated_final_folds_BIRADS_num_1_report.csv
├── clip_pretrain_100.csv
└── DICOM/images_png_CC_MLO/
    ├── Patient_100/
    │   ├── 1.png
    │   ├── 2.png
    └── Patient_200/
        ├── 3.png
        ├── 4.png
        ├── 53.png
        ├── 6.png
        └── 7.png
        

VinDr

.
├── breast-level_annotations.csv
├── finding_annotations.csv
├── vindr_detection_v1_folds.csv 
├── clip_vindr_final.csv
└── images_png/
    ├── c7811f4575c1229ad4a7606de49ea68f/
    │   ├── 9eb4650a2b630e44074c403f6127c5a1.png
    │   ├── cc3fdc5d733a671f3000e20838e192d9.png
    │   ├── 181fd193d3b785dc9faafdaa8e1695fc.png
    │   └── 55eb5ea616abacd225e584ffc8be57da.png
    └── a1dd219b28806fc295fac20ceb147870/
        ├── 887cdcc99ebed66bd062ada6c8210152.png
        ├── 36f2921a2ac19eba7420c591c4c07ae4.png
        ├── 12dc17dfd9d30ea7c0c1ccb33a505085.png
        └── e22e4f297b4c82279e7b78a98417a6cd.png

RSNA

.
├── train_folds.csv
├── train_images_png/
    ├── 59549/
    │   ├── 1154694388.png
    │   ├── 1192817932.png
    │   ├── 1979035704.png
    │   ├── 2022274082.png
    │   ├── 431013616.png
    │   ├── 457600713.png
    │   ├── 78005871.png
    │   └── 856162422.png
    └── 28242/
        ├── 1966298736.png
        ├── 233201459.png
        ├── 349787619.png
        └── 98615814.png

Mammo-CLIP checkpoints

Following are the pre-training checkpoints of Mammo-CLIP:

We have also uploaded the downstream checkpoints for classification and localization (both linear probe and finetuning) with the image encoder of Efficient-Net B5 Mammo-CLIP for fold 0 here.

Pretraining Mammo-CLIP

For pretraining Efficient-Net B5 Mammo-CLIP with a single GPU, use the following command:

python ./src/codebase/train.py --config-name pre_train_b5_clip.yaml

For pretraining Efficient-Net B5 Mammo-CLIP with a 4 GPUs using pytorch-ddp, use the following command:

torchrun --nproc_per_node=4 ./src/codebase/train.py --config-name pre_train_b5_clip.yaml

All the yaml files for the config are found here.

• Use pre_train_b5_clip.yaml for pre-training image-text variant of Efficient-Net B5 Mammo-CLIP
• Use pre_train_b2_clip.yaml for pre-training image-text variant of Efficient-Net B2 Mammo-CLIP
• Use pre_train_b5_w_vindr_clip.yaml for pre-training image-text + image-label variant of Efficient-Net B5 Mammo-CLIP

Creating classifiers and detectors for downstream evaluations

• For creating classifiers for downstream evaluations using the image encoder of Mammo-CLIP, use the class BreastClipClassifier in breast-clip-classifier.py file.
• For creating detectors for downstream evaluations using the image encoder of Mammo-CLIP, use the function RetinaNet_efficientnet in detector_model.py file.

Evaluation

Zero-shot evaluation of Mammo-CLIP

FOLD=0
CKPT="b2-model-best-epoch-10.tar"
DIR="./Mammo-CLIP/src/codebase/outputs/img_text_clip/b2_detector_period_n"
FULL_CKPT="$DIR/checkpoints/fold_$FOLD/$CKPT"

python ./src/codebase/eval_zero_shot_clip.py \
  --config-name zs_clip.yaml hydra.run.dir=$DIR model.clip_check_point=$FULL_CKPT

Adjust the CKPT and DIR variables according to your setup.

Linear probe vision encoder Mammo-CLIP on target classification task

python ./src/codebase/train_classifier.py \
  --data-dir '/restricted/projectnb/batmanlab/shawn24/PhD/RSNA_Breast_Imaging/Dataset' \
  --img-dir 'External/Vindr/vindr-mammo-a-large-scale-benchmark-dataset-for-computer-aided-detection-and-diagnosis-in-full-field-digital-mammography-1.0.0/images_png' \
  --csv-file 'External/Vindr/vindr-mammo-a-large-scale-benchmark-dataset-for-computer-aided-detection-and-diagnosis-in-full-field-digital-mammography-1.0.0/vindr_detection_v1_folds.csv' \
  --clip_chk_pt_path "/restricted/projectnb/batmanlab/shawn24/PhD/Mammo-CLIP/src/codebase/outputs/img_text_clip/b5_detector_period_n/checkpoints/fold_0/b5-model-best-epoch-7.tar" \
  --data_frac 1.0 \
  --dataset 'ViNDr' \
  --arch 'breast_clip_det_b5_period_n_lp' \
  --label "Mass" \
  --epochs 30 \
  --batch-size 8 \
  --num-workers 0 \
  --print-freq 10000 \
  --log-freq 500 \
  --running-interactive 'n' \
  --n_folds 1 \
  --lr 5.0e-5 \
  --weighted-BCE 'y' \
  --balanced-dataloader 'n' 

• data-dir: root directory of the dataset
• img-dir: directory containing images, absolute path: data-dir/img-dir
• csv-file: csv file containing image paths and labels, absolute path: data-dir/csv-file
• clip_chk_pt_path: path to the checkpoint of the pre-trained Mammo-CLIP model
• dataset: dataset name, e.g., ViNDr or RSNA
• data_frac: fraction of the dataset to use for training, e.g., 1.0, 0.5 etc
• arch: architecture of the model, e.g., breast_clip_det_b5_period_n_lp for Efficient-Net B5 or breast_clip_det_b2_period_n_lp for Efficient-Net B2, pretrained on img_text dataset. Also, breast_clip_det_b5_period_n_lp for Efficient-Net B5 or breast_clip_det_b2_period_n_lp for Efficient-Net B2, pretrained on img_text and VinDr datasets.
• label: target label for classification, e.g., Mass, Suspicious_Calcificationor density for ViNDr dataset; cancer for RSNA dataset
• running-interactive: running on interactive mode. In this mode,the training will be done using 100 samples for sanity check

Finetune vision encoder Mammo-CLIP on target classification task

python ./src/codebase/train_classifier.py \
  --data-dir '/restricted/projectnb/batmanlab/shawn24/PhD/RSNA_Breast_Imaging/Dataset' \
  --img-dir 'External/Vindr/vindr-mammo-a-large-scale-benchmark-dataset-for-computer-aided-detection-and-diagnosis-in-full-field-digital-mammography-1.0.0/images_png' \
  --csv-file 'External/Vindr/vindr-mammo-a-large-scale-benchmark-dataset-for-computer-aided-detection-and-diagnosis-in-full-field-digital-mammography-1.0.0/vindr_detection_v1_folds.csv' \
  --clip_chk_pt_path "/restricted/projectnb/batmanlab/shawn24/PhD/Mammo-CLIP/src/codebase/outputs/img_text_clip/b5_detector_period_n/checkpoints/fold_0/b5-model-best-epoch-7.tar" \
  --data_frac 1.0 \
  --dataset 'ViNDr' \
  --arch 'breast_clip_det_b5_period_n_ft' \
  --label "Mass" \
  --epochs 30 \
  --batch-size 8 \
  --num-workers 0 \
  --print-freq 10000 \
  --log-freq 500 \
  --running-interactive 'n' \
  --n_folds 1 \
  --lr 5.0e-5 \
  --weighted-BCE 'y' \
  --balanced-dataloader 'n'

• data-dir: root directory of the dataset
• img-dir: directory containing images, absolute path: data-dir/img-dir
• csv-file: csv file containing image paths and labels, absolute path: data-dir/csv-file
• clip_chk_pt_path: path to the checkpoint of the pre-trained Mammo-CLIP model
• dataset: dataset name, e.g., ViNDr or RSNA
• data_frac: fraction of the dataset to use for training, e.g., 1.0, 0.5 etc
• arch: arch: architecture of the model, e.g., breast_clip_det_b5_period_n_ft for Efficient-Net B5 or breast_clip_det_b2_period_n_ft for Efficient-Net B2, pretrained on img_text dataset. Also, breast_clip_det_b5_period_n_ft for Efficient-Net B5 or breast_clip_det_b2_period_n_ft for Efficient-Net B2, pretrained on img_text and VinDr datasets.
• label: target label for classification, e.g., Mass, Suspicious_Calcificationor density for ViNDr dataset; cancer for RSNA dataset
• running-interactive: running on interactive mode. In this mode,the training will be done using 100 samples for sanity check

Linear probe vision encoder Mammo-CLIP on target detection task

python ./src/codebase/train_detector.py \
  --data-dir '/restricted/projectnb/batmanlab/shawn24/PhD/RSNA_Breast_Imaging/Dataset' \
  --img-dir 'External/Vindr/vindr-mammo-a-large-scale-benchmark-dataset-for-computer-aided-detection-and-diagnosis-in-full-field-digital-mammography-1.0.0/images_png' \
  --csv-file 'External/Vindr/vindr-mammo-a-large-scale-benchmark-dataset-for-computer-aided-detection-and-diagnosis-in-full-field-digital-mammography-1.0.0/vindr_detection_v1_folds.csv' \
  --clip_chk_pt_path "/restricted/projectnb/batmanlab/shawn24/PhD/Mammo-CLIP/src/codebase/outputs/img_text_clip/b5_detector_period_n/checkpoints/fold_0/b5-model-best-epoch-7.tar" \
  --dataset 'ViNDr' \
  --arch 'breast_clip_b5' \
  --epochs 120 \
  --batch-size 7 \
  --freeze_backbone "y" \
  --data_frac 1.0 \
  --concepts 'Mass' \
  --print-freq 5000 \
  --log-freq 300 \
  --running-interactive 'n' \
  --focal-alpha 0.25 \
  --focal-gamma 2.0 \
  --score-threshold 0.2

• data-dir: root directory of the dataset
• img-dir: directory containing images, absolute path: data-dir/img-dir
• csv-file: csv file containing image paths and labels, absolute path: data-dir/csv-file
• clip_chk_pt_path: path to the checkpoint of the pre-trained Mammo-CLIP model
• dataset: dataset name, e.g., ViNDr
• data_frac: fraction of the dataset to use for training, e.g., 1.0, 0.5 etc
• arch: architecture of the model, e.g., breast_clip_b5 for Efficient-Net B5 or breast_clip_b2 for Efficient-Net B2, pretrained on img_text dataset. Similarly, breast_clip_b5_vindr for Efficient-Net B5 or breast_clip_b2 for Efficient-Net B2, pretrained on img_text and VinDr datasets.
• concepts: target label for classification, e.g., Mass, Suspicious Calcification for ViNDr dataset
• running-interactive: running on interactive mode. In this mode,the training will be done using 100 samples for sanity check
• freeze_backbone: freeze the backbone of the model, for linear probe, set to y

Finetune vision encoder Mammo-CLIP on target detection task

python ./src/codebase/train_detector.py \
  --data-dir '/restricted/projectnb/batmanlab/shawn24/PhD/RSNA_Breast_Imaging/Dataset' \
  --img-dir 'External/Vindr/vindr-mammo-a-large-scale-benchmark-dataset-for-computer-aided-detection-and-diagnosis-in-full-field-digital-mammography-1.0.0/images_png' \
  --csv-file 'External/Vindr/vindr-mammo-a-large-scale-benchmark-dataset-for-computer-aided-detection-and-diagnosis-in-full-field-digital-mammography-1.0.0/vindr_detection_v1_folds.csv' \
  --clip_chk_pt_path "/restricted/projectnb/batmanlab/shawn24/PhD/Mammo-CLIP/src/codebase/outputs/img_text_clip/b5_detector_period_n/checkpoints/fold_0/b5-model-best-epoch-7.tar" \
  --dataset 'ViNDr' \
  --arch 'breast_clip_b5' \
  --epochs 120 \
  --batch-size 7 \
  --freeze_backbone "n" \
  --data_frac 1.0 \
  --concepts 'Mass' \
  --print-freq 5000 \
  --log-freq 300 \
  --running-interactive 'n' \
  --focal-alpha 0.25 \
  --focal-gamma 2.0 \
  --score-threshold 0.2

• data-dir: root directory of the dataset
• img-dir: directory containing images, absolute path: data-dir/img-dir
• csv-file: csv file containing image paths and labels, absolute path: data-dir/csv-file
• clip_chk_pt_path: path to the checkpoint of the pre-trained Mammo-CLIP model
• dataset: dataset name, e.g., ViNDr
• data_frac: fraction of the dataset to use for training, e.g., 1.0, 0.5 etc
• arch: architecture of the model, e.g., breast_clip_b5 for Efficient-Net B5 or breast_clip_b2 for Efficient-Net B2, pretrained on img_text dataset. Similarly, breast_clip_b5 for Efficient-Net B5 or breast_clip_b2 for Efficient-Net B2, pretrained on img_text and VinDr datasets.
• concepts: target label for classification, e.g., Mass, Suspicious Calcification for ViNDr dataset
• running-interactive: running on interactive mode. In this mode,the training will be done using 100 samples for sanity check
• freeze_backbone: freeze the backbone of the model, for finetune, set to n

Tutorial Notebooks

• For a quick look at setting up the downstream classifier, follow the notebook: Downstream_classifier_tutorial.ipynb
• For a quick look at downloading the image embeddings from the vision encoder of Mammo-CLIP, follow the notebook: Get_Embedding_Vision_encoder_Mammo_CLIP_tutorial.ipynb

Additional scripts

For all the training scripts, we add them in the scripts directory:

Mammo-FActOR

For training Mammo-FActOR, refer to the following notebook.

Citation

@InProceedings{10.1007/978-3-031-72390-2_59,
author="Ghosh, Shantanu
and Poynton, Clare B.
and Visweswaran, Shyam
and Batmanghelich, Kayhan",
editor="Linguraru, Marius George
and Dou, Qi
and Feragen, Aasa
and Giannarou, Stamatia
and Glocker, Ben
and Lekadir, Karim
and Schnabel, Julia A.",
title="Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammography",
booktitle="Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024",
year="2024",
publisher="Springer Nature Switzerland",
address="Cham",
pages="632--642",
abstract="The lack of large and diverse training data on Computer-Aided Diagnosis (CAD) in breast cancer detection has been one of the concerns that impedes the adoption of the system. Recently, pre-training with large-scale image text datasets via Vision-Language models (VLM) (e.g., CLIP) partially addresses the issue of robustness and data efficiency in computer vision (CV). This paper proposes Mammo-CLIP, the first VLM pre-trained on a substantial amount of screening mammogram-report pairs, addressing the challenges of dataset diversity and size. Our experiments on two public datasets demonstrate strong performance in classifying and localizing various mammographic attributes crucial for breast cancer detection, showcasing data efficiency and robustness similar to CLIP in CV. We also propose Mammo-FActOR, a novel feature attribution method, to provide spatial interpretation of representation with sentence-level granularity within mammography reports. Code is available publicly: https://github.com/batmanlab/Mammo-CLIP.",
isbn="978-3-031-72390-2"
}

License and Copyright

This project is licensed under the Creative Commons Attribution–NonCommercial–ShareAlike 4.0 International (CC BY-NC-SA 4.0) license.

You may use, share, and adapt this work for non-commercial research and educational purposes only, provided that you:

• Give appropriate credit to the original authors.
• Indicate if changes were made.
• Distribute any derivative works under the same license (CC BY-NC-SA 4.0).

Commercial use, including model deployment, integration into for-profit products or services, or sale of derivatives, is strictly prohibited.

Copyright © Batman Lab, 2024
License text: Creative Commons BY-NC-SA 4.0

Contact

For any queries, contact Shantanu Ghosh (email: shawn24@bu.edu)

Acknowledgements

Special thanks to Boston University Masters students Abhishek Varshney & Akshat Gurbuxani for enabling multi-GPU support to Mammo-CLIP.

Contributing

Did you try Mammo-CLIP on other datasets containing 2D-Mammograms and want to report the results? Feel free to send a pull request.