Dataset Card: Harvard-FairVLMed

Dataset Summary

Harvard-FairVLMed is the first fair vision-language medical dataset designed for studying fairness in medical vision-language (VL) foundation models. It contains 10,000 SLO fundus images paired with de-identified clinical notes and comprehensive demographic annotations, enabling in-depth fairness analysis across four protected attributes: race, gender, ethnicity, and preferred language.

This dataset was introduced at CVPR 2024: FairCLIP: Harnessing Fairness in Vision-Language Learning.

Dataset Details

Dataset Description

• Curated by: Yan Luo, Min Shi, Muhammad Osama Khan, Muhammad Muneeb Afzal, Hao Huang, Shuaihang Yuan, Yu Tian, Luo Song, Ava Kouhana, Tobias Elze, Yi Fang, Mengyu Wang
• License: CC BY-NC-ND 4.0 — non-commercial research only
• Paper: CVPR 2024
• Contact: harvardophai@gmail.com, harvardairobotics@gmail.com

Data Fields

Each subject is stored as a .npz file (data_xxxxx.npz) containing:

A metadata overview is provided in data_summary.csv. GPT-4-summarized versions of all notes are available in gpt-4_summarized_notes.csv.

Demographics

6 demographic attributes are annotated per patient: age, gender, race, ethnicity, preferred language, and marital status.

Uses

Direct Use

• Fairness benchmarking for medical vision-language foundation models (e.g., CLIP, BLIP-2)
• Analysis of demographic disparities in ophthalmic AI across race, gender, ethnicity, and language
• Development and evaluation of bias-mitigation methods for VL models in healthcare
• Medical note summarization and multimodal learning research

Out-of-Scope Use

Clinical decisions, patient care, or any commercial application. This dataset shall not be used for clinical decisions at any time.

Access

The "Harvard" designation indicates the dataset originates from the Department of Ophthalmology at Harvard Medical School. It does not imply endorsement, sponsorship, or legal responsibility by Harvard University or Harvard Medical School.

Citation

BibTeX:

@inproceedings{luo2024fairclip,
  title={Fairclip: Harnessing fairness in vision-language learning},
  author={Luo, Yan and Shi, Min and Khan, Muhammad Osama and Afzal, Muhammad Muneeb and Huang, Hao and Yuan, Shuaihang and Tian, Yu and Song, Luo and Kouhana, Ava and Elze, Tobias and others},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={12289--12301},
  year={2024}
}

APA:

Luo, Y., Shi, M., Khan, M. O., Afzal, M. M., Huang, H., Yuan, S., Tian, Y., Song, L., Kouhana, A., Elze, T., Fang, Y., & Wang, M. (2024). FairCLIP: Harnessing Fairness in Vision-Language Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024), 12289–12301.