Diagnosing Our Datasets: How does my language model learn clinical information?

The official code for Diagnosing Our Datasets: How does my language model learn clinical information? (CHIL 2025).

Install

1. Create conda environment

conda create --name eval python=3.9

2. install packages

• install torch: conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia
• install other requirments: pip install -r requirements.txt

Datasets

• CASI (The Clinical Acronym Sense Inventory): cleaned subset in datasets/casi
• MedLingo: created dataset in datasets/MedLingo/questions.csv
• Disputed Medical Claims: created two dataset with different prompts but for the same claims in datasets/disputed_medical_claims/questions.csv and datasets/disputed_medical_claims/questions_2.csv

Run evaluation on CASI

Run for CASI:

• Alpaca: scripts/Alpaca.sh
• Flan T5: scripts/Flan_T5.sh
• OLMo Instruct: scripts/OLMo_Instruct_0724.sh
• LLaMA 7B: scripts/LLaMA.sh
• LLaMA 13B: scripts/LLaMA_13b.sh
• LLaMA 33B: scripts/LLaMA_30b.sh
• LLaMA 65B: scripts/LLaMA_65b.sh
• LLaMA3 Instruct: scripts/LLaMA3_Instruct.sh
• LLaMA3.3 70B Instruct: scripts/LLaMA3_70B_Instruct.sh

Run for MedLingo questions: scripts/questions.sh

Run for disputed medical claims evaluations: scripts/disputed_medical_claims_1.sh and scripts/disputed_medical_claims_2.sh

Disputed Medical Claims Evaluation Results

In the excel file with one sheet for each prompting type: disputed_claims_evaluation.xlsx

Prompts used in this work

The Prompts we used in this work are included in the Prompts.md file.