VLM Medical VQA Benchmark

A rigorous, reproducible benchmark comparing medical-domain VLMs against general-purpose VLMs on medical visual question answering (VQA). The project evaluates five models across four datasets using a six-metric evaluation suite, with post-hoc statistical significance testing.

Internship research project — all evaluations are zero-shot, run on Kaggle T4 GPUs.

---

Models Evaluated

Datasets

---

Repository Structure

vlm_benchmark/
│
├── README.md
├── requirements.txt
│
├── docs/                              # All reports and analysis documents
│   ├── report.md                      # Main benchmark report (start here)
│   ├── error_analysis_report.md
│   ├── dicom_sample_dataset_report.md
│   ├── research_gap_analysis.md
│   └── statistical_significance_results.md
│
├── notebooks/                         # Jupyter / Kaggle notebooks (numbered pipeline)
│   ├── 01_dataset_verification.ipynb
│   ├── 02_inference_harness.ipynb     # v1 baseline inference
│   ├── 03_inference_harness_v2.ipynb  # v2 prompt-engineered inference (primary)
│   ├── 04_llm_judge.ipynb             # LLM-as-a-Judge pipeline
│   ├── 05_error_analysis.ipynb
│   ├── 06_statistical_significance.ipynb
│   ├── 07-medical-on-general-2.ipynb
│   ├── LLMasaJudgeMedonGen.ipynb
│   ├── dicom-evaluation-multislice-huatuogpt.ipynb
│   └── dicom/                         # DICOM-specific experiments
│       ├── dicom.ipynb
│       └── dicom-evaluation.ipynb
│
├── scripts/                           # Standalone Python scripts
│   ├── make_notebook.py
│   ├── score_new_outputs.py
│   └── statistical_significance.py   # Post-hoc bootstrap CI + permutation tests
│
├── outputs/
│   ├── inference/                     # Active v2 JSONL predictions (14 files)
│   ├── judge/                         # LLM-judge JSONL files + summary CSV
│   ├── error_analysis_csvs/           # Per-error-type analysis CSVs (A1–C3)
│   ├── dicom_evaluation/              # DICOM clinical VQA outputs
│   └── _archive/                      # Superseded files (v1, rescored, S-CoT)
│       ├── v1_predictions/
│       ├── rescored/
│       ├── scot_experiment/
│       └── judge_dry_run/
│
├── results/                           # Final computed outputs
│   ├── findings.json
│   ├── statistical_significance_results.json
│   └── statistical_significance_summary.md
│
└── data/                              # Local dataset assets (see .gitignore)
    ├── clinical_metadata.csv
    ├── clinical_vqa_dataset.jsonl
    ├── dicom_files/                   # ⚠ Not tracked — patient DICOM data
    └── slake_imgs/                    # ⚠ Not tracked — 212 MB image archive

---

Key Results (v2 Protocol, Zero-Shot)

All differences verified with paired permutation tests (n=10,000). See results/statistical_significance_results.json.

Notable Findings

• Domain pre-training > parameter count: MedGemma-4B outperforms all 7B models on medical VQA (p<0.001).
• LLaVA-Med ≡ LLaVA-1.6 on SLAKE F1 (p=0.963, ns): medical fine-tuning confers zero advantage over the general-purpose model.
• MedGemma vs HuatuoGPT on VQA-RAD is only marginally significant (p=0.022); indistinguishable by Judge Accuracy (p=0.067, ns).

---

Evaluation Metrics

---

Running the Statistical Analysis

# From the repo root
pip install numpy
python scripts/statistical_significance.py
# → results/statistical_significance_results.md
# → results/statistical_significance_results.json

---

Infrastructure

• Hardware: Kaggle NVIDIA Tesla T4 (16 GB VRAM)
• Quantization: 4-bit NF4 via bitsandbytes for 7B models
• Inference speed: ~4s/sample (4B), ~6s/sample (7B quantized)
• Full pipeline runtime: ~9 hours for all 7,500+ LLM-judge evaluations

Limitations

• All evaluations are zero-shot — not comparable to supervised fine-tuning results
• VQA-RAD uses the public split which may have training contamination
• LLM judge uses an 8B model; a frontier judge (GPT-4o) would be more reliable

---

Citation / Report

See docs/report.md for the full technical report including pipeline evolution, engineering challenges, and diagnostic analysis.