CEBG Score: Fast Medical VQA Hallucination Metrics

This repository reproduces and extends VASE-style hallucination analysis on VQA-RAD with fast metrics:

• SE, VASE, RadFlag (baseline uncertainty/confidence)
• fast_metric_score (image vs null-image gain)
• gain_score (image vs text-only gain)
• ITBG (Important-Token Bayesian Gain)
• CEBG (Concentrated-Evidence Bayesian Gain with model-adaptive lambda sweep)

Key findings encoded in this repo

• Metrics are model-adaptable via --model_id.
• For google/medgemma-1.5-4b-it, concise prompting materially improves fast-metric stability.
• CEBG should use a lambda sweep before reporting final numbers (rather than fixed lambda).
• GREEN is used only as evaluation ground truth, not as an input to the metric formula itself.

One-command default reproduction (recommended)

bash scripts/reproduce_table.sh

This runs:

• model: google/medgemma-1.5-4b-it
• prompt style: concise
• CEBG lambda sweep + best-lambda selection by AUC

Primary outputs:

• outputs/itbg_comparison_table_medgemma15_concise.csv
• outputs/cebg_lambda_sweep_medgemma15_concise.csv
• outputs/cebg_best_medgemma15_concise.json

General experiment runner

bash scripts/run_experiment.sh <model_id> <prompt_style> <tag> <num_gpus>

Examples:

# MedGemma 1.5 concise (default behavior)
bash scripts/run_experiment.sh google/medgemma-1.5-4b-it concise medgemma15_concise 8

# MedGemma 1.5 standard prompt
bash scripts/run_experiment.sh google/medgemma-1.5-4b-it standard medgemma15_standard 8

# Legacy MedGemma 4b baseline
bash scripts/run_experiment.sh google/medgemma-4b-it standard medgemma4b_standard 8

Environment variables for lambda sweep:

LAMBDA_MIN=-1.5 LAMBDA_MAX=1.5 LAMBDA_STEP=0.125 LAMBDA_SELECT_BY=auc \
  bash scripts/run_experiment.sh google/medgemma-1.5-4b-it concise medgemma15_concise 8

Environment notes

• Python with torch, transformers, datasets, torchvision, pandas, scikit-learn, Pillow.
• Access to requested model checkpoint on Hugging Face.
• 8 GPUs recommended for fast end-to-end runs.