CEBaG: Confidence-Evidence Bayesian Gain

CEBaG is a deterministic, training-free hallucination-detection score for medical visual question answering (VQA). Existing detectors such as VASE and semantic entropy need many stochastic generations and an external NLI model; CEBaG instead reads two teacher-forced forward passes of the same model and combines per-token confidence with the evidence the image actually contributes:

CEBaG = sigma * (1 + E)
  sigma = std of the per-token log-probabilities of the generated answer
  E     = |G| / L
  G     = logP(r | x_v, x_q) - logP(r | x_q)   (image-conditioned gain)
  L     = number of answer tokens

Here r is the generated answer, x_v the image, and x_q the question. sigma captures token-level uncertainty; G measures how much the image changed the answer's likelihood relative to a text-only pass. A high score flags a likely hallucination.

Key result. Across 16 (model x dataset) settings, CEBaG attains the best AUC in 13/16 settings against the SE / VASE / RadFlag baselines, with a mean AUC of 67.9 vs 59.6 for VASE (+8.2 on average), while needing only 2 deterministic forward passes and no external model (see outputs/all_results_cebg_pp.csv, method CEBG_pp).

Repo layout

.
|-- dataset_loader.py          # unified VQA-RAD / SLAKE / PathVQA loader
|-- requirements.txt
|-- VASE/                       # answer generation + SE/VASE/RadFlag + GREEN driver
|   |-- main_hall_det_medgemma.py   # step 1, MedGemma family
|   |-- main_hall_det_llava.py      # step 1, LLaVA-Med / HuatuoGPT
|   |-- llava_utils.py              # LLaVA-family loading / log-prob helpers
|   |-- green_eval_generic.py       # GREEN scoring of one chunk
|   |-- run_green_for_prefix.py     # step 2 driver (chunks + merge)
|   |-- SeEntLib/                   # semantic-entropy + DeBERTa NLI
|   `-- LICENSE                     # original VASE license
|-- GREEN/                      # GREEN package (ground-truth labeller)
|   |-- green_score/                # importable `green_score` package
|   |-- setup.py
|   `-- README.md
|-- scripts/
|   |-- run_experiment.sh           # one (model,dataset,mode,prompt) tuple
|   |-- run_all.sh                  # all 16 runs + table scripts
|   |-- lambda_analysis.py          # step 3: per-sample sigma + gain
|   |-- compute_cebg_pp.py          # main results + sigma/E/CEBaG_lambda ablation
|   |-- compute_token_baselines.py  # extra token baselines (AvgProb, MaxProb, ...)
|   |-- computational_overhead.py   # efficiency / forward-pass counts
|   `-- green_threshold_sensitivity.py  # GREEN-threshold robustness
`-- outputs/                    # generated results (gitignored; see results zip)

Setup

pip install -r requirements.txt
pip install -e GREEN          # installs the `green_score` package

Requirements / access:

• GPUs. Generation and the per-sample passes are GPU-bound; the pipeline is written for multi-GPU machines (default 8). Set NUM_GPUS to your count.
• Model access. Gated HF checkpoints: google/medgemma-4b-it, google/medgemma-1.5-4b-it, chaoyinshe/llava-med-v1.5-mistral-7b-hf, FreedomIntelligence/HuatuoGPT-Vision-7B-hf.
• GREEN. Ground-truth labels use StanfordAIMI/GREEN-radllama2-7b (downloaded automatically on first use).

Reproduce

Each run produces, for <prefix> = <dataset>_<mode>_<model_tag>: outputs/<prefix>_hallscore.csv (answers + SE/VASE/RadFlag), outputs/<prefix>_green.csv (GREEN labels), and outputs/lambda_analysis/<prefix>_persample.csv (CEBaG sigma + gain).

# one tuple
bash scripts/run_experiment.sh google/medgemma-4b-it vqa_rad open \
     medgemma standard 8 medgemma

# all 16 tuples, then the four table scripts
bash scripts/run_all.sh

The table scripts then write to outputs/: all_results_cebg_pp.csv (per setting, AUC/AUG for SE / VASE / RadFlag and the CEBaG variants sigma_only, E_only, CEBG_pp = CEBaG, CEBG_tuned = the tuned CEBaG_lambda upper bound); token_baseline_results.csv (AvgProb / MaxProb / AvgEnt / MaxEnt); computational_overhead.csv (forward-pass / cost summary); and green_threshold_sensitivity.csv (AUC vs the GREEN hallucination threshold).

Precomputed results

To rebuild every result without any GPU or model access, unzip the companion archive into outputs/ and run only the table scripts:

unzip cebag_results.zip -d outputs/
python scripts/compute_cebg_pp.py                          # main results + ablation
python scripts/compute_token_baselines.py --aggregate_only # token baselines (no GPU)
python scripts/computational_overhead.py                   # efficiency
python scripts/green_threshold_sensitivity.py              # GREEN-threshold robustness

--aggregate_only rebuilds the token baselines from the precomputed token-stat CSVs without loading any model. computational_overhead.py (Table 3) reports timings that were measured from the original GPU run logs; those logs are not in the archive, so the script prints the shipped computational_overhead.csv and does not overwrite it.

The archive ships every *_hallscore.csv, *_green.csv, and lambda_analysis/*_persample.csv plus the aggregated result CSVs (large logs and per-chunk *_part_*.csv shards are excluded). See RESULTS_README.txt inside the archive.

Reproducibility notes

• Headline numbers (Table 1 + ablation) are exactly reproducible from the released CSVs: the scoring/aggregation layer is fully deterministic, and rerunning the table scripts regenerates all_results_cebg_pp.csv and reproduces every AUC/AUG value.
• From-scratch GPU regeneration is approximate. Answer generation uses sampling (temperature=0.1 for the answer; T=1.0 for the SE/VASE/RadFlag draws). Generation is seeded (set_seed(42)), but GPU/library nondeterminism means the per-sample *_hallscore.csv / *_persample.csv are not bit-identical run to run; the deterministic CEBaG/baseline scoring on top still reproduces the reported numbers.
• Table 3 timings are measured (NVIDIA H100, LLaVA-Med on PathVQA) and shipped in computational_overhead.csv; recomputing them requires the original run logs.
• AUG tie-breaks. For highly-tied baseline scores, per-cell AUG depends on the sort tie-break; the released numbers use NumPy's default argsort. All AUC values and the reported averages are tie-independent.

Models and datasets

Citation

@article{asadi2026cebag,
  title   = {Deterministic Hallucination Detection in Medical VQA via
             Confidence-Evidence Bayesian Gain},
  author  = {Asadi, Mohammad and Nedaee, Tahoura and O'Sullivan, Jack W. and
             Ashley, Euan and Adeli, Ehsan},
  journal = {arXiv preprint arXiv:2603.21693},
  year    = {2026}
}

Paper: https://arxiv.org/abs/2603.21693

License and acknowledgements

Code released under the MIT License (see LICENSE), © 2026 Mohammad Asadi, Tahoura Nedaee, Jack W. O'Sullivan, Euan Ashley, Ehsan Adeli. The accompanying paper (arXiv:2603.21693) is distributed under CC BY 4.0. This work builds on and vendors:

• VASE (visual-augmented semantic entropy) - VASE/, see VASE/LICENSE;
• the semantic-entropy library - VASE/SeEntLib/;
• GREEN (StanfordAIMI), used only as evaluation ground truth - GREEN/, see GREEN/README.md.

We thank the authors of these projects and of the MedGemma, LLaVA-Med, and HuatuoGPT-Vision models and the VQA-RAD, SLAKE, and PathVQA datasets.