MedHall-Bench

MedHall-Bench is a field-grounded hallucination detection benchmark for medical AI assistants. It decomposes each clinical response into verifiable structured fields (dose value, unit, reference range, ICD/LOINC code, entity relation, ...) and evaluates AI outputs via per-field programmatic matching in addition to sentence-level LLM-as-Judge. Designed for use with the HolyEval framework.

⚠️ Research use only. Content is for benchmarking AI agents and should not be used for diagnosis or treatment decisions.

Motivation

A single clinical sentence can embed many independently-lethal structured fields. Existing hallucination benchmarks (HaluBench, FActScore, HALoGEN) treat each sentence as one claim and produce a single correctness score, losing the ability to localize which field went wrong. MedHall-Bench drills hallucination evaluation down to the field level across numerical, unit, code, temporal, reference-range, structural, and entity-relation dimensions — enabling field-level localization, programmatic verification, and weighted scoring of lethal errors.

Dataset Summary

112 cases across 5 hallucination types:

Dataset Structure

├── manifest.json                                # Change-detection entry point
├── README.md
└── data/
    └── 202604/
        ├── full-20260420.jsonl                  # All 112 cases
        ├── factual-20260420.jsonl               # Per-type splits
        ├── contextual-20260420.jsonl
        ├── citation-20260420.jsonl
        ├── numerical-20260420.jsonl
        ├── relational-20260420.jsonl
        └── <email_id>/                          # Virtual user (e.g. user110_AT_demo)
            ├── profile.json                     # Demographics, history, family history
            ├── exam_data.json                   # Clinical exam records
            └── timeline.json                    # Event + indicator timeline

Contextual cases reference 20 virtual users. Each case's user.target_overrides.*.email field points to the corresponding user directory. User data is included in this repository — no external dataset required.

Item Schema

Each JSONL line is a BenchItem compatible with the HolyEval framework:

{
  "id": "dh_d1_0000",
  "title": "...",
  "description": "...",
  "user": {
    "type": "manual",
    "strict_inputs": ["..."],
    "target_overrides": { "theta_api": { "email": "user110@demo" } }
  },
  "eval": {
    "evaluator": "hallucination",
    "categories": ["data_hallucination"],
    "data_hallu_type": "d1_numerical",
    "context": "...",
    "ground_truth_fields": [ { "field_name": "...", "expected_value": "...", "expected_unit": "...", "verification": "numeric_tolerance", "tolerance": 0.1 } ],
    "known_facts": ["..."],
    "threshold": 0.7
  },
  "tags": ["hallu_type:numerical", "subtype:d1_numerical", "difficulty:l1"]
}

Evaluation

Scored by the hallucination evaluator in HolyEval, which routes by type:

• factual / contextual / citation → LLM-as-Judge (0–1 score, per-item threshold)
• numerical / relational → Per-field extraction + programmatic matching against ground_truth_fields (numeric tolerance, unit normalization, code whitelist, date equality, etc.)

Citation cases additionally use NCBI PubMed/PMC + CrossRef DOI + DuckDuckGo multi-source verification (30% API weight + 70% LLM weight).

Data Access

Download the full dataset + user data

from huggingface_hub import snapshot_download

path = snapshot_download(repo_id="healthmemoryarena/MedHall-Bench", repo_type="dataset")
# path/manifest.json, path/data/202604/...

Fetch a single type

from huggingface_hub import hf_hub_download

p = hf_hub_download(
    repo_id="healthmemoryarena/MedHall-Bench",
    filename="data/202604/numerical-20260420.jsonl",
    repo_type="dataset",
)

Run with HolyEval

# Place dataset under benchmark/data/medhall/ (already mirrored in the HolyEval repo)
python -m benchmark.basic_runner medhall full-20260420 --target-model gpt-4.1
python -m benchmark.basic_runner medhall contextual-20260420 --target-type theta_api

License

Apache 2.0

Citation

@software{holyeval,
  title = {HolyEval: Virtual User Evaluation Framework for Medical AI Assistants},
  author = {Theta Health},
  url = {https://github.com/healthmemoryarena/holyeval},
  year = {2026}
}