CV-MedBench: Clear-to-Vague Medical Question Answering Benchmark

CV-MedBench is a benchmark designed to study the effect of input ambiguity in Medical Question Answering (QA).
It provides paired clear and ambiguous (vague) versions of real medical exam questions, enabling controlled evaluation of how QA systems—especially large language models—respond to underspecified user inputs in safety-critical medical settings.

The benchmark is motivated by the observation that ambiguity in user queries induces aleatoric uncertainty, which cannot be resolved by model capacity alone and often leads to unsafe or unreliable medical responses. CV-MedBench isolates this factor by ensuring that paired questions differ only in clarity while sharing the same underlying medical intent and correct answer.

---

Citation

If you use CV-MedBench in academic work, please cite the following paper:

Mind the Ambiguity: Aleatoric Uncertainty Quantification in LLMs for Safe Medical Question Answering
Yaokun Liu et al.
Proceedings of The Web Conference (WWW) 2026.

The dataset construction, ambiguity generation process, and validation protocol are described in detail in the paper.

---

Source Datasets

CV-MedBench is constructed from three publicly available medical QA datasets:

• MedQA
  Jin et al., What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams.

• MedMCQA
  Pal et al., MedMCQA: A Large-scale Multi-subject Multi-choice Dataset for Medical Domain Question Answering.

• MedExQA
  Li et al., MedExQA: Medical Licensing Examination Question-Answering Dataset across Multiple Languages.

For each source dataset, original exam-style questions are treated as clear inputs. Corresponding ambiguous versions are generated through controlled linguistic rewriting that removes or weakens essential clinical information while preserving the original answer.

---

Dataset Organization

CV-MedBench is organized by source dataset, each exposed as a separate configuration on Hugging Face:

CV-MedBench/
├── cv_medqa/
│ ├── train/
│ ├── train_clear/
│ ├── train_vague/
│ ├── test/
│ ├── test_clear/
│ └── test_vague/
├── cv_medmcqa/
│ ├── train/
│ ├── train_clear/
│ ├── train_vague/
│ ├── test/
│ ├── test_clear/
│ └── test_vague/
└── cv_medexqa/
├── test/
├── test_clear/
└── test_vague/

Each question appears in both clear and ambiguous forms.
Samples originating from the same exam question share the same id, allowing direct alignment between clarity conditions.

The *_clear and *_vague subsets contain exclusively one version of the question text, which supports evaluation settings where models are tested on a single clarity condition without contrastive pairing.

---

Data Schema

All subsets follow a unified Arrow-based schema compatible with the Hugging Face datasets library.

Samples with the same id represent the same underlying medical question under different clarity conditions.

---

Intended Use

CV-MedBench is intended for research on:

• Robust Medical QA under underspecified inputs
• Aleatoric uncertainty estimation and calibration
• Ambiguity detection and clarification strategies
• Safety evaluation of LLM-based medical assistants

The dataset is not intended for direct clinical deployment or medical decision-making.

---