ClinSeek-Bench

ClinSeek-Bench is the evaluation suite introduced in ClinSeekAgent: Automating Multimodal Evidence Seeking for Agentic Clinical Reasoning. It evaluates clinical reasoning under two paired settings with the same task definitions and answer labels:

• Curated Input: the model answers from the evidence package provided by the source benchmark.
• Automated Evidence-Seeking: the curated context is removed, and the model must retrieve evidence from raw clinical data using ClinSeekAgent tools.

This Hugging Face dataset release provides the metadata needed to reconstruct ClinSeek-Bench. It is not a fully materialized benchmark package.

To rebuild the full benchmark locally from this released metadata, see Reconstructing The Full Benchmark.

Why This Release Contains Metadata Only

ClinSeek-Bench is built from credentialed clinical datasets, including MIMIC-IV, MIMIC-IV-Note, MIMIC-IV-ED, MIMIC-CXR, MIMIC-CXR-JPG, EHRXQA, and MedMod. These sources contain protected clinical information and must be obtained under each user's own credentialed access and data-use agreements.

For privacy and licensing reasons, this repository does not redistribute:

• raw MIMIC tables;
• patient-level SQLite databases;
• chest X-ray image files;
• radiology report text;
• experiment logs or model trajectories.

Instead, this repository releases the metadata needed to rebuild the complete runtime benchmark locally from the official source datasets.

Released Metadata

The metadata files are the source of truth for reconstruction and evaluation. During local reconstruction, the ClinSeekAgent scripts materialize protected runtime assets such as patient databases, rendered EHR contexts, linked CXR files, and radiology reports from the official source datasets.

Benchmark Composition

Text-only EHR tasks

The text-only split is derived from EHR-Bench in EHR-R1. It contains 45 EHR subtasks covering risk-prediction and decision-making scenarios. We sample 40 examples per subtask, resulting in 1,800 text-only examples. The released metadata contains 1,563 unique patients.

Multimodal tasks

The multimodal split adapts EHRXQA and MedMod, both grounded in MIMIC-IV EHRs and MIMIC-CXR chest radiographs. It contains 989 image-grounded examples:

• 497 EHRXQA-derived CXR question-answering rows;
• 492 MedMod-derived ICU/CXR prediction rows.

The six multimodal task groups are CXR finding presence, CXR finding enumeration, CXR temporal change comparison, 24-hour decompensation prediction, in-hospital mortality prediction, and phenotype prediction.

Experimental Results

We evaluate ClinSeekAgent under the Automated Evidence-Seeking setting and compare it with the paired Curated Input setting. All numbers below are F1 scores in percentage points. Δ is Automated Evidence-Seeking minus Curated Input.

Text-only EHR tasks

Multimodal tasks

These results show that active evidence acquisition is most helpful when the answer depends on sparse, longitudinal, or multimodal signals that may be missed by fixed curated inputs. The gains are especially clear for risk prediction in the text-only split and for CXR-related multimodal tasks, where models can combine EHR retrieval, image-tool outputs, and task-relevant medical knowledge.

Files

ClinSeek-Bench/
+-- README.md
+-- ClinSeek-Bench_text.json
+-- ClinSeek-Bench_multimodal.jsonl

The paths above are the released metadata paths in this dataset repository. If you clone this dataset next to the ClinSeekAgent repository, reference the metadata as:

ClinSeek-Bench/ClinSeek-Bench_text.json
ClinSeek-Bench/ClinSeek-Bench_multimodal.jsonl

If you use ClinSeekAgent's default local data layout, you may copy or symlink the text manifest to data/text/ClinSeek-Bench_text.json. That location is a local workspace convenience, not a released metadata path.

Reconstructing The Full Benchmark

Reconstruction code and detailed instructions are maintained in the ClinSeekAgent GitHub repository:

• Code: https://github.com/UCSC-VLAA/ClinSeekAgent
• Text data preparation: https://github.com/UCSC-VLAA/ClinSeekAgent/blob/main/docs/ClinSeek-Bench_text_data_prepare.md
• Automated Evidence-Seeking text evaluation: https://github.com/UCSC-VLAA/ClinSeekAgent/blob/main/docs/ClinSeek-Bench_text_evaluation.md
• Curated Input text evaluation: https://github.com/UCSC-VLAA/ClinSeekAgent/blob/main/docs/ClinSeek-Bench_text_curated_input_evaluation.md
• Multimodal data preparation: https://github.com/UCSC-VLAA/ClinSeekAgent/blob/main/docs/ClinSeek-Bench_multimodal_data_prepare.md

At a high level:

git clone https://huggingface.co/datasets/UCSC-VLAA/ClinSeek-Bench
git clone https://github.com/UCSC-VLAA/ClinSeekAgent.git

Then obtain the required official source datasets, prepare local paths, and follow the text and multimodal reconstruction guides above.

Required Source Data

Download clinical source data only from official sources under your own credentialed access and data-use agreements.

Validation

After cloning ClinSeekAgent, validate the released multimodal metadata without requiring protected assets:

python ClinSeekAgent/scripts/data_build/validate_multimodal_release.py \
  --bench-root ClinSeek-Bench \
  --input ClinSeek-Bench/ClinSeek-Bench_multimodal.jsonl \
  --manifest-only

Expected released metadata counts:

• 1,800 text-only rows;
• 45 text-only EHR subtasks;
• 1,563 unique text-only patients;
• 989 multimodal rows;
• 497 EHRXQA-derived multimodal rows;
• 492 MedMod-derived multimodal rows;
• 350 unique EHRXQA linked CXR JPG paths;
• 477 unique MedMod linked CXR JPG paths;
• 356 unique EHRXQA linked CXR report paths.

After local reconstruction, the same validation script can also check the materialized runtime assets, including patient DBs, image paths, and report paths. See the ClinSeekAgent reconstruction documentation for the full commands.

Evaluation

Evaluation code and instructions are maintained in the ClinSeekAgent repository: https://github.com/UCSC-VLAA/ClinSeekAgent.

After reconstructing the protected runtime assets locally, follow the ClinSeekAgent documentation for Automated Evidence-Seeking evaluation, Curated Input evaluation, and scoring.

Responsible Use

ClinSeek-Bench is for research on clinical evidence seeking. It is not a medical device and must not be used for clinical diagnosis, treatment, triage, or patient management without separate validation, governance, and regulatory review.