Dataset and Benchmark for Quantitative Medical Image Analysis

🎉 Paper accepted @ EMNLP 2026 Main Conference 🎉

| 🌏 Project | 🧑🏻‍💻 GitHub | 📚 Docs | 🩻 Dataset | 🔎 Data Explorer | 🐳 Docker | 🤗 Models | 🚀 Demo | 📖 arXiv | 💼 LinkedIn |

🔎 Benchmarking VLMs for medical vision tasks: detection and measurement 📏

💿 v1.4.0 | 33.2K 3D images | 12.0M 2D slices | 25.7M single-instance / 50.5M multi-instance annotations | multi-modality | multi-anatomy 💿

📏 Annotation: segmentation mask | landmark coordinate | bounding box | tumor/lesion size | distance | angle 📏

🎯 Post-training: SFT, RFT (RL), CoT, LoRA | Framework: TRL, verl 🎯

@misc{yao2026medvisionbenchmarkingquantitativemedical,
      title={MedVision: Benchmarking Quantitative Medical Image Analysis}, 
      author={Yongcheng Yao and Yongshuo Zong and Raman Dutt and Yongxin Yang and Sotirios A Tsaftaris and Timothy Hospedales},
      year={2026},
      eprint={2511.18676},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.18676}, 
}

📑 Table of Contents

• 🏆 Benchmarked Models
• 🔥 News
• 🌟 Quick Start
• 🐳 Use Docker
• 💿 Data
• 📊 Benchmark
• 🎯 Training: SFT
• 🎯 Training: RFT
• 📚 New Tasks/Models Guide
• 🤖 Agent Skills
• 📖 Essential Dataset Concept
• 💿 Batch Download (Optional)
• 📜 License
• 🩵 Acknowledgement

🏆 Benchmarked Models

This codebase supports open-weight VLMs and API models. The leaderboard published full scores for 18 VLMs, evaluated with MedVision dataset v1.0.0-v1.1.1 and 1000-sample-per-subtask. Pilot study include scored for API models with the same dataset but with 100-sample-per-subtask.

The live open leaderboard (per-task score tables + a frontier API-model pilot study) lives on the project page.

Ours

MedVision-V0

General-purpose VLMs (open-weight)

Qwen2.5-VL, Qwen3-VL-Thinking  ·  InternVL3  ·  Gemma-3, Gemma-4  ·  Llama-3.2-Vision  ·  LLaVA-OneVision  ·  GLM-4.6V, GLM-4.6V-Flash

Medical VLMs (open-weight)

MedGemma  ·  LLaVA-Med, Lingshu, MedDr, HuatuoGPT-Vision, HealthGPT-L14

Proprietary / API

Claude-Fable-5  ·  GPT-5.5-Pro  ·  Gemini-3.1-Pro  ·  Kimi-K2.6

The project-page leaderboard currently publishes full score tables for the 12 off-the-shelf VLMs + MedVision-V0, plus a Claude-Fable-5 / Gemini-3.1-Pro API pilot on tumor/lesion size. Newer entries (Qwen3-VL-Thinking, Gemma-4, GLM-4.6V/-Flash, GPT-5.5-Pro, Kimi-K2.6) have eval scripts wired up and are being rolled into the leaderboard.

🔥 News

• [Sep 1, 2026] Update paper, leaderboard, dataset v1.4.0 blog, and RFT recipe

• [Aug 31, 2026] Released the benchmarking/fine-tuning codebase medvision_bm v1.2.0 — release notes

  Details (Click to expand)

  • 🧑‍⚖️ New LLM-as-Judge output parsing: a second, independent reading of every benchmark response, so format-following failure is separable from measurement failure.
  • 🩺 New Clinical Decision Agreement (CDA) analysis: re-scores existing outputs into clinical categories (SNA/SNB position bands, AJCC kidney tumor staging) and reports Cohen's and weighted kappa.
  • 🐛 Key evaluation bugfixes: HuatuoGPT-Vision's decoding config never reached generate() (512-token CoT truncation); explicit output-token budgets on every launcher; GLM-4.6V no longer reinstalls medvision_ds mid-run.
  • 📏 New metrics: COCO-grid Acc@IoU for Detection, cross-dataset Distance and Angle group rows for A/D.
  • Dataset v1.2.0–v1.4.0 integration with ceiling-based plan resolution.

• [Aug 19, 2026] 🚀 Release MedVision dataset v1.4.0 [release-v1.4.0]

  Details (Click to expand)

  • Regenerate Tumor-Lesion-Size annotations for all 12 tumour/lesion datasets: multi-instance annotations 75K --> 3.8M

• [Aug 9, 2026] 🚀 Release MedVision dataset v1.3.0 [release-v1.3.0]

  Details (Click to expand)

  • New dataset: MSWAL (484 abdominal CT cases; tumor/lesion labels: liver tumour, kidney tumour, pancreatic cancer, liver cyst, and kidney cyst).

• [Aug 3, 2026] 🚀 Release MedVision dataset v1.2.1 [release-v1.2.1]

  Details (Click to expand)

  • ⚠️ Corrects MAMA-MIA and PI-CAI, whose v1.2.0 annotations were recorded in the source orientation instead of RAS+ — the loader reoriented the images at load time without renumbering the coordinates. Their v1.2.0 annotations are withdrawn. If you have used either dataset, clear that cache once.
  • No other dataset is affected — the other 28 resolve to exactly the same annotation files as at v1.2.0.
  • New: scripts/gen-annotations/ rebuilds the preprocessed images and annotations of any dataset from its original source -- For the record only, you never use it to load data.

• [Jul 28, 2026] 🚀 Release MedVision dataset v1.2.0 [release-v1.2.0]

  Details (Click to expand)

  • Highlight: 8 new datasets (130 configs) — AFIDs, DEEP-PSMA, LIDC-IDRI, LNQ2023, MAMA-MIA, PDDCA, PI-CAI, VerSe.
  • No existing annotation changed. Annotation versions now resolve per dataset: the version you set is a ceiling, and each dataset loads the newest annotation it published at or before it. Pinning '1.1.1' or older keeps working for every pre-existing dataset (check Annotation Version Control).
  • ⚠️ Fixes a stale-cache defect present in all earlier versions. The cache key used the version you requested rather than the annotation actually loaded, so load_dataset could silently return previously cached rows after the annotations changed — which really happened, to the v1.1.0 T/L train/test split. See Fixed: cached data could be stale for who is affected and how to clear it. The data root is now part of the key too, which matters only if your HuggingFace cache is not already co-located with it. Because the cache key changed, existing Arrow caches rebuild once on next use (reads the annotation file, no re-download)

• [Jul 21, 2026] Updated leaderboard and data explorer

• [Jul 4, 2026] Released the benchmarking/fine-tuning codebase medvision_bm v1.1.1 — release notes

  Details (Click to expand)

  • 📚 New documentation site on Read the Docs: installation, dataset, benchmarking, and fine-tuning guides plus the full CLI and Python API reference.

• [Jun 29, 2026] 🚀 Released the MedVision dataset (medvision_ds) v1.1.1 — release notes

  Details (Click to expand)

  • Highlight: corrected T/L ellipse fit — fixes a transposed in-plane voxel-spacing bug (wrong axis lengths and major/minor labelling on anisotropic slices, e.g. sagittal/coronal); ~22% fewer T/L samples on anisotropic data, isotropic data (e.g., axial slices) essentially unchanged
  • Backward compatibility: The codebase medvision_ds will be automatically updated to the latest (v1.1.1). MedVision_PLANNER_VERSION='latest' now resolves to '1.1.1'; pin '1.1.0' or '1.0.0' for earlier annotations.
  • ⚠️ New env var MedVision_ACK_RELEASE: required only when you pin an older version (MedVision_PLANNER_VERSION below the latest) — set it to the latest version (1.1.1) to acknowledge you have read this release note and unblock loading legacy data.
  • Always set MedVision_FORCE_INSTALL_CODE='True' to receive notification of future releases. See Environment Variables.

• [Jun 9, 2026] Released MedVision-V0, RFT code, preprint v2, project page with interactive case viewer.

• [May 15, 2026] Released the benchmarking/fine-tuning codebase medvision_bm v1.1.0 — release notes

• [May 14, 2026] Released the MedVision dataset (medvision_ds) v1.1.0 — release notes

  Details (Click to expand)

  • Highlight: new T/L sample filtering (with ambiguous cases removed), more T/L samples with a single small target (cluster size > 20)
  • Backward compatibility: The codebase medvision_ds will be automatically updated to the latest (v1.1.0). MedVision_PLANNER_VERSION is required (v1.1.0+) to specify the annotation data version. Setting MedVision_PLANNER_VERSION='1.0.0' will fall back to MedVision dataset v1.0.0.
  • 🧪 Test backward compatibility:

  python unit-test/medvision-ds-planner-version/test_planner_switch_medvision_ds_v1.1.0.py --data_dir <local-data-folder>
  

• [Dec 10, 2025] Added preprint, training code, docker images, released models, new tasks/models guide

• [Oct 8, 2025] Released MedVision dataset v1.0.0

🌟 Quick Start

📚 Read the Docs: Installation · Quickstart walkthrough

Option 1 — run the full pipeline (benchmarking, SFT/RFT). Clone the repo and install from the local copy. Use this when you rely on the repo's folder structure (e.g. script/, tasks_list/, Results/), since the scripts and configs live there.

git clone https://github.com/YongchengYAO/MedVision.git MedVision
cd MedVision
pip install .
pip show medvision_bm

Option 2 — import the package in your own project. Install medvision_bm from PyPI. Use this when you only want to import its modules/functions (e.g. from medvision_bm.utils import parse_utils) and do not need the repo's folder structure.

Stable release (PyPI):

# Install the benchmark codebase
pip install medvision-bm
pip show medvision_bm

# Install the dataset codebase
# "Data" is the local dataset folder
mvbm install mvds -d Data

Or the nightly build (latest commit on GitHub master):

pip install "git+https://github.com/YongchengYAO/MedVision.git"
pip show medvision_bm

🐳 Use Docker

📚 Read the Docs: Installation → Docker

Docker images are built from these dockerfiles

• Base docker: docker pull vincentycyao/medvision:base
• You can choose model-specific docker and skip env setup in scripts, such as docker pull vincentycyao/medvision:eval_medvision-v0, then in eval_*.sh:

  python -m medvision_bm.benchmark.install_medvision_ds --data_dir "${data_dir}" # always keep
  python -m medvision_bm.benchmark.install_vendored_lmms_eval --lmms_eval_opt_deps medvision_v0 # always keep
  #pip install -r "${benchmark_dir}/requirements/requirements_eval_medvision-v0.txt" --no-deps # can skip if using model-specific docker
  
  python -m medvision_bm.benchmark.eval__medvision-model-rft \
    --skip_env_setup \
    ...
  
  

1. Choose the docker image for a specific model: https://hub.docker.com/r/vincentycyao/medvision/tags

   docker pull vincentycyao/medvision:<tag>
   

2. Map local volumes and GPUs, then use the docker image vincentycyao/medvision:<tag>

   # NOTE: replace </path/to/working/folder>, <tag>
   docker run -it --rm \
       --gpus all \
       -v </path/to/working/folder>:/root/Documents/MedVision \
       vincentycyao/medvision:<tag> \
       bash
   

   # In the container
   git clone https://github.com/YongchengYAO/MedVision.git /root/Documents/MedVision
   cd /root/Documents/MedVision
   
   # Check existing Conda env and activate 
   conda env list
   conda activate <env-name>
   
   # Install the latest medvision_bm
   pip install .
   pip show medvision_bm
   
   # Install the latest medvision_ds
   mvbm install mvds -d Data
   pip show medvision_ds
   

[!TIP] Treat the MedVision folder as the working directory for benchmarking and fine-tuning.

File structure: imaging data, benchmark results, and model checkpoints are automatically saved

💿 Data

📚 Read the Docs: Dataset concepts · Loading data

[!IMPORTANT] Leaderboard results use annotation v1.0.0. All leaderboard numbers are computed on the v1.0.0 annotations. We removed ambiguous cases (multi-instance targets) in metric calculation as a workaround. For new studies we recommend the latest annotation version (currently v1.4.0).

💫 Quick start: use MedVision Data Explorer

• Dataset. For the full description of the MedVision dataset (source datasets, modalities, anatomies, annotation types, and returned fields), see the Hugging Face dataset repo.

• Benchmark subtasks ↔ dataset subsets. Each subtask in this benchmark links to a subset of the MedVision dataset. The per-subtask sample sizes are listed for every dataset version 1.0.0–1.4.0:

  • all_tasks__ds_v1.1.1: we use this for OOD ablation since we fixed the annotation error for sagittal and coronal slices from v1.1.1
  • all_tasks__ds_v1.0.0: we use single instance annotations from v1.0.0 in all benchmarking tasks

• Pixel size (physical spacing) distribution. Because the quantitative tasks require pixel→mm arithmetic, the distribution of pixel sizes across subtasks is provided in pixel_sizes__ds_v1.0.0.

• Image size distribution. The distribution of image sizes across subtasks is provided in image_sizes__ds_v1.0.0.

• Multi-instance and single-instance annotations. Each benchmark sample is a (2D slice, target) pair, several instances of the same target on one slice still count as one annotation.

  • Single-instance ⊆ multi-instance
  • Multi-instance (unfiltered) — every target carrying ≥ 1 annotation is kept, however many instances (boxes / clusters) it has on the slice and whatever their size.
  • Single-instance (filtered) — a target is kept only when it is a single, large-enough instance. Per benchmark task, a sample is dropped when:
  Details (Click to expand)

  | Benchmark task | Single-instance drops the sample when… | |---|---| | **Box** — detection | the slice has **more than one** box for the target (`len(boxes) > 1`), **or** a box is **< 10 px** on any side | | **T/L** — tumor / lesion size | the target has **more than one** cluster on the slice (`n_clusters > 1`; `len(biometric_profile) > 1` on the v1.0.0 fallback) | | **A/D** — biometrics (angle / distance) | *never dropped* — every angle and distance sample is kept (the loader only splits them by `metric_type`) |

[!TIP] To load the unfiltered (multi-instance) samples, set the environment variable MedVision_DISABLE_SAMPLE_FILTERING=true (default off) — it bypasses the per-sample quality/size filters and returns every planner sample.

[!WARNING] Multi-instance annotations are not for leaderboard comparison. Do not use multi-instance annotations to compare models on the leaderboad. The current MedVision-V0 SFT/RFT training is not optimized for multi-instance detection and measurement tasks.

Per-dataset statistics

Computed from the local benchmark plans by script/misc/summarize_datasets.sh. The modality / image / slice / segmentation stats are the same for every dataset version; only the benchmark-annotation counts (Box / T/L / A/D) depend on it — and among those, only T/L changes (Box and A/D are identical across versions). Each row is version-invariant, but the set of rows is not: the table lists the 31 datasets present in v1.4.0, so its Total row is specific to that release.

Benchmark annotations by dataset version. Each block below breaks the annotation counts down by dataset and by task — Box (detection), T/L (tumor/lesion size) and A/D (biometrics) — under both the single-instance (filtered) and multi-instance (unfiltered) views. Across releases, Box and A/D counts are byte-identical; T/L is the only task ever regenerated, in v1.1.0, v1.1.1 and most recently v1.4.0, which rebuilt all 12 T/L datasets. A total therefore moves for exactly two reasons: a T/L regeneration, or a release that adds datasets (v1.2.0 added 8, v1.3.0 added MSWAL). Each block totals only the datasets its own release shipped — 31 for v1.3.0 and v1.4.0, 30 for v1.2.0, 22 for the rest:

MedVision v1.4.0 (default)

T/L regenerated. Clusters are now selected by a physical size floor in millimetres — max(2.0 mm, 2 × the coarser in-plane spacing) of the measured plane — instead of a raw pixel count; a gate that silently discarded rotated ellipses is removed; and the ellipse fit is guarded against degenerate results. Published T/L landmarks grow 75,840 → 3,801,540 (50×). Every other task and every previously published annotation file is unchanged. See the v1.4.0 release note and the blog post (MedVision v1.4.0: 50× More Tumor/Lesion Measurements and Their Annotation Recall).

Single-instance	Multi-instance

Dataset	Single-instance Annotation			Multi-instance Annotation
	Box	T/L	A/D	Box	T/L	A/D
ACDC	14,271	0	0	94,160	0	0
AFIDs	0	0	432	0	0	432
AMOS22	666,532	0	0	1,215,776	0	0
AbdomenAtlas1.0Mini	9,748,290	0	0	13,770,398	0	0
AbdomenCT-1K	1,041,588	0	0	1,549,325	0	0
BCV15	68,543	0	0	125,870	0	0
BraTS24	1,115,524	427,339	0	3,767,594	1,645,550	0
CAMUS	951,370	0	0	1,341,433	0	0
Ceph-Biometrics-400	0	0	7,600	0	0	7,600
CrossMoDA	4,076	0	0	16,623	0	0
DEEP-PSMA	2,179	10,740	0	50,341	156,439	0
FLARE22	104,211	0	0	152,954	0	0
FeTA24	49,087	0	325	153,599	0	325
HNTSMRG24	32,029	43,920	0	62,424	69,093	0
ISLES24	9,774	0	0	97,228	0	0
KiPA22	37,647	13,061	0	74,690	13,864	0
KiTS23	114,491	62,225	0	291,550	76,016	0
LIDC-IDRI	10,379	43,488	0	73,976	59,179	0
LNQ2023	1,653	5,702	0	25,331	37,898	0
MAMA-MIA	67,767	90,725	0	214,408	369,419	0
MSD	277,451	115,764	0	1,438,472	712,247	0
MSWAL	60,106	78,586	0	125,126	145,136	0
OAIZIB-CM	648,048	0	0	922,989	0	0
PDDCA	15,076	0	132	37,119	0	132
PI-CAI	5,455	22,764	0	42,857	28,324	0
SKM-TEA	262,338	0	0	475,828	0	0
ToothFairy2	1,413,979	0	0	2,131,223	0	0
TopCoW24	41,829	0	0	251,901	0	0
TotalSegmentator	7,603,455	0	0	16,979,575	0	0
VerSe	350,311	0	1,500	874,896	0	1,500
autoPET-III	31,794	51,875	0	360,638	488,375	0
Total (31)	24,749,253	966,189	9,989	46,718,304	3,801,540	9,989
Total (all tasks)	25,725,431			50,529,833

MedVision v1.3.0

Adds MSWAL (CT, 484 volumes). No annotation logic changed — every v1.2.0 count is carried over unmodified.

Single-instance	Multi-instance

Dataset	Single-instance Annotation			Multi-instance Annotation
	Box	T/L	A/D	Box	T/L	A/D
ACDC	14,271	0	0	94,160	0	0
AFIDs	0	0	432	0	0	432
AMOS22	666,532	0	0	1,215,776	0	0
AbdomenAtlas1.0Mini	9,748,290	0	0	13,770,398	0	0
AbdomenCT-1K	1,041,588	0	0	1,549,325	0	0
BCV15	68,543	0	0	125,870	0	0
BraTS24	1,115,524	15,880	0	3,767,594	26,183	0
CAMUS	951,370	0	0	1,341,433	0	0
Ceph-Biometrics-400	0	0	7,600	0	0	7,600
CrossMoDA	4,076	0	0	16,623	0	0
DEEP-PSMA	2,179	94	0	50,341	753	0
FLARE22	104,211	0	0	152,954	0	0
FeTA24	49,087	0	325	153,599	0	325
HNTSMRG24	32,029	2,272	0	62,424	3,188	0
ISLES24	9,774	0	0	97,228	0	0
KiPA22	37,647	3,077	0	74,690	3,142	0
KiTS23	114,491	7,048	0	291,550	8,034	0
LIDC-IDRI	10,379	417	0	73,976	517	0
LNQ2023	1,653	45	0	25,331	238	0
MAMA-MIA	67,767	3,240	0	214,408	5,071	0
MSD	277,451	6,126	0	1,438,472	12,914	0
MSWAL	60,106	8,254	0	125,126	12,260	0
OAIZIB-CM	648,048	0	0	922,989	0	0
PDDCA	15,076	0	132	37,119	0	132
PI-CAI	5,455	393	0	42,857	409	0
SKM-TEA	262,338	0	0	475,828	0	0
ToothFairy2	1,413,979	0	0	2,131,223	0	0
TopCoW24	41,829	0	0	251,901	0	0
TotalSegmentator	7,603,455	0	0	16,979,575	0	0
VerSe	350,311	0	1,500	874,896	0	1,500
autoPET-III	31,794	879	0	360,638	3,118	0
Total (31)	24,749,253	47,725	9,989	46,718,304	75,827	9,989
Total (all tasks)	24,806,967			46,804,120

MedVision v1.2.0

Single-instance	Multi-instance

Dataset	Single-instance Annotation			Multi-instance Annotation
	Box	T/L	A/D	Box	T/L	A/D
ACDC	14,271	0	0	94,160	0	0
AFIDs	0	0	432	0	0	432
AMOS22	666,532	0	0	1,215,776	0	0
AbdomenAtlas1.0Mini	9,748,290	0	0	13,770,398	0	0
AbdomenCT-1K	1,041,588	0	0	1,549,325	0	0
BCV15	68,543	0	0	125,870	0	0
BraTS24	1,115,524	15,880	0	3,767,594	26,183	0
CAMUS	951,370	0	0	1,341,433	0	0
Ceph-Biometrics-400	0	0	7,600	0	0	7,600
CrossMoDA	4,076	0	0	16,623	0	0
DEEP-PSMA	2,179	94	0	50,341	753	0
FLARE22	104,211	0	0	152,954	0	0
FeTA24	49,087	0	325	153,599	0	325
HNTSMRG24	32,029	2,272	0	62,424	3,188	0
ISLES24	9,774	0	0	97,228	0	0
KiPA22	37,647	3,077	0	74,690	3,142	0
KiTS23	114,491	7,048	0	291,550	8,034	0
LIDC-IDRI	10,379	417	0	73,976	517	0
LNQ2023	1,653	45	0	25,331	238	0
MAMA-MIA	67,767	3,327	0	214,408	5,117	0
MSD	277,451	6,126	0	1,438,472	12,914	0
OAIZIB-CM	648,048	0	0	922,989	0	0
PDDCA	15,076	0	132	37,119	0	132
PI-CAI	5,455	395	0	42,857	410	0
SKM-TEA	262,338	0	0	475,828	0	0
ToothFairy2	1,413,979	0	0	2,131,223	0	0
TopCoW24	41,829	0	0	251,901	0	0
TotalSegmentator	7,603,455	0	0	16,979,575	0	0
VerSe	350,311	0	1,500	874,896	0	1,500
autoPET-III	31,794	879	0	360,638	3,118	0
Total (30)	24,689,147	39,560	9,989	46,593,178	63,614	9,989
Total (all tasks)	24,738,696			46,666,781

MedVision v1.1.1

Single-instance	Multi-instance

Dataset	Single-instance Annotation			Multi-instance Annotation
	Box	T/L	A/D	Box	T/L	A/D
ACDC	14,271	0	0	94,160	0	0
AMOS22	666,532	0	0	1,215,776	0	0
AbdomenAtlas1.0Mini	9,748,290	0	0	13,770,398	0	0
AbdomenCT-1K	1,041,588	0	0	1,549,325	0	0
BCV15	68,543	0	0	125,870	0	0
BraTS24	1,115,524	15,880	0	3,767,594	26,183	0
CAMUS	951,370	0	0	1,341,433	0	0
Ceph-Biometrics-400	0	0	7,600	0	0	7,600
CrossMoDA	4,076	0	0	16,623	0	0
FLARE22	104,211	0	0	152,954	0	0
FeTA24	49,087	0	325	153,599	0	325
HNTSMRG24	32,029	2,272	0	62,424	3,188	0
ISLES24	9,774	0	0	97,228	0	0
KiPA22	37,647	3,077	0	74,690	3,142	0
KiTS23	114,491	7,048	0	291,550	8,034	0
MSD	277,451	6,126	0	1,438,472	12,914	0
OAIZIB-CM	648,048	0	0	922,989	0	0
SKM-TEA	262,338	0	0	475,828	0	0
ToothFairy2	1,413,979	0	0	2,131,223	0	0
TopCoW24	41,829	0	0	251,901	0	0
TotalSegmentator	7,603,455	0	0	16,979,575	0	0
autoPET-III	31,794	879	0	360,638	3,118	0
Total (22)	24,236,327	35,282	7,925	45,274,250	56,579	7,925
Total (all tasks)	24,279,534			45,338,754

MedVision v1.1.0

Single-instance	Multi-instance

Dataset	Single-instance Annotation			Multi-instance Annotation
	Box	T/L	A/D	Box	T/L	A/D
ACDC	14,271	0	0	94,160	0	0
AMOS22	666,532	0	0	1,215,776	0	0
AbdomenAtlas1.0Mini	9,748,290	0	0	13,770,398	0	0
AbdomenCT-1K	1,041,588	0	0	1,549,325	0	0
BCV15	68,543	0	0	125,870	0	0
BraTS24	1,115,524	19,139	0	3,767,594	30,357	0
CAMUS	951,370	0	0	1,341,433	0	0
Ceph-Biometrics-400	0	0	7,600	0	0	7,600
CrossMoDA	4,076	0	0	16,623	0	0
FLARE22	104,211	0	0	152,954	0	0
FeTA24	49,087	0	325	153,599	0	325
HNTSMRG24	32,029	3,129	0	62,424	4,475	0
ISLES24	9,774	0	0	97,228	0	0
KiPA22	37,647	3,077	0	74,690	3,142	0
KiTS23	114,491	12,471	0	291,550	14,148	0
MSD	277,451	9,152	0	1,438,472	16,620	0
OAIZIB-CM	648,048	0	0	922,989	0	0
SKM-TEA	262,338	0	0	475,828	0	0
ToothFairy2	1,413,979	0	0	2,131,223	0	0
TopCoW24	41,829	0	0	251,901	0	0
TotalSegmentator	7,603,455	0	0	16,979,575	0	0
autoPET-III	31,794	1,246	0	360,638	3,869	0
Total (22)	24,236,327	48,214	7,925	45,274,250	72,611	7,925
Total (all tasks)	24,292,466			45,354,786

MedVision v1.0.0

Single-instance	Multi-instance

Dataset	Single-instance Annotation			Multi-instance Annotation
	Box	T/L	A/D	Box	T/L	A/D
ACDC	14,271	0	0	94,160	0	0
AMOS22	666,532	0	0	1,215,776	0	0
AbdomenAtlas1.0Mini	9,748,290	0	0	13,770,398	0	0
AbdomenCT-1K	1,041,588	0	0	1,549,325	0	0
BCV15	68,543	0	0	125,870	0	0
BraTS24	1,115,524	11,071	0	3,767,594	11,093	0
CAMUS	951,370	0	0	1,341,433	0	0
Ceph-Biometrics-400	0	0	7,600	0	0	7,600
CrossMoDA	4,076	0	0	16,623	0	0
FLARE22	104,211	0	0	152,954	0	0
FeTA24	49,087	0	325	153,599	0	325
HNTSMRG24	32,029	1,392	0	62,424	1,416	0
ISLES24	9,774	0	0	97,228	0	0
KiPA22	37,647	3,095	0	74,690	3,095	0
KiTS23	114,491	8,484	0	291,550	8,540	0
MSD	277,451	7,472	0	1,438,472	7,674	0
OAIZIB-CM	648,048	0	0	922,989	0	0
SKM-TEA	262,338	0	0	475,828	0	0
ToothFairy2	1,413,979	0	0	2,131,223	0	0
TopCoW24	41,829	0	0	251,901	0	0
TotalSegmentator	7,603,455	0	0	16,979,575	0	0
autoPET-III	31,794	735	0	360,638	749	0
Total (22)	24,236,327	32,249	7,925	45,274,250	32,567	7,925
Total (all tasks)	24,276,501			45,314,742

📊 Benchmark

📚 Read the Docs: Pipeline overview · Running evaluations · Parsing & summarizing · CLI reference

Benchmark Setting

• Proprietary/API models are evaluated in a pilot study that caps each subtask at 100 samples; all other (open-weight) models use a limit of 1000 samples per subtask.
• The subtasks are defined in tasks_list/:
  • A/D → tasks_MedVision-AD-CoT.json
  • T/L → tasks_MedVision-TL-CoT.json
  • Detection → tasks_MedVision-detect-CoT.json.

Steps

1. The scripts in script/benchmark-*/ should be sufficient for dependency installation, data processing, and benchmarking

[!TIP] Set these variables:

• benchmark_dir: the working directory
• model_hf_id: Hugging Face ID (<user>/<model>) of the tested model
• model_name: user-defined identifier for the tested model, used as folder name in Results/MedVision-*/
• resource-constrained configs, such as batch_size_per_gpu

[!NOTE] Crash-safe resume.

During evaluation each finished output is written immediately to Results/MedVision-*/<model_name>/response_cache/<task>_rank<N>.jsonl, so re-running an interrupted eval skips already-completed samples instead of regenerating them — only the in-flight sample is lost. The cache key includes a hash of the prompt, so editing a prompt/config automatically invalidates stale entries (no need to clear the folder). Set the environment variable MEDVISION_RESP_CACHE=0 to disable this layer entirely and reproduce the original (no-cache) behavior.

2. After evaluating all models in step 1, parse model outputs and calculate metrics (e.g., MRE, MAE, nMAE, IoU, F1, Precision, Recall, Success Rate). Base command:

[!TIP] Command: python -m medvision_bm.benchmark.parse_outputs

Arguments:

• --task_type: one of ["AD", "TL", "Detection"]
• --task_dir: task folder
• --model_dir: model folder
• --limit: limit sample size in the parsed files
• --skip_existing: (store_true) skip parsed files
• --processes, -p: number of processes
• --rm_old: remove existing parsed folder for each model

Example 1 — parse all models for the T/L task:

python -m medvision_bm.benchmark.parse_outputs \
--task_type TL \
--task_dir Results/MedVision-TL \
-p 32

Example 2 — parse all models for the A/D task (remove existing parsed folder):

python -m medvision_bm.benchmark.parse_outputs \
--task_type AD \
--task_dir Results/MedVision-AD \
-p 32 \
--rm_old

Example 3 — parse one model for the detection task and skip existing parsed files:

python -m medvision_bm.benchmark.parse_outputs \
--task_type Detection \
--model_dir Results/MedVision-detect/Qwen2.5-VL-32B-Instruct \
--skip_existing \
-p 32

3. Summarize model performance for each task

[!TIP] If medvision_ds is missing, install with:

python -m medvision_bm.benchmark.install_medvision_ds --data_dir

[!TIP] Command:

python -m medvision_bm.benchmark.summarize_{AD,TL,detection}_task

Arguments:

• --task_dir: task folder
• --model_dir: model folder
• --limit: limit sample size in the parsed files
• --skip_model_wo_parsed_files: skip model directories that don't have a parsed folder
• --processes, -p: number of processes
• --removed_samples_dir: (TL task only) root directory with per-dataset removed_samples JSON files, used to filter ambiguous cases

Example 1 — summarize all models for the A/D task:

python -m medvision_bm.benchmark.summarize_AD_task \
--task_dir Results/MedVision-AD \
-p 32

Example 2 — summarize all models for the T/L task:

python -m medvision_bm.benchmark.summarize_TL_task \
--task_dir Results/MedVision-TL \
--removed_samples_dir <local-data-folder>/Datasets \
-p 32

Example 3 — summarize one model for the detection task:

python -m medvision_bm.benchmark.summarize_detection_task \
--model_dir Results/MedVision-detect/Qwen2.5-VL-32B-Instruct \
-p 32

4. (Recommended) LLM-judge parsing — the format-robust second pass

The regex parser in step 2 only accepts answers written inside <answer>…</answer>, so a response that states a correct answer in any other form (\boxed{…}, **Answer:** …, plain prose) is scored as a miss — mixing "the model can't measure" with "the model didn't follow the output format". The LLM-judge pipeline re-reads every response with a judge model whose only job is to find and quote the answer, wherever it was written. Regex parsing (steps 2–3) and LLM-judge parsing together are the complete parsing scheme — run both: the diff between the two reports is how much of a model's apparent failure was formatting.

• [File structure] after steps 1-4

  ├── MedVision
  │   ├── completed_tasks 
  │   │   ├── completed_tasks_MedVision-AD.json       # <== tasks status tracker
  │   │   ├── ...
  │   ├── Results                                     # <== benchmark results
  │   │   ├── MedVision-AD
  │   │   │   ├── ...
  │   │   │   ├── summary_AD_task.txt                 # <== [step 3] summary
  │   │   ├── MedVision-detect
  │   │   │   ├── Qwen2.5-VL-32B-Instruct
  │   │   │   │   ├── parsed                               
  │   │   │   │   │   ├── *.jsonl                     # <== [step 2] parsed model outputs
  │   │   │   │   │   ├── *.json                      # <== [step 2] parsed summary file
  │   │   │   │   │   ├── summary_*                   # <== [step 3] mean metrics, values
  │   │   │   │   ├── llm-parsed_<judge>              # <== [step 4] format-robust re-parse; parsed/ is never touched
  │   │   │   │   │   ├── *.jsonl                     #        re-parsed records: outcome label + quoted answer span
  │   │   │   │   │   ├── summary_*                   #        mean metrics, values (same summarizer as step 3)
  │   │   │   │   │   ├── summary_metrics_judge_Task.json    # per-model failure decomposition
  │   │   │   │   ├── response_cache                  # <== [step 1] per-sample resume cache (auto; MEDVISION_RESP_CACHE=0 to disable)
  │   │   │   │   │   ├── *_rank*.jsonl               #        one line per finished sample, written as produced
  │   │   │   │   ├── *.jsonl                         # <== [step 1] model outputs
  │   │   │   │   ├── *.json                          # <== [step 1] summary file
  │   │   │   ├── ...
  │   │   │   ├── summary_detection_task.txt          # <== [step 3] summary
  │   │   │   ├── judge-queue_Detection.jsonl         # <== [step 4] work list, one row per response
  │   │   │   ├── judge-out_Detection_<judge>.jsonl   # <== [step 4] the judge's raw answers
  │   │   │   ├── summary_detection_task__llm-parsed_<judge>.txt  # <== [step 4] format-robust summary (diff against summary_detection_task.txt)
  │   │   │   ├── summary_judge_task__llm-parsed_<judge>.txt      # <== [step 4] wrong-format vs. no-answer breakdown + judge validity
  │   │   ├── MedVision-TL
  │   │   │   ├── ...
  │   │   │   ├── summary_TL_task.txt                 # <== [step 3] summary
  

• [Analysis & Visualization] (optional) Scripts in script/visualization:

  • Radar charts (viz_radar.sh, viz_radar_batch.sh): cross-model comparison across metrics.
  • Per-model radar grids (viz_radar_grid.sh): one row per model, six radars per row (Detection Recall/Precision/F1, A/D Angle/Distance MRE, T/L MRE), each spoke overlaid with that model's per-sample violin + box plots.
  • Leaderboard timeline (viz_benchmark_leaderboard_timeline.sh): benchmark accuracy against model release date, one panel per task (Detection IoU, plus T/L, Distance and Angle as 1/MRE).
  • Detection label × box-size (viz_detection_sampleSize_per_label_x_boxSize.sh): detection metrics and sample distribution per label × box-to-image ratio group.
  • A/D landmark overlays (viz_ad_landmarks.sh): per-sample GT vs. predicted landmarks and lines.
  • A/D response panels (viz_ad_responses.sh): per-sample prompt/response/GT panels.
  • T/L axis overlays (viz_tl_axes.sh): per-sample predicted vs. GT axes with mask contour.
  • T/L response panels (viz_tl_responses.sh): per-sample prompt/response/GT panels.
  • T/L ellipse fit: image vs. real space (viz_ellipse_fit_comparison.py): overlay the T/L ellipse axes fit in image (pixel) space vs. real (physical) space on one coronal/sagittal slice, illustrating how anisotropic pixel spacing makes the two fits diverge (rendered with and without the physical aspect-ratio correction).
  • Detection box overlays (viz_detection_boxes.sh): per-sample GT vs. predicted bounding boxes.
  • Detection response panels (viz_detection_responses.sh): per-sample prompt/response/GT panels.
  • Ground-truth annotation figure (viz_gt_annotations.sh): GT-only panels rendered straight from the on-disk benchmark plans (no model output, no HuggingFace build), compiled into one figure with a labelled row-block per task.
  • Plane-OOD sample grids (viz_planeOOD_samples.sh): the same volume and target shown in the in-distribution axial plane and in both OOD planes (coronal, sagittal); Detection and T/L only, since A/D has no plane-OOD split.
  • Label clouds (viz_label_cloud.sh): 2×2 target × modality label clouds contrasting each task's in-distribution roster against its target-OOD roster.
  • Comparison grids (viz_compile_grid_batch.sh): tile per-sample overlays across models.

• [Analysis] (optional) Scripts in script/analyze:

  • Process accuracy (process-accuracy/analyze_process_accuracy_TL.py, process-accuracy/analyze_process_accuracy_AD.py): step-by-step CoT accuracy for T/L (4 steps: major/minor axis endpoint norm-L2 → axis length MRE) and A/D (3 steps: landmark coordinate norm-L2 → scalar MRE), evaluated against ground truth.
  • Equation accuracy (equation-accuracy/analyze_equation_accuracy_TL.py, equation-accuracy/analyze_equation_accuracy_AD.py): arithmetic correctness independent of ground truth — extracts the equation the model wrote, evaluates it in Python, and computes MRE between that result and the model's own reported answer.
  • Detection × target size (detection--target-size/run_analysis.sh): detection metrics (F1, IoU, etc.) stratified by box-to-image ratio, revealing performance trends across small, medium, and large targets.
  • Clinical Decision Agreement (CDA) (clinical-decision-analysis/run_CDA_analysis.sh): asks whether a measurement error would change the clinical decision — each prediction and its ground truth are pushed through a published cutoff table into a clinical category, and agreement is scored with Cohen's / weighted kappa. Re-reads existing parsed/ records only: no re-inference, no GPU, seconds per model. Check the CDA pipeline.
  • Compare with Segmentation Specialist (script/ablation/biomedparse): Evaludate and fine-tune BiomedParse v2 on our dataset

• [Troubleshooting] here

🎯 Training: SFT

📚 Read the Docs: Supervised fine-tuning (SFT)

• [Script] script/sft/train*.sh handles dependency installation, data processing, and training.

[!TIP] Set these variables in the script:

• benchmark_dir: the working directory
• base_model_hf: Hugging Face ID (<user>/<model>) of the base model, or the path to a local model folder.
• run_name: an identifier for the current training
• merged_model_hf: Hugging Face model name (<model>) of the merged model
• resource-constrained configs, such as
  • per_device_train_batch_size
  • gradient_accumulation_steps
  • CUDA_VISIBLE_DEVICES=0,1,2,3 and --num_processes=4

• [Blog] Supervised Fine-Tuning (SFT) for VLMs on Medical Image Data

🎯 Training: RFT

📚 Read the Docs: Reinforcement fine-tuning (RFT)

RL fine-tuning uses the verl framework. MedVision provides parquet dataset builders that turn the MedVision tasks into verl-ready parquet datasets.

• [Data Processing] Build the verl parquet dataset with the scripts in script/rft, which call:

  • medvision_bm.rft.verl.build_parquet_ds: normal parquet dataset builder
  • medvision_bm.rft.verl.build_parquet_ds__checkpointed: checkpointed builder to avoid OOM, recommended for large datasets (e.g. ~1M detection samples)

  Available scripts:

  • build_parquet_ds__verl__D0k-AD5.5k-TL0k__512x512.sh: A/D task only (5.5K train / 45 val)
  • build_parquet_ds__verl__D0k-AD0k-TL5.5k__512x512.sh: T/L task only (5.5K train / 50 val)
  • build_parquet_ds__verl__D110k-AD0k-TL0k__512x512.sh: Detection task only (110K train / 105 val)
  • build_parquet_ds__verl__D110k-AD5.5k-TL5.5k__512x512.sh: all 3 tasks combined (121K train / 200 val)
  • build_parquet_ds__verl__D1000k-AD0k-TL0k__512x512__checkpointed.sh: Detection task only, large scale (1M train / 500 val); uses the checkpointed builder

• [RFT] RL fine-tuning (GRPO) runs in our verl fork, branch medvision-rl, which implements the RFT stage of the paper:

  • Rewards (verl/utils/reward_score/medvision_rewards/): format, process (per-CoT-step landmark / measurement accuracy) and answer rewards with the exp(-error) mapping; CIoU overlap reward for detection; multiplicative composition r = r_format + r_process * r_answer (default) and the additive ablation r = r_format + r_process + r_answer; options and CLI overrides in REWARDS.md.
  • Temperature-scaled task mixing (verl/utils/dataset/temperature_sampler.py): task probability proportional to count^(1/T) to rebalance the 110K / 5.5K / 5.5K multi-task mixture (T=8).
  • Epoch-level curriculum learning (verl/utils/dataset/curriculum.py, CURRICULUM_FILTERING.md): per-task easy / hard pools, EMA-gated promotion of solved samples, retention mix-in, rotating audits with hysteresis-guarded demotion, and a per-task floor.
  • Recipes (examples/grpo_trainer/): sequential A/D → T/L → detection RFT (train__rft-sequential__{1-AD,2-TL,3-detection}.sh; MedVision-V0) and single-stage multi-task RFT with curriculum (train__rft-multitask.sh; additive-reward twin train__rft-multitask__additive-reward.sh); each script takes DATASET_ROOT plus either BASE_MODEL_PATH (a local checkpoint) or BASE_MODEL_HF (a Hub id, downloaded locally before training).

• [Evaluation] Evaluate the trained model with eval__MedVision-V0-7B__detect.sh (in script/benchmark-*/).

📚 New Tasks/Models Guide

📚 Read the Docs: Adding a new model · Adding a new task

New tasks guide | New models guide

🖼️ Model Image Processing

For the quantitative tasks (TL/AD), the image size and pixel size stated in each prompt must match the resolution the model's vision encoder actually perceives after its internal resize. Model image processing documents the per-model strategy (fixed perceived size, dynamic processor probe, or API resize formula), with code references, validation status, and known caveats for every supported model.

🤖 Agent Skills

MedVision ships Agent Skills — plain-Markdown instructions that teach an AI coding agent how to operate this repository. They live in skills/ and use the cross-harness <name>/SKILL.md layout, so the same files work in Claude Code, Codex, OpenCode and Pi.

Installation

Every harness discovers skills at <skills-root>/<name>/SKILL.md. Codex, OpenCode and Pi all read the shared ~/.agents/skills root; Claude Code reads ~/.claude/skills. Pick your root and link the three skills:

git clone https://github.com/YongchengYAO/MedVision.git && cd MedVision

SKILLS_ROOT="${CLAUDE_CONFIG_DIR:-$HOME/.claude}/skills"   # Claude Code
# SKILLS_ROOT=~/.agents/skills                             # Codex · OpenCode · Pi (shared root)
# SKILLS_ROOT=~/.config/opencode/skills                    # OpenCode (native root)
# SKILLS_ROOT=~/.pi/agent/skills                           # Pi (native root)

mkdir -p "$SKILLS_ROOT"
for s in medvision medvision-paper medvision-pipeline; do
  ln -s "$PWD/skills/$s" "$SKILLS_ROOT/$s"    # use `cp -r skills/$s "$SKILLS_ROOT/$s"` to pin a copy
done

To scope the skills to this checkout only, link them into the project instead — .claude/skills/ for Claude Code, .agents/skills/ for Codex, OpenCode and Pi:

mkdir -p .claude/skills
for s in medvision medvision-paper medvision-pipeline; do
  ln -s "../../skills/$s" ".claude/skills/$s"
done

Restart the agent afterwards so it re-scans the skills root.

[!NOTE] medvision sets disable-model-invocation: true, so Claude Code and Pi load it only when you type /medvision. OpenCode ignores that field and will advertise it automatically.

Understand the paper: /medvision-paper

Ask about the technical details behind the benchmark and get the paper's own definitions rather than a guess:

/medvision-paper  How are tumor/lesion sizes annotated, and which ellipse fits are discarded?
/medvision-paper  What is nMAE, how does it differ from MRE, and what is the denominator?
/medvision-paper  What data and hyperparameters produced MedVision-V0?

It is the source of truth for terminology and scale numbers — annotation-generation rules, the real-world-units convention (annotations are mm and degrees, never pixels), metric definitions, the 70/30 subject-level split, and the SFT/RFT recipe.

Run the benchmark: /medvision-pipeline

Describe the run you want and the skill returns the ordered steps with the right module names and flags:

/medvision-pipeline  Evaluate Qwen-2.5-VL on the T/L task, then parse and summarize the results.

The pipeline it walks you through:

# 1 — evaluate (or use a launcher in script/benchmark-{detect,TL,AD}/)
python -m medvision_bm.benchmark.eval__qwen2_5_vl --task_type TL ...

# 2 — parse model outputs into structured predictions
python -m medvision_bm.benchmark.parse_outputs --task_type TL --task_dir <dir> --model_dir <dir>

# 3 — summarize into metrics
python -m medvision_bm.benchmark.summarize_TL_task --task_dir <dir>

# 4 — optional: re-parse with the LLM judge for format-robust extraction
bash script/llm-parsing/run_llm_parsing.sh

Use it the same way for dataset download, environment setup and the SFT launchers.

Go deeper: /medvision

For maintainer-level work — the exact flag, pin or failure mode — the repo skill routes to one of ten sub-skills (environment-setup, dataset-and-tasks, benchmark-evaluation, results-parsing-and-metrics, llm-judge-parsing, sft, rft, analysis, extending-models-and-tasks, biomedparse-ablation):

/medvision  How do I add a new model across every site that needs registering?
/medvision  Why did my detection run OOM while building the plan?

See skills/README.md for the routing map, the 39 bundled helper scripts, and how to use the skills as documentation without installing anything.

📖 Essential Dataset Concept

📚 Read the Docs: Dataset concepts

We cover some essential concepts that help you use the MedVision dataset with ease.

Concepts: Dataset & Data Configuration

• MedVision: the collection of public imaging data and our annotations
• dataset: name of the public datasets, such as BraTS24, MSD, OAIZIB-CM
• data-config: name of predefined subsets
  • naming convention: {dataset}_{annotation-type}_{task-ID}_{slice}_{split}
    • dataset: details
    • annotation-type:
      • BoxSize: detection annotations (bounding box)
      • TumorLesionSize: tumor/lesion size annotations
      • BiometricsFromLandmarks: angle/distance annotations
      • MaskSize: area / mask-size annotations
    • task-ID: Task[xx] (Note: this is a local ID in the dataset, not a global ID in MedVision.)
      • For datasets with multiple image-mask pairs, we defined tasks in medvision_ds/datasets/*/preprocess_*.py
      • source: medvision_ds
      • e.g., detection tasks for the BraTS24 dataset are defined in the benchmark_plan in medvision_ds/datasets/BraTS24/preprocess_detection.py
    • slice: [Sagittal, Coronal, Axial]
    • split: [Train, Test]

What's returned from MedVision Dataset?

We only share the annotations (https://huggingface.co/datasets/YongchengYAO/MedVision/tree/main/Datasets). The data loading script MedVision.py will handle raw image downloading and processing. The returned fields in each sample are defined as follows.

In MedVision.py, the class MedVision(GeneratorBasedBuilder) defines the feature dict and the method _generate_examples() builds the dataset.

"""
MedVision dataset.

NOTE: To update the features returned by the load_dataset() method, the followings should be updated:
        - the feature dict in this class 
        - the dict yielded by the _generate_examples() method 
"""

# The feature dict for the task:
# - Mask-Size
features_dict_MaskSize = {
    "dataset_name": Value("string"),
    "taskID": Value("string"),
    "taskType": Value("string"),
    "image_file": Value("string"),
    "mask_file": Value("string"),
    "slice_dim": Value("uint8"),
    "slice_idx": Value("uint16"),
    "label": Value("uint16"),
    "image_size_2d": Sequence(Value("uint16"), length=2),
    "pixel_size": Sequence(Value("float16"), length=2),
    "image_size_3d": Sequence(Value("uint16"), length=3),
    "voxel_size": Sequence(Value("float16"), length=3),
    "pixel_count": Value("uint32"),
    "ROI_area": Value("float16"),
}

# The feature dict for the task:
# - Box-Size
features_dict_BoxSize = {
    "dataset_name": Value("string"),
    "taskID": Value("string"),
    "taskType": Value("string"),
    "image_file": Value("string"),
    "mask_file": Value("string"),
    "slice_dim": Value("uint8"),
    "slice_idx": Value("uint16"),
    "label": Value("uint16"),
    "image_size_2d": Sequence(Value("uint16"), length=2),
    "pixel_size": Sequence(Value("float16"), length=2),
    "image_size_3d": Sequence(Value("uint16"), length=3),
    "voxel_size": Sequence(Value("float16"), length=3),
    "bounding_boxes": Sequence(
        {
            "min_coords": Sequence(Value("uint16"), length=2),
            "max_coords": Sequence(Value("uint16"), length=2),
            "center_coords": Sequence(Value("uint16"), length=2),
            "dimensions": Sequence(Value("uint16"), length=2),
            "sizes": Sequence(Value("float16"), length=2),
        },
    ),
}

features_dict_BiometricsFromLandmarks = {
    "dataset_name": Value("string"),
    "taskID": Value("string"),
    "taskType": Value("string"),
    "image_file": Value("string"),
    "landmark_file": Value("string"),
    "slice_dim": Value("uint8"),
    "slice_idx": Value("uint16"),
    "image_size_2d": Sequence(Value("uint16"), length=2),
    "pixel_size": Sequence(Value("float16"), length=2),
    "image_size_3d": Sequence(Value("uint16"), length=3),
    "voxel_size": Sequence(Value("float16"), length=3),
    "biometric_profile": {
        "metric_type": Value("string"),
        "metric_map_name": Value("string"),
        "metric_key": Value("string"),
        "metric_value": Value("float16"),
        "metric_unit": Value("string"),
        "slice_dim": Value("uint8"),
    },
}

features_dict_TumorLesionSize = {
    "dataset_name": Value("string"),
    "taskID": Value("string"),
    "taskType": Value("string"),
    "image_file": Value("string"),
    "landmark_file": Value("string"),
    "mask_file": Value("string"),
    "slice_dim": Value("uint8"),
    "slice_idx": Value("uint16"),
    "label": Value("uint16"),
    "image_size_2d": Sequence(Value("uint16"), length=2),
    "pixel_size": Sequence(Value("float16"), length=2),
    "image_size_3d": Sequence(Value("uint16"), length=3),
    "voxel_size": Sequence(Value("float16"), length=3),
    "biometric_profile": Sequence(
        {
            "metric_type": Value("string"),
            "metric_map_name": Value("string"),
            "metric_key_major_axis": Value("string"),
            "metric_value_major_axis": Value("float16"),
            "metric_key_minor_axis": Value("string"),
            "metric_value_minor_axis": Value("float16"),
            "metric_unit": Value("string"),
        },
    ),
}

# Env var to disable per-sample quality filtering. When true, the size/cluster
# exclusion filters below are bypassed so every sample in the planner is returned.
# The distance/angle metric_type split (task partitioning) is always preserved.
disable_sample_filtering = (
    os.environ.get("MedVision_DISABLE_SAMPLE_FILTERING", "False").lower()
    == "true"
)
if disable_sample_filtering:
    logger.info(
        "MedVision_DISABLE_SAMPLE_FILTERING=true — quality/size sample filters bypassed"
    )

# Task type: Mask-Size
if taskType == "Mask-Size":
    flatten_slice_profiles = (
        MedVision_BenchmarkPlannerSegmentation.flatten_slice_profiles_2d
    )
    if imageSliceType.lower() == "sagittal":
        slice_dim = 0
    elif imageSliceType.lower() == "coronal":
        slice_dim = 1
    elif imageSliceType.lower() == "axial":
        slice_dim = 2
    slice_profile_flattened = flatten_slice_profiles(biometricData, slice_dim)
    for idx, case in enumerate(slice_profile_flattened):
        # Skip cases with a mask size smaller than 200 pixels
        if not disable_sample_filtering and case["pixel_count"] < 200:
            continue
        else:
            yield idx, {
                "dataset_name": dataset_name,
                "taskID": taskID,
                "taskType": taskType,
                "image_file": os.path.join(dataset_dir, case["image_file"]),
                "mask_file": os.path.join(dataset_dir, case["mask_file"]),
                "slice_dim": case["slice_dim"],
                "slice_idx": case["slice_idx"],
                "label": case["label"],
                "image_size_2d": case["image_size_2d"],
                "pixel_size": case["pixel_size"],
                "image_size_3d": case["image_size_3d"],
                "voxel_size": case["voxel_size"],
                "pixel_count": case["pixel_count"],
                "ROI_area": case["ROI_area"],
            }

# Task type: Box-Size
if taskType == "Box-Size":
    if imageType.lower() == "2d":
        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerDetection.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            # Skip cases with multiple bounding boxes in the same slice
            if not disable_sample_filtering and len(case["bounding_boxes"]) > 1:
                continue
            # Skip cases with a bounding box size smaller than 10 pixels in any dimension
            elif not disable_sample_filtering and (
                case["bounding_boxes"][0]["dimensions"][0] < 10
                or case["bounding_boxes"][0]["dimensions"][1] < 10
            ):
                continue
            else:
                yield idx, {
                    "dataset_name": dataset_name,
                    "taskID": taskID,
                    "taskType": taskType,
                    "image_file": os.path.join(dataset_dir, case["image_file"]),
                    "mask_file": os.path.join(dataset_dir, case["mask_file"]),
                    "slice_dim": case["slice_dim"],
                    "slice_idx": case["slice_idx"],
                    "label": case["label"],
                    "image_size_2d": case["image_size_2d"],
                    "pixel_size": case["pixel_size"],
                    "image_size_3d": case["image_size_3d"],
                    "voxel_size": case["voxel_size"],
                    "bounding_boxes": case["bounding_boxes"],
                }

# Task type: Biometrics-From-Landmarks
if taskType == "Biometrics-From-Landmarks":
    if imageType.lower() == "2d":
        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerBiometry.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            yield idx, {
                "dataset_name": dataset_name,
                "taskID": taskID,
                "taskType": taskType,
                "image_file": os.path.join(dataset_dir, case["image_file"]),
                "landmark_file": os.path.join(
                    dataset_dir, case["landmark_file"]
                ),
                "slice_dim": case["slice_dim"],
                "slice_idx": case["slice_idx"],
                "image_size_2d": case["image_size_2d"],
                "pixel_size": case["pixel_size"],
                "image_size_3d": case["image_size_3d"],
                "voxel_size": case["voxel_size"],
                "biometric_profile": case["biometric_profile"],
            }

# Task type: Biometrics-From-Landmarks-Distance
if taskType == "Biometrics-From-Landmarks-Distance":
    if imageType.lower() == "2d":
        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerBiometry.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            if case["biometric_profile"]["metric_type"] == "distance":
                yield idx, {
                    "dataset_name": dataset_name,
                    "taskID": taskID,
                    "taskType": taskType,
                    "image_file": os.path.join(dataset_dir, case["image_file"]),
                    "landmark_file": os.path.join(
                        dataset_dir, case["landmark_file"]
                    ),
                    "slice_dim": case["slice_dim"],
                    "slice_idx": case["slice_idx"],
                    "image_size_2d": case["image_size_2d"],
                    "pixel_size": case["pixel_size"],
                    "image_size_3d": case["image_size_3d"],
                    "voxel_size": case["voxel_size"],
                    "biometric_profile": case["biometric_profile"],
                }

# Task type: Biometrics-From-Landmarks-Angle
if taskType == "Biometrics-From-Landmarks-Angle":
    if imageType.lower() == "2d":
        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerBiometry.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            if case["biometric_profile"]["metric_type"] == "angle":
                yield idx, {
                    "dataset_name": dataset_name,
                    "taskID": taskID,
                    "taskType": taskType,
                    "image_file": os.path.join(dataset_dir, case["image_file"]),
                    "landmark_file": os.path.join(
                        dataset_dir, case["landmark_file"]
                    ),
                    "slice_dim": case["slice_dim"],
                    "slice_idx": case["slice_idx"],
                    "image_size_2d": case["image_size_2d"],
                    "pixel_size": case["pixel_size"],
                    "image_size_3d": case["image_size_3d"],
                    "voxel_size": case["voxel_size"],
                    "biometric_profile": case["biometric_profile"],
                }

# Task type: Tumor-Lesion-Size
if taskType == "Tumor-Lesion-Size":
    if imageType.lower() == "2d":
        # Get the target label for the task
        target_label = benchmark_plan["tasks"][int(taskID) - 1]["target_label"]

        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerBiometry_fromSeg.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            if not disable_sample_filtering:
                n_total_clusters = case["n_total_clusters"]
                if n_total_clusters is not None:
                    # New JSON (v1.1.0+): filter on raw cluster count
                    if n_total_clusters > 1:
                        continue
                else:
                    # Old JSON (v1.0.0): fall back to above-threshold cluster count
                    if len(case["biometric_profile"]) > 1:
                        continue
            yield idx, {
                "dataset_name": dataset_name,
                "taskID": taskID,
                "taskType": taskType,
                "image_file": os.path.join(dataset_dir, case["image_file"]),
                "mask_file": os.path.join(dataset_dir, case["mask_file"]),
                "landmark_file": os.path.join(
                    dataset_dir, case["landmark_file"]
                ),
                "slice_dim": case["slice_dim"],
                "slice_idx": case["slice_idx"],
                "label": target_label,
                "image_size_2d": case["image_size_2d"],
                "pixel_size": case["pixel_size"],
                "image_size_3d": case["image_size_3d"],
                "voxel_size": case["voxel_size"],
                "biometric_profile": case["biometric_profile"],
            }

Dataset Building Workflow

Workflow

There are a few ways to control the dataset loading and building behavior:

• Rebuild Dataset (Arrow files): Use the download_mode argument in load_dataset() (docs).
  • [1] Set download_mode="force_redownload" to ignore the cached Arrow files and trigger the data loading script MedVision.py to rebuild the dataset.
• Redownload Raw Data:
  • [2] MedVision_FORCE_DOWNLOAD_DATA: Set this environment variable to True to force re-downloading raw images and annotations.
  • [3] .downloaded_datasets.json: This tracker file records downloaded status. Removing a dataset's entry here will trigger a re-download of the raw data for that dataset.

[!TIP] How to properly update/redownload raw data?

If you need to update raw data (images, masks, landmarks) using [2] or [3], you MUST ALSO use [1] (download_mode="force_redownload").

Why? Because if Hugging Face finds a valid cached dataset (Arrow files), it will load it directly and skip running the script entirely. Without running the script, the environment variable [2] or tracker file [3] will never be checked.

Summary:

• Update Arrow/Fields only: Use [1].
• Update Raw Data: Use [1] AND ([2] or [3]).

🔥 We will maintain a change log for essential updates.

Examples

import os
from datasets import load_dataset

# Set data folder
wd = os.path.join(os.getcwd(), "Data-testing")
os.makedirs(wd, exist_ok=True)
os.environ["MedVision_DATA_DIR"] = wd

# Pick a dataset config name and split
config = "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
    )

import os
from datasets import load_dataset

# Set data folder
wd = os.path.join(os.getcwd(), "Data-testing")
os.makedirs(wd, exist_ok=True)
os.environ["MedVision_DATA_DIR"] = wd

# Pick a dataset config name and split
config = "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
    )

import os
from datasets import load_dataset

# Set data folder
wd = os.path.join(os.getcwd(), "Data-testing")
os.makedirs(wd, exist_ok=True)
os.environ["MedVision_DATA_DIR"] = wd

# Pick a dataset config name and split
config = "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
        download_mode="force_redownload",
    )

import os
from datasets import load_dataset

# Set data folder
wd = os.path.join(os.getcwd(), "Data-testing")
os.makedirs(wd, exist_ok=True)
os.environ["MedVision_DATA_DIR"] = wd

# Pick a dataset config name and split
config = "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Force redownload
os.environ["MedVision_FORCE_DOWNLOAD_DATA"] = "True"

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
        download_mode="force_redownload",
    )

Download Mode in MedVision Dataset

🔥 Summary: Understanding the download mode of MedVision dataset

💿 Batch Download (Optional)

📚 Read the Docs: Loading data → batch download · CLI reference

Since data downloading and processing take time, you can download datasets from the tasks list or configs list in advance.

[!WARNING] You need to set an API token for these datasets (see detailed instructions): FeTA24, SKM-TEA, and ToothFairy2

[!NOTE] Per-slice QC figures are opt-in (MedVision_DOWNLOAD_QC_FIGURES, default False).

export MedVision_DOWNLOAD_QC_FIGURES="True"   # default: "False"

The QC figures are review material — nothing in the loader reads them, and no task needs them — but they are ~99% of the annotation payload (298 GB of PNG against 3 GB of annotation). Until v1.4.0 they shipped inside Datasets/<dataset>.zip, which pushed BraTS24.zip to 72.6 GB and MSD.zip to 51.4 GB — past Hugging Face's 50 GB per-file limit, a hard publish failure. Since v1.4.0 they ship in their own archives: Datasets/<dataset>_fig.zip, or Datasets/<dataset>_fig.partNN.zip where a single archive would again clear 50 GB.

Set the variable to True and the figures are restored to exactly the paths they occupied before v1.4.0 — the archives carry the same arcnames, so nothing is relocated. Details worth knowing:

• Checked on every load. Setting the flag on a machine whose annotations are already present still fetches the figures — and only the figures: the image, landmark and planner downloads stay step 3's decision, so nothing else is re-pulled.
• Tracked per annotation version. .downloaded_datasets.json records "qc_figures_<dataset>": "1.4.0" — the biometry version the figures belong to. A release that regenerates a dataset's biometry re-fetches its figures; a release that leaves it alone does not.
• Never re-downloads what is already on disk. With no usable recorded version — any pre-v1.4.0 install already holds every figure — the directory itself is read and its version written back. Datasets that publish no figures (roughly half) record the attempt too, so they do not re-query the Hub on every load.
• Retry with MedVision_FORCE_DOWNLOAD_DATA=True, which also forces figures already on disk to be fetched again. Shards are independent zips (not zip -s volumes), so they extract in any order and a missing one costs only its own figures.

[!TIP] Command:

python -m medvision_bm.benchmark.download_datasets

Arguments:

• --data_dir: (required) data folder
• --tasks_json: task json file
• --configs_csv: config csv file
• --force_download_data: (store_true) force redownload raw imaging data ⚠️ for debugging only; it will repeatedly download data for tasks/configs of the same dataset

Download from a task-list JSON (replace <task-list-json>, <data-folder>):

python -m medvision_bm.benchmark.download_datasets \
--tasks_json <task-list-json> \
--data_dir <data-folder>

Or from a configs CSV (replace <config-list-csv>, <data-folder>):

python -m medvision_bm.benchmark.download_datasets \
--configs_csv <config-list-csv> \
--data_dir <data-folder>

📜 License

MedVision is released under the Creative Commons Attribution 4.0 International (CC-BY 4.0) license. Users are permitted to utilize, adapt, and build upon this dataset for both academic and commercial purposes, provided that appropriate credit is given. MedVision is a meta-dataset built upon various publicly available source datasets. While the annotations provided by MedVision are covered by the CC-BY 4.0 license, any downstream application must continue to comply with the specific usage terms and licensing requirements stipulated by the curators of the original raw imaging data. It is the responsibility of the user to ensure that their application of this data aligns with the license agreements of all constituent source datasets.

🩵 Acknowledgement

This work was supported by

• The United Kingdom Research and Innovation (grant EP/S02431X/1), UKRI Centre for Doctoral Training in Biomedical AI at the University of Edinburgh, School of Informatics
• The Edinburgh International Data Facility (EIDF) and the Data-Driven Innovation Programme at the University of Edinburgh.

MedVision is based on some open-source projects:

• EvolvingLMMs-Lab/lmms-eval: VLM evaluation framework
• EleutherAI/lm-evaluation-harness: LLM evaluation framework
• vllm-project/vllm: LLM/VLM inference
• volcengine/verl: Volcano Engine Reinforcement Learning for LLMs