Shadowgraph
About
Sign in

YongchengYAO/MedVision

Dataset

23

stars

115

commits

2

linked in READMEs

Sep 3, 2026

updated

angle
detection
distance
image
measurement
medical

README

MedVision Logo

MedVision Dataset

| 🌏 Project | 🧑🏻‍💻 Code | 🩻 Dataset | 🔎 Data Explorer | 🤗 Models | 📖 arXiv | 💼 LinkedIn |

💿 v1.4.0 | 33.2K 3D images | 12.0M 2D slices | 25.7M single-instance / 50.5M multi-instance annotations | multi-modality | multi-anatomy 💿

📏 Annotation: segmentation mask | landmark coordinate | bounding box | tumor/lesion size | distance | angle 📏

@misc{yao2026medvisionbenchmarkingquantitativemedical,
      title={MedVision: Benchmarking Quantitative Medical Image Analysis}, 
      author={Yongcheng Yao and Yongshuo Zong and Raman Dutt and Yongxin Yang and Sotirios A Tsaftaris and Timothy Hospedales},
      year={2026},
      eprint={2511.18676},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.18676}, 
}

News

  • [Aug 19, 2026] 🚀 Release MedVision dataset v1.4.0 [release-v1.4.0]

    • Regenerate Tumor-Lesion-Size annotations for all 12 tumour/lesion datasets: multi-instance annotations 75K --> 3.8M

  • [Aug 9, 2026] 🚀 Release MedVision dataset v1.3.0 [release-v1.3.0]

    • New dataset: MSWAL (484 abdominal CT cases; tumor/lesion labels: liver tumour, kidney tumour, pancreatic cancer, liver cyst, and kidney cyst).

  • [Aug 3, 2026] 🚀 Release MedVision dataset v1.2.1 [release-v1.2.1]

    • ⚠️ Corrects MAMA-MIA and PI-CAI, whose v1.2.0 annotations were recorded in the source orientation instead of RAS+ — the loader reoriented the images at load time without renumbering the coordinates. Their v1.2.0 annotations are withdrawn. If you have used either dataset, clear that cache once.
    • No other dataset is affected — the other 28 resolve to exactly the same annotation files as at v1.2.0.
    • New: scripts/gen-annotations/ rebuilds the preprocessed images and annotations of any dataset from its original source -- For the record only, you never use it to load data.

  • [Jul 28, 2026] 🚀 Release MedVision dataset v1.2.0 [release-v1.2.0]

    • Highlight: 8 new datasets (130 configs) — AFIDs, DEEP-PSMA, LIDC-IDRI, LNQ2023, MAMA-MIA, PDDCA, PI-CAI, VerSe.
    • No existing annotation changed. Annotation versions now resolve per dataset: the version you set is a ceiling, and each dataset loads the newest annotation it published at or before it. Pinning '1.1.1' or older keeps working for every pre-existing dataset (check Annotation Version Control).
    • ⚠️ Fixes a stale-cache defect present in all earlier versions. The cache key used the version you requested rather than the annotation actually loaded, so load_dataset could silently return previously cached rows after the annotations changed — which really happened, to the v1.1.0 T/L train/test split. See Fixed: cached data could be stale for who is affected and how to clear it. The data root is now part of the key too, which matters only if your HuggingFace cache is not already co-located with it. Because the cache key changed, existing Arrow caches rebuild once on next use (reads the annotation file, no re-download)

  • [Jun 29, 2026] 🚀 Release MedVision dataset v1.1.1 [release-v1.1.1]

    • Highlight: corrected T/L ellipse fit — fixes a transposed in-plane voxel-spacing bug (wrong axis lengths and major/minor labelling on anisotropic slices, e.g. sagittal/coronal); ~22% fewer T/L samples on anisotropic data, isotropic data (e.g. KiPA22) essentially unchanged
    • The codebase medvision_ds will be automatically updated to the latest (v1.1.1)
    • Backward compatibility: MedVision_PLANNER_VERSION='latest' now resolves to v1.1.1; pin '1.1.0' or '1.0.0' for earlier annotations. Only the Tumor-Lesion-Size task changed — all other tasks fall back to v1.0.0.

  • [May 14, 2026] 🚀 Release MedVision dataset v1.1.0 [release-v1.1.0]

    • Highlight: new T/L samples filtering (with ambiguous cases removed), more T/L samples with a single small target (cluster size > 20)
    • The codebase medvision_ds will be automatically updated to the latest (v1.1.0)
    • Backward compatibility: the env var MedVision_PLANNER_VERSION is required (v1.1.0+) to specify the annotation data version. Setting MedVision_PLANNER_VERSION='1.0.0' will fall back to MedVision dataset v1.0.0.

  • [Oct 8, 2025] 🚀 Release MedVision dataset v1.0.0


Change Log

For essential updates, check the change log.


Datasets

File structure: raw data will be automatically downloaded and processed, and our annotations are in each dataset folder

📝 The MedVision dataset consists of public medical images and quantitative annotations from this study. MRI: Magnetic Resonance Imaging; CT: Computed Tomography; PET: positron emission tomography; US: Ultrasound; b-box: bounding box; T/L: tumor/lesion size; A/D: angle/distance; HF: HuggingFace; GC: Grand-Challenge; * redistributed. Sample counts are for annotation v1.2.0 — b-box and A/D are identical in every release, and only T/L was ever regenerated (in 1.1.0 and 1.1.1). MSWAL was introduced at v1.3.0; its counts are for that release.

DatasetAnatomyModalityAnnotationAvailabilitySource# Sample (Train/Test)Status
b-boxT/LA/D
AbdomenAtlasabdomenCTb-boxopenHF6.8 / 2.9M00✅
AbdomenCT-1KabdomenCTb-boxopenZenodo0.7 / 0.3M00✅
ACDCheartMRIb-boxopenHF*, others9.5 / 4.8K00✅
AFIDsbrainMRIA/DopenHF*, OpenNeuro00300 / 132✅
AMOS22abdomenCT, MRIb-boxopenHF*, Zenodo0.5 / 0.2M00✅
autoPET-IIIwhole bodyCT, PETb-box, T/LopenHF*, others22 / 9.9K570 / 3090✅
BCV15abdomenCTb-boxopenHF*, Synapse48 / 20K00✅
BraTS24brainMRIb-box, T/LopenHF*, Synapse0.8 / 0.3M11 / 4.6K0✅
CAMUSheartUSb-boxopenHF*, others0.7 / 0.3M00✅
Ceph-Bio-400head and neckX-rayA/DopenHF*, others005.3 / 2.3K✅
CrossMoDAbrainMRIb-boxopenHF*, Zenodo3.0 / 1.0K00✅
DEEP-PSMAwhole bodyPETb-box, T/LopenHF*, Zenodo1.3 / 0.9K34 / 600✅
FeTA24fetal brainMRIb-box, A/DregistrationSynapse34 / 15K0225 / 100✅
FLARE22abdomenCTb-boxopenHF*, others72 / 33K00✅
HNTSMRG24head and neckMRIb-box, T/LopenHF*, Zenodo23 / 9.4K1.6 / 0.6K0✅
ISLES24brainMRIb-boxopenHF*, GC7.2 / 2.6K00✅
KiPA22kidneyCTb-box, T/LopenHF*, GC26 / 11K2.0 / 1.0K0✅
KiTS23kidneyCTb-box, T/LopenHF*, GC80 / 35K5.0 / 2.1K0✅
LIDC-IDRIlungCTb-box, T/LopenHF*, TCIA7.3 / 3.0K314 / 1030✅
LNQ2023mediastinumCTb-box, T/LopenHF*, TCIA1.2 / 0.5K34 / 110✅
MAMA-MIAbreastMRIb-box, T/LopenHF*, Synapse47 / 21K2.3 / 1.0K0✅
MSDmultipleCT, MRIb-box, T/LopenHF*, others0.2 / 0.1M4.3 / 1.8K0✅
MSWALabdomenCTb-box, T/LopenHF*42 / 18K5.8 / 2.5K0✅
OAIZIB-CMkneeMRIb-boxopenHF0.5 / 0.2M00✅
PDDCAhead and neckCTb-box, A/DopenHF*, others10 / 4.8K092 / 40✅
PI-CAIprostateMRIb-box, T/LopenHF*, Zenodo3.9 / 1.6K238 / 1570✅
SKM-TEAkneeMRIb-boxregistrationothers0.2 / 0.1M00✅
ToothFairy2toothCTb-boxregistrationothers1.0 / 0.4M00✅
TopCoW24brainCT, MRIb-boxopenHF*, Zenodo29 / 13K00✅
TotalSegmentatormultipleCT, MRIb-boxopenHF*, Zenodo5.4 / 2.2M00✅
VerSespineCTb-box, A/DopenHF*, others0.2 / 0.1M01.1 / 0.5K✅
Total17 / 7.3M34 / 15K7.0 / 3.0K

⚠️ For the following datasets, which do not allow redistribution, you need to apply for access from data owners, (optionally) upload to your private HF dataset repo, and set corresponding environment variables.

DatasetSourceHost PlatformEnv Var
FeTA24https://www.synapse.org/Synapse:syn25649159/wiki/610007SynapseSYNAPSE_TOKEN
SKM-TEAhttps://aimi.stanford.edu/datasets/skm-tea-knee-mriHuggingfaceMedVision_SKMTEA_HF_ID
ToothFairy2https://ditto.ing.unimore.it/toothfairy2/HuggingfaceMedVision_ToothFairy2_HF_ID

📝 For SKM-TEA and ToothFairy2, you need to process the raw data and upload the preprocessed data to your private HF dataset repo. To use HF private dataset, you need to set HF_TOKEN and login with hf auth login --token $HF_TOKEN --add-to-git-credential

📝 AbdomenAtlas is fetched from the official AbdomenAtlas/AbdomenAtlas1.0Mini repo, which is gated — the imaging files return 401 for anonymous requests. Accept the dataset terms on that page, then set HF_TOKEN and login with hf auth login --token $HF_TOKEN --add-to-git-credential; the download works for a user who has accepted the terms.


New Datasets Guide

To add new datasets, check this blog for an introduction of MedVision dataset.

Requirement

📝 Note: trust_remote_code is no longer supported in datasets>=4.0.0, install dataset with pip install datasets==3.6.0


Use

import os
from datasets import load_dataset

# Set data folder
os.environ["MedVision_DATA_DIR"] = <your/data/folder>

# Required: annotation version. No default — loading raises without it.
os.environ["MedVision_PLANNER_VERSION"] = "latest"

# Pick a dataset config name and split
config = <config-name> # e.g., "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
    )

📝 List of config names here (./info/v1.4.0, 1002 configs). Config lists are versioned, one snapshot per release, and a config is listed only when it resolves at that pin and yields at least one sample there — so a config that is empty at a given pin appears only in the lists where it has data: ./info/v1.0.0-v1.1.1 (818 configs), ./info/v1.2.0 (920), ./info/v1.3.0 (998). Use ./info/v1.0.0-v1.1.1 when pinning an annotation version below 1.2.0, since datasets added in v1.2.0 cannot be loaded at an earlier version.


Environment Variables

# Set where data will be saved, requires ~1T for the complete dataset
export MedVision_DATA_DIR=<your/data/folder>

# Required: the newest annotations you are willing to load (no default; unset
# raises an error). Accepted: 'latest', any published annotation version
# (1.0.0 | 1.1.0 | 1.1.1 | 1.2.0), or the medvision_ds release version.
# Anything else — a malformed value like 'v1.1.1' or '1.2', or a well-formed but
# unpublished one like '1.1.5' — is refused, with the accepted set listed.
#
# The value is a CEILING, not a selection: each dataset loads the newest
# annotation published at or before it. So a release that did not regenerate a
# dataset never changes what that dataset loads, and datasets introduced after
# the version you pin cannot be loaded at that version.
export MedVision_PLANNER_VERSION=latest

# Acknowledges that you are deliberately loading an older annotation. Required
# ONLY when you pin a version older than the newest one published FOR THE DATASET
# you are loading, for ANY task. Two values are accepted: that dataset's newest
# annotation version, or the release version as a blanket acknowledgement. The
# error tells you both. Use the release value for a catalogue sweep — one env var
# cannot hold several per-dataset values. Pinning an older version is a valid
# choice when the latest fix does not affect your task or slices;
# see doc/release-v1.2.0.md for what changed.
export MedVision_ACK_RELEASE=1.2.0

# Force download and process raw images, default to "False"
export MedVision_FORCE_DOWNLOAD_DATA="False"

# Force install dataset codebase, default to "True"
export MedVision_FORCE_INSTALL_CODE="True"

# Download the per-slice QC figures too, default to "False". The figures are review
# material — no task reads them — and they are ~99% of the annotation payload
# (298 GB of PNG against 3 GB of annotation), so since v1.4.0 they ship in their own
# archives: <dataset>_fig.zip, or <dataset>_fig.partNN.zip for the two sets that
# would otherwise exceed Hugging Face's 50 GB per-file limit. Set this to "True" and
# they are restored to exactly the paths they occupied before v1.4.0.
#
# This fetches the FIGURES ONLY when the rest of the dataset is already present:
# the flag never re-triggers the image, landmark or planner download, and figures
# already on disk are left alone -- including those from a pre-v1.4.0 install,
# where they arrived inside Datasets/<dataset>.zip.
#
# What counts as "already on disk" is per ANNOTATION VERSION, not a yes/no:
# .downloaded_datasets.json records qc_figures_<dataset>: "1.4.0", the biometry
# version the figures belong to. Regenerating a dataset's biometry raises that
# version and its figures are re-fetched; a release that leaves the dataset alone
# re-fetches nothing. MedVision_FORCE_DOWNLOAD_DATA="True" forces them either way.
export MedVision_DOWNLOAD_QC_FIGURES="False"

📖 Essential Dataset Concept

We cover some essential concepts that help we use the MedVision dataset with ease.

Concepts: Dataset & Data Configuration

  • MedVision: the collection of public imaging data and our annotations
  • dataset: name of the public datasets, such BraTS24, MSD, OAIZIB-CM
  • data-config: name of predefined subsets
    • naming convention: {dataset}_{annotation-type}_{task-ID}_{slice}_{split}
      • dataset: details
      • annotation-type:
        • BoxSize: detection annotations (bounding box)
        • TumorLesionSize: tumor/lesion size annotations
        • BiometricsFromLandmarks: angle/distance annotations
      • task-ID: Task[xx] (Note, this is a local ID in the dataset, not a glocal ID in MedVision.)
        • For datasets with multiple image-mask pairs, we defined tasks in medvision_ds/datasets/*/preprocess_*.py
        • source: medvision_ds
        • e.g., detection tasks for the BraTS24 dataset is defined in the benchmark_plan in medvision_ds/datasets/BraTS24/preprocess_detection.py
      • slice: [Sagittal, Coronal, Axial]
      • split: [Train, Test]

What's returned from MedVision Dataset?

We only share the annotations (https://huggingface.co/datasets/YongchengYAO/MedVision/tree/main/Datasets). The data loading script MedVision.py will handle raw image downloading and processing. The returned fields in each sample is defined as followed.

⚠️ In MedVision.py, the class MedVision(GeneratorBasedBuilder) defines the feature dict and the method _generate_examples() builds the dataset.

Code block in `MedVision(GeneratorBasedBuilder)` (Click to expand) 
"""
MedVision dataset.

NOTE: To update the features returned by the load_dataset() method, the followings should be updated:
        - the feature dict in this class 
        - the dict yielded by the _generate_examples() method 
"""

# The feature dict for the task:
# - Mask-Size
features_dict_MaskSize = {
    "dataset_name": Value("string"),
    "taskID": Value("string"),
    "taskType": Value("string"),
    "image_file": Value("string"),
    "mask_file": Value("string"),
    "slice_dim": Value("uint8"),
    "slice_idx": Value("uint16"),
    "label": Value("uint16"),
    "image_size_2d": Sequence(Value("uint16"), length=2),
    "pixel_size": Sequence(Value("float16"), length=2),
    "image_size_3d": Sequence(Value("uint16"), length=3),
    "voxel_size": Sequence(Value("float16"), length=3),
    "pixel_count": Value("uint32"),
    "ROI_area": Value("float16"),
}

# The feature dict for the task:
# - Box-Size
features_dict_BoxSize = {
    "dataset_name": Value("string"),
    "taskID": Value("string"),
    "taskType": Value("string"),
    "image_file": Value("string"),
    "mask_file": Value("string"),
    "slice_dim": Value("uint8"),
    "slice_idx": Value("uint16"),
    "label": Value("uint16"),
    "image_size_2d": Sequence(Value("uint16"), length=2),
    "pixel_size": Sequence(Value("float16"), length=2),
    "image_size_3d": Sequence(Value("uint16"), length=3),
    "voxel_size": Sequence(Value("float16"), length=3),
    "bounding_boxes": Sequence(
        {
            "min_coords": Sequence(Value("uint16"), length=2),
            "max_coords": Sequence(Value("uint16"), length=2),
            "center_coords": Sequence(Value("uint16"), length=2),
            "dimensions": Sequence(Value("uint16"), length=2),
            "sizes": Sequence(Value("float16"), length=2),
        },
    ),
}

features_dict_BiometricsFromLandmarks = {
    "dataset_name": Value("string"),
    "taskID": Value("string"),
    "taskType": Value("string"),
    "image_file": Value("string"),
    "landmark_file": Value("string"),
    "slice_dim": Value("uint8"),
    "slice_idx": Value("uint16"),
    "image_size_2d": Sequence(Value("uint16"), length=2),
    "pixel_size": Sequence(Value("float16"), length=2),
    "image_size_3d": Sequence(Value("uint16"), length=3),
    "voxel_size": Sequence(Value("float16"), length=3),
    "biometric_profile": {
        "metric_type": Value("string"),
        "metric_map_name": Value("string"),
        "metric_key": Value("string"),
        "metric_value": Value("float16"),
        "metric_unit": Value("string"),
        "slice_dim": Value("uint8"),
    },
}

features_dict_TumorLesionSize = {
    "dataset_name": Value("string"),
    "taskID": Value("string"),
    "taskType": Value("string"),
    "image_file": Value("string"),
    "landmark_file": Value("string"),
    "mask_file": Value("string"),
    "slice_dim": Value("uint8"),
    "slice_idx": Value("uint16"),
    "label": Value("uint16"),
    "image_size_2d": Sequence(Value("uint16"), length=2),
    "pixel_size": Sequence(Value("float16"), length=2),
    "image_size_3d": Sequence(Value("uint16"), length=3),
    "voxel_size": Sequence(Value("float16"), length=3),
    "biometric_profile": Sequence(
        {
            "metric_type": Value("string"),
            "metric_map_name": Value("string"),
            "metric_key_major_axis": Value("string"),
            "metric_value_major_axis": Value("float16"),
            "metric_key_minor_axis": Value("string"),
            "metric_value_minor_axis": Value("float16"),
            "metric_unit": Value("string"),
        },
    ),
}
Code block in `_generate_examples` (Click to expand) 
# Task type: Mask-Size
if taskType == "Mask-Size":
    flatten_slice_profiles = (
        MedVision_BenchmarkPlannerSegmentation.flatten_slice_profiles_2d
    )
    if imageSliceType.lower() == "sagittal":
        slice_dim = 0
    elif imageSliceType.lower() == "coronal":
        slice_dim = 1
    elif imageSliceType.lower() == "axial":
        slice_dim = 2
    slice_profile_flattened = flatten_slice_profiles(biometricData, slice_dim)
    for idx, case in enumerate(slice_profile_flattened):
        # Skip cases with a mask size smaller than 200 pixels
        if case["pixel_count"] < 200:
            continue
        else:
            yield idx, {
                "dataset_name": dataset_name,
                "taskID": taskID,
                "taskType": taskType,
                "image_file": os.path.join(dataset_dir, case["image_file"]),
                "mask_file": os.path.join(dataset_dir, case["mask_file"]),
                "slice_dim": case["slice_dim"],
                "slice_idx": case["slice_idx"],
                "label": case["label"],
                "image_size_2d": case["image_size_2d"],
                "pixel_size": case["pixel_size"],
                "image_size_3d": case["image_size_3d"],
                "voxel_size": case["voxel_size"],
                "pixel_count": case["pixel_count"],
                "ROI_area": case["ROI_area"],
            }

# Task type: Box-Size
if taskType == "Box-Size":
    if imageType.lower() == "2d":
        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerDetection.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            # Skip cases with multiple bounding boxes in the same slice
            if len(case["bounding_boxes"]) > 1:
                continue
            # Skip cases with a bounding box size smaller than 10 pixels in any dimension
            elif (
                case["bounding_boxes"][0]["dimensions"][0] < 10
                or case["bounding_boxes"][0]["dimensions"][1] < 10
            ):
                continue
            else:
                yield idx, {
                    "dataset_name": dataset_name,
                    "taskID": taskID,
                    "taskType": taskType,
                    "image_file": os.path.join(dataset_dir, case["image_file"]),
                    "mask_file": os.path.join(dataset_dir, case["mask_file"]),
                    "slice_dim": case["slice_dim"],
                    "slice_idx": case["slice_idx"],
                    "label": case["label"],
                    "image_size_2d": case["image_size_2d"],
                    "pixel_size": case["pixel_size"],
                    "image_size_3d": case["image_size_3d"],
                    "voxel_size": case["voxel_size"],
                    "bounding_boxes": case["bounding_boxes"],
                }

# Task type: Biometrics-From-Landmarks
if taskType == "Biometrics-From-Landmarks":
    if imageType.lower() == "2d":
        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerBiometry.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            yield idx, {
                "dataset_name": dataset_name,
                "taskID": taskID,
                "taskType": taskType,
                "image_file": os.path.join(dataset_dir, case["image_file"]),
                "landmark_file": os.path.join(
                    dataset_dir, case["landmark_file"]
                ),
                "slice_dim": case["slice_dim"],
                "slice_idx": case["slice_idx"],
                "image_size_2d": case["image_size_2d"],
                "pixel_size": case["pixel_size"],
                "image_size_3d": case["image_size_3d"],
                "voxel_size": case["voxel_size"],
                "biometric_profile": case["biometric_profile"],
            }

# Task type: Biometrics-From-Landmarks-Distance
if taskType == "Biometrics-From-Landmarks-Distance":
    if imageType.lower() == "2d":
        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerBiometry.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            if case["biometric_profile"]["metric_type"] == "distance":
                yield idx, {
                    "dataset_name": dataset_name,
                    "taskID": taskID,
                    "taskType": taskType,
                    "image_file": os.path.join(dataset_dir, case["image_file"]),
                    "landmark_file": os.path.join(
                        dataset_dir, case["landmark_file"]
                    ),
                    "slice_dim": case["slice_dim"],
                    "slice_idx": case["slice_idx"],
                    "image_size_2d": case["image_size_2d"],
                    "pixel_size": case["pixel_size"],
                    "image_size_3d": case["image_size_3d"],
                    "voxel_size": case["voxel_size"],
                    "biometric_profile": case["biometric_profile"],
                }

# Task type: Biometrics-From-Landmarks-Angle
if taskType == "Biometrics-From-Landmarks-Angle":
    if imageType.lower() == "2d":
        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerBiometry.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            if case["biometric_profile"]["metric_type"] == "angle":
                yield idx, {
                    "dataset_name": dataset_name,
                    "taskID": taskID,
                    "taskType": taskType,
                    "image_file": os.path.join(dataset_dir, case["image_file"]),
                    "landmark_file": os.path.join(
                        dataset_dir, case["landmark_file"]
                    ),
                    "slice_dim": case["slice_dim"],
                    "slice_idx": case["slice_idx"],
                    "image_size_2d": case["image_size_2d"],
                    "pixel_size": case["pixel_size"],
                    "image_size_3d": case["image_size_3d"],
                    "voxel_size": case["voxel_size"],
                    "biometric_profile": case["biometric_profile"],
                }

# Task type: Tumor-Lesion-Size
if taskType == "Tumor-Lesion-Size":
    if imageType.lower() == "2d":
        # Get the target label for the task
        target_label = benchmark_plan["tasks"][int(taskID) - 1]["target_label"]

        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerBiometry_fromSeg.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            # Skip cases with multiple fitted ellipses in the same slice
            if len(case["biometric_profile"]) > 1:
                continue
            else:
                yield idx, {
                    "dataset_name": dataset_name,
                    "taskID": taskID,
                    "taskType": taskType,
                    "image_file": os.path.join(dataset_dir, case["image_file"]),
                    "mask_file": os.path.join(dataset_dir, case["mask_file"]),
                    "landmark_file": os.path.join(
                        dataset_dir, case["landmark_file"]
                    ),
                    "slice_dim": case["slice_dim"],
                    "slice_idx": case["slice_idx"],
                    "label": target_label,
                    "image_size_2d": case["image_size_2d"],
                    "pixel_size": case["pixel_size"],
                    "image_size_3d": case["image_size_3d"],
                    "voxel_size": case["voxel_size"],
                    "biometric_profile": case["biometric_profile"],
                }

Dataset Building Workflow

Workflow

MedVision Dataset Building Workflow


There are a few venues to control the dataset loading and building behavior:

  • Rebuild Dataset (Arrow files): Use the download_mode argument in load_dataset() (docs).
    • [1] Set download_mode="force_redownload" to ignore the cached Arrow files and trigger the data loading script MedVision.py to rebuild the dataset.
  • Redownload Raw Data:
    • [2] MedVision_FORCE_DOWNLOAD_DATA: Set this environment variable to True to force re-downloading raw images and annotations.
    • [3] .downloaded_datasets.json: This tracker file records downloaded status. Removing a dataset's entry here will trigger a re-download of the raw data for that dataset.

[!Note] ⚠️ How to properly update/redownload raw data?

If you need to update raw data (images, masks, landmarks) using [2] or [3], you MUST ALSO use [1] (download_mode="force_redownload").

Why? Because if Hugging Face finds a valid cached dataset (Arrow files), it will load it directly and skip running the script entirely. Without running the script, the environment variable [2] or tracker file [3] will never be checked.

Summary:

  • Update Arrow/Fields only: Use [1].
  • Update Raw Data: Use [1] AND ([2] or [3]).

[!Important] When would I need this? Normally never — v1.2.0 keys the Arrow cache on the annotation version actually loaded, so a cache is invalidated whenever the annotations behind it change.

There is one historical exception. Versions before v1.2.0 keyed the cache on the version you requested, and the v1.1.1 release re-aligned the already-published v1.1.0 T/L train/test split in place without a version bump. A cache built for a Tumor-Lesion-Size config at MedVision_PLANNER_VERSION=1.1.0 before that release still holds the old partition, at both the Arrow and annotation-file layers. Clear it with [1] AND [2] once. See Fixed: cached data could be stale.

🔥 We will maintain a change log for essential updates.

Examples

Run this for the first time will download the raw data and build the dataset 
import os
from datasets import load_dataset

# Set data folder
wd = os.path.join(os.getcwd(), "Data-testing")
os.makedirs(wd, exist_ok=True)
os.environ["MedVision_DATA_DIR"] = wd

# Pick a dataset config name and split
config = "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
    )
Run the same script again will use the cached dataset 
import os
from datasets import load_dataset

# Set data folder
wd = os.path.join(os.getcwd(), "Data-testing")
os.makedirs(wd, exist_ok=True)
os.environ["MedVision_DATA_DIR"] = wd

# Pick a dataset config name and split
config = "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
    )
Adding `download_mode="force_redownload"` will skip raw data downloading and rebuild the dataset 
import os
from datasets import load_dataset

# Set data folder
wd = os.path.join(os.getcwd(), "Data-testing")
os.makedirs(wd, exist_ok=True)
os.environ["MedVision_DATA_DIR"] = wd

# Pick a dataset config name and split
config = "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
        download_mode="force_redownload",
    )
Adding `download_mode="force_redownload"` and `os.environ["MedVision_FORCE_DOWNLOAD_DATA"] = "True"` will redownload raw data and rebuild the dataset 
import os
from datasets import load_dataset

# Set data folder
wd = os.path.join(os.getcwd(), "Data-testing")
os.makedirs(wd, exist_ok=True)
os.environ["MedVision_DATA_DIR"] = wd

# Pick a dataset config name and split
config = "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Force redownload
os.environ["MedVision_FORCE_DOWNLOAD_DATA"] = "True"

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
        download_mode="force_redownload",
    )

Download Mode in MedVision Dataset

(Advanced) Understand how the customized dataset loading script `MedVision.py` changes the behavior of `download_mode` in `load_dataset()`

  • download_mode can be one of these: "reuse_dataset_if_exists" (default), "reuse_cache_if_exists", "force_redownload"

  • Default behavior of download_mode in load_dataset():

    DownloadsDataset
    reuse_dataset_if_exists (default)ReuseReuse
    reuse_cache_if_existsReuseFresh
    force_redownloadFreshFresh

  • download_mode in MedVision dataset:

    DownloadsDataset
    reuse_dataset_if_exists (default)ReuseReuse
    reuse_cache_if_existsReuseFresh
    force_redownload (MedVision_FORCE_DOWNLOAD_DATA=False)ReuseFresh
    force_redownload (MedVision_FORCE_DOWNLOAD_DATA=True)FreshFresh

Advanced Usage

The dataset codebase medvision_ds can be used to scale the dataset, including adding new annotation types and datasets.

🛠️ Install

pip install "git+https://huggingface.co/datasets/YongchengYAO/MedVision.git#subdirectory=src"
pip show medvision_ds

or

# First, install the benchmark codebase: medvision_bm
pip install "git+https://github.com/YongchengYAO/MedVision.git" 
 
# Install the dataset codebase: medvision_ds
python -m medvision_bm.benchmark.install_medvision_ds --data_dir <local-data-folder>

🧑🏻‍💻 Use utility functions for image processing

from medvision_ds.utils.data_conversion import (
    convert_nrrd_to_nifti,
    convert_mha_to_nifti,
    convert_nii_to_niigz,
    convert_bmp_to_niigz,
    copy_img_header_to_mask,
    reorient_niigz_RASplus_batch_inplace,
)

from medvision_ds.utils.preprocess_utils import (
    split_4d_nifti,
)

👩🏼‍💻Examples of dataset scaling:


License: CC-BY-4.0

MedVision is released under the Creative Commons Attribution 4.0 International (CC-BY 4.0) license. Users are permitted to utilize, adapt, and build upon this dataset for both academic and commercial purposes, provided that appropriate credit is given. MedVision is a meta-dataset built upon various publicly available source datasets. While the annotations provided by MedVision are covered by the CC-BY 4.0 license, any downstream application must continue to comply with the specific usage terms and licensing requirements stipulated by the curators of the original raw imaging data. It is the responsibility of the user to ensure that their application of this data aligns with the license agreements of all constituent source datasets.

Acknowledgement

This work was supported by the United Kingdom Research and Innovation (grant EP/S02431X/1), UKRI Centre for Doctoral Training in Biomedical AI at the University of Edinburgh, School of Informatics.

Contributors

YongchengYAO

98 commits

CO
Claude Opus 4.8

5 commits

CF
Claude Fable 5

5 commits

CO
Claude Opus 5 (1M context)

3 commits

Similar repos

alexanderdann/CTSpine1K

Dataset

14

shared topics

huggingface/CADS-dataset

Dataset

4

shared topics

YongchengYAO/MedVision

The official codebase of "MedVision: Benchmarking Quantitative Medical Image Analysis"

15

General-Medical-AI/GMAI-VL-5.5M

Dataset

6

FreedomIntelligence/Med-MAT

Dataset

18

foreverbeliever/OmniMedVQA

Dataset

21

openmedlab/Awesome-Medical-Dataset

Collection of awesome medical dataset resources.

2,118

GoodBaiBai88/M3D-Seg

Dataset

12

Linked in READMEs

FreedomIntelligence/Awesome-AI4Med

A curated list of medical LLMs, multimodal systems, datasets, benchmarks, and more. 🏥

2,877

README badge

YongchengYAO/MedVision

The official codebase of "MedVision: Benchmarking Quantitative Medical Image Analysis"

15

README badge

YongchengYAO/MedVision

Dataset

23

stars

115

commits

2

linked in READMEs

Sep 3, 2026

updated

angle
detection
distance
image
measurement
medical

README

MedVision Logo

MedVision Dataset

| 🌏 Project | 🧑🏻‍💻 Code | 🩻 Dataset | 🔎 Data Explorer | 🤗 Models | 📖 arXiv | 💼 LinkedIn |

💿 v1.4.0 | 33.2K 3D images | 12.0M 2D slices | 25.7M single-instance / 50.5M multi-instance annotations | multi-modality | multi-anatomy 💿

📏 Annotation: segmentation mask | landmark coordinate | bounding box | tumor/lesion size | distance | angle 📏

@misc{yao2026medvisionbenchmarkingquantitativemedical,
      title={MedVision: Benchmarking Quantitative Medical Image Analysis}, 
      author={Yongcheng Yao and Yongshuo Zong and Raman Dutt and Yongxin Yang and Sotirios A Tsaftaris and Timothy Hospedales},
      year={2026},
      eprint={2511.18676},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.18676}, 
}

News

  • [Aug 19, 2026] 🚀 Release MedVision dataset v1.4.0 [release-v1.4.0]

    • Regenerate Tumor-Lesion-Size annotations for all 12 tumour/lesion datasets: multi-instance annotations 75K --> 3.8M

  • [Aug 9, 2026] 🚀 Release MedVision dataset v1.3.0 [release-v1.3.0]

    • New dataset: MSWAL (484 abdominal CT cases; tumor/lesion labels: liver tumour, kidney tumour, pancreatic cancer, liver cyst, and kidney cyst).

  • [Aug 3, 2026] 🚀 Release MedVision dataset v1.2.1 [release-v1.2.1]

    • ⚠️ Corrects MAMA-MIA and PI-CAI, whose v1.2.0 annotations were recorded in the source orientation instead of RAS+ — the loader reoriented the images at load time without renumbering the coordinates. Their v1.2.0 annotations are withdrawn. If you have used either dataset, clear that cache once.
    • No other dataset is affected — the other 28 resolve to exactly the same annotation files as at v1.2.0.
    • New: scripts/gen-annotations/ rebuilds the preprocessed images and annotations of any dataset from its original source -- For the record only, you never use it to load data.

  • [Jul 28, 2026] 🚀 Release MedVision dataset v1.2.0 [release-v1.2.0]

    • Highlight: 8 new datasets (130 configs) — AFIDs, DEEP-PSMA, LIDC-IDRI, LNQ2023, MAMA-MIA, PDDCA, PI-CAI, VerSe.
    • No existing annotation changed. Annotation versions now resolve per dataset: the version you set is a ceiling, and each dataset loads the newest annotation it published at or before it. Pinning '1.1.1' or older keeps working for every pre-existing dataset (check Annotation Version Control).
    • ⚠️ Fixes a stale-cache defect present in all earlier versions. The cache key used the version you requested rather than the annotation actually loaded, so load_dataset could silently return previously cached rows after the annotations changed — which really happened, to the v1.1.0 T/L train/test split. See Fixed: cached data could be stale for who is affected and how to clear it. The data root is now part of the key too, which matters only if your HuggingFace cache is not already co-located with it. Because the cache key changed, existing Arrow caches rebuild once on next use (reads the annotation file, no re-download)

  • [Jun 29, 2026] 🚀 Release MedVision dataset v1.1.1 [release-v1.1.1]

    • Highlight: corrected T/L ellipse fit — fixes a transposed in-plane voxel-spacing bug (wrong axis lengths and major/minor labelling on anisotropic slices, e.g. sagittal/coronal); ~22% fewer T/L samples on anisotropic data, isotropic data (e.g. KiPA22) essentially unchanged
    • The codebase medvision_ds will be automatically updated to the latest (v1.1.1)
    • Backward compatibility: MedVision_PLANNER_VERSION='latest' now resolves to v1.1.1; pin '1.1.0' or '1.0.0' for earlier annotations. Only the Tumor-Lesion-Size task changed — all other tasks fall back to v1.0.0.

  • [May 14, 2026] 🚀 Release MedVision dataset v1.1.0 [release-v1.1.0]

    • Highlight: new T/L samples filtering (with ambiguous cases removed), more T/L samples with a single small target (cluster size > 20)
    • The codebase medvision_ds will be automatically updated to the latest (v1.1.0)
    • Backward compatibility: the env var MedVision_PLANNER_VERSION is required (v1.1.0+) to specify the annotation data version. Setting MedVision_PLANNER_VERSION='1.0.0' will fall back to MedVision dataset v1.0.0.

  • [Oct 8, 2025] 🚀 Release MedVision dataset v1.0.0


Change Log

For essential updates, check the change log.


Datasets

File structure: raw data will be automatically downloaded and processed, and our annotations are in each dataset folder

📝 The MedVision dataset consists of public medical images and quantitative annotations from this study. MRI: Magnetic Resonance Imaging; CT: Computed Tomography; PET: positron emission tomography; US: Ultrasound; b-box: bounding box; T/L: tumor/lesion size; A/D: angle/distance; HF: HuggingFace; GC: Grand-Challenge; * redistributed. Sample counts are for annotation v1.2.0 — b-box and A/D are identical in every release, and only T/L was ever regenerated (in 1.1.0 and 1.1.1). MSWAL was introduced at v1.3.0; its counts are for that release.

DatasetAnatomyModalityAnnotationAvailabilitySource# Sample (Train/Test)Status
b-boxT/LA/D
AbdomenAtlasabdomenCTb-boxopenHF6.8 / 2.9M00✅
AbdomenCT-1KabdomenCTb-boxopenZenodo0.7 / 0.3M00✅
ACDCheartMRIb-boxopenHF*, others9.5 / 4.8K00✅
AFIDsbrainMRIA/DopenHF*, OpenNeuro00300 / 132✅
AMOS22abdomenCT, MRIb-boxopenHF*, Zenodo0.5 / 0.2M00✅
autoPET-IIIwhole bodyCT, PETb-box, T/LopenHF*, others22 / 9.9K570 / 3090✅
BCV15abdomenCTb-boxopenHF*, Synapse48 / 20K00✅
BraTS24brainMRIb-box, T/LopenHF*, Synapse0.8 / 0.3M11 / 4.6K0✅
CAMUSheartUSb-boxopenHF*, others0.7 / 0.3M00✅
Ceph-Bio-400head and neckX-rayA/DopenHF*, others005.3 / 2.3K✅
CrossMoDAbrainMRIb-boxopenHF*, Zenodo3.0 / 1.0K00✅
DEEP-PSMAwhole bodyPETb-box, T/LopenHF*, Zenodo1.3 / 0.9K34 / 600✅
FeTA24fetal brainMRIb-box, A/DregistrationSynapse34 / 15K0225 / 100✅
FLARE22abdomenCTb-boxopenHF*, others72 / 33K00✅
HNTSMRG24head and neckMRIb-box, T/LopenHF*, Zenodo23 / 9.4K1.6 / 0.6K0✅
ISLES24brainMRIb-boxopenHF*, GC7.2 / 2.6K00✅
KiPA22kidneyCTb-box, T/LopenHF*, GC26 / 11K2.0 / 1.0K0✅
KiTS23kidneyCTb-box, T/LopenHF*, GC80 / 35K5.0 / 2.1K0✅
LIDC-IDRIlungCTb-box, T/LopenHF*, TCIA7.3 / 3.0K314 / 1030✅
LNQ2023mediastinumCTb-box, T/LopenHF*, TCIA1.2 / 0.5K34 / 110✅
MAMA-MIAbreastMRIb-box, T/LopenHF*, Synapse47 / 21K2.3 / 1.0K0✅
MSDmultipleCT, MRIb-box, T/LopenHF*, others0.2 / 0.1M4.3 / 1.8K0✅
MSWALabdomenCTb-box, T/LopenHF*42 / 18K5.8 / 2.5K0✅
OAIZIB-CMkneeMRIb-boxopenHF0.5 / 0.2M00✅
PDDCAhead and neckCTb-box, A/DopenHF*, others10 / 4.8K092 / 40✅
PI-CAIprostateMRIb-box, T/LopenHF*, Zenodo3.9 / 1.6K238 / 1570✅
SKM-TEAkneeMRIb-boxregistrationothers0.2 / 0.1M00✅
ToothFairy2toothCTb-boxregistrationothers1.0 / 0.4M00✅
TopCoW24brainCT, MRIb-boxopenHF*, Zenodo29 / 13K00✅
TotalSegmentatormultipleCT, MRIb-boxopenHF*, Zenodo5.4 / 2.2M00✅
VerSespineCTb-box, A/DopenHF*, others0.2 / 0.1M01.1 / 0.5K✅
Total17 / 7.3M34 / 15K7.0 / 3.0K

⚠️ For the following datasets, which do not allow redistribution, you need to apply for access from data owners, (optionally) upload to your private HF dataset repo, and set corresponding environment variables.

DatasetSourceHost PlatformEnv Var
FeTA24https://www.synapse.org/Synapse:syn25649159/wiki/610007SynapseSYNAPSE_TOKEN
SKM-TEAhttps://aimi.stanford.edu/datasets/skm-tea-knee-mriHuggingfaceMedVision_SKMTEA_HF_ID
ToothFairy2https://ditto.ing.unimore.it/toothfairy2/HuggingfaceMedVision_ToothFairy2_HF_ID

📝 For SKM-TEA and ToothFairy2, you need to process the raw data and upload the preprocessed data to your private HF dataset repo. To use HF private dataset, you need to set HF_TOKEN and login with hf auth login --token $HF_TOKEN --add-to-git-credential

📝 AbdomenAtlas is fetched from the official AbdomenAtlas/AbdomenAtlas1.0Mini repo, which is gated — the imaging files return 401 for anonymous requests. Accept the dataset terms on that page, then set HF_TOKEN and login with hf auth login --token $HF_TOKEN --add-to-git-credential; the download works for a user who has accepted the terms.


New Datasets Guide

To add new datasets, check this blog for an introduction of MedVision dataset.

Requirement

📝 Note: trust_remote_code is no longer supported in datasets>=4.0.0, install dataset with pip install datasets==3.6.0


Use

import os
from datasets import load_dataset

# Set data folder
os.environ["MedVision_DATA_DIR"] = <your/data/folder>

# Required: annotation version. No default — loading raises without it.
os.environ["MedVision_PLANNER_VERSION"] = "latest"

# Pick a dataset config name and split
config = <config-name> # e.g., "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
    )

📝 List of config names here (./info/v1.4.0, 1002 configs). Config lists are versioned, one snapshot per release, and a config is listed only when it resolves at that pin and yields at least one sample there — so a config that is empty at a given pin appears only in the lists where it has data: ./info/v1.0.0-v1.1.1 (818 configs), ./info/v1.2.0 (920), ./info/v1.3.0 (998). Use ./info/v1.0.0-v1.1.1 when pinning an annotation version below 1.2.0, since datasets added in v1.2.0 cannot be loaded at an earlier version.


Environment Variables

# Set where data will be saved, requires ~1T for the complete dataset
export MedVision_DATA_DIR=<your/data/folder>

# Required: the newest annotations you are willing to load (no default; unset
# raises an error). Accepted: 'latest', any published annotation version
# (1.0.0 | 1.1.0 | 1.1.1 | 1.2.0), or the medvision_ds release version.
# Anything else — a malformed value like 'v1.1.1' or '1.2', or a well-formed but
# unpublished one like '1.1.5' — is refused, with the accepted set listed.
#
# The value is a CEILING, not a selection: each dataset loads the newest
# annotation published at or before it. So a release that did not regenerate a
# dataset never changes what that dataset loads, and datasets introduced after
# the version you pin cannot be loaded at that version.
export MedVision_PLANNER_VERSION=latest

# Acknowledges that you are deliberately loading an older annotation. Required
# ONLY when you pin a version older than the newest one published FOR THE DATASET
# you are loading, for ANY task. Two values are accepted: that dataset's newest
# annotation version, or the release version as a blanket acknowledgement. The
# error tells you both. Use the release value for a catalogue sweep — one env var
# cannot hold several per-dataset values. Pinning an older version is a valid
# choice when the latest fix does not affect your task or slices;
# see doc/release-v1.2.0.md for what changed.
export MedVision_ACK_RELEASE=1.2.0

# Force download and process raw images, default to "False"
export MedVision_FORCE_DOWNLOAD_DATA="False"

# Force install dataset codebase, default to "True"
export MedVision_FORCE_INSTALL_CODE="True"

# Download the per-slice QC figures too, default to "False". The figures are review
# material — no task reads them — and they are ~99% of the annotation payload
# (298 GB of PNG against 3 GB of annotation), so since v1.4.0 they ship in their own
# archives: <dataset>_fig.zip, or <dataset>_fig.partNN.zip for the two sets that
# would otherwise exceed Hugging Face's 50 GB per-file limit. Set this to "True" and
# they are restored to exactly the paths they occupied before v1.4.0.
#
# This fetches the FIGURES ONLY when the rest of the dataset is already present:
# the flag never re-triggers the image, landmark or planner download, and figures
# already on disk are left alone -- including those from a pre-v1.4.0 install,
# where they arrived inside Datasets/<dataset>.zip.
#
# What counts as "already on disk" is per ANNOTATION VERSION, not a yes/no:
# .downloaded_datasets.json records qc_figures_<dataset>: "1.4.0", the biometry
# version the figures belong to. Regenerating a dataset's biometry raises that
# version and its figures are re-fetched; a release that leaves the dataset alone
# re-fetches nothing. MedVision_FORCE_DOWNLOAD_DATA="True" forces them either way.
export MedVision_DOWNLOAD_QC_FIGURES="False"

📖 Essential Dataset Concept

We cover some essential concepts that help we use the MedVision dataset with ease.

Concepts: Dataset & Data Configuration

  • MedVision: the collection of public imaging data and our annotations
  • dataset: name of the public datasets, such BraTS24, MSD, OAIZIB-CM
  • data-config: name of predefined subsets
    • naming convention: {dataset}_{annotation-type}_{task-ID}_{slice}_{split}
      • dataset: details
      • annotation-type:
        • BoxSize: detection annotations (bounding box)
        • TumorLesionSize: tumor/lesion size annotations
        • BiometricsFromLandmarks: angle/distance annotations
      • task-ID: Task[xx] (Note, this is a local ID in the dataset, not a glocal ID in MedVision.)
        • For datasets with multiple image-mask pairs, we defined tasks in medvision_ds/datasets/*/preprocess_*.py
        • source: medvision_ds
        • e.g., detection tasks for the BraTS24 dataset is defined in the benchmark_plan in medvision_ds/datasets/BraTS24/preprocess_detection.py
      • slice: [Sagittal, Coronal, Axial]
      • split: [Train, Test]

What's returned from MedVision Dataset?

We only share the annotations (https://huggingface.co/datasets/YongchengYAO/MedVision/tree/main/Datasets). The data loading script MedVision.py will handle raw image downloading and processing. The returned fields in each sample is defined as followed.

⚠️ In MedVision.py, the class MedVision(GeneratorBasedBuilder) defines the feature dict and the method _generate_examples() builds the dataset.

Code block in `MedVision(GeneratorBasedBuilder)` (Click to expand) 
"""
MedVision dataset.

NOTE: To update the features returned by the load_dataset() method, the followings should be updated:
        - the feature dict in this class 
        - the dict yielded by the _generate_examples() method 
"""

# The feature dict for the task:
# - Mask-Size
features_dict_MaskSize = {
    "dataset_name": Value("string"),
    "taskID": Value("string"),
    "taskType": Value("string"),
    "image_file": Value("string"),
    "mask_file": Value("string"),
    "slice_dim": Value("uint8"),
    "slice_idx": Value("uint16"),
    "label": Value("uint16"),
    "image_size_2d": Sequence(Value("uint16"), length=2),
    "pixel_size": Sequence(Value("float16"), length=2),
    "image_size_3d": Sequence(Value("uint16"), length=3),
    "voxel_size": Sequence(Value("float16"), length=3),
    "pixel_count": Value("uint32"),
    "ROI_area": Value("float16"),
}

# The feature dict for the task:
# - Box-Size
features_dict_BoxSize = {
    "dataset_name": Value("string"),
    "taskID": Value("string"),
    "taskType": Value("string"),
    "image_file": Value("string"),
    "mask_file": Value("string"),
    "slice_dim": Value("uint8"),
    "slice_idx": Value("uint16"),
    "label": Value("uint16"),
    "image_size_2d": Sequence(Value("uint16"), length=2),
    "pixel_size": Sequence(Value("float16"), length=2),
    "image_size_3d": Sequence(Value("uint16"), length=3),
    "voxel_size": Sequence(Value("float16"), length=3),
    "bounding_boxes": Sequence(
        {
            "min_coords": Sequence(Value("uint16"), length=2),
            "max_coords": Sequence(Value("uint16"), length=2),
            "center_coords": Sequence(Value("uint16"), length=2),
            "dimensions": Sequence(Value("uint16"), length=2),
            "sizes": Sequence(Value("float16"), length=2),
        },
    ),
}

features_dict_BiometricsFromLandmarks = {
    "dataset_name": Value("string"),
    "taskID": Value("string"),
    "taskType": Value("string"),
    "image_file": Value("string"),
    "landmark_file": Value("string"),
    "slice_dim": Value("uint8"),
    "slice_idx": Value("uint16"),
    "image_size_2d": Sequence(Value("uint16"), length=2),
    "pixel_size": Sequence(Value("float16"), length=2),
    "image_size_3d": Sequence(Value("uint16"), length=3),
    "voxel_size": Sequence(Value("float16"), length=3),
    "biometric_profile": {
        "metric_type": Value("string"),
        "metric_map_name": Value("string"),
        "metric_key": Value("string"),
        "metric_value": Value("float16"),
        "metric_unit": Value("string"),
        "slice_dim": Value("uint8"),
    },
}

features_dict_TumorLesionSize = {
    "dataset_name": Value("string"),
    "taskID": Value("string"),
    "taskType": Value("string"),
    "image_file": Value("string"),
    "landmark_file": Value("string"),
    "mask_file": Value("string"),
    "slice_dim": Value("uint8"),
    "slice_idx": Value("uint16"),
    "label": Value("uint16"),
    "image_size_2d": Sequence(Value("uint16"), length=2),
    "pixel_size": Sequence(Value("float16"), length=2),
    "image_size_3d": Sequence(Value("uint16"), length=3),
    "voxel_size": Sequence(Value("float16"), length=3),
    "biometric_profile": Sequence(
        {
            "metric_type": Value("string"),
            "metric_map_name": Value("string"),
            "metric_key_major_axis": Value("string"),
            "metric_value_major_axis": Value("float16"),
            "metric_key_minor_axis": Value("string"),
            "metric_value_minor_axis": Value("float16"),
            "metric_unit": Value("string"),
        },
    ),
}
Code block in `_generate_examples` (Click to expand) 
# Task type: Mask-Size
if taskType == "Mask-Size":
    flatten_slice_profiles = (
        MedVision_BenchmarkPlannerSegmentation.flatten_slice_profiles_2d
    )
    if imageSliceType.lower() == "sagittal":
        slice_dim = 0
    elif imageSliceType.lower() == "coronal":
        slice_dim = 1
    elif imageSliceType.lower() == "axial":
        slice_dim = 2
    slice_profile_flattened = flatten_slice_profiles(biometricData, slice_dim)
    for idx, case in enumerate(slice_profile_flattened):
        # Skip cases with a mask size smaller than 200 pixels
        if case["pixel_count"] < 200:
            continue
        else:
            yield idx, {
                "dataset_name": dataset_name,
                "taskID": taskID,
                "taskType": taskType,
                "image_file": os.path.join(dataset_dir, case["image_file"]),
                "mask_file": os.path.join(dataset_dir, case["mask_file"]),
                "slice_dim": case["slice_dim"],
                "slice_idx": case["slice_idx"],
                "label": case["label"],
                "image_size_2d": case["image_size_2d"],
                "pixel_size": case["pixel_size"],
                "image_size_3d": case["image_size_3d"],
                "voxel_size": case["voxel_size"],
                "pixel_count": case["pixel_count"],
                "ROI_area": case["ROI_area"],
            }

# Task type: Box-Size
if taskType == "Box-Size":
    if imageType.lower() == "2d":
        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerDetection.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            # Skip cases with multiple bounding boxes in the same slice
            if len(case["bounding_boxes"]) > 1:
                continue
            # Skip cases with a bounding box size smaller than 10 pixels in any dimension
            elif (
                case["bounding_boxes"][0]["dimensions"][0] < 10
                or case["bounding_boxes"][0]["dimensions"][1] < 10
            ):
                continue
            else:
                yield idx, {
                    "dataset_name": dataset_name,
                    "taskID": taskID,
                    "taskType": taskType,
                    "image_file": os.path.join(dataset_dir, case["image_file"]),
                    "mask_file": os.path.join(dataset_dir, case["mask_file"]),
                    "slice_dim": case["slice_dim"],
                    "slice_idx": case["slice_idx"],
                    "label": case["label"],
                    "image_size_2d": case["image_size_2d"],
                    "pixel_size": case["pixel_size"],
                    "image_size_3d": case["image_size_3d"],
                    "voxel_size": case["voxel_size"],
                    "bounding_boxes": case["bounding_boxes"],
                }

# Task type: Biometrics-From-Landmarks
if taskType == "Biometrics-From-Landmarks":
    if imageType.lower() == "2d":
        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerBiometry.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            yield idx, {
                "dataset_name": dataset_name,
                "taskID": taskID,
                "taskType": taskType,
                "image_file": os.path.join(dataset_dir, case["image_file"]),
                "landmark_file": os.path.join(
                    dataset_dir, case["landmark_file"]
                ),
                "slice_dim": case["slice_dim"],
                "slice_idx": case["slice_idx"],
                "image_size_2d": case["image_size_2d"],
                "pixel_size": case["pixel_size"],
                "image_size_3d": case["image_size_3d"],
                "voxel_size": case["voxel_size"],
                "biometric_profile": case["biometric_profile"],
            }

# Task type: Biometrics-From-Landmarks-Distance
if taskType == "Biometrics-From-Landmarks-Distance":
    if imageType.lower() == "2d":
        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerBiometry.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            if case["biometric_profile"]["metric_type"] == "distance":
                yield idx, {
                    "dataset_name": dataset_name,
                    "taskID": taskID,
                    "taskType": taskType,
                    "image_file": os.path.join(dataset_dir, case["image_file"]),
                    "landmark_file": os.path.join(
                        dataset_dir, case["landmark_file"]
                    ),
                    "slice_dim": case["slice_dim"],
                    "slice_idx": case["slice_idx"],
                    "image_size_2d": case["image_size_2d"],
                    "pixel_size": case["pixel_size"],
                    "image_size_3d": case["image_size_3d"],
                    "voxel_size": case["voxel_size"],
                    "biometric_profile": case["biometric_profile"],
                }

# Task type: Biometrics-From-Landmarks-Angle
if taskType == "Biometrics-From-Landmarks-Angle":
    if imageType.lower() == "2d":
        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerBiometry.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            if case["biometric_profile"]["metric_type"] == "angle":
                yield idx, {
                    "dataset_name": dataset_name,
                    "taskID": taskID,
                    "taskType": taskType,
                    "image_file": os.path.join(dataset_dir, case["image_file"]),
                    "landmark_file": os.path.join(
                        dataset_dir, case["landmark_file"]
                    ),
                    "slice_dim": case["slice_dim"],
                    "slice_idx": case["slice_idx"],
                    "image_size_2d": case["image_size_2d"],
                    "pixel_size": case["pixel_size"],
                    "image_size_3d": case["image_size_3d"],
                    "voxel_size": case["voxel_size"],
                    "biometric_profile": case["biometric_profile"],
                }

# Task type: Tumor-Lesion-Size
if taskType == "Tumor-Lesion-Size":
    if imageType.lower() == "2d":
        # Get the target label for the task
        target_label = benchmark_plan["tasks"][int(taskID) - 1]["target_label"]

        flatten_slice_profiles = (
            MedVision_BenchmarkPlannerBiometry_fromSeg.flatten_slice_profiles_2d
        )
        if imageSliceType.lower() == "sagittal":
            slice_dim = 0
        elif imageSliceType.lower() == "coronal":
            slice_dim = 1
        elif imageSliceType.lower() == "axial":
            slice_dim = 2
        slice_profile_flattened = flatten_slice_profiles(
            biometricData, slice_dim
        )
        for idx, case in enumerate(slice_profile_flattened):
            # Skip cases with multiple fitted ellipses in the same slice
            if len(case["biometric_profile"]) > 1:
                continue
            else:
                yield idx, {
                    "dataset_name": dataset_name,
                    "taskID": taskID,
                    "taskType": taskType,
                    "image_file": os.path.join(dataset_dir, case["image_file"]),
                    "mask_file": os.path.join(dataset_dir, case["mask_file"]),
                    "landmark_file": os.path.join(
                        dataset_dir, case["landmark_file"]
                    ),
                    "slice_dim": case["slice_dim"],
                    "slice_idx": case["slice_idx"],
                    "label": target_label,
                    "image_size_2d": case["image_size_2d"],
                    "pixel_size": case["pixel_size"],
                    "image_size_3d": case["image_size_3d"],
                    "voxel_size": case["voxel_size"],
                    "biometric_profile": case["biometric_profile"],
                }

Dataset Building Workflow

Workflow

MedVision Dataset Building Workflow


There are a few venues to control the dataset loading and building behavior:

  • Rebuild Dataset (Arrow files): Use the download_mode argument in load_dataset() (docs).
    • [1] Set download_mode="force_redownload" to ignore the cached Arrow files and trigger the data loading script MedVision.py to rebuild the dataset.
  • Redownload Raw Data:
    • [2] MedVision_FORCE_DOWNLOAD_DATA: Set this environment variable to True to force re-downloading raw images and annotations.
    • [3] .downloaded_datasets.json: This tracker file records downloaded status. Removing a dataset's entry here will trigger a re-download of the raw data for that dataset.

[!Note] ⚠️ How to properly update/redownload raw data?

If you need to update raw data (images, masks, landmarks) using [2] or [3], you MUST ALSO use [1] (download_mode="force_redownload").

Why? Because if Hugging Face finds a valid cached dataset (Arrow files), it will load it directly and skip running the script entirely. Without running the script, the environment variable [2] or tracker file [3] will never be checked.

Summary:

  • Update Arrow/Fields only: Use [1].
  • Update Raw Data: Use [1] AND ([2] or [3]).

[!Important] When would I need this? Normally never — v1.2.0 keys the Arrow cache on the annotation version actually loaded, so a cache is invalidated whenever the annotations behind it change.

There is one historical exception. Versions before v1.2.0 keyed the cache on the version you requested, and the v1.1.1 release re-aligned the already-published v1.1.0 T/L train/test split in place without a version bump. A cache built for a Tumor-Lesion-Size config at MedVision_PLANNER_VERSION=1.1.0 before that release still holds the old partition, at both the Arrow and annotation-file layers. Clear it with [1] AND [2] once. See Fixed: cached data could be stale.

🔥 We will maintain a change log for essential updates.

Examples

Run this for the first time will download the raw data and build the dataset 
import os
from datasets import load_dataset

# Set data folder
wd = os.path.join(os.getcwd(), "Data-testing")
os.makedirs(wd, exist_ok=True)
os.environ["MedVision_DATA_DIR"] = wd

# Pick a dataset config name and split
config = "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
    )
Run the same script again will use the cached dataset 
import os
from datasets import load_dataset

# Set data folder
wd = os.path.join(os.getcwd(), "Data-testing")
os.makedirs(wd, exist_ok=True)
os.environ["MedVision_DATA_DIR"] = wd

# Pick a dataset config name and split
config = "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
    )
Adding `download_mode="force_redownload"` will skip raw data downloading and rebuild the dataset 
import os
from datasets import load_dataset

# Set data folder
wd = os.path.join(os.getcwd(), "Data-testing")
os.makedirs(wd, exist_ok=True)
os.environ["MedVision_DATA_DIR"] = wd

# Pick a dataset config name and split
config = "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
        download_mode="force_redownload",
    )
Adding `download_mode="force_redownload"` and `os.environ["MedVision_FORCE_DOWNLOAD_DATA"] = "True"` will redownload raw data and rebuild the dataset 
import os
from datasets import load_dataset

# Set data folder
wd = os.path.join(os.getcwd(), "Data-testing")
os.makedirs(wd, exist_ok=True)
os.environ["MedVision_DATA_DIR"] = wd

# Pick a dataset config name and split
config = "OAIZIB-CM_BoxSize_Task01_Axial_Test"
split_name = "test" # use "test" for testing set config; use "train" for training set config 

# Force redownload
os.environ["MedVision_FORCE_DOWNLOAD_DATA"] = "True"

# Get dataset
ds = load_dataset(
        "YongchengYAO/MedVision",
        name=config,
        trust_remote_code=True,
        split=split_name,
        download_mode="force_redownload",
    )

Download Mode in MedVision Dataset

(Advanced) Understand how the customized dataset loading script `MedVision.py` changes the behavior of `download_mode` in `load_dataset()`

  • download_mode can be one of these: "reuse_dataset_if_exists" (default), "reuse_cache_if_exists", "force_redownload"

  • Default behavior of download_mode in load_dataset():

    DownloadsDataset
    reuse_dataset_if_exists (default)ReuseReuse
    reuse_cache_if_existsReuseFresh
    force_redownloadFreshFresh

  • download_mode in MedVision dataset:

    DownloadsDataset
    reuse_dataset_if_exists (default)ReuseReuse
    reuse_cache_if_existsReuseFresh
    force_redownload (MedVision_FORCE_DOWNLOAD_DATA=False)ReuseFresh
    force_redownload (MedVision_FORCE_DOWNLOAD_DATA=True)FreshFresh

Advanced Usage

The dataset codebase medvision_ds can be used to scale the dataset, including adding new annotation types and datasets.

🛠️ Install

pip install "git+https://huggingface.co/datasets/YongchengYAO/MedVision.git#subdirectory=src"
pip show medvision_ds

or

# First, install the benchmark codebase: medvision_bm
pip install "git+https://github.com/YongchengYAO/MedVision.git" 
 
# Install the dataset codebase: medvision_ds
python -m medvision_bm.benchmark.install_medvision_ds --data_dir <local-data-folder>

🧑🏻‍💻 Use utility functions for image processing

from medvision_ds.utils.data_conversion import (
    convert_nrrd_to_nifti,
    convert_mha_to_nifti,
    convert_nii_to_niigz,
    convert_bmp_to_niigz,
    copy_img_header_to_mask,
    reorient_niigz_RASplus_batch_inplace,
)

from medvision_ds.utils.preprocess_utils import (
    split_4d_nifti,
)

👩🏼‍💻Examples of dataset scaling:


License: CC-BY-4.0

MedVision is released under the Creative Commons Attribution 4.0 International (CC-BY 4.0) license. Users are permitted to utilize, adapt, and build upon this dataset for both academic and commercial purposes, provided that appropriate credit is given. MedVision is a meta-dataset built upon various publicly available source datasets. While the annotations provided by MedVision are covered by the CC-BY 4.0 license, any downstream application must continue to comply with the specific usage terms and licensing requirements stipulated by the curators of the original raw imaging data. It is the responsibility of the user to ensure that their application of this data aligns with the license agreements of all constituent source datasets.

Acknowledgement

This work was supported by the United Kingdom Research and Innovation (grant EP/S02431X/1), UKRI Centre for Doctoral Training in Biomedical AI at the University of Edinburgh, School of Informatics.

Similar repos

alexanderdann/CTSpine1K

Dataset

14

shared topics

huggingface/CADS-dataset

Dataset

4

shared topics

YongchengYAO/MedVision

The official codebase of "MedVision: Benchmarking Quantitative Medical Image Analysis"

15

General-Medical-AI/GMAI-VL-5.5M

Dataset

6

FreedomIntelligence/Med-MAT

Dataset

18

foreverbeliever/OmniMedVQA

Dataset

21

openmedlab/Awesome-Medical-Dataset

Collection of awesome medical dataset resources.

2,118

GoodBaiBai88/M3D-Seg

Dataset

12

Linked in READMEs

FreedomIntelligence/Awesome-AI4Med

A curated list of medical LLMs, multimodal systems, datasets, benchmarks, and more. 🏥

2,877

README badge

YongchengYAO/MedVision

The official codebase of "MedVision: Benchmarking Quantitative Medical Image Analysis"

15

README badge

Contributors

YongchengYAO

98 commits

CO
Claude Opus 4.8

5 commits

CF
Claude Fable 5

5 commits

CO
Claude Opus 5 (1M context)

3 commits