Welcome!

This repository contains tutorial materials (for the most part, as Python notebooks) that are developed to help you learn about NCI Imaging Data Commons and utilize it in your work.

If this is the first time you hear about IDC, you may want to check out our Getting Started documentation page. Here are some highlights about what IDC has to offer:

• ~100 TB of data: IDC contains radiology, brightfield (H&E) and fluorescence slide microscopy images, along with image-derived data (annotations, segmentations, quantitative measurements) and accompanying clinical data

• free: all of the data in IDC is publicly available: no registration, no access requests

• commercial-friendly: >95% of the data in IDC is covered by the permissive CC-BY license, which allows commercial reuse (small subset of data is covered by the CC-NC license); each file in IDC is tagged with the license to make it easier for you to understand and follow the rules

• cloud-based: all of the data in IDC is available from both Google and AWS public buckets: fast and free to download, no out-of-cloud egress fees

• harmonized: all of the images and image-derived data in IDC is harmonized into standard DICOM representation

The tutorial notebooks are located in the notebooks, and are organized in the following folders.

Zenodo permanent citeable record: https://zenodo.org/records/22691908.

Recommended Learning Paths

Beginner (no GCP account needed):

1. Basics of using IDC data programmatically - start here!
2. Searching clinical data

Intermediate (GCP account needed):

3. BigQuery prerequisites and setup - set up Google Cloud for BigQuery access
4. Advanced searching using BigQuery
5. Working with IDC clinical data

Domain-specific paths:

• Radiology: NLST exploration → TotalSegmentator features → MedSAM on IDC
• Digital Pathology: Getting started with digital pathology → Slide microscopy metadata → DICOM annotations
• Cloud Access: Direct GCS/AWS access → DICOMweb access → IDC on AWS

getting_started

"Getting Started" python notebooks are intended to introduce the users to IDC.

• Basics of using IDC data programmatically: learn how to use idc-index python package to programmatically search and download IDC data, visualize images and annotations (including interactive 3D rendering of an image with its segmentation overlay directly in the notebook using ipyniivue), build cohorts and checking acknowledgments and licenses for the data included in your cohort.
• Searching clinical data: identify clinical and other non-imaging data accompanying imaging collections in IDC using idc-index python package and duckdb.
• BigQuery prerequisites and setup: set up the prerequisites for working with IDC data via Google BigQuery. This notebook is only needed if you plan to use BigQuery for advanced metadata queries (Part 3 and beyond).
• Advanced searching using BigQuery: access all of the metadata to build comprehensive queries and detailed cohort selection criteria.

advanced_topics

Notebooks in this folder focus on topics that will require understanding of the basics, and aim to address more narrow use cases of IDC usage.

• Searching DICOM private tags: all of DICOM attributes for the imaging data in IDC are searchable using BigQuery. DICOM private tags often contain critical information, such as diffusion b-values, but are a bit more tricky to access from BigQuery. In this tutorial you will learn how to accomplish this.
• Working with IDC clinical data: this tutorial will help you become familiar with how clinical and other tabular data accompanying images in IDC is organized, and how you can use it to build cohorts.
• Direct access to tags/frames from GCS/AWS buckets: learn how to access individual frames or tags of large DICOM files from the bucket without having to download the entire file (this notebook accompanies documentation article here: https://learn.canceridc.dev/data/downloading-data/direct-loading)
• Using DICOMweb to access IDC data: both metadata and pixel data can be access using DICOMweb, which is particularly important while working with digital pathology, as it enables granular access to the individual pyramid tiles (frames)
• IDC on AWS: learn how to work with IDC data using AWS services, including AWS HealthImaging.
• Visualizing IDC data in 3D with trame-slicer: view IDC images and segmentations interactively in 3D inside a Colab cell using trame-slicer, which brings the 3D Slicer rendering engine (VTK-grade volume rendering, multi-planar reformat, segmentation overlay) to the browser. For a lighter-weight WebGL viewer embedded directly in the notebook, see the ipyniivue section of the Getting started Part 2 tutorial.

viewers_deployment

These notebooks can be used to deploy your own cloud-based instance of OHIF or Slim viewers using Google Firebase, which you can use to visualize analysis results you generated for IDC data, or to work with your own images. These tutorials utilize free tier of Firebase, and so there is no cost to keep the deployed viewers available in the cloud.

• OHIF Firebase deployment
• Slim Firebase deployment
• Setting up Google Healthcare DICOM store: once you have your viewers deployed, you can use this tutorial to create a DICOM store with your data, which you can then access from the viewers deployed

collections_demos

This folder contains notebooks that demonstrate the usage of the data in the specific IDC collections. The notebooks in this folder will always have the prefix of the collection_id they correspond to, for easier navigation.

• Using hiplot for exploring prostate MRI in IDC: this notebook demonstrates how hiplot, an open source package for high-dimensional parameter visualization, for examining various MRI acquisition parameters for the prostate MRI images available in IDC.
• Visible Human Project exploration: demonstration of searching and visualizing images from the National Library of Medicine Visible Human Project available on IDC.
• RMS-Mutation-Prediction collection exploration: notebooks in this folder demonstrate selecting images from the RMS-Mutation-Prediction collection based on various attributes of images and expert annotations.
• NLST collection exploration: explanation of the content included in the IDC NLST collection, and how it is different from the NLST collection you will find in TCIA.
• Working with NLST clinical data in IDC: demonstration of how to access and search clinical data tables accompanying the IDC NLST collection, and how to combine clinical data with imaging metadata.
• Working with bone marrow smears from the BoneMarrowWSI-PediatricLeukemia collection: demonstration of how to access and use the bone marrow smear WSIs and extensive expert annotations (regions-of-interest, > 40 000 cells with and without cell type information) contained in this collection.

pathomics

This folder is dedicated to the notebooks focused on the digital pathology (pathomics) applications. The use of DICOM standard is relatively new in digital pathology, and this field is being actively developed, thus a dedicated folder for this.

• Getting started with pathology images in IDC: all of the pathology images in IDC are in DICOM Slide Microscopy format; this notebook will help you get started with using this representation and also searching IDC pathology images.
• Exploring IDC slide microscopy images metadata: introduction to the key metadata accompanying IDC slide microscopy images that can be used for subsetting data and building cohorts.
• Working with DICOM Microscopy Bulk Simple annotations: introduction to working with bulk annotations encoded in DICOM using the example of nuclei annotations.
• Working with DICOM Segmentations: introduction to working with segmentations encoded in DICOM using the example of nuclei segmentations.

analysis

Demonstrations/examples of analyses of images from IDC.

• MedSAM on IDC: learn how to experiment with MedSAM on the images available from IDC.
• MHub.ai with IDC data: MHub.ai is a platform for Deep Learning models in medical imaging, which are interoperable with IDC and can be applied directly to the IDC DICOM images. Learn how to get started from this notebook!

idc_api

Notebooks demonstrating the IDC REST API.

• How to use the IDC V2 API: introduction to the IDC REST API v2 for programmatic access to IDC data, including cohort creation, manifest generation, and data download.

labs

Here you will find an archive of the notebooks that were used in tutorials, which at times may demonstrate experimental features. By design, the notebooks presented at specific events may not be updated after the event, and are stored in this folder for archival purposes.

• Running the full 3D Slicer desktop in Colab with IDC data (⚠️ experimental, not actively supported): launch the complete 3D Slicer desktop application headlessly and stream its live GUI into a Colab cell using the experimental desktopia project, loading an IDC DICOM image and DICOM SEG natively. This is a developer/community proof-of-concept that depends on fast-moving, unversioned components and may break without notice — for a robust, supported in-notebook viewer use the ipyniivue section of Getting started Part 2.

deprecated

IDC is an actively evolving resource. As we develop new and improved capabilities, we improve our recommended usage practices, and may deprecate notebooks that are no longer maintained and may no longer work. You will find these in the deprecated folder.

testing

This directory is used for the maintenance of the repository to support testing of the actively supported notebooks.

Support

If you have any questions about the notebooks in this repository, please open a discussion thread in IDC user forum, or open the issue in this repository.