NV-Generate-CTMR

3D Latent Diffusion Models (LDM) for generating large CT and MRI images with corresponding segmentation masks. Supports variable volume size and voxel spacing with precise control of organ/tumor size.

Please cite all the following papers if you are using code or model from this repo.

• MAISI-v1 Paper (WACV 2025)
• MAISI-v2 Paper (AAAI 2026)

Overview

NV-Generate-CTMR generates high-resolution synthetic 3D medical volumes using latent diffusion models built on the MAISI (Medical AI for Synthetic Imaging) framework. It produces CT images with paired segmentation masks and MRI volumes across multiple contrasts — enabling synthetic training data generation, data augmentation for rare pathologies, and privacy-preserving data sharing.

Key capabilities:

• CT generation with paired 132-class segmentation masks, supporting volumes up to 512x512x768 voxels with controllable organ and tumor size
• MRI generation across T1, T2, FLAIR, and additional contrasts for brain, abdomen, breast, and prostate anatomy
• Brain MRI synthesis with cross-sequence ControlNet for generating matched multi-contrast brain volumes (T1w, T2w, FLAIR, SWI)
• Variable resolution with configurable volume size and voxel spacing for each generation

Live Demo (no GPU required)

News

• [October 2026] — Released NV-Generate-Body, which generates body mask with controllable anatomy, tumor, and demographic information.
• [August 2026] — Updated NV-Generate-MR-Brain model weights from v0 to v1. Code usage is unchanged.
• [May 2026] — Added operator-focused inference skills — ask an AI coding agent (Claude Code, Cursor, Codex, etc.) to use the matching skill and it will run the inference workflow end-to-end.
• 🎆 March 2026 🎇 — Released NV-Generate-MR-Brain v0 models rflow-mr-brain for fast high-resolution 3D MR brain image generation, which covers both whole brain and skull-stripped brain generation for T1w, T2w, FLAIR, SWI images. The training data of this version v0 is MR-RATE.
• [October 2025] — Released rectified flow models rflow-mr for fast high-resolution 3D MR image generation. Upgraded previous MAISI repo to this NV-Generate-CTMR repo.
• [March 2025] — Released rectified flow models rflow-ct for fast high-resolution 3D CT image generation and paired CT image/mask synthesis. rflow-ct is 33x faster than ddpm-ct and generates better quality images for the head region and small output volumes.
• [August 2024] — Initial release ddpm-ct supporting 3D latent diffusion (DDPM) for CT image generation and paired CT image/mask synthesis.

Table of Contents

• Overview
• News
• 1. Model Variants
• 2. Quick Start
  • 2.1 Installation
  • 2.2 MR Brain Image Generation
  • 2.3 CT Paired Image/Mask Generation
  • 2.4 Body Mask Generation
  • 2.5 CT Image Generation
  • 2.6 MR Image Generation
  • 2.7 CT Image Generation from Your Own Mask
  • 2.8 Example Applications (Community)
• 3. Documentation: details of data preparation, training, and inference tutorials
• 4. Performance: accuracy, speed, and GPU memory usage
• 5. License
• 6. Citation
• 7. Resources
• 8. Acknowledgements

1. Model Variants

This repository provides five model variants for medical image generation: rflow-mr-brain, rflow-mr, rflow-ct, ddpm-ct, and rflow-mask.

Summary: Use rflow-ct for CT image/mask pairs (whole-body). Use rflow-mask for CT body mask with controllable anatomy size, tumor, and demographics. Use rflow-mr-brain for brain MRI (multi-contrast). Use rflow-mr for other MRI anatomies (fine-tune on your own data).

Detailed comparison

2. Quick Start (requires at least a 16G GPU)

💡 Using an AI coding agent (Claude Code, Cursor, Codex, etc.)? Each subsection below links to a per-workflow skill in skills/. Ask the agent to read this README then use the corresponding skill file — it will run the matching workflow end-to-end, including model download, config edits, and the inference command.

Example prompt — copy into your agent:

Read the README in this repo and use the matching skill to generate a T1 whole-brain MR image. Please first summarize what you plan to do, and let me know where the output will be saved.

⚠️ Run inference manually? dim and spacing are the single biggest factor in output quality — their product dim × spacing defines the field of view (FOV). Each model variant has only ever seen FOVs in the training-data distribution for its target anatomy; asking it to synthesize at a numerically-valid but out-of-distribution FOV (e.g. a 128 mm-cube whole-body CT) produces unusable output. Start from the recommended (dim, spacing) per anatomy:

• CT (rflow-ct, ddpm-ct): docs/inference.md#recommended-spacing-for-ct
• MR (non-brain) (rflow-mr): docs/inference.md#recommended-fov-for-mr-rflow-mr-model
• MR Brain (rflow-mr-brain): docs/inference.md#recommended-fov-for-mr-rflow-mr-brain-model

2.1 Installation

pip install -r requirements.txt

2.2 MR Brain Image Generation

Skill: infer_image-only — feed this file to an AI coding agent to run the workflow below end-to-end.

Please refer to inference_diff_unet_tutorial.ipynb for the inference tutorial that generates CT or MR image without mask.

You can also run it in command line to generate MR image without mask. Please change "modality" in configs/config_maisi_diff_model_rflow-mr-brain.json according to configs/modality_mapping.json to control the output MR contrast. Currently we support both whole brain and skull-stripped brain generation for T1w, T2w, FLAIR, SWI images.

"mri":8, # MRI without specifying contrast or skull condition, can be any of them
"mri_t1":9, # T1w whole-brain MRI
"mri_t2":10, # T2w whole-brain MRI
"mri_flair":11, # FLAIR whole-brain MRI
"mri_swi":20, # SWI whole-brain MRI
"mri_t1_skull_stripped":29, # T1w skull-stripped brain MRI
"mri_t2_skull_stripped":30, # T2w skull-stripped brain MRI
"mri_flair_skull_stripped":31, # FLAIR skull-stripped brain MRI
"mri_swi_skull_stripped":32, # SWI skull-stripped brain MRI

export MONAI_DATA_DIRECTORY="./temp_work_dir"
network="rflow"
generate_version="rflow-mr-brain"
python -m scripts.diff_model_infer \
    -t ./configs/config_network_${network}.json \
    -e ./configs/environment_maisi_diff_model_${generate_version}.json \
    -c ./configs/config_maisi_diff_model_${generate_version}.json \
    --version ${generate_version}

2.3 CT Paired Image/Mask Generation

Skill: infer_mask-image-paired — feed this file to an AI coding agent to run the workflow below end-to-end.

export MONAI_DATA_DIRECTORY="./temp_work_dir"
network="rflow"
generate_version="rflow-ct" # can change to "ddpm-ct"
python -m scripts.inference -t ./configs/config_network_${network}.json -i ./configs/config_infer.json -e ./configs/environment_${generate_version}.json --random-seed 0 --version ${generate_version}

See also: inference_tutorial.ipynb

2.4 Body Mask Generation

Skill: infer_mask-only — feed this file to an AI coding agent to run the workflow below end-to-end.

Generate a 3D CT body mask using the v2 mask diffusion model (rflow-mask). The mask always contains all 132 MAISI labels. Control anatomy size, tumors, and patient demographics via controllable_anatomy_size and controllable_demographics (Path A), or retrieve a training mask matching body_region + anatomy_list (Path B).

# Set MONAI_DATA_DIRECTORY — weights and datasets (~10 GB) are downloaded here automatically on first run.
export MONAI_DATA_DIRECTORY="./temp_work_dir"

# Edit config_infer.json: set controllable_anatomy_size / controllable_demographics for Path A,
# or body_region / anatomy_list for Path B (leave controllable_* empty).
python -m scripts.inference \
    -t ./configs/config_network_rflow.json \
    -i ./configs/config_infer.json \
    -e ./configs/environment_rflow-ct.json \
    --random-seed 0 --version rflow-ct

The paired image is also generated (mask stage + image stage cannot be separated in the current CLI). The mask is saved as sample_<timestamp>_label.nii.gz.

Path A example — control bone-lesion size and patient demographics:

{
  "controllable_anatomy_size": [["bone lesion", 0.5]],
  "controllable_demographics": [["age", 55], ["sex", "M"], ["bmi", 24.5]],
  "output_size": [256, 256, 256],
  "spacing": [1.5, 1.5, 2.0]
}

⚠️ controllable_anatomy_size accepts conditioning slot names ("liver", "spleen", "lung", "bone lesion", …) — not anatomy_list organ names. "lung" is a valid conditioning name (slot 7) but is not valid in anatomy_list. See ANATOMY_SIZE_IDX in scripts/sample_mask.py for all 14 valid names.

2.5 CT Image Generation

Skill: infer_image-only — feed this file to an AI coding agent to run the workflow below end-to-end.

export MONAI_DATA_DIRECTORY="./temp_work_dir"
network="rflow"
generate_version="rflow-ct" # can change to "ddpm-ct"
python -m scripts.diff_model_infer \
    -t ./configs/config_network_${network}.json \
    -e ./configs/environment_maisi_diff_model_${generate_version}.json \
    -c ./configs/config_maisi_diff_model_${generate_version}.json \
    --version ${generate_version}

2.6 MR Image Generation

Skill: infer_image-only — feed this file to an AI coding agent to run the workflow below end-to-end.

Change "modality" in configs/config_maisi_diff_model_rflow-mr.json according to configs/modality_mapping.json to control the output MR contrast. Supported contrasts: T1/T2 brain, FLAIR skull-stripped brain, T2 prostate, T1 breast, T1/T2 abdomen. But if you are going to synthesize brain images, we recommend using rflow-mr-brain model instead. Please see 2.2 MR Brain Image Generation. Different body region has different recommended FOV, please see detailed inference guide.

export MONAI_DATA_DIRECTORY="./temp_work_dir"
network="rflow"
generate_version="rflow-mr"
python -m scripts.diff_model_infer \
    -t ./configs/config_network_${network}.json \
    -e ./configs/environment_maisi_diff_model_${generate_version}.json \
    -c ./configs/config_maisi_diff_model_${generate_version}.json \
    --version ${generate_version}

2.7 CT Image Generation from Your Own Mask

Skill: infer_image-from-mask — feed this file to an AI coding agent to run the workflow below end-to-end (including mask preprocessing).

If you already have a 3D label mask in the MAISI 132-class vocabulary with the body envelope (label 200) added, you can feed it directly to the CT ControlNet to synthesize a paired CT image — no mask diffusion step needed:

network="rflow"
generate_version="rflow-ct" # can change to "ddpm-ct"
python -m scripts.download_model_data --version ${generate_version} --root_dir "./"
python -m scripts.infer_image_from_mask \
  -t ./configs/config_network_${network}.json \
  -i ./configs/config_infer.json \
  -e ./configs/environment_${generate_version}.json \
  --mask /path/to/your_mask.nii.gz

⚠️ The mask must be in the MAISI 132-class label vocabulary AND include the body envelope (label 200). In concrete terms, the MAISI 132-class vocabulary is the same as the nv-segment-ct output label definition plus the body envelope (label 200). The authoritative reference is configs/label_dict.json. Two practical ways to produce a valid mask:

• From nv-segment-ct (recommended — already in MAISI vocabulary): run nv-segment-ct on the CT, then add the body envelope via scripts.utils.add_body_envelope (label 200 is never emitted by nv-segment-ct).
• From another segmenter: remap the output labels to the MAISI 132-class IDs in configs/label_dict.json, then add the body envelope.

See the infer_image-from-mask skill for the full preprocessing chain and the complete spec of "valid mask format".

For batch generation from many masks listed in a JSON, see scripts.infer_image_from_mask_batch.

2.8 Example Applications (Community)

⚠️ Independently developed by their respective authors; not endorsed, maintained, or audited by NVIDIA or the NV-Generate-CTMR maintainers — use at your own risk and review each project's license + clinical disclaimers. See docs/applications.md for the full disclaimer.

• MR-to-CT image synthesis — reference implementation based on rflow-ct: https://github.com/brudfors/maisi-mr-to-ct.
• Educational toolkit for radiotherapy research — Jupyter notebooks plus DICOM RT-STRUCT export so synthetic CT + mask pairs are loadable in clinical tools (3D Slicer, RayStation, Eclipse): https://github.com/MustafaKadhim/MAISI-v2-Educational-for-Radiotherapy-Research.

If you've adapted NV-Generate-CTMR for other imaging tasks or applications and would like to share your work, please feel free to open an issue or contact the maintainers — we'd love to link to your repo.

3. Documentation: details of data preparation, training, and inference tutorials

Training, inference, data preparation, and evaluation details are covered in the guides linked above.

4. Performance: accuracy, speed, and GPU memory usage

On the unseen autoPET 2023 benchmark:

For inference parameters, see Documentation. For GPU memory and timing, see Performance.

5. License

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

6. Citation

@article{zhao2026maisi,
  title={MAISI-v2: Accelerated 3D high-resolution medical image synthesis with rectified flow and region-specific contrastive loss},
  author={Zhao, Can and Guo, Pengfei and Yang, Dong and Tang, Yucheng and He, Yufan and Simon, Benjamin and Belue, Mason and Harmon, Stephanie and Turkbey, Baris and Xu, Daguang},
  journal={Proceedings of the 40th AAAI Conference on Artificial Intelligence (AAAI 2026)},
  year={2026}
}

@inproceedings{guo2025maisi,
  title={MAISI: Medical AI for synthetic imaging},
  author={Guo, Pengfei and Zhao, Can and Yang, Dong and Xu, Ziyue and Nath, Vishwesh and Tang, Yucheng and Simon, Benjamin and Belue, Mason and Harmon, Stephanie and Turkbey, Baris and others},
  booktitle={2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  pages={4430--4441},
  year={2025},
  organization={IEEE}
}

7. Resources

• NV-Generate-CT on HuggingFace -- CT model weights and model card
• NV-Generate-MR on HuggingFace -- MR model weights and model card
• NV-Generate-MR-Brain on HuggingFace -- Brain MRI model weights and model card
• MR-RATE on HuggingFace -- Brain MRI model training data MR-RATE
• MAISI Live Demo -- Try online without GPU
• MAISI-v1 Paper (WACV 2025)
• MAISI-v2 Paper (AAAI 2026)
• Built with MONAI -- Medical Open Network for AI

8. Acknowledgements

This project was conducted by NVIDIA in collaboration with the University of Zurich, Istanbul Medipol University, and Forithmus.

We would like to thank the following people for their contributions to the development of the NV-Generate-MR-Brain models: Bjoern Menze, Ibrahim Ethem Hamamci, Sezgin Er, Suprosanna Shit, Utku Türkbey, etc.