IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation (NeurIPS 2025)

Official implementation of "IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation"

Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Ronald Clark, Ming-Hsuan Yang

💡 Method

📣 News

• Release the training code.
• Release IllumiCraft dataset.
• Release the model and the inference code.
• Set up the project page.

❤️ Support IllumiCraft

If you find this repository useful, please consider giving it a star ⭐.

🚀 Installation

git clone https://github.com/yuanze-lin/IllumiCraft.git
cd IllumiCraft

conda create -n illumicraft python=3.10 -y
conda activate illumicraft
pip install torch==2.6.0+cu118 torchvision==0.21.0+cu118 torchaudio==2.6.0+cu118 --index-url https://download.pytorch.org/whl/cu118
conda env update -n illumicraft -f environment.yml

# For automatic foreground extraction (SAM3 + MatAnyone); matanyone isn't on PyPI:
pip install --no-deps -e 'git+https://github.com/pq-yang/MatAnyone#egg=matanyone'

📂 Dataset Preparation

Download the IllumiCraft training dataset and demo examples:

python utils/download_illumicraft_dataset.py

The script will automatically download the dataset from Hugging Face and organize it into two parts, train and demo_examples, for training and inference, respectively:

dataset/
├── train/
└── demo_examples/

The training dataset will be stored in:

dataset/train/
├── foreground_videos/
├── background_videos/
├── tracking_videos/
├── lighting_videos/
├── videos/
├── prompt.txt
├── videos.txt
├── foreground_videos.txt
├── background_videos.txt
├── tracking_videos.txt
└── lighting_videos.txt

Use dataset/train/ as the DATA_ROOT in train.sh and dataset/demo_examples/ as the DATA_ROOT in inference.sh.

📥 Download Pretrained Weights

Before running training and inference, download both the base Wan2.1-Fun-1.3B-Control model and the released IllumiCraft checkpoint:

python utils/download_illumicraft_weights.py

The script will automatically download the checkpoints to:

checkpoints/
├── Wan2.1-Fun-1.3B-Control/
└── illumicraft_pretrained_weights/

After downloading, verify that the model paths in inference.sh are correctly configured:

WAN_MODEL_PATH="checkpoints/Wan2.1-Fun-1.3B-Control"
ILLUMICRAFT_CKPT_PATH="checkpoints/illumicraft_pretrained_weights"

WAN_MODEL_PATH points to the base Wan2.1 model and is shared by both train.sh and inference.sh. ILLUMICRAFT_CKPT_PATH points to the pretrained IllumiCraft checkpoint used during inference.

🏋️ Training

Edit the following fields in train.sh:

DATA_ROOT=/path/to/train_dataset
WAN_MODEL_PATH=/path/to/Wan2.1-Fun-1.3B-Control

DATA_ROOT=/path/to/train

Launch training:

bash train.sh

🎥 Inference

IllumiCraft supports both dataset-style inference and single-sample inference.

Both accept either an already-prepared gray-background foreground video or a raw RGB video that is converted into one on the fly — via SAM3 (text-prompted segmentation of the first frame) + MatAnyone (video matting), composited onto a fixed gray background (implemented in utils/prepare_foreground_video.py). To use the raw-video path, first download the SAM3 checkpoint into checkpoints/sam3 (facebook/sam3 on Hugging Face — access may need to be requested); MatAnyone's weights download automatically on first use.

Dataset-style inference

Run video generation using a trained IllumiCraft checkpoint.

Edit the following fields in inference.sh:

ILLUMICRAFT_CKPT_PATH="checkpoints/illumicraft_pretrained_weights"
DATA_ROOT=/path/to/demo_examples

Launch inference:

bash inference.sh

Inference directly from raw input videos (dataset-style)

To skip preparing foreground videos yourself, add --input_video_column (a txt file of raw input video paths, parallel to --foreground_column). For any row whose --foreground_column entry is missing or doesn't resolve to an existing file, the foreground video is auto-generated from the corresponding raw input video via SAM3 + MatAnyone and cached under <DATA_ROOT>/generated_foreground_videos/:

python testing/inference.py \
    --data_root $DATA_ROOT \
    --config_path config/wan.yaml \
    --model_path $ILLUMICRAFT_CKPT_PATH \
    --caption_column $CAPTION_COLUMN \
    --lighting_caption_column $LIGHT_CAPTION_COLUMN \
    --input_video_column input_videos.txt \
    --background_column $BACKGROUND_COLUMN \
    --output_path $OUTPUT_PATH

Single-sample inference

For quick testing, IllumiCraft also supports direct inference on a single foreground video without requiring dataset text files.

Launch:

bash inference_single_sample.sh

Example configuration:

#!/bin/bash
export CUDA_VISIBLE_DEVICES=0

ILLUMICRAFT_CKPT_PATH="checkpoints/illumicraft_pretrained_weights"
OUTPUT_PATH="demo/single_sample_outputs"

FOREGROUND_VIDEO_PATH="demo/eval/foreground_videos_00000.mp4"
FOREGROUND_PROMPT="A majestic waterfall cascades down a rugged cliff into a serene pool."
LIGHTING_PROMPT="Cool-blue spotlights beam through mist onto a central pool of light, creating high-contrast cinematic depth and a moody, immersive atmosphere."

# Optional background-conditioned generation
BACKGROUND_PATH=""

python testing/inference_single_sample.py \
    --config_path config/wan.yaml \
    --model_path "$ILLUMICRAFT_CKPT_PATH" \
    --foreground_video_path "$FOREGROUND_VIDEO_PATH" \
    --foreground_prompt "$FOREGROUND_PROMPT" \
    --lighting_prompt "$LIGHTING_PROMPT" \
    --output_path "$OUTPUT_PATH" \
    ${BACKGROUND_PATH:+--background_path "$BACKGROUND_PATH"}

Inference directly from a raw input video

If you don't already have a prepared (gray-background) foreground video, pass a raw input video via --input_video_path instead of --foreground_video_path. The foreground video is then extracted automatically with SAM3 (text-prompted segmentation on the first frame, using --foreground_prompt as the text prompt) + MatAnyone (video matting), and composited onto the same fixed gray background used elsewhere in the pipeline:

bash inference_single_sample_from_original_video.sh

This example config auto-generates its foreground video from demo/eval/00000.mp4 (with demo/eval/custom_background.jpg as the background image) — the same raw clip used elsewhere in demo/eval/.

Example configuration:

#!/bin/bash
export CUDA_VISIBLE_DEVICES=7

ILLUMICRAFT_CKPT_PATH="checkpoints/illumicraft_pretrained_weights"
OUTPUT_PATH="demo/single_sample_from_video_outputs"

# Raw input video with a real (non-gray) background -- the foreground video is
# auto-extracted from this via SAM3 (text-prompted segmentation) + MatAnyone
# (video matting), instead of requiring an already-prepared foreground video.
INPUT_VIDEO_PATH="demo/eval/00000.mp4"
FOREGROUND_PROMPT="A majestic waterfall cascades down a rugged cliff into a serene pool."
LIGHTING_PROMPT="Cool-blue spotlights beam through mist onto a central pool of light, creating high-contrast cinematic depth and a moody, immersive atmosphere."

# Optional background-conditioned generation
BACKGROUND_PATH="demo/eval/custom_background.jpg"

python testing/inference_single_sample.py \
    --config_path config/wan.yaml \
    --model_path "$ILLUMICRAFT_CKPT_PATH" \
    --input_video_path "$INPUT_VIDEO_PATH" \
    --foreground_prompt "$FOREGROUND_PROMPT" \
    --lighting_prompt "$LIGHTING_PROMPT" \
    --output_path "$OUTPUT_PATH" \
    ${BACKGROUND_PATH:+--background_path "$BACKGROUND_PATH"}

Outputs

When a background image and lighting prompt are provided, IllumiCraft generates background-conditioned results:

sample_bg.mp4
sample_bg_concat.mp4       # foreground video | background | generated video
                           # (input video instead of foreground video, if it was used to auto-generate one)

For comparison, it also generates results without background conditioning:

sample_nobg.mp4
sample_nobg_concat.mp4     # foreground video | generated video
                           # (input video instead of foreground video, if it was used to auto-generate one)

🖥️ Gradio Demo

IllumiCraft also provides an interactive Gradio demo for relighting custom videos.

Before launching the demo, set the model paths in run_gradio_demo.sh:

ILLUMICRAFT_CKPT_PATH="checkpoints/illumicraft_pretrained_weights"
WAN_MODEL_PATH="${WAN_MODEL_PATH:-$ILLUMICRAFT_CKPT_PATH}"

# WAN_MODEL_PATH="checkpoints/Wan2.1-Fun-1.3B-Control"  # Set if not using our pretrained weights.

Then launch:

bash run_gradio_demo.sh

The demo runs the full pipeline end-to-end: upload a raw input video, click Generate foreground video to auto-extract the gray-background foreground (SAM3 + MatAnyone, see Foreground extraction), then click Generate relit video to relight it with your foreground/lighting prompts and an optional background image. Preloaded examples from demo/eval/ can be used directly or freely replaced with your own inputs. The IllumiCraft Gradio interface for video relighting is shown below:

🎬 Sample Results

❓ FAQ

Q: Why do you use background videos during training but background images during inference?

During training, we only use the first frame of each background video. Therefore, a background image is sufficient during inference. If you have a background image, you can simply repeat it to create a background video with the same length as the input foreground video.

We originally used background videos in the dataset for training because we also explored background-video-conditioned video generation.

Q: Why does inference use both foreground_prompt.txt and lighting_prompt.txt?

🏋️ Training

• prompt.txt describes the entire video, including both foreground and background content.

🎥 Inference

• foreground_prompt.txt describes the foreground object and its appearance.
• lighting_prompt.txt describes the background scene and lighting conditions associated with the selected background image.

Since the background images used during inference are independently collected and can be freely replaced with custom images, they are not paired with the foreground videos. Therefore, lighting_prompt.txt is used to provide scene and illumination information that is not contained in foreground_prompt.txt.

Note: For paired data (e.g., formal evaluation), where the foreground, background, caption, and ground-truth video correspond to the same scene, a single caption describing the entire scene can be stored in prompt.txt.

For arbitrary background image customization, we recommend using foreground_prompt.txt to describe the foreground and lighting_prompt.txt to describe the background scene and lighting conditions.

📚 Citation

If you find IllumiCraft useful for your research, please consider citing:

@article{lin2026illumicraft,
  title={Illumicraft: Unified geometry and illumination diffusion for controllable video generation},
  author={Lin, Yuanze and Chen, Yi-Wen and Tsai, Yi-Hsuan and Clark, Ronald and Yang, Ming-Hsuan},
  journal={Advances in Neural Information Processing Systems},
  volume={38},
  pages={27798--27829},
  year={2026}
}

🙏 Acknowledgement

• Wan2.1: IllumiCraft is built upon the Wan2.1 framework and uses Wan2.1-Fun-1.3B-Control as its foundation. We thank the Wan team for open-sourcing their codebase and pretrained models.