1
stars
45
commits
1
linked in READMEs
Jun 7, 2026
updated
This repository contains the dataset released with:
IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation
Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Ronald Clark, Ming-Hsuan Yang
The IllumiCraft dataset is designed for controllable video generation under varying geometry and illumination conditions. Each training sample contains multiple modalities used by IllumiCraft, including foreground appearance videos, background videos, tracking videos, lighting videos, and paired text descriptions.
The IllumiCraft dataset provides paired geometry, appearance, and illumination control signals for video generation and can support a wide range of research directions, including:
We hope the dataset will facilitate future research on controllable, physically plausible, and temporally consistent video generation.
Download the dataset using:
python utils/download_illumicraft_dataset.py
The dataset will be organized as:
dataset/
βββ train/
βββ demo_examples/
Example inputs for training are provided in:
dataset/train/
βββ foreground_videos/
βββ background_videos/
βββ tracking_videos/
βββ lighting_videos/
βββ videos/
βββ prompt.txt
βββ videos.txt
βββ foreground_videos.txt
βββ background_videos.txt
βββ tracking_videos.txt
βββ lighting_videos.txt
Demo spotlight examples for inference are provided in:
dataset/demo_examples/
βββ background_images/
βββ foreground_videos/
βββ background_images.txt
βββ foreground_videos.txt
βββ light.txt
βββ prompt.txt
Please refer to the official GitHub repository for installation, dataset preparation, training, and inference instructions:
https://github.com/yuanze-lin/IllumiCraft
If you find IllumiCraft useful for your research, please consider citing:
@article{lin2026illumicraft,
title={IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation},
author={Lin, Yuanze and Chen, Yi-Wen and Tsai, Yi-Hsuan and Clark, Ronald and Yang, Ming-Hsuan},
journal={Advances in Neural Information Processing Systems},
volume={38},
pages={27798--27829},
year={2026}
}
45 commits
1
stars
45
commits
1
linked in READMEs
Jun 7, 2026
updated
This repository contains the dataset released with:
IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation
Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Ronald Clark, Ming-Hsuan Yang
The IllumiCraft dataset is designed for controllable video generation under varying geometry and illumination conditions. Each training sample contains multiple modalities used by IllumiCraft, including foreground appearance videos, background videos, tracking videos, lighting videos, and paired text descriptions.
The IllumiCraft dataset provides paired geometry, appearance, and illumination control signals for video generation and can support a wide range of research directions, including:
We hope the dataset will facilitate future research on controllable, physically plausible, and temporally consistent video generation.
Download the dataset using:
python utils/download_illumicraft_dataset.py
The dataset will be organized as:
dataset/
βββ train/
βββ demo_examples/
Example inputs for training are provided in:
dataset/train/
βββ foreground_videos/
βββ background_videos/
βββ tracking_videos/
βββ lighting_videos/
βββ videos/
βββ prompt.txt
βββ videos.txt
βββ foreground_videos.txt
βββ background_videos.txt
βββ tracking_videos.txt
βββ lighting_videos.txt
Demo spotlight examples for inference are provided in:
dataset/demo_examples/
βββ background_images/
βββ foreground_videos/
βββ background_images.txt
βββ foreground_videos.txt
βββ light.txt
βββ prompt.txt
Please refer to the official GitHub repository for installation, dataset preparation, training, and inference instructions:
https://github.com/yuanze-lin/IllumiCraft
If you find IllumiCraft useful for your research, please consider citing:
@article{lin2026illumicraft,
title={IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation},
author={Lin, Yuanze and Chen, Yi-Wen and Tsai, Yi-Hsuan and Clark, Ronald and Yang, Ming-Hsuan},
journal={Advances in Neural Information Processing Systems},
volume={38},
pages={27798--27829},
year={2026}
}
45 commits