YuanzeLin/IllumiCraft

Dataset

1

stars

45

commits

1

linked in READMEs

Jun 7, 2026

updated

computer-vision
controllable-video-generation
diffusion-models
relighting
video-generation
video-relighting
Browse cluster: Video Generation with Diffusion Models β†’

README

IllumiCraft Dataset

This repository contains the dataset released with:

IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation

Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Ronald Clark, Ming-Hsuan Yang

✨ Overview

The IllumiCraft dataset is designed for controllable video generation under varying geometry and illumination conditions. Each training sample contains multiple modalities used by IllumiCraft, including foreground appearance videos, background videos, tracking videos, lighting videos, and paired text descriptions.

🌟 Potential Applications

The IllumiCraft dataset provides paired geometry, appearance, and illumination control signals for video generation and can support a wide range of research directions, including:

  • πŸŽ₯ Controllable video generation
  • πŸ’‘ Video relighting and illumination editing
  • 🎭 Appearance-preserving video synthesis
  • πŸ“ Geometry-aware video generation
  • 🎬 Video-to-video generation and editing
  • 🌍 Virtual cinematography and lighting design
  • πŸ€– Multimodal video foundation models
  • πŸ•ΆοΈ AR/VR content creation and scene manipulation

We hope the dataset will facilitate future research on controllable, physically plausible, and temporally consistent video generation.

πŸ“₯ Download Dataset

Download the dataset using:

python utils/download_illumicraft_dataset.py

The dataset will be organized as:

dataset/
β”œβ”€β”€ train/
└── demo_examples/

πŸ“‚ Training Dataset

Example inputs for training are provided in:

dataset/train/
β”œβ”€β”€ foreground_videos/
β”œβ”€β”€ background_videos/
β”œβ”€β”€ tracking_videos/
β”œβ”€β”€ lighting_videos/
β”œβ”€β”€ videos/
β”œβ”€β”€ prompt.txt
β”œβ”€β”€ videos.txt
β”œβ”€β”€ foreground_videos.txt
β”œβ”€β”€ background_videos.txt
β”œβ”€β”€ tracking_videos.txt
└── lighting_videos.txt

Demo spotlight examples for inference are provided in:

dataset/demo_examples/
β”œβ”€β”€ background_images/
β”œβ”€β”€ foreground_videos/
β”œβ”€β”€ background_images.txt
β”œβ”€β”€ foreground_videos.txt
β”œβ”€β”€ light.txt
└── prompt.txt

πŸš€ Usage

Please refer to the official GitHub repository for installation, dataset preparation, training, and inference instructions:

https://github.com/yuanze-lin/IllumiCraft

πŸ“– Citation

If you find IllumiCraft useful for your research, please consider citing:

@article{lin2026illumicraft,
  title={IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation},
  author={Lin, Yuanze and Chen, Yi-Wen and Tsai, Yi-Hsuan and Clark, Ronald and Yang, Ming-Hsuan},
  journal={Advances in Neural Information Processing Systems},
  volume={38},
  pages={27798--27829},
  year={2026}
}

Contributors

YuanzeLin

45 commits

YuanzeLin/IllumiCraft

Dataset

1

stars

45

commits

1

linked in READMEs

Jun 7, 2026

updated

computer-vision
controllable-video-generation
diffusion-models
relighting
video-generation
video-relighting
Browse cluster: Video Generation with Diffusion Models β†’

README

IllumiCraft Dataset

This repository contains the dataset released with:

IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation

Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Ronald Clark, Ming-Hsuan Yang

✨ Overview

The IllumiCraft dataset is designed for controllable video generation under varying geometry and illumination conditions. Each training sample contains multiple modalities used by IllumiCraft, including foreground appearance videos, background videos, tracking videos, lighting videos, and paired text descriptions.

🌟 Potential Applications

The IllumiCraft dataset provides paired geometry, appearance, and illumination control signals for video generation and can support a wide range of research directions, including:

  • πŸŽ₯ Controllable video generation
  • πŸ’‘ Video relighting and illumination editing
  • 🎭 Appearance-preserving video synthesis
  • πŸ“ Geometry-aware video generation
  • 🎬 Video-to-video generation and editing
  • 🌍 Virtual cinematography and lighting design
  • πŸ€– Multimodal video foundation models
  • πŸ•ΆοΈ AR/VR content creation and scene manipulation

We hope the dataset will facilitate future research on controllable, physically plausible, and temporally consistent video generation.

πŸ“₯ Download Dataset

Download the dataset using:

python utils/download_illumicraft_dataset.py

The dataset will be organized as:

dataset/
β”œβ”€β”€ train/
└── demo_examples/

πŸ“‚ Training Dataset

Example inputs for training are provided in:

dataset/train/
β”œβ”€β”€ foreground_videos/
β”œβ”€β”€ background_videos/
β”œβ”€β”€ tracking_videos/
β”œβ”€β”€ lighting_videos/
β”œβ”€β”€ videos/
β”œβ”€β”€ prompt.txt
β”œβ”€β”€ videos.txt
β”œβ”€β”€ foreground_videos.txt
β”œβ”€β”€ background_videos.txt
β”œβ”€β”€ tracking_videos.txt
└── lighting_videos.txt

Demo spotlight examples for inference are provided in:

dataset/demo_examples/
β”œβ”€β”€ background_images/
β”œβ”€β”€ foreground_videos/
β”œβ”€β”€ background_images.txt
β”œβ”€β”€ foreground_videos.txt
β”œβ”€β”€ light.txt
└── prompt.txt

πŸš€ Usage

Please refer to the official GitHub repository for installation, dataset preparation, training, and inference instructions:

https://github.com/yuanze-lin/IllumiCraft

πŸ“– Citation

If you find IllumiCraft useful for your research, please consider citing:

@article{lin2026illumicraft,
  title={IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation},
  author={Lin, Yuanze and Chen, Yi-Wen and Tsai, Yi-Hsuan and Clark, Ronald and Yang, Ming-Hsuan},
  journal={Advances in Neural Information Processing Systems},
  volume={38},
  pages={27798--27829},
  year={2026}
}

Contributors

YuanzeLin

45 commits