Relit-LiVE: Relight Video by Jointly Learning Environment Video

     

This repo contains the official code of our paper: Relit-LiVE: Relight Video by Jointly Learning Environment Video.

📊 Overview

We present Relit-LiVE, a novel video relighting framework that produces physically consistent and temporally stable results without needing prior knowledge of camera pose. This is achieved by jointly generating relighting videos and environment videos. Additionally, by integrating real-world lighting effects with intrinsic constraints, the relighting videos demonstrate remarkable physical plausibility, showcasing realistic reflections and shadows.

✨ News

• May 8, 2026: Release project page and infer pipeline.

📝 Check list

• Release the arxiv and project page.
• Release inference code and model checkpoints.
• Release gradio code and full inference pipeline (inverse-forward).
• Release training code and data pipeline.
• Release training dataset.

---

🛠️ Installation

Minimum requirements

• Python 3.10
• NVIDIA GPU, with at least 24 GB VRAM recommended
• CUDA 12.4 or a compatible version
• Model weights prepared under checkpoints/ and models/Wan-AI/Wan2.1-T2V-1.3B/

Recommended environment:

• Ubuntu 20.04 or newer
• Single-GPU CUDA inference setup

Conda environment

conda create -n diffsynth python=3.10
conda activate diffsynth
pip install -e .
pip install -U deepspeed
pip install transformers==4.50.0
pip install gradio==6.14.0

Optional for full inference pipeline

The cosmos-transfer1-diffusion-renderer repository is essential for full pipeline inference. Install the conda environment named cosmos-predict1 following the instructions in its README.md.

cd third_party
git clone https://github.com/nv-tlabs/cosmos-transfer1-diffusion-renderer.git
...

📦 Checkpoints

Download the Relit-LiVE checkpoints from HuggingFace and place them under checkpoints/.

In addition, inference loads the Wan2.1 base model from models/Wan-AI/Wan2.1-T2V-1.3B/. Make sure all weights are in place before running inference.

If you want to reproduce the MIT metrics reported in the paper, you should load the model_frame57_480_832.ckpt and perform single-frame inference directly on the test set.

(Optional for full inference pipeline) Download the cosmos-transfer1-diffusion-renderer checkpoints from HuggingFace and place them under third_party/cosmos-transfer1-diffusion-renderer/checkpoints/ following the instructions in its README.md.

🚀 Inference

By default, generated results are written to inference_output/.

Basic 25-frame relighting

python relit_inference.py \
    --dataset_path datasets/demos \
    --ckpt_path checkpoints/model_frame25_480_832.ckpt \
    --output_dir inference_output \
    --cfg_scale 1.0 \
    --height 480 \
    --width 832 \
    --num_frames 25 \
    --padding_resolution \
    --use_ref_image \
    --env_map_path datasets/envs/Pink_Sunrise \
    --frame_interval 1 \
    --num_inference_steps 50 \
    --quality 10

25-frame rotating-light relighting

python relit_inference.py \
    --dataset_path datasets/demos \
    --ckpt_path checkpoints/model_frame25_480_832.ckpt \
    --output_dir inference_output \
    --cfg_scale 1.0 \
    --height 480 \
    --width 832 \
    --num_frames 25 \
    --padding_resolution \
    --use_ref_image \
    --env_map_path datasets/envs/Pink_Sunrise \
    --frame_interval 1 \
    --num_inference_steps 50 \
    --use_rotate_light \
    --quality 10

Fixed-frame relighting with width-axis light rotation

python relit_inference.py \
    --dataset_path datasets/demos \
    --ckpt_path checkpoints/model_frame25_480_832.ckpt \
    --output_dir inference_output \
    --cfg_scale 1.0 \
    --height 480 \
    --width 832 \
    --num_frames 25 \
    --padding_resolution \
    --use_ref_image \
    --env_map_path datasets/envs/Pink_Sunrise \
    --frame_interval 1 \
    --num_inference_steps 50 \
    --use_fixed_frame_and_w_rotate_light \
    --quality 10

Fixed-frame relighting with height-axis light rotation

python relit_inference.py \
    --dataset_path datasets/demos \
    --ckpt_path checkpoints/model_frame25_480_832.ckpt \
    --output_dir inference_output \
    --cfg_scale 1.0 \
    --height 480 \
    --width 832 \
    --num_frames 25 \
    --padding_resolution \
    --use_ref_image \
    --env_map_path datasets/envs/Pink_Sunrise \
    --frame_interval 1 \
    --num_inference_steps 50 \
    --use_fixed_frame_and_h_rotate_light \
    --quality 10

57-frame video relighting

python relit_inference.py \
    --dataset_path datasets/demos \
    --ckpt_path checkpoints/model_frame57_480_832.ckpt \
    --output_dir inference_output \
    --cfg_scale 1.0 \
    --height 480 \
    --width 832 \
    --num_frames 57 \
    --padding_resolution \
    --use_ref_image \
    --env_map_path datasets/envs/Pink_Sunrise \
    --frame_interval 1 \
    --num_inference_steps 50 \
    --quality 10

Single-frame high-resolution relighting

python relit_inference.py \
    --dataset_path datasets/demos \
    --ckpt_path checkpoints/model_frame1_1024_1472.ckpt \
    --output_dir inference_output \
    --cfg_scale 1.0 \
    --height 1024 \
    --width 1472 \
    --num_frames 1 \
    --padding_resolution \
    --use_ref_image \
    --env_map_path datasets/envs/Pink_Sunrise \
    --frame_interval 1 \
    --num_inference_steps 50 \
    --quality 10

📋 Argument reference

The following arguments are defined in parse_args() inside relit_inference.py.

Notes

• Output filenames automatically include parts of the checkpoint name, sequence name, resolution, reference-image mode, environment lighting information, inference steps, frame count, and cfg_scale.
• When --num_frames 1 is used, the script writes a png. When --num_frames > 1, it writes an mp4.

🚀 Full inference pipeline (gradio)

Please make sure you have the following items ready:

1. conda environment named diffsynth.
2. conda environment named cosmos-predict1.
3. ./checkpoints/*.ckpt.
4. ./third_party/cosmos-transfer1-diffusion-renderer.
5. ./third_party/cosmos-transfer1-diffusion-renderer/checkpoints/Cosmos-Tokenize1-CV8x8x8-720p and ./third_party/cosmos-transfer1-diffusion-renderer/checkpoints/Diffusion_Renderer_Inverse_Cosmos_7B.

Then,

conda activate diffsynth
python run_full_inference_gradio.py

📌 Future plans

This project will be continuously maintained. We welcome users to try it out and share their feedback (15770575681@163.com).

The current plan includes a model version specifically designed for portraits and another that is better suited for handling motion (including camera and scene dynamics).

🤝 Citation

If you find this repository helpful, please consider citing our paper:

@article{xiao2026relit,
  title={Relit-LiVE: Relight Video by Jointly Learning Environment Video},
  author={Xiao, Weiqing and Li, Hong and Yang, Xiuyu and Chen, Houyuan and Li, Wenyi and Liu, Tianqi and Xu, Shaocong and Ye, Chongjie and Zhao, Hao and Wang, Beibei},
  journal={arXiv preprint arXiv:2605.06658},
  year={2026}
}

📝 Acknowledgements

Code is built on DiffSynth-Studio and diffusion-renderer. Thanks all the authors for their excellent contributions!