Learning Zero-Shot Subject-Driven Video Generation Using 1% Compute (ECCV 2026)

This repository provides the code for the paper "Learning Zero-Shot Subject-Driven Video Generation Using 1% Compute". The method enables the generation of high-quality videos based on a subject image and a text prompt, without requiring large annotated video datasets. By leveraging an image customization dataset and a small set of unannotated videos, this approach achieves robust subject consistency and temporal coherence in a zero-shot setting.

Note: This repository currently only includes the inference code for version 1 release. Release for version 3 (Including CogVideoX and Wan) Inference checkpoint and Fine-tuning code is not provided at this time but is planned for a future release.

🔥 Latest News

• Jun 18, 2026 Our paper has been accepted to ECCV 2026! Details will be released doon!
• May 06, 2026 Updated Version (v3) Released to arXiv!
• Jan 10, 2026 Updated Version (v2) Released to arXiv!
• Apr 28, 2025 Initial Version (v1) Released to arXiv!
• Apr 27, 2025 Inference Code Release!

TODOs

• Release inference code.
• Release v2 arXiv release
• Release v3 arXiv Release
• Release v3 inference code with Wan and CogVideoX
• Release fine-tuning code.
• Add more features and improvements.

Quick Start

To set up the environment and install dependencies, first start with:

conda create -n disentangled_s2v python=3.12
conda activate disentangled_s2v

You can install all the required packages by:

bash installer.sh

Or follow these steps:

1. Install PyTorch:

   pip install torch==2.4.1+cu121 -f https://download.pytorch.org/whl/torch_stable.html
   

2. Install diffusers in editable mode:

   cd diffusers
   pip install -e .
   cd ..
   

3. Install other dependencies:

   pip install sentencepiece huggingface peft transformers accelerate
   pip install opencv-python imageio ffmpeg imageio-ffmpeg
   

Get Checkpoint

• Download the pre-trained model checkpoint: Download the checkpoint from Google Drive Link and extract it to a directory, e.g., ./disentangled_s2v_ckpt.

  Place the ckpt file as follows:

  # Directory structure of the checkpoint folder:
  ./disentangled_s2v_ckpt
  ├── optimizer.bin
  ├── pytorch_lora_weights_transformer.safetensors
  ├── random_states_0.pkl
  └── scheduler.bin
  

Usage

• Run the inference script:

  python src/inference.py \
    --reference_image_path <REFERENCE IMAGE PATH>
    --output_dir "./test_output.mp4" \
    --checkpoint_path <CHECKPOINT_PATH> \
    --prompt <PROMPT>
  

Alternatively, you can use the provided shell script for quick demo:

bash s2v_inference_demo.sh

Acknowledgements

We built our work based on CogVideoX, with dataset from OminiControl and Pexels.

BibTeX

@article{kim2025subject,
  author    = {Kim, Daneul and Zhang, Jingxu and Jin, Wonjoon and Cho, Sunghyun and Dai, Qi and Park, Jaesik and Luo, Chong},
  title     = {Subject-driven Video Generation via Disentangled Identity and Motion},
  journal   = {arXiv},
  year      = {2025},
}