🔥 News

• Feb 08, 2026: 🎉 Our Video-As-Prompt has been accepted by ICLR 2026! See you in Brazil! 🇧🇷
• Jan 31, 2026: 🚀 Thanks to @Artiprocher, Video-As-Prompt has been integrated into DiffSynth-Studio (PR #1034) with support for Inference, Full Training, Full Training Validation, LoRA Training, and LoRA Training Validation. Check it out!
• Jan 31, 2026: 🎉 Community contribution: @okdalto released a ComfyUI implementation of Video-As-Prompt!
• Oct 31, 2025: ❤️ Thanks to @gangxu822 reminder, we managed to lower the memory cost of both variant CogVideoX and Wan2.1 from 40GB/64GB to max around 8GB!
• Oct 24, 2025: 📖 We release the first unified semantic video generation model, Video-As-Prompt (VAP)!
• Oct 24, 2025: 🤗 We release the VAP-Data, the largest semantic-controlled video generation datasets with more than $100K$ samples!
• Oct 24, 2025: 👋 We present the technical report of Video-As-Prompt, please check out the details and spark some discussion!

🖌️ Video-As-Prompt

Core idea: Given a reference video with wanted semantics as a video prompt, Video-As-Prompt animate a reference image with the same semantics as the reference video.

Your browser does not support HTML5 video. Here is a link to the video instead.
E.g., Different Reference Videos + Same Reference Image → New Videos with Different Semantics

Welcome to see our project page for more interesting results!

Architecture

We introduce Video-As-Prompt (VAP), a new paradigm that reframes unified and generalizable semantic-controlled video generation as in-context generation. VAP leverages a reference video as a direct semantic prompt, guiding a frozen Video Diffusion Transformer (DiT) via a plug-and-play Mixture-of-Transformers (MoT) expert. This architecture prevents catastrophic forgetting and is guided by a temporally biased position embedding that eliminates spurious mapping priors for robust context retrieval. As a single unified model, VAP sets a new state-of-the-art for open-source methods, achieving a 38.7% user preference rate that rivals leading condition-specific commercial models. VAP's strong zero-shot generalization and support for various downstream applications mark a significant advance toward general-purpose, controllable video generation.

Performance

We have evaluated Video-As-Prompt (VAP) with other open-source as well as close-source commercial models (Kling / Vidu). The numerical results indicate that Video-As-Prompt (VAP) surpasses all non-unified baselines under various semantic conditions as the first unified and generalizable semantic-controlled video generation model!

🎁 Models Zoo

To demonstrate cross-architecture generality, Video-As-Prompt provides two variants, each with distinct trade-offs:

• CogVideoX-I2V-5B

  • Strengths: Fewer backbone parameters let us train more steps under limited resources, yielding strong stability on most semantic conditions.
  • Limitations: Due to backbone ability limitation, it is weaker on human-centric generation and on concepts underrepresented in pretraining (e.g., ladudu, Squid Game, Minecraft).

• Wan2.1-I2V-14B

  • Strengths: Strong performance on human actions and novel concepts, thanks to a more capable base model.
  • Limitations: Larger model size reduced feasible training steps given our resources, lowering stability on some semantic conditions.

👏👏👏 Contributions and further optimization from the community are welcome.

Please download the pre-trained video DiTs and our corresponding Video-As-Prompt models, and structure them as follows

ckpts/
  ├── Video-As-Prompt-CogVideoX-5B/
      ├── scheduler
      ├── vae
      ├── transformer
      ├── ...
  ├── Video-As-Prompt-Wan2.1-14B/ 
      ├── scheduler
      ├── vae
      ├── transformer
      ├── ...

🌐 Community Implementations

We appreciate the community's efforts in extending Video-As-Prompt to different platforms!

💡 If you've built an implementation for other platforms, feel free to open a PR to add it here!

🤗 Get Started with Video-As-Prompt

Video-As-Prompt supports Macos, Windows, Linux. You may follow the next steps to use Video-As-Prompt via:

Install Requirements

We test our model with Python 3.10 and PyTorch 2.7.1+cu124.

conda create -n video_as_prompt python=3.10 -y
conda activate video_as_prompt
pip install -r requirements.txt
pip install -e ./diffusers
conda install -c conda-forge ffmpeg -y

Data

We have published the VAP-Data dataset used in our paper on VAP-Data. Please download it and put it in the data folder. The structure should look like:

data/
  ├── VAP-Data/
  │   ├── vfx_videos/
  │   ├── vfx_videos_hq/
  │   ├── vfx_videos_hq_camera/
  │   ├── benchmark/benchmark.csv
  │   ├── vap_data.csv

Code Usage

We mainly implement our code based on Diffusers and Finetrainers for their modular design.

Minimal Demo

Below is a minimal demo of our CogVideoX-I2V-5B variant. The full code can be found in infer/cog_vap.py. The WAN2.1-I2V-14B variant is similar and can be found in infer/wan_vap.py.

import torch
from diffusers import (
    AutoencoderKLCogVideoX,
    CogVideoXImageToVideoMOTPipeline,
    CogVideoXTransformer3DMOTModel,
)
from diffusers.utils import export_to_video, load_video
from PIL import Image

vae = AutoencoderKLCogVideoX.from_pretrained("ByteDance/Video-As-Prompt-CogVideoX-5B", subfolder="vae", torch_dtype=torch.bfloat16)
transformer = CogVideoXTransformer3DMOTModel.from_pretrained("ByteDance/Video-As-Prompt-CogVideoX-5B", torch_dtype=torch.bfloat16)
pipe = CogVideoXImageToVideoMOTPipeline.from_pretrained(
    "ByteDance/Video-As-Prompt-CogVideoX-5B", vae=vae, transformer=transformer, torch_dtype=torch.bfloat16
).to("cuda")

ref_video = load_video("assets/videos/demo/object-725.mp4")
image = Image.open("assets/images/demo/animal-2.jpg").convert("RGB")
idx = torch.linspace(0, len(ref_video) - 1, 49).long().tolist()
ref_frames = [ref_video[i] for i in idx]

output_frames = pipe(
    image=image,
    ref_videos=[ref_frames],
    prompt="A chestnut-colored horse stands on a grassy hill against a backdrop of distant, snow-dusted mountains. The horse begins to inflate, its defined, muscular body swelling and rounding into a smooth, balloon-like form while retaining its rich, brown hide color. Without changing its orientation, the now-buoyant horse lifts silently from the ground. It begins a steady vertical ascent, rising straight up and eventually floating out of the top of the frame. The camera remains completely static throughout the entire sequence, holding a fixed shot on the landscape as the horse transforms and departs, ensuring the verdant hill and mountain range in the background stay perfectly still.",
    prompt_mot_ref=[
      "A hand holds up a single beige sneaker decorated with gold calligraphy and floral illustrations, with small green plants tucked inside. The sneaker immediately begins to inflate like a balloon, its shape distorting as the decorative details stretch and warp across the expanding surface. It rapidly transforms into a perfectly smooth, matte beige sphere, inheriting the primary color from the original shoe. Once the transformation is complete, the new balloon-like object quickly ascends, moving straight up and exiting the top of the frame. The camera remains completely static and the plain white background is unchanged throughout the entire sequence."
    ],
    height=480,
    width=720,
    num_frames=49,
    frames_selection="evenly",
    use_dynamic_cfg=True,
).frames[0]

Inference Memory Optimization

Based on diffusers' pipe.enable_sequential_cpu_offload() function，we lower the memory cost of CogVideoX version on NVIDIA A100 from the current 40GB to max around 7.5GB, and the memory cost of Wan2.1 version on A100 from the current 64GB to max around 8GB. Detailed update:

1. delete the.to("cuda") initialization
2. use the pipe.enable_model_cpu_offload() for module offload (load one module at a time, medium memory save) or pipe.enable_sequential_cpu_offload() for layer offload (load one layer at a time, minimum memory save)

• CogVideoX Current Version：

  pipe = WanImageToVideoMOTPipeline.from_pretrained(
          model_id,
          vae=vae,
          image_encoder=image_encoder,
          transformer=transformer,
          torch_dtype=torch.bfloat16,
      ).to("cuda")
  

  Lower Memory Version：

  pipe = CogVideoXImageToVideoMOTPipeline.from_pretrained(
      model_id, 
      vae=vae, 
      transformer=transformer, 
      torch_dtype=torch.bfloat16
  )
  # offload base on module, max around 30GB
  # pipe.enable_model_cpu_offload()
  # offload base on layer, max around 7.5GB
  pipe.enable_sequential_cpu_offload()
  

• Wan2.1 Current Version：

  pipe = WanImageToVideoMOTPipeline.from_pretrained(
          model_id,
          vae=vae,
          image_encoder=image_encoder,
          transformer=transformer,
          torch_dtype=torch.bfloat16,
      ).to("cuda")
  

  Lower Memory Version：

  pipe = WanImageToVideoMOTPipeline.from_pretrained(
          model_id,
          vae=vae,
          image_encoder=image_encoder,
          transformer=transformer,
          torch_dtype=torch.bfloat16,
          low_cpu_mem_usage=True,
      )
  # offload base on modules, max around 44GB
  # pipe.enable_model_cpu_offload()
  # offload base on layers, max around 8GB
  pipe.enable_sequential_cpu_offload()
  

Benchmark Inference

You can alse refer the following code for benchmark inference. Then you can use Vbench to evaluate the results.

python infer/cog_vap_bench.py
python infer/wan_vap_bench.py

Welcome to modify the scripts to see more results in our dataset VAP-Data and even in-the-wild reference videos or images.

Training

Pick a recipe, then run the corresponding script. Each script sets sensible defaults; override as needed.

Recipes — CogVideoX-I2V-5B

DPO and multi-reference SFT are just our exploration. We provide the code for boost of the community research.

Recipes — Wan2.1-I2V-14B (SFT only)

Quick start (CogVideoX-5B, single-node SFT)

bash examples/training/sft/cogvideox/vap_mot/train_single_node.sh

Quick start (Wan2.1-14B, single-node SFT)

bash examples/training/sft/wan/vap_mot/train_single_node.sh

Multi-node launch (example)

# 6 nodes
bash examples/training/sft/cogvideox/vap_mot/train_multi_node.sh xxx:xxx:xxx:xxx:xxx(MASTER_ADDR) 0
bash examples/training/sft/cogvideox/vap_mot/train_multi_node.sh xxx:xxx:xxx:xxx:xxx(MASTER_ADDR) 1
...
bash examples/training/sft/cogvideox/vap_mot/train_multi_node.sh xxx:xxx:xxx:xxx:xxx(MASTER_ADDR) 5
# or for Wan:
# examples/training/sft/wan/vap_mot/train_multi_node.sh xxx:xxx:xxx:xxx:xxx(MASTER_ADDR) 0
# examples/training/sft/wan/vap_mot/train_multi_node.sh xxx:xxx:xxx:xxx:xxx(MASTER_ADDR) 1
...
# examples/training/sft/wan/vap_mot/train_multi_node.sh xxx:xxx:xxx:xxx:xxx(MASTER_ADDR) 5

Notes

• CogVideoX supports SFT, DPO, and a ≤3-reference SFT variant; Wan currently supports standard SFT only.
• All scripts read shared config (datasets, output dir, batch size, etc.); edit the script to override.
• Please edit train_multi_node*.sh base on your environment if you want to change the distributed settings (e.g., gpu num, node num, master addr/port, etc.).

🔗 BibTeX

❤️ If you found this repository helpful, please give us a star and cite our report:

@article{bian2025video,
  title={Video-As-Prompt: Unified Semantic Control for Video Generation},
  author={Bian, Yuxuan and Chen, Xin and Li, Zenan and Zhi, Tiancheng and Sang, Shen and Luo, Linjie and Xu, Qiang},
  journal={arXiv preprint arXiv:2510.20888},
  year={2025}
}

Acknowledgements

We would like to thank the contributors to the Finetrainers, Diffusers, CogVideoX, and Wan repositories for their open research and exploration.

We also thank:

• @Artiprocher and the DiffSynth-Studio team for integrating Video-As-Prompt with comprehensive training and inference support.
• @okdalto for creating the ComfyUI implementation, making Video-As-Prompt more accessible to the ComfyUI community.

License

Copyright 2025 Bytedance Ltd. and/or its affiliates

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Star History