VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control

       

Sixiao Zheng^1,2    Minghao Yin³    Wenbo Hu^4†    Xiaoyu Li⁴    Ying Shan⁴    Yanwei Fu^1,2†

¹Fudan University    ²Shanghai Innovation Institute    ³HKU    ⁴ARC Lab, Tencent PCG

^†Corresponding authors

CVPR 2026

✨ A controllable video world model with explicit 4D geometric control over camera and multi-object motion.

📚 Table of Contents

• News
• TODO
• TL;DR
• Installation
• Download Checkpoints
• Usage
• Building the VerseControl4D Dataset
• Training
• Batch Inference
• Acknowledgements
• License
• Star History
• Citation

🔥 News

• [Jul 20, 2026] 🚀 We release the training code for VerseCrafter!
• [Jul 19, 2026] 🚀 We release the data processing code for building VerseControl4D!
• [Feb 21, 2026] 🎉 VerseCrafter is accepted to CVPR 2026!
• [Jan 9, 2026] 🚀 VerseCrafter is released! We publish the arXiv preprint, inference code, and model checkpoints.

✅ TODO

• Inference code
• Training code
• Data processing code

TL;DR

• Dynamic Realistic Video World Model: VerseCrafter learns a realistic and controllable video world prior from large-scale in-the-wild data, handling challenging dynamic scenes with strong spatial-temporal coherence.
• 4D Geometric Control: A unified 4D control state provides direct, interpretable control over camera motion, multi-object motion, and their joint coordination, improving geometric faithfulness.
• Frozen Video Prior + GeoAdapter: We attach a geometry-aware GeoAdapter to a frozen Wan2.1 backbone, injecting 4D controls into diffusion blocks for precise control without sacrificing video quality.
• VerseControl4D Dataset: We introduce a large-scale real-world dataset with automatically rendered camera trajectories and multi-object 3D Gaussian trajectories to supervise 4D controllable generation.

Installation

1. Clone the repository:

   git clone --recursive https://github.com/TencentARC/VerseCrafter.git
   # If you have already cloned the repo, you can update the submodules manually:
   git submodule update --init --recursive
   
   cd VerseCrafter
   

2. Create and activate the Conda environment:

   conda create -n versecrafter python=3.11 -y
   conda activate versecrafter
   
   # Install PyTorch
   conda install pytorch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y 
   
   # Install Python dependencies
   pip install -r requirements.txt
   
   # Install MoGe
   pip install git+https://github.com/microsoft/MoGe.git
   
   # Install Grounded-SAM-2
   cd third_party/Grounded-SAM-2
   pip install -e .
   pip install --no-build-isolation -e grounding_dino
   
   # Install flash attention
   pip install flash-attn --no-build-isolation
   
   # Install pytorch3d
   cd ../../
   git clone https://github.com/facebookresearch/pytorch3d.git
   cd pytorch3d
   pip install --no-build-isolation .
   cd ../VerseCrafter
   

Download Checkpoints

1. Download VerseCrafter and Wan2.1 models:

   pip install --upgrade huggingface_hub
   mkdir -p model
   hf download --local-dir model/VerseCrafter TencentARC/VerseCrafter
   hf download --local-dir model/Wan2.1-T2V-14B Wan-AI/Wan2.1-T2V-14B
   

2. Download Grounded-SAM-2 and Grounding DINO checkpoints:

   cd third_party/Grounded-SAM-2/checkpoints
   bash download_ckpts.sh
   
   cd ../gdino_checkpoints
   bash download_ckpts.sh
   cd ../../../
   

Usage

We provide two ways to use VerseCrafter:

💡 Tip: We recommend the Blender Addon for most users. It supports proxy authentication for secure server access. If you cannot connect to a remote GPU server, use the Script Pipeline instead.

---

Option 1: Blender Addon (Recommended)

For detailed instructions, see README_BLENDER.md.

Prerequisites

• Blender 4.0+ (4.5+ recommended)
• A remote GPU server running the VerseCrafter API

Quick Start

1. Install the addon:

   cd VerseCrafter
   zip -r blender_addon.zip blender_addon/
   

   In Blender: Edit → Preferences → Add-ons → ↓ → Install from Disk... → Select blender_addon.zip → Enable "VerseCrafter Workflow"

2. Start the API server (on GPU server):

   python api_server.py --port 8188 --num_gpus 8
   

3. Configure connection in Blender:

   • Press N to open the sidebar → VerseCrafter tab
   • Set Server URL (e.g., http://<server-ip>:8188)
   • Click Test Connection

4. Run the workflow:

   • Step 1: Select input image, set workflow directory, enter object prompt (e.g., "person . car ."), click "Run Preprocessing"
   • Step 2: Edit camera and object trajectories visually, click "Export Trajectories"
   • Step 3: Enter video prompt, click "Generate Video"

---

Option 2: Script Pipeline

The inference.sh script provides a complete pipeline for generating videos. You can run the steps individually or use the script as a reference.

1. Configuration

Edit inference.sh to set your input image, output directory, and prompt.

INPUT_IMAGE=demo_data/y57HgqX1uGc_0039750_0041550_0000635_0000716/0001.jpg
OUTPUT_DIR=demo_data/y57HgqX1uGc_0039750_0041550_0000635_0000716
MODEL_PATH="model/VerseCrafter"

2. Run the Pipeline

The pipeline consists of the following steps:

Step 1: Depth Estimation

Generate depth maps using MoGE-V2.

python inference/moge-v2_infer.py  -i $INPUT_IMAGE -o $OUTPUT_DIR/estimated_depth --maps

Step 2: Segmentation

Segment objects using Grounded-SAM-2.

python inference/grounded_sam2_infer.py \
    --image_path "$INPUT_IMAGE" \
    --text_prompt "person . car ." \
    --output_dir "$OUTPUT_DIR/object_mask" \
    --min_area_ratio 0.003 \
    --max_area_ratio 0.2

Step 3: Fit 3D Gaussian

Fit 3D Gaussians to the segmented objects.

python inference/fit_3D_gaussian.py \
    --image_path $INPUT_IMAGE \
    --npz_path $OUTPUT_DIR/estimated_depth/depth_intrinsics.npz \
    --masks_dir $OUTPUT_DIR/object_mask/masks \
    --output_dir $OUTPUT_DIR/fitted_3D_gaussian

The following are the input image and its corresponding results:

Step 4: Customize Trajectory (Manual Operation in Blender)

This step requires Blender to interactively edit the 4D control scene. We also provide a demonstration video that shows step-by-step Blender operations for this process:
Watch the Blender operation video here

1. Prepare Scripts:

   • Open inference/blender_script/build_4d_control_scene.py and inference/blender_script/export_blender_custom_trajectories.py.
   • Crucial: Update the ROOT_DIR variable in both scripts to the absolute path of your input directory (e.g., /absolute/path/to/demo_data/your_folder).

2. Build Scene:

   • Open Blender.
   • Go to the Scripting tab.
   • Open or paste the content of build_4d_control_scene.py.
   • Run the script to load the scene (point cloud, camera, objects).

3. Customize Trajectories:

   • Switch to the Layout tab.
   • Camera Trajectory:
     • Create a curve (e.g., Shift+A → Curve → Bezier).
     • Switch to Edit Mode to draw or adjust the curve.
     • Select the Camera, add a Follow Path constraint targeting the curve.
     • Check Fixed Position.
     • Set the animation duration to 81 frames.
   • 3D Gaussian (Object) Trajectory:
     • Select the object (Ellipsoid).
     • Use the same Follow Path method as the camera, or insert Keyframes (I key) for location/rotation/scale.

4. Export Trajectories:

   • Go back to the Scripting tab.
   • Open or paste the content of export_blender_custom_trajectories.py.
   • Run the script to export custom_camera_trajectory.npz and custom_3D_gaussian_trajectory.json.

This is an animation of custom trajectories in Blender:

Step 5: Render 4D Control Maps

Render the 4D control maps for the diffusion model.

python inference/rendering_4D_control_maps.py \
    --png_path $INPUT_IMAGE \
    --npz_path $OUTPUT_DIR/estimated_depth/depth_intrinsics.npz \
    --mask_dir $OUTPUT_DIR/object_mask/masks \
    --trajectory_npz $OUTPUT_DIR/camera_object_0/custom_camera_trajectory.npz \
    --ellipsoid_json $OUTPUT_DIR/camera_object_0/custom_3D_gaussian_trajectory.json \
    --output_dir $OUTPUT_DIR/camera_object_0/rendering_4D_maps

The following are the 4D control maps rendered from this step:

Step 6: VerseCrafter Inference

Generate the final video.

torchrun --nproc-per-node=8 inference/versecrafter_inference.py \
  --transformer_path $MODEL_PATH \
  --num_inference_steps 30 \
  --sample_size "720,1280" \
  --ulysses_degree 2 \
  --ring_degree 4 \
  --prompt "A sun-drenched street in Valletta, Malta, showcasing towering honey-colored limestone buildings adorned with traditional wrought-iron balconies and arched doorways. On the left-hand sidewalk, a man in a bright orange T-shirt and a woman in a beige summer dress walk side-by-side. Several cars are parked in the distance. The vibrant Mediterranean sunlight casts soft shadows, illuminating the weathered textures of the ancient architecture, which stretches towards distant city fortifications under a clear, pale blue sky." \
  --input_image_path $INPUT_IMAGE \
  --save_path $OUTPUT_DIR/camera_object_0 \
  --rendering_maps_path $OUTPUT_DIR/camera_object_0/rendering_4D_maps

Building the VerseControl4D Dataset

We provide integrated filtering/rendering entry scripts under data_processing/ that turn raw Sekai / SpatialVID-HQ videos into VerseControl4D clips and 4D control maps:

• build_versecontrol4d_spatialvid.sh
• build_versecontrol4d_sekai.sh

Download datasets first:

• Sekai: Lixsp11/Sekai
• SpatialVID-HQ: FelixYuan/SpatialVID-HQ

System dependency: ffmpeg and ffprobe must be available in PATH.

Quick Start

# 1) make sure submodules are initialized
git submodule update --init --recursive

# 2) after dataset download + external preprocessing, enter data_processing
cd data_processing

# 3) SpatialVID pipeline
SPATIALVID_ROOT=/path/to/SpatialVID-HQ \
bash build_versecontrol4d_spatialvid.sh

# 4) Sekai pipeline
SEKAI_ROOT=/path/to/sekai-codebase \
bash build_versecontrol4d_sekai.sh

Optional environment variables:

• GSAM2_HOME: Grounded-SAM-2 root path (default: third_party/Grounded-SAM-2)

build_versecontrol4d_sekai.sh renders the scattered and moderate subsets separately.

Pipeline Steps

Each build script runs the same step order. Steps marked (existing code) rely on external repositories.

Sekai flow (build_versecontrol4d_sekai.sh)

• Step 0: Split metadata by crowdDensity (scattered/moderate).
• Step 1 (existing code): Use sekai-codebase for clip extraction.
• Step 2: Filter clips with 2_filter_sekai_with_grounded_sam2.py.
• Step 3 (existing code): Use TAPIP3D to predict depth, camera pose, and export *_depth_pose.npz files.
• Step 4: Render control maps with render_video_pipeline_sekai.py.

SpatialVID-HQ flow (build_versecontrol4d_spatialvid.sh)

• Step 0: Split metadata by crowdDensity (Sparse/Moderate).
• Step 1 (existing code): Run clip extraction with 1_run_scenedetect.py.
• Step 2: Filter clips with 2_filter_spatialvid_with_grounded_sam2.py.
• Step 3 (existing code): Use TAPIP3D to predict depth, camera pose, and export *_depth_pose.npz files.
• Step 4: Render control maps with render_video_pipeline_spatialvid.py.

The scripts assume these artifacts already exist at expected paths (for Sekai, clips under vstreams_scattered / vstreams_moderate; for SpatialVID under SpatialVid/HQ), and Grounded-SAM-2 defaults to third_party/Grounded-SAM-2 via GSAM2_HOME.

Output layout

Step 4 renders 5 control/mask maps per clip: background_RGB.mp4, background_depth.mp4, 3D_gaussian_RGB.mp4, 3D_gaussian_depth.mp4, merged_mask.mp4. Together with the source clips, they are consumed by both Training and Batch Inference.

• Sekai clips: vstreams_{scattered|moderate}_clip/<video_id>/<clip_stem>.mp4
• Sekai maps: vstreams_{scattered|moderate}_clip_annotation/<video_id>/<clip_stem>/*.mp4
• SpatialVID clips: SpatialVid/HQ/{Sparse|Moderate}_clip/<clip_stem>.mp4
• SpatialVID maps: clip_annotation/<clip_stem>/*.mp4

Training

We train VerseCrafter on VerseControl4D by attaching a geometry-aware GeoAdapter to a frozen Wan2.1-T2V-14B backbone: the VAE, text encoder, and the Wan2.1 diffusion backbone are all frozen, and only the GeoAdapter modules (--trainable_modules=geoada) are trained. The training entry point is:

• train/train.py: training script
• train/train.sh: multi-node launch script (Accelerate + DeepSpeed ZeRO-2)

Prerequisites

1. Finish the data build/rendering first (see Building the VerseControl4D Dataset).
2. Prepare a merged training CSV (this repo does not ship training CSV files) with at least:
   • clipPath
   • crowdDensity
   • qwen_prompt or prompt
3. Download the base model to model/Wan2.1-T2V-14B (the same checkpoint fetched in Download Checkpoints).

Expected dataset layout

--train_data_dir (default dataset) is the common root that holds both sub-datasets, each providing the source clips and the 5 control/mask maps:

• Sekai clips: sekai_train_10k/vstreams_{scattered|moderate}_clip/<video_id>/<clip_stem>.mp4
• Sekai maps: sekai_train_10k/vstreams_{scattered|moderate}_clip_annotation/<video_id>/<clip_stem>/*.mp4
• SpatialVID maps: spatialvid/clip_annotation/<clip_stem>/*.mp4

Run Training

Edit the placeholders at the top of train/train.sh (Python environment, MASTER_ADDR, dataset/model paths), then run from the repository root:

conda activate versecrafter
bash train/train.sh

For multi-node training, set these environment variables per machine before launching (or edit them in train/train.sh):

export MASTER_ADDR=<rank-0 machine IP>   # shared across all machines
export MASTER_PORT=29500
export WORLD_SIZE=<number of machines>
export NUM_PROCESS=<WORLD_SIZE * GPUs per node>
export RANK=<rank of this machine, 0 .. WORLD_SIZE-1>
bash train/train.sh

Checkpoints and validation samples are written to --output_dir (default results/exp_ours_14B). Training auto-resumes from the latest checkpoint (--resume_from_checkpoint="latest") and restarts on crash.

Batch Inference

We provide release-ready batch inference scripts under batch_inference/:

• batch_inference/predict_v2v_control_sekai_batch.py
• batch_inference/predict_v2v_control_spatialvid_batch.py
• batch_inference/run_batch_inference_sekai.sh
• batch_inference/run_batch_inference_spatialvid.sh

These scripts run VerseCrafter over prepared VerseControl4D clips and export:

• {clip_stem}_generated.mp4: generated result
• {clip_stem}_comparison.mp4: 3x3 comparison grid (controls + mask + generated + GT)

This section produces generated videos and qualitative comparison grids only; it does not compute quantitative metrics.

Prerequisites

1. Finish the data build/rendering first (see Building the VerseControl4D Dataset); the scripts read clips and maps from its output layout.
2. Prepare your own test split CSV (this repo does not ship test CSV files).
3. Ensure your CSV includes:
   • clipPath
   • crowdDensity
   • qwen_prompt or prompt

Run Sekai Batch Inference

Update placeholders in batch_inference/run_batch_inference_sekai.sh and run:

conda activate versecrafter
bash batch_inference/run_batch_inference_sekai.sh 0

Run SpatialVID Batch Inference

Update placeholders in batch_inference/run_batch_inference_spatialvid.sh and run:

conda activate versecrafter
bash batch_inference/run_batch_inference_spatialvid.sh 0

Acknowledgements

Our codes are built upon MoGe, Grounded-SAM-2, VideoX-Fun, Wan2.1 and diffusers.

License

This project is released under the VerseCrafter License. It is intended for academic/research purposes only and commercial use is not permitted.

Star History

Citation

If you find this work useful, please consider citing:

@article{zheng2026versecrafter,
  title={VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control},
  author={Zheng, Sixiao and Yin, Minghao and Hu, Wenbo and Li, Xiaoyu and Shan, Ying and Fu, Yanwei},
  journal={arXiv preprint arXiv:2601.05138},
  year={2026}
}