---

Clone This Repo

This repository has multiple branches. The page branch contains the project website and many large asset files, so for inference usage you should clone only the main branch:

git clone --single-branch --branch main https://github.com/ZhaoJingjing713/Spatia.git
cd Spatia

If you have already cloned the full repository and want to avoid fetching other branches later, keep working on main and avoid checking out page.

---

Overview

This release provides the inference pipeline for Spatia, built on top of:

• Wan2.2-TI2V-5B as the base video generator
• a control / VACE checkpoint for spatial guidance
• a LoRA checkpoint for autoregressive long-horizon generation

Main entry point:

python inference.py

At a high level, the pipeline:

1. Reconstructs and updates a 3D scene point cloud with MapAnything.
2. Renders point-guided control videos from target camera trajectories.
3. Generates spatially consistent clips with Wan2.2.
4. Reuses history frames and matched reference views for long-horizon rollout.

---

Repository Layout

.
├── inference.py
├── install.sh
├── download.sh
├── requirements.txt
├── test_cases/
├── utils/
└── wan/

Key files:

• inference.py: end-to-end inference entry
• install.sh: environment setup script
• download.sh: base Wan2.2 checkpoint download helper
• test_cases/: example prompts, intrinsics, and camera trajectories

---

Installation

1. Create the environment

First create and activate a Conda environment with Python 3.12:

conda create -n spatia python=3.12 -y
conda activate spatia

Then run the install script. It supports both cuda and rocm.

bash install.sh cuda

or

bash install.sh rocm

You can also specify a custom build directory:

bash install.sh cuda ./env

The script installs:

• PyTorch
• FlashAttention
• DiffSynth-Studio
• PyTorch3D
• MapAnything
• Python dependencies from requirements.txt

2. Download the base Wan2.2 model

Download Wan2.2-TI2V-5B to model_weights/Wan2.2-TI2V-5B:

bash download.sh

Equivalent manual command:

pip install -U "huggingface_hub[cli]<1.0.0"
hf download Wan-AI/Wan2.2-TI2V-5B --local-dir ./model_weights/Wan2.2-TI2V-5B

3. Download Spatia weights

Download the Spatia checkpoints from Hugging Face:

hf download Jinjing713/Spatia --local-dir ./checkpoints/Spatia

You need:

• one control / VACE checkpoint
• one LoRA checkpoint

Then point --vace_path and --lora_path to the downloaded files.

Example:

python inference.py \
  --vace_path ./checkpoints/Spatia/step-8500.safetensors \
  --lora_path ./checkpoints/Spatia/lora_weights_10000.safetensors

If the filenames in your local download differ, just update the paths accordingly.

---

Input Format

Each run requires:

• one starting image via --img_path
• one camera trajectory file via --camera_w2c_path
• one intrinsics file via --camera_intrinsics_path
• one prompt or prompt file

w2c.txt

w2c.txt stores one camera extrinsic per line in JSON-style list format.

• Coordinate convention: OpenCV camera coordinates
• Matrix type: world-to-camera
• Supported shapes per line:
  • 3x4
  • homogeneous 4x4

Examples:

[[r11, r12, r13, t1], [r21, r22, r23, t2], [r31, r32, r33, t3]]

[[r11, r12, r13, t1], [r21, r22, r23, t2], [r31, r32, r33, t3], [0, 0, 0, 1]]

intrinsics.txt

intrinsics.txt stores normalized intrinsics:

[fx fy cx cy]

Here:

• fx, cx are normalized by image width
• fy, cy are normalized by image height

Pixel-space intrinsics are reconstructed internally using runtime --width and --height.

--mask_path

--mask_path specifies foreground object masks for dynamic/static disentanglement and controllable generation.

The foreground mask marks dynamic or editable foreground regions in the input image or video. It is used during spatial memory construction so the model can better separate:

• static scene structure
• foreground dynamic content

Supported formats:

• one or more binary mask images
• one .npy file with shape [N, H, W]

Typical usage:

• a single foreground mask for the first frame
• frame-aligned masks for a full sequence

Notes:

• Image masks should be binary foreground masks.
• For .npy, each slice should be one binary mask frame.
• If the masks are frame-aligned, use --per_frame_mask.

prompt.txt

prompt.txt can contain:

• a single prompt line, reused for all rounds
• multiple prompt lines, one prompt per round

---

Quick Start

Minimal example:

python inference.py \
  --img_path path/to/input.jpg \
  --camera_w2c_path test_cases/case_2/w2c.txt \
  --camera_intrinsics_path test_cases/case_2/intrinsics.txt \
  --prompt_path test_cases/case_2/prompt.txt \
  --save_path test_cases/case_2/output.mp4 \
  --vace_path ./checkpoints/Spatia/step-8500.safetensors \
  --lora_path ./checkpoints/Spatia/lora_weights_10000.safetensors

---

CLI Arguments

Core paths

Prompt and masks

Resolution and rollout

Reference-frame selection

Diffusion settings

Subprocess and reconstruction

---

Test Cases

The repository includes three example camera/prompt setups:

• test_cases/case_1
• test_cases/case_2
• test_cases/case_3

Each case includes:

• prompt.txt
• w2c.txt
• intrinsics.txt

case_3 additionally uses:

• mask.png

The starting image is not included. Before running the examples below, place an input image in each case directory, for example:

test_cases/case_1/img.jpg
test_cases/case_2/img.jpg
test_cases/case_3/img.jpg

Case 1

python inference.py \
  --img_path test_cases/case_1/img.jpg \
  --camera_w2c_path test_cases/case_1/w2c.txt \
  --camera_intrinsics_path test_cases/case_1/intrinsics.txt \
  --prompt_path test_cases/case_1/prompt.txt \
  --save_path test_cases/case_1/output.mp4 \
  --work_dir test_cases/case_1/output \
  --vace_path checkpoints/Spatia/control_weight_8500.safetensors \
  --lora_path checkpoints/Spatia/lora_weights_10000.safetensors

Case 2

python inference.py \
  --img_path test_cases/case_2/img.jpg \
  --camera_w2c_path test_cases/case_2/w2c.txt \
  --camera_intrinsics_path test_cases/case_2/intrinsics.txt \
  --prompt_path test_cases/case_2/prompt.txt \
  --save_path test_cases/case_2/output.mp4 \
  --work_dir test_cases/case_2/output \
  --vace_path checkpoints/Spatia/control_weight_8500.safetensors \
  --lora_path checkpoints/Spatia/lora_weights_10000.safetensors

Case 3

python inference.py \
  --img_path test_cases/case_3/img.jpg \
  --camera_w2c_path test_cases/case_3/w2c.txt \
  --camera_intrinsics_path test_cases/case_3/intrinsics.txt \
  --prompt_path test_cases/case_3/prompt.txt \
  --mask_path test_cases/case_3/mask.png \
  --save_path test_cases/case_3/output.mp4 \
  --work_dir test_cases/case_3/output \
  --vace_path checkpoints/Spatia/control_weight_8500.safetensors \
  --lora_path checkpoints/Spatia/lora_weights_10000.safetensors

---

Notes

• model_weights/Wan2.2-TI2V-5B must exist before running inference.
• --vace_path and --lora_path must point to valid Spatia checkpoints.
• Intermediate reconstruction and rendering results are cached in --work_dir.
• If prompts, masks, or camera files change, use --force_rebuild_intermediate for a clean rerun.

---

Citation

If you find this project useful, please cite the paper.

@inproceedings{zhao2026spatia,
  title={Spatia: Video Generation with Updatable Spatial Memory},
  author={Zhao, Jinjing and Wei, Fangyun and Liu, Zhening and Zhang, Hongyang and Xu, Chang and Lu, Yan},
  booktitle={Proceedings of the IEEE/cvf conference on computer vision and pattern recognition},
  year={2026}
}