UniGeo(ECCV'2026)

Hong Jiang · Wensong Song · Zongxin Yang · Ruijie Quan · Yi Yang


Zhejiang University   |   Harvard University  

⚠️ Important Notice on Input Images:

Due to the inherent constraints of the VGGT and Wan models, please ensure your input images satisfy the condition: Width ≥ Height (i.e., landscape or square aspect ratios). Using portrait images (where height > width) may lead to suboptimal results.

🔥 News

• [2026.4.28] Update inference demo to support 23GB VRAM.
• [2026.4.26] Release HF Space. Welcome to try it out!
• [2026.4.24] Release inference demo and pretrained checkpoint.

💡 Demo

The parameters in our examples are normalized to a unified scale via VGGT. For more demos and detailed examples, check out our project page:

🛠️ Installation

Begin by cloning the repository:

git clone https://github.com/mo230761/UniGeo
cd UniGeo

Installation Guide for Linux

Conda's installation instructions are available here.

conda create -n unigeo python==3.9

conda activate unigeo

pip install -r requirements.txt

📦 Install PyTorch3D (Required)

This project strictly requires pytorch3d, which must be installed via a conda package.

⚠️ CRITICAL Note: The exact PyTorch3D package you need depends on your local Python, CUDA, and PyTorch versions. DO NOT just copy the command below blindly. You must find and use the specific package that matches your environment.

Example Installation: If your environment is Python 3.9, CUDA 12.1, and PyTorch 2.4.1, your installation command would look like this:

# Please replace the .tar.bz2 file with the one matching your specific environment!
conda install /path/to/your/download/pytorch3d-0.7.8-py39_cu121_pyt241.tar.bz2

⏬ Download Checkpoints

• UniGeo Lora: Download the main checkpoint from HuggingFace. This checkpoint is optimized for real-world images. Since real-world images often feature highly complex compositions and diverse scene layouts, we disabled the Geometric Anchor Attention to avoid over-constraining the generation.

• Wan and VGGT Model: This project relies on Wan2.2-TI2V-5B and VGGT-1B. Download its checkpoint(s) as well.

🎥 Inference

💡 Quick Start: We have provided an example dataset in the example_dataset/ folder. You can directly use it!

To run inference on your images, please follow this step-by-step pipeline:

Step 1: Data Preparation

Organize your images and the corresponding prompt.json directly into a dataset folder :

/path/to/dataset/
├── image1.png
├── image2.png
├── ...
└── prompt.json

The prompt.json should map each image filename to its corresponding camera motion prompt.

Example:

{
  "image1.png": "Camera pans left by 16 degrees; Camera tilts up by 7 degrees",
  "image2.png": "Camera moves forward by 2.5 meters"
}

Step 2: Generate Point Clouds

Before running the main generation, translate your prompts into point clouds :

cd vggt
python Prompt_to_Point_Cloud.py \
    --model_path /path/to/VGGT \
    --dataset_path /path/to/dataset

💡 Tip: The generated point cloud serves as an explicit visual preview of the camera motion. You can iteratively adjust the camera parameters in your prompt.json and re-run this step until the geometric trajectory perfectly matches your expectations before proceeding to the final image generation.

Step 3: Final Generation

Once the point clouds are ready, run the core inference pipeline to generate the final camera-controllable results :

cd ../DiffSynth-Studio
python infer.py \
    --dataset_path /path/to/dataset (point cloud already included) \
    --wan_model_dir /path/to/Wan2.2-TI2V-5B \
    --lora_path /path/to/lora \
    --wan_config_path /path/to/config.json (see `my_config.json` in this repo)

💡 Tip: We added offload support in the code. You can enable it as needed, with a minor trade-off in running speed.

🤝 Acknowledgement

We appreciate the open source of the following projects:

• Diffusers
• Wan
• VGGT

Citation

@misc{jiang2026unigeounifyinggeometricguidance,
      title={UniGeo: Unifying Geometric Guidance for Camera-Controllable Image Editing via Video Models}, 
      author={Hong Jiang and Wensong Song and Zongxing Yang and Ruijie Quan and Yi Yang},
      year={2026},
      eprint={2604.17565},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.17565}, 
}