Go-with-the-Track: Video Compositing and Motion Control with Point Tracking

SIGGRAPH 2026

Koichi Namekata^1,2,3 · Yash Kant^1,2† · Zhizheng Liu^1,4 · Ryan D Burgert^1,2,5 · Yuancheng Xu^1,2 . Kuan Heng Lin^1,6 · Emmett Steven² . Julien Philip¹ . Li Ma¹ . Andrea Vedaldi³ . Paul Debevec^1,2 . Ning Yu^1,2†

¹Eyeline Labs · ²Netflix · ³University of Oxford · ⁴University of California, Los Angeles · ⁵Stony Brook University · ⁶Columbia University

^†Equal supervision.

_{K.N., Z.L., R.B., and K.H.L. performed this work during an internship at Netflix and Eyeline Labs.}

---

Table of Contents

This is the official implementation of Go-with-the-Track: Video Compositing and Motion Control with Point Tracking.

• 💡 TL;DR
• 🔧 Setup
  • Environment Setup
  • Downloading Checkpoints
• 🖌️ Quick Inference
  • Downloading the Evaluation Dataset
  • Inference with wan22 (480P)
  • Inference with wan22 (720P)
  • Output Structure
• 💡 Understanding Input Format
• 🖌️ Creating Customized Mesh Stylization
• 🖌️ Creating Customized Video Stylization
• ✏️ Acknowledgements

---

💡 TL;DR

Go-with-the-Track is a video generation framework that unifies spatio-temporal motion control and reference image insertion via point-tracks. While conventional point-tracks are defined as the 2D flow of a point strictly within the generated video sequence, we extend this definition by anchoring the point-tracks to the reference images, explicitly establishing point correspondences between the generated video frames and the reference content. Treating these correspondences as integral to point-track conditioning, Go-with-the-Track enables fine-grained compositing and motion control throughout the video.

---

🔧 Setup

The code has been tested on Ubuntu 22.04.5 LTS.

Environment Setup

Make sure to run the following commands in a GPU-enabled environment.

For CUDA 11.8 (x86):

conda create -n gwtt python=3.10.*
conda activate gwtt
python -m pip install -e . --no-build-isolation

Downloading Checkpoints

Download the checkpoints from Eyeline-Labs/Go-with-the-Track and place them under ./checkpoints/.

For example:

pip install -U "huggingface_hub"

#set up cache directory
export SCRATCHDIR="/path/to/your/scratchdir" # <--- CHANGE THIS to your scratch directory
export HF_HOME=$SCRATCHDIR/hf_home
export HF_HUB_CACHE=$HF_HOME/hub
mkdir -p "$HF_HUB_CACHE"

#download checkpoints
mkdir -p "$SCRATCHDIR/checkpoints/Go-with-the-Track"
HF_XET_HIGH_PERFORMANCE=1  hf download Eyeline-Labs/Go-with-the-Track \
  --repo-type model \
  --local-dir "$SCRATCHDIR/checkpoints/Go-with-the-Track"

ln -s "$SCRATCHDIR/checkpoints/Go-with-the-Track" checkpoints

Expected Folder Structure:

./checkpoints
    ./ckpt_wan
        ./Wan-AI/
          ./Wan2.2-T2V-A14B/
            high_noise_model/
            low_noise_model/
    ./480P
        low.safetensors
        high.safetensors
    ./720P
        low.safetensors
        high.safetensors

🖌️ Quick Inference

Downloading the Evaluation Dataset

Download the evaluation dataset from Eyeline-Labs/Go-with-the-Track and save it as eval_data.

For example:

# Set up cache directory
export HF_HOME=$SCRATCHDIR/hf_home
export HF_HUB_CACHE=$HF_HOME/hub
mkdir -p "$HF_HUB_CACHE"

# Download dataset
mkdir -p "$SCRATCHDIR/eval_data/Go-with-the-Track"
HF_XET_HIGH_PERFORMANCE=1 hf download Eyeline-Labs/Go-with-the-Track \
  --repo-type dataset \
  --local-dir "$SCRATCHDIR/eval_data/Go-with-the-Track"

ln -s "$SCRATCHDIR/eval_data/Go-with-the-Track" eval_data

Expected Folder Structure:

./eval_data/
      {Video name 1}/
          stylized_prompt.txt
          track.npy
          video.mp4 (<- strictly 49 frames; 480P or 720P)
          video720.mp4 (<- strictly 49 frames; guaranteed 720P)
          ref/
              0.png (up to 720P)
              0.npy
              1.png
              1.npy
              ...
              3.png
              3.npy
      {Video name 2}/
        ...

Inference with wan22 (480P)

By default, the model is trained to generate 49 frames at 480P resolution, with the number of reference frames ranging between 1 and 4. The following demonstrates how to run inference on the evaluation dataset downloaded above.

Using 1 GPU:

python run_inference_dataset.py \
  --config_path "./configs/480P.sh" \
  --path_to_dataset ./eval_data \
  --output_folder ./tmp_output/wan22_480P \
  --fps 15 \
  --skip_exist 1 \
  --save_minimal 0 \
  --max_trajectory_num 10000 \
  --cfg_scale 3.0 \
  --gpus 1

Inference with wan22 (720P)

This variant uses the same base model, further fine-tuned on the 720P dataset for 4,000 iterations starting from the 480P checkpoint.

Using 4 GPUs:

python run_inference_dataset.py \
  --config_path "./configs/720P.sh" \
  --path_to_dataset ./eval_data \
  --output_folder ./tmp_output/wan22_720P \
  --fps 15 \
  --skip_exist 1 \
  --save_minimal 0 \
  --max_trajectory_num 10000 \
  --cfg_scale 3.0 \
  --gpus 4

Output Structure

After running the inference commands above, the results will be stored in your output directory:

{output_folder}/
    {video_name}.mp4 <- Generated video (MP4 format)
    {video_name}.gif <- Generated video (GIF format)
    Visualize/
        {video_name}/
            combined.gif <- Visualization of point-track conditions, reference images, and source videos (if available)

---

💡 Understanding Input Format

Go-with-the-Track generates video by establishing direct, point-based correspondences between Reference Frames and generated videos. To achieve this, the model requires a specific directory structure and data format for inputting reference images and the point-track conditions for both reference frames and generated sequences.

📂 Example Directory Structure

eval_data/
    {Example 1}/
        ref/
          {reference_id 1}.png
          {reference_id 1}.npy
          {reference_id 2}.png
          {reference_id 2}.npy
          ...
        track.npy
        stylized_prompt.txt
        (Optional) video.mp4
        
    {Example 2}/
          ...

Explanation

Each video example must be contained in its own folder within the dataset directory.

• track.npy: The point-track conditions for generated frames.

• ref/: A folder containing 1 to 4 reference images (.png) and their corresponding reference point-tracks (.npy).

  • reference_id: This can be any arbitrary integer, but both the reference image and its corresponding reference point-track must share the same ID (e.g., 100.png and 100.npy).
  • Resolution: Reference images can be 480 x 832 or 720 x 1280 and will be resized automatically during inference.
  • Scaling: The resolution of point-tracks is typically computed differently from the resolution of the reference image (e.g., point-track conditions are typically 480 x 832, while reference images are 720 x 1280). Therefore, the point-track condition matches the reference image only up to scaling.

• video.mp4: (Optional) The source video (original video) for visualization purposes. It can be 480 x 832 or 720 x 1280.

---

Details of point-track files (.npy files) The .npy files are saved as Python dictionaries. While the generated video might be 720P, point-tracks are typically defined at a resolution of 480 x 832 for efficiency, as specified by H and W.

[!IMPORTANT]

• Track Consistency: track.npy and all files in ref/*.npy must contain the same number of point-tracks $N$. The $i$-th point-track in a reference must correspond to the $i$-th point-track in the generated frames.
• Note on Indexing: In ref/*.npy files, the shape of uvz is $(1, N, 3)$ and vis is $(1, N)$ because the reference frame is treated as a single-frame video.
• Handling Invisible Point-tracks: If a specific point-track is not visible in a particular reference/generated frame, simply set its corresponding vis value to False. In such cases, the values in uvz are ignored and can be arbitrary.

---

---

🖌️ Creating Customized Mesh Stylization Examples

Here we explain how to create your own stylization examples.

Prerequisite: Download Blender

For x86:

mkdir -p ./tmp_data
cd ./tmp_data
wget https://download.blender.org/release/Blender5.0/blender-5.0.0-linux-x64.tar.xz -O blender-5.0.0-linux-x64.tar.xz 
tar -xJf blender-5.0.0-linux-x64.tar.xz
cd blender-5.0.0-linux-x64/
./5.0/python/bin/python3.11 -m pip install tqdm joblib
cd ../../

This installs Blender in the following directory structure:

./tmp_data/
    blender-5.0.0-linux-x64/
        blender

Verify the installation:

./tmp_data/blender-5.0.0-linux-x64/blender -b

Step 1: Save Customized .blend File

Import or create an animated mesh using Blender and save it as a .blend file.

Example:

./tmp_data/
  ./mesh_example/
    mesh.blend
    mesh1.abc (make sure to save all associated files)
    mesh2.abc 

We will now convert the .blend file into our inference format. In the following scripts, intermediate results are saved at ./tmp_data/blender_output.

Step 2: Run Preprocessing (Part 1)

cd ./tmp_data/blender-5.0.0-linux-x64
./blender -b ../mesh_example/mesh.blend -P ../../data_preprocess/preprocess_blender_part1.py
cd ../../

This saves the mesh output under ./tmp_data/blender_output.

./tmp_data/
  ./blender_output/
    blender_results.pkl 
    final_video.mp4
    frame_{idx}.png

Step 3: Run Preprocessing (Part 2)

python ./data_preprocess/preprocess_blender_part2.py ./tmp_data/blender_output

This saves additional information to ./tmp_data/blender_output.

./tmp_data/
  ./blender_output/
    pred_track.pkl <- newly saved file

Step 4: Run Preprocessing (Part 3)

Here, --num_frames specifies the number of frames to extract (49 by default), and --max_tracks specifies the number of point-tracks to keep (15000 by default). These default values match the training data configurations. For longer videos, please subsample frames using the --stride flag.

Finally, --ref_idx specifies the index of the reference keyframes. For instance, --ref_idx="0,48" means we use two reference keyframes extracted from the 0th and 48th frames.

mkdir -p ./tmp_data/blender_unprocessed
mv ./tmp_data/blender_output ./tmp_data/blender_unprocessed/
python ./data_preprocess/preprocess_blender_part3.py \
  --input_folder ./tmp_data/blender_unprocessed \
  --output_folder ./tmp_data/blender_preprocessed \
  --num_frames 49 \
  --max_tracks 15000 \
  --skip_exist 1 \
  --stride 1 \
  --ref_idx "0"

This creates the preprocessed Blender data at ./tmp_data/blender_preprocessed.

Folder Structure:

./tmp_data/
  blender_preprocessed/
    blender_{start_frame_idx}/
      original.mp4 (<- Extracted video saved at the original resolution)
      video720p.mp4 (<- Extracted video saved at 720x1280 resolution)
      video.mp4 (<- Extracted video saved at 480x832 resolution)
      track.npy (<- Storing point-track conditions)
      ref/ (<- Storing reference images and point-track conditions)
        0.png
        0.npy
        ... 
      tmp_traj/ 
        traj_ref0.gif (<- Visualization)

(Note: start_frame_idx indicates the starting frame index of the extracted video.)

Step 5: Replace Reference Frames and Add Captions

Currently, ref/*.png stores the original frames. Please replace them with your customized stylized keyframes.

Our model also requires a text caption describing the video. Place this as a text file named stylized_prompt.txt inside each video example directory.

Final Inference-Ready Folder Structure:

./tmp_data/
  blender_preprocessed/
    blender_output_{start_frame_idx}/
      original.mp4
      video720p.mp4
      video.mp4
      track.npy
      stylized_prompt.txt (<- Newly created)
      ref/
        0.png (<- Updated with stylized frame)
        0.npy 

Step 6: Run Inference

You can now run inference as described in previous sections by adjusting the --path_to_dataset and --output_folder flags.

Inference with wan22 (480P) using 1 GPU:

python run_inference_dataset.py \
  --config_path "./configs/480P.sh" \
  --path_to_dataset ./tmp_data/blender_preprocessed \
  --output_folder ./tmp_output/wan22_480P \
  --fps 15 \
  --skip_exist 1 \
  --save_minimal 0 \
  --max_trajectory_num 10000 \
  --cfg_scale 3.0 \
  --gpus 1

---

🖌️ Creating Customized Video Stylization Examples

To perform video stylization, we need to estimate the underlying point-tracks. In this section, we demonstrate how to obtain point-tracks using DELTA and Video-Depth-Anything.

Prerequisite 1: Set up DELTA

Download Checkpoints:

mkdir -p ./data_preprocess/delta/checkpoints/
pip install gdown
gdown --fuzzy https://drive.google.com/file/d/18d5M3nl3AxbG4ZkT7wssvMXZXbmXrnjz/view?usp=sharing -O ./data_preprocess/delta/checkpoints/

Build Environment (CUDA 11.8, x86):

conda create -n densetrack3d python=3.10 cmake=3.14.0 -y 
conda activate densetrack3d 
conda install pytorch torchvision pytorch-cuda=11.8 -c pytorch -c nvidia -y  # use the correct version of cuda for your system
pip install pip==24.0 # downgrade pip to install pytorch_lightning==1.6.0
pip install -r ./data_preprocess/delta/requirements.txt # This may take some time
conda install ffmpeg -c conda-forge # to write .mp4 video

Install UniDepth (CUDA 11.8, x86):

pip install ninja
pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"

Prerequisite 2: Set up Video-Depth-Anything

Download Checkpoints:

mkdir -p ./data_preprocess/Video-Depth-Anything/checkpoints
wget https://huggingface.co/depth-anything/Video-Depth-Anything-Large/resolve/main/video_depth_anything_vitl.pth?download=true -O ./data_preprocess/Video-Depth-Anything/checkpoints/video_depth_anything_vitl.pth

Build Environment (CUDA 11.8, x86):

conda create -n VideoDepthAnything python=3.10
conda activate VideoDepthAnything
pip install numpy==1.24.0
pip install torch==2.1.0 torchvision --index-url https://download.pytorch.org/whl/cu118
pip install -r ./data_preprocess/Video-Depth-Anything/requirements.txt
pip install xformers==0.0.22.post4 --index-url https://download.pytorch.org/whl/cu118

Step 0: Save Source Videos

Please prepare your source video(s). Since our model is trained on 49 frames, the source videos must contain at least 49 frames. In this demo, we save all source videos under ./tmp_data/source_videos/.

./tmp_data/source_videos/
  a.mp4
  b.mp4
  {video_name}.mp4
  ...

Step 1: Pre-compute Point Tracks

Run the following commands sequentially:

# 1. Extract frames from the source videos and save to a temporary directory.
# (Specify --stride > 1 if subsampling videos)
conda activate gwtt
python ./data_preprocess/preprocess_video_part1.py --input_folder "./tmp_data/source_videos/" --output_folder "./tmp_data/tmp_videos/" --stride 1 --min_frames 49 --max_frames 49 

# 2. Compute relative video depth (required for DELTA tracking)
conda activate VideoDepthAnything
CUDA_VISIBLE_DEVICES=0 python ./data_preprocess/preprocess_video_part2.py --input_folder "./tmp_data/tmp_videos/" --input_video "./video.mp4" --encoder vitl 

# 3. Compute metric depth via UniDepth2 (required for DELTA)
conda activate densetrack3d
CUDA_VISIBLE_DEVICES=0 python ./data_preprocess/preprocess_video_part3.py --input_folder "./tmp_data/tmp_videos/" --input_video "./video.mp4" --input_depth "./video_depth_anything_disp.npy"

# 4. Run DELTA for point tracking
# --max_limit indicates the maximum number of point-tracks to estimate (default is 3000 for efficiency).
# We use the iterative query-point strategy (--iter 5) introduced in our paper to uniformly sample point-tracks.
# Lower --iter for speed, or increase it for more accurate uniform point-track coverage.
conda activate densetrack3d
CUDA_VISIBLE_DEVICES=0 python ./data_preprocess/preprocess_video_part4.py --input_folder "./tmp_data/tmp_videos/" --input_video "./video.mp4" --input_depth "./unidepth2_metric_depth.npy" --max_limit 3000 --iter 5

Current Folder Structure:

./tmp_data/
  tmp_videos/
      {video_name}/
        video.mp4 (<- 480 x 832)
        video720p.mp4 (<- 720 x 1280)
        original.mp4 (original resolution)
        track.npy (<- estimated track file)
        
        ...

        disparity_VideoDepthAnything.mp4  (<- visualization of relative disparity video depth)
        unidepth2_aligned_depth.mp4 (<- visualization of estimated metric video depth)
        traj.gif (<- visualization of estimated point-tracks)

Step 2: Convert to Inference Format

Given the estimated point-tracks and source video, convert them into the inference format:

conda activate gwtt
python ./data_preprocess/preprocess_video_part5.py --input_folder "./tmp_data/tmp_videos" --output_folder "./tmp_data/video_processed" --ref_idx "0,48"

(Note: --ref_idx="0,48" means we use keyframes extracted from the 0th and 48th frames. --ref_idx="0" means we exclusively use the first frame).

Current Folder Structure:

./tmp_data/
  video_processed/
   {video_name}/
      video720p.mp4
      video.mp4
      track.npy
      ref/
        0.png (<- updated)
        0.npy 
        ...

Step 3: Replace Reference Frames and Add Captions

Just like the mesh stylization steps, replace the original frames in ref/*.png with your stylized frames and create a stylized_prompt.txt file for each video directory.

Final inference-ready folder structure:

./tmp_data/
  video_processed/
    {video_name}/
      video720p.mp4
      video.mp4
      track.npy
      stylized_prompt.txt (<- newly created)
      ref/
        0.png (<- updated)
        0.npy 

Step 4: Run Inference

You can now run inference by pointing --path_to_dataset to ./tmp_data/video_processed.

Inference with wan22 (480P) using 1 GPU:

python run_inference_dataset.py --config_path "./configs/480P.sh" --path_to_dataset ./tmp_data/video_processed --output_folder ./output/test/wan22_480P --fps 15 --skip_exist 1 --save_minimal 0 --max_trajectory_num 10000 --cfg_scale 3.0 --gpus 1

---

✏️ Acknowledgements

We would like to thank Pablo Salamanca, Simon Su, and Nick Abrahan for their technical support; Nhat Phong Tran and Mingming He for their production support; Jeffrey Shapiro, Ritwik Kumar, and Hossein Taghavi for their executive support; Jennifer Lao and Lianette Alnaber for their operational support. Koichi acknowledges support from the Clarendon Scholarship.

This codebase is partially built on Wan2.2, DiffSynth-Studio, DELTA, Video-Depth-Anything, CoTracker, and Diffusion as Shader. We thank the authors for their open-source contributions.

📖 Citation

If you find our paper and code useful, please cite us:

@inproceedings{namekata2026gowiththetrack,
  author = {Namekata, Koichi and Kant, Yash and Liu, Zhizheng and Burgert, Ryan D. and Xu, Yuancheng and Lin, Kuan Heng and Steven, Emmett and Philip, Julien and Ma, Li and Vedaldi, Andrea and Debevec, Paul and Yu, Ning},
  title = {Go-with-the-Track: Video Compositing and Motion Control with Point Tracking},
  year = {2026},
  isbn = {979-8-4007-2554-8/2026/07},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3799902.3811093},
  doi = {10.1145/3799902.3811093},
  booktitle = {Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (SIGGRAPH Conference Papers '26), July 19--23, 2026, Los Angeles, CA, USA},
  series = {SIGGRAPH Conference Papers '26}
}