Feat2GS: Probing Visual Foundation Models with Gaussian Splatting

Yue Chen, Xingyu Chen, Anpei Chen, Gerard Pons-Moll, Yuliang Xiu

https://github.com/user-attachments/assets/07ebb8e1-6001-47bf-bf74-984b0032cc17

Updates

• [July 10, 2025] Add a new evaluation of VGGT encoder and decoder features. See the results here.

Get Started

Installation

1. Clone Feat2GS and download pre-trained model from DUSt3R/MASt3R.

git clone https://github.com/fanegg/Feat2GS.git
cd Feat2GS/submodules/mast3r/
mkdir -p checkpoints/
wget https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth -P checkpoints/
wget https://download.europe.naverlabs.com/ComputerVision/MASt3R/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth -P checkpoints/
cd ../../

2. Create the environment, here we show an example using conda.

conda create -n feat2gs python=3.11 cmake=3.14.0
conda activate feat2gs
pip install "torch==2.5.1" "torchvision==0.20.1" "numpy<2" --index-url https://download.pytorch.org/whl/cu121  # use the correct version of cuda for your system
cd Feat2GS/
pip install -r requirements.txt
pip install submodules/simple-knn

3. Optional but highly suggested, compile the cuda kernels for RoPE (as in CroCo v2).

# DUST3R relies on RoPE positional embeddings for which you can compile some cuda kernels for faster runtime.
cd submodules/mast3r/dust3r/croco/models/curope/
python setup.py build_ext --inplace
cd ../../../../../../

4. (Optional) follow this instruction to install requirements for probing Zero123.

Usage

1. Data preparation (We provide our evaluation and inference datasets: link)

  cd <data_root>/Feat2GS/

If you want to build custom datasets, please follow and edit:

build_dataset/0_create_json.py ## create dataset_split.json to split train/test set
build_dataset/1_create_feat2gs_dataset.py ## use dataset_split.json to create dataset

2. Evaluate Visual Foundation Models:

  # Run evaluation for all datasets, all VFM features, all probing modes
  bash scripts/run_feat2gs_eval_parallel.sh

  # (Example) Run evaluation for a single scene, DINO feature, Geometry mode
  bash scripts/run_feat2gs_eval.sh

[!NOTE] To run experiments in parallel, we added a GPU lock feature to ensure only one evaluation experiment runs per GPU. Once an experiment finishes, the GPU is automatically unlocked. If interrupted by Ctrl+C, the GPU won’t be unlocked automatically. To fix this, manually delete the .lock files in your LOCK_DIR. To disable this feature, comment out these lines in the script: L4-L5, L9-L22, L223-L233, L330-L331.

  # Evaluate Visual Foundation Models on DTU dataset
  bash scripts/run_feat2gs_eval_dtu_parallel.sh

  # Run InstantSplat for evaluation
  bash scripts/run_instantsplat_eval_parallel.sh

3. After training, render RGB/depth/normal video with generated trajectory.

  # If render depth/normal, set RENDER_DEPTH_NORMAL=true
  # Set type of generated trjectory by editing <TRAJ_SCENES>
  bash scripts/run_video_render.sh

  # Render video on DTU dataset
  bash scripts/run_video_render_dtu.sh

🎮 Interactive demo

🚀 Quickstart

1. Input Images

• Upload 2 or more images of the same scene from different views
• For best results, ensure images have good overlap

2. Step 1: DUSt3R Initialization & Feature Extraction

• Click "RUN Step 1" to process your images
• This step estimates initial DUSt3R point cloud and camera poses, and extracts DUSt3R features for each pixel

3. Step 2: Readout 3DGS from Features

• Set the number of training iterations, larger number leads to better quality but longer time (default: 2000, max: 8000)
• Click "RUN Step 2" to optimize the 3D model

4. Step 3: Video Rendering

• Choose a camera trajectory
• Click "RUN Step 3" to generate a video of your 3D model

gradio demo.py

💡 Tips

• Processing time depends on image resolution and quantity
• For optimal performance, test on high-end GPUs (A100/4090)
• Use the mouse to interact with 3D models:
  • Left button: Rotate
  • Scroll wheel: Zoom
  • Right button: Pan

Acknowledgement

This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!

• Gaussian-Splatting and diff-gaussian-rasterization
• gsplat
• DUSt3R and MASt3R
• InstantSplat
• Probe3D
• FeatUp
• Shape of Motion
• Splatt3R
• VGGT

Citation

If you find our work useful in your research, please consider giving a star :star: and citing the following paper :pencil:.

@inproceedings{chen2025feat2gs,
  title={Feat2gs: Probing visual foundation models with gaussian splatting},
  author={Chen, Yue and Chen, Xingyu and Chen, Anpei and Pons-Moll, Gerard and Xiu, Yuliang},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={6348--6361},
  year={2025}
}

Contact

For feedback, questions, or press inquiries please contact Yue Chen and Xingyu Chen.