Official implementation of Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairs, a feed-forward model that can directly predict 3D Gaussians from uncalibrated images.

News

• [2024/08/27] 🔥 We release the initial version of the codebase, the paper, the project webpage, and the Gradio demo!!

Installation

1. Clone Splatt3R

git clone https://github.com/btsmart/splatt3r.git
cd splatt3r

2. Setup Anaconda Environment

conda env create -f environment.yml
pip install git+https://github.com/dcharatan/diff-gaussian-rasterization-modified

3. (Optional) Compile the CUDA kernels for RoPE (as in MASt3R and CroCo v2)

cd src/mast3r_src/dust3r/croco/models/curope/
python setup.py build_ext --inplace
cd ../../../../../../

Checkpoints

We train our model using the pretrained MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric checkpoint from the MASt3R authors, available from the MASt3R GitHub repo. This checkpoint is placed at the file path checkpoints/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth.

A pretrained Splatt3R model can be downloaded here.

Data

We use ScanNet++ to train our model. We download the data from the official ScanNet++ homepage and process the data using SplaTAM's modified version of the ScanNet++ toolkit. We save the processed data to the 'processed' subfolder of the ScanNet++ root directory.

Our generated test coverage files, and our training and testing splits, can be downloaded here, and placed in data/scannetpp.

Demo

The Gradio demo can be run using python demo.py, which loads our trained checkpoint from Hugging Face.

This demo generates a .ply file that represents the scene, which can be downloaded and rendered using online 3D Gaussian Splatting viewers such as here or here.

Our example images and .ply files are available for download here.

Training

Our training run can be recreated by running python main.py configs/main.yaml. Other configurations, such as those for the ablations, can be found in the configs folder.

BibTeX

If you find Splatt3R useful for your research and applications, please cite us using this BibTex:

@article{smart2024splatt3r,
      title={Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairs}, 
      author={Brandon Smart and Chuanxia Zheng and Iro Laina and Victor Adrian Prisacariu},
      year={2024},
      eprint={2408.13912},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2408.13912}, 
}

License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.