🖥️ Tested Configuration

SpatialFusion-LM has been tested on:

• 🐧 Ubuntu: 24.04
• 🧠 GPU: NVIDIA RTX A6000
• ⚙️ CUDA: 12.8
• 🧊 Environment: Docker container with GPU support

Other modern Ubuntu + CUDA setups may work, but this is the validated reference configuration.

A GPU with ≥ 24 GB of VRAM is recommended to ensure stable real-time inference and efficient handling of high-resolution inputs across all components.

🚀 Quick Start

1. Clone the repo

git clone --recursive https://github.com/jagennath-hari/SpatialFusion-LM.git && cd SpatialFusion-LM

2. Download model weights and sample dataset

bash scripts/download_weights.sh && bash scripts/download_sample.sh

3. Run the demo inside Docker

bash run_container.sh
ros2 launch llm_ros llm_demo.launch.py

Change modes using mode:=mono/mono+/stereo

🎮 Supported Modes

📖 Overview

🔧 Features

• 📷 Supports monocular, monocular+ and stereo vision
• 🔍 Neural depth estimation with metric 3D reconstruction
• 🧱 Differentiable point cloud generation in the camera frame
• 🧠 Language-conditioned spatial layout prediction
• 🧩 Modular ROS 2 architecture (plug-and-play components)
• 🌀 Real-time inference and visualization
• 📊 Integrated logging via Rerun

✅ TODO

• Create end-to-end inference using Triton Inference Server via ensemble models
• Quantize UniK3D and FoundationStereo to TensorRT engines
• Fix UniK3D bug in CUDA Memeory stream for async error while running SpatialLM in parallel
• (2025-04-24) Expand evaluation to TUM, Replica, and with one custom recorded dataset

🗃️ Download TUM and Replica ROS 2 datasets

This script will prompt you to select one or more datasets to download:

bash scripts/download_dataset.sh

⚙️ Launch Configuration Options

The llm_demo.launch.py file accepts the following arguments:

📸 Mono, 📷 Mono+, 📷 📷 Stereo?

+---------------------------------+
|              mode=?             |
+---------------------------------+
                |
  ┌─────────────┴──────────────┐
  │             │              │
 mono         mono+          stereo
  │             │              │
 rgb           rgb           camera
              intr.           intr.
                +              +
               rgb           stereo
                              pair
                               +
                            baseline

🤖 Mode Descriptions

• mono – Only RGB image is provided.
  UniK3D internally estimates camera intrinsics and uses them to predict metric (absolute) depth.
  While this enables 3D reconstruction without calibration, the accuracy depends on the quality of intrinsic estimation.
  🚀 Suitable for quick deployment or uncalibrated cameras.

• mono+ – RGB image and accurate camera intrinsics are provided.
  UniK3D uses the supplied intrinsics to produce more accurate metric depth, with better scale alignment.
  🧪 Ideal for calibrated cameras (e.g., using /camera_info).

• stereo – Left and right rectified images, intrinsics, and baseline are required.
  FoundationStereo uses a ViT-based architecture to predict dense disparity maps from stereo pairs.
  Metric depth is then computed using the stereo baseline and intrinsics, and converted to a 3D point cloud.
  🛡️ This mode provides the most robust and accurate depth, especially in structured or texture-rich environments.

🖼️ Demo Gallery

Below are example configurations showing how SpatialFusion-LM behaves with different launch options.

---

📸 Mono 🧠 SpatialLM Disabled (mono, rerun)

ros2 launch llm_ros llm_demo.launch.py mode:=mono spatialLM:=false rerun:=true rviz:=false

SpatialFusion-LM performing monocular estimation and 3D reconstruction on TUM scene xyz.

📷 📷 Stereo 🧠 SpatialLM Disabled (mono, rerun)

ros2 launch llm_ros llm_demo.launch.py mode:=stereo spatialLM:=false rerun:=true rviz:=false

SpatialFusion-LM performing stereo estimation and 3D reconstruction on indoor scene indoor_0.

🧪 Run with TUM Dataset

SpatialFusion-LM supports pre-recorded ROS 2 bags from the TUM RGB-D dataset. The llm_demo_tum.launch.py launch file is preconfigured and mono or mono+ modes depending on intrinsics.

ros2 launch llm_ros llm_demo_tum.launch.py \
  mode:=mono+ \
  bag_path:=/datasets/tum_office \
  spatialLM:=true \
  rerun:=true \
  rviz:=true

This assumes you have already downloaded the ROS 2 TUM dataset. If not, you can follow the provided script scripts/download_dataset.sh to do this.

🧪 Run with Replica Dataset

SpatialFusion-LM supports pre-recorded ROS 2 bags from the Replica dataset. The llm_demo_replica.launch.py launch file is preconfigured and mono or mono+ modes depending on intrinsics.

ros2 launch llm_ros llm_demo_replica.launch.py \
  mode:=mono+ \
  bag_path:=/datasets/replica_office2 \
  spatialLM:=true \
  rerun:=true \
  rviz:=true

This assumes you have already downloaded the ROS 2 Replica dataset. If not, you can follow the provided script scripts/download_dataset.sh to do this.

🛠️ Using SpatialFusion-LM with Your Own ROS 2 Topics

To run SpatialFusion-LM on a live ROS 2 system or your own dataset:

1️⃣ Use llm.launch.py for direct topic-level control

This version of the launch file allows you to specify raw topic names directly (no bag playback or auto setup). Examples:

This simulates mode:=mono as there is no rgb_info provided.

ros2 launch llm_ros llm.launch.py \
  rgb_image:=/your_camera/image_rect \
  rerun:=true \
  spatialLM:=true

This simulates mode:=mono+ as rgb_info is provided.

ros2 launch llm_ros llm.launch.py \
  rgb_image:=/your_camera/image_rect \
  rgb_info:=/your_camera/camera_info \
  rerun:=true \
  spatialLM:=true

This simulates mode:=stereo as left and right topics, left_info and right_info, and baseline are provided.

ros2 launch llm_ros llm.launch.py \
  left_image:=/stereo/left/image_rect \
  right_image:=/stereo/right/image_rect \
  left_info:=/stereo/left/camera_info \
  right_info:=/stereo/right/camera_info \
  baseline:=0.12 \
  rerun:=true \
  spatialLM:=true

2️⃣ Parameter Descriptions

ros2 launch llm_ros llm.launch.py -s

📤 Output Topics

These are the outputs published by the core_node.py:

⚖️ Depth Comparison

Comparison of predicted depth maps from stereo, mono+, and mono modes respectively.

📈 Performance Benchmarks

Measured on a single NVIDIA RTX A6000 (48 GB VRAM) at 1920×1080 resolution. Input images are automatically resized to model-specific inference resolution, and values may vary depending on hardware, backend load, and ROS 2 message overhead.

Measured using Headless mode without Rerun or RVIZ.

A signficant VRAM of ~8192 MiB is needed when spatialLM:=true.

🤝 Contributing

I welcome pull requests and suggestions! If you want to add a new dataset, model backend, or visualization utility, open an issue or fork this repo!

If you find any bug in the code, please report to jh7454@nyu.edu

📖 Citation

If you found this code/work to be useful in your own research, please considering citing the following:

@article{wen2025stereo,
  title={FoundationStereo: Zero-Shot Stereo Matching},
  author={Bowen Wen and Matthew Trepte and Joseph Aribido and Jan Kautz and Orazio Gallo and Stan Birchfield},
  journal={CVPR},
  year={2025}
}

@inproceedings{piccinelli2025unik3d,
    title     = {{U}ni{K3D}: Universal Camera Monocular 3D Estimation},
    author    = {Piccinelli, Luigi and Sakaridis, Christos and Segu, Mattia and Yang, Yung-Hsu and Li, Siyuan and Abbeloos, Wim and Van Gool, Luc},
    booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year      = {2025}
}

@misc{spatiallm,
  title        = {SpatialLM: Large Language Model for Spatial Understanding},
  author       = {ManyCore Research Team},
  howpublished = {\url{https://github.com/manycore-research/SpatialLM}},
  year         = {2025}
}

📄 License

This software is released under the GNU General Public License v3.0 (GPL-3.0). You are free to use, modify, and distribute this code under the terms of the license, but derivative works must also be open-sourced under GPL-3.0.

🙏 Acknowledgement

This work integrates several powerful research papers, libraries, and open-source tools:

• FoundationStereo
• UniK3D
• SpatialLM
• Rerun