Spatial Mosaic

This repository contains training and evaluation code for Spatial Mosaic.

Use /spatial_mosaic below as the repository root.

Data Format and Examples

We provide two scene sources for Spatial Mosaic:

Here, {scene_id} denotes each actual scene folder name, such as 0a5c013435.

The Hugging Face dataset linked below provides only the Spatial Mosaic VQA annotations. It does not redistribute the original ScanNet++ or Waymo scene images. Download the image data from the official dataset sources and follow their access requirements, licenses, terms of use, citation rules, and privacy

The VQA annotation license does not grant any rights to redistribute or use the underlying ScanNet++ or Waymo images outside their original dataset terms.

Our dataset directory structure is:

spatial_mosaic_dataset/
├── scannetpp/
│   └── {scene_id}/
│       └── images/
├── waymo/
│   └── {scene_id}/
│       └── images/
└── spatial_mosaic_vqa/
    ├── train/
    └── test/

Download VQA Annotations

The Spatial Mosaic VQA annotations are hosted on Hugging Face: https://huggingface.co/datasets/anonymoussubmmision/spatial_mosaic_vqa

This archive contains the VQA JSON annotations only; follow the license and usage terms listed on the Hugging Face dataset card. Install or update the Hugging Face CLI, then download and extract the VQA archive under the dataset root:

cd spatial_mosaic
python -m pip install -U huggingface_hub

mkdir -p spatial_mosaic_dataset
hf download anonymoussubmmision/spatial_mosaic_vqa \
  spatial_mosaic_vqa.tar.gz SHA256SUMS \
  --repo-type dataset \
  --local-dir spatial_mosaic_dataset

cd spatial_mosaic_dataset
sha256sum -c SHA256SUMS
tar -xzf spatial_mosaic_vqa.tar.gz

After extraction, the VQA JSON files should be available under spatial_mosaic_dataset/spatial_mosaic_vqa/train/ and spatial_mosaic_dataset/spatial_mosaic_vqa/test/.

Train

Environmental Setup

Create and activate the training environment:

cd spatial_mosaic
conda env create -f scripts/environment.yaml
conda activate spatial_mosaic

Install VGGT from the vendored source:

cd spatial_mosaic
cd vggt
pip install -e .
cd ..

Set PYTHONPATH from both the spatial_mosaic root and the llava directory:

PYTHONPATH=$(pwd):$PYTHONPATH
cd llava
PYTHONPATH=$(pwd):$PYTHONPATH
cd ..

How To Train

Training scripts are provided for both indoor and outdoor scenes, with standard and VGGT variants. The example below shows indoor LLaVA-NeXT training.

Before launching training, update the indoor config and script with your data paths:

• scripts/model/spatial_mosaic/indoor/indoor.yaml
• scripts/model/spatial_mosaic/indoor/train_llavanext.sh

In train_llavanext.sh, set the required paths such as FRAME_FOLDER, IMAGE_FOLDER, and VIDEO_FOLDER.

Then run training from the spatial_mosaic root:

CUDA_VISIBLE_DEVICES=2,3 NUM_GPUS_PER_NODE=2 bash scripts/model/spatial_mosaic/indoor/train_llavanext.sh

Eval

Environmental Setup

Create the evaluation environment:

cd spatial_mosaic
conda env create -f thinking-in-space/spatial_mosaic/environment.yaml
conda activate vsibench

How To Eval

Evaluation scripts are provided for both indoor and outdoor scenes, with standard and VGGT variants. The example below shows indoor LLaVA-NeXT evaluation.

Before launching evaluation, update the indoor evaluation script and task files with your checkpoint and data paths:

• /spatial_mosaic/thinking-in-space/spatial_mosaic/indoor/eval_llavanext.sh Set pretrained to your checkpoint path.
• /spatial_mosaic/thinking-in-space/lmms_eval/tasks/spatial_mosaic/indoor/indoor.yaml Set the VQA data path.
• /spatial_mosaic/thinking-in-space/lmms_eval/tasks/spatial_mosaic/indoor/utils.py Set FRAMES_ROOT = "path_to_img".

Then run evaluation from the indoor evaluation directory:

cd thinking-in-space/spatial_mosaic/indoor
bash eval_llavanext.sh