RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics

🌐 Homepage | 📖 arXiv | 🛠️ Data Gen | 🧪 Eval Code

🔥 Core spatial understanding benchmark used by Qwen3-VL and Gemini Robotics!

⚠️ Important Note

Annotation Correction (08/14/25): In the context category, the microwave question was corrected from “below” to “above” to fix an annotation error.

Annotation Correction (05/13/26): In the configuration category, the speaker/cup question answer was corrected from “Yes” to “No” to fix an annotation error.

📌 Dataset Versions

What changed in v1: The spatial context questions use a more explicit prompt template that better reflects the placement intent. For example:

• main: "Pinpoint several points within the vacant space situated to the left of the bowl."
• v1: "Point to several empty spots to the left of the bowl where another object could be placed."

Dataset Description

We introduce RoboSpatial-Home: a new spatial reasoning benchmark designed to evaluate vision-language models (VLMs) in real-world indoor environments for robotics. It consists of 350 spatial reasoning questions paired with crowd-sourced RGBD images captured using a handheld iPhone camera equipped with a depth sensor. Each image is annotated with three types of spatial relationship questions—spatial configuration, spatial context, and spatial compatibility—providing a comprehensive evaluation of spatial understanding in robotic applications.

Dataset Structure

RoboSpatial-Home consists of QA annotations paired with RGB and depth images. The dataset is organized as follows:

• category: The spatial reasoning category for the entry (configuration, context, or compatibility).
• question: The spatial reasoning question.
• answer: The human annotated answer.
• img: The RGB image from iPhone 13 Pro Max.
• depth_image: The corresponding depth image from iPhone 13 Pro Max.
• mask: (If available) The corresponding segmentation mask for spatial context questions.

QA types:

• Spatial Configuration: Determines the relative positioning of objects (e.g., "Is the mug to the left of the laptop?").
• Spatial Context: Identifies vacant areas in relation to a reference object (e.g., "Identify empty space to the left of the bowl.").
• Spatial Compatibility: Assesses whether an object can fit within a specified area (e.g., "Can the chair be placed in front of the desk?").

Load Dataset

You can load the RoboSpatial-Home dataset in two ways:

1. Using the Hugging Face datasets library

import datasets

dataset_name = 'chanhee-luke/RoboSpatial-Home'
data = load_dataset(dataset_name, CATEGORY)

where CATEGORY is one of the spatial reasoning categories: configuration, context, compatibility. If not specified, the entire dataset will be loaded.

2. Downloading locally with the script If you prefer to work with local files, the RoboSpatial-Eval repo provides a script:

python download_benchmark.py [OUTPUT_FOLDER_PATH]

This downloads the dataset locally for debugging or for setups that don’t use the Hugging Face datasets library. If no output path is provided, the dataset will be saved to ./RoboSpatial-Home by default.

Dataset Creation

The data for RoboSpatial-Home was manually collected and annotated by graduate-level students in computer science.

Disclaimers

⚠️ Disclaimer: The images in this dataset were collected from real homes of real individuals. When using or distributing this dataset, ensure that privacy and ethical considerations are upheld. Redistribution of images should be done with caution to respect the privacy of the original contributors.

Contact

• Luke Song: song.1855@osu.edu

Citation

BibTex:

@inproceedings{song2025robospatial,
  author    = {Song, Chan Hee and Blukis, Valts and Tremblay, Jonathan and Tyree, Stephen and Su, Yu and Birchfield, Stan},
  title     = {{RoboSpatial}: Teaching Spatial Understanding to {2D} and {3D} Vision-Language Models for Robotics},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2025},
  note      = {Oral Presentation},
}