🎯 Spatial Perception And Reasoning Benchmark (SPAR-Bench)

A benchmark to evaluate spatial perception and reasoning in vision-language models (VLMs), with high-quality QA across 20 diverse tasks.

SPAR-Bench is a high-quality benchmark for evaluating spatial perception and reasoning in vision-language models (VLMs). It covers 20 diverse spatial tasks across single-view, multi-view, and video settings, with a total of 7,207 manually verified QA pairs.

SPAR-Bench is derived from the large-scale SPAR-7M dataset, and specifically designed to support zero-shot evaluation and task-specific analysis

📌 SPAR-Bench at a glance:

• ✅ 7,207 manually verified QA pairs
• 🧠 20 spatial tasks (depth, distance, relation, imagination, etc.)
• 🎥 Supports single-view, multi-view inputs
• 📏 Two evaluation metrics: Accuracy & MRA
• 📷 Available in RGB-only and RGB-D versions

🧱 Available Variants

We provide four versions of SPAR-Bench, covering both RGB-only and RGB-D settings, as well as full-size and lightweight variants:

🔎 Tiny versions are designed for quick evaluation (e.g., APIs, human studies).
💡 RGBD versions include depths, poses, and intrinsics, suitable for 3D-aware models.

To load a different version via datasets, simply change the dataset name:

from datasets import load_dataset
spar = load_dataset("jasonzhango/SPAR-Bench")
spar_rgbd = load_dataset("jasonzhango/SPAR-Bench-RGBD")
spar_tiny = load_dataset("jasonzhango/SPAR-Bench-Tiny")
spar_tiny_rgbd = load_dataset("jasonzhango/SPAR-Bench-Tiny-RGBD")

🕹️ Evaluation

SPAR-Bench supports two evaluation metrics, depending on the question type:

• Accuracy – for multiple-choice questions (exact match)
• Mean Relative Accuracy (MRA) – for numerical-answer questions (e.g., depth, distance)

🧠 The MRA metric is inspired by the design in Thinking in Space, and is tailored for spatial reasoning tasks involving quantities like distance and depth.

We provide an evaluation pipeline in our GitHub repository, built on top of lmms-eval.

📚 Bibtex

If you find this project or dataset helpful, please consider citing our paper:

@article{zhang2025from,
    title={From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D},
    author={Zhang, Jiahui and Chen, Yurui and Zhou, Yanpeng and Xu, Yueming and Huang, Ze and Mei, Jilin and Chen, Junhui and Yuan, Yujie and Cai, Xinyue and Huang, Guowei and Quan, Xingyue and Xu, Hang and Zhang, Li},
    year={2025},
    journal={arXiv preprint arXiv:2503.22976},
}