[!IMPORTANT] [Aug. 9, 2026] PROVENANCE UPDATE: The existing "Debiased" subset is VSI-Bench-Debiased v1, a designer-in-the-loop manual pilot created with bespoke per-question-type filtering heuristics. It predates and was not generated by the automated Iterative Bias Pruning (IBP) algorithm. We retain v1 for reproducibility and will version any future automated subset separately.

Visual-Spatial Intelligence Benchmark (VSI-Bench & VSI-Bench-Debiased v1)

This repository contains the visual spatial intelligence benchmark (VSI-Bench), introduced in Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces, and the designer-in-the-loop pilot VSI-Bench-Debiased v1, analyzed in our follow-up work Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts.

Overview

VSI-Bench evaluates visual-spatial intelligence of multimodal models through egocentric video understanding, comprising over 5,000 question-answer pairs from real-world indoor scenes.

VSI-Bench-Debiased v1 is a manual proof-of-concept subset produced before the automated IBP method was formalized. Its per-question-type heuristics preferentially remove samples with non-visual shortcuts, but they also change the question-type distribution materially. Treat v1 as a separate evaluation artifact, not a drop-in replacement for the original benchmark or an automated-IBP output.

Description

VSI-Bench quantitatively evaluates the visual-spatial intelligence of MLLMs from egocentric video. VSI-Bench comprises over 5,000 question-answer pairs derived from 288 real videos. These videos are sourced from the validation sets of the public indoor 3D scene reconstruction datasets ScanNet, ScanNet++, and ARKitScenes, and represent diverse environments -- including residential spaces, professional settings (e.g., offices, labs), and industrial spaces (e.g., factories) and multiple geographic regions. By repurposing these existing 3D reconstruction and understanding datasets, VSI-Bench benefits from accurate object-level annotations, which are used in question generation and could support future studies exploring the connection between MLLMs and 3D reconstruction.

Fields

The dataset contains the following fields:

Why VSI-Bench-Debiased?

While the original VSI-Bench was designed to require visual understanding, our follow-up analysis revealed that a portion of questions could be answered using non-visual shortcuts—such as statistical biases in answer distributions or world knowledge priors—without actually processing the visual input.

VSI-Bench-Debiased v1 addresses this through a designer-in-the-loop pilot: each question type has a bespoke statistical filtering strategy and a manually chosen removal budget. The resulting subset removes 2,768 of 5,130 questions and retains 2,362. All ten question types remain, but retention varies substantially by type; this limitation is documented in the paper.

Key improvements in VSI-Bench-Debiased:

• Reduced non-visual solvability: Blind models (text-only, no vision) perform closer to chance
• Wider vision-blind gap: Greater performance difference between vision-enabled and vision-disabled models
• Better isolation of visual reasoning: Fine-tuning on in-distribution data improves vision-enabled performance much more than blind performance, confirming reduced shortcut reliance

For researchers interested in robust evaluation of visual-spatial intelligence, we recommend reporting results on both the full and v1 subsets, clearly labeling v1 and avoiding direct leaderboard comparisons between differently sized subsets.

Usage

Dataset Configurations

This dataset provides three configurations for flexible evaluation:

Loading the Dataset Annotations

Load specific configuration

If you want to load just a specific subset, you can use the config name with the load_dataset function as follows:

from datasets import load_dataset

# Load full dataset (default)
vsi_bench_full = load_dataset("nyu-visionx/VSI-Bench")
# or use the config name "full"
vsi_bench_full = load_dataset("nyu-visionx/VSI-Bench", "full")

# Load debiased version only
vsi_bench_debiased = load_dataset("nyu-visionx/VSI-Bench", "debiased")

# Load pruned examples only
vsi_bench_pruned = load_dataset("nyu-visionx/VSI-Bench", "pruned")

Load full dataset and filter using pruned column (recommended)

[!TIP] For LMMS-Eval users: We have updated the vsi-bench task to automatically report scores on both full and debiased subsets. (TODO: LINK).

We recommend loading the "full" set, evaluating on all samples, and then using the pruned column to compute scores on both the full and debiased subsets.

from datasets import load_dataset

# Load full dataset with pruned annotations
vsi_bench_full = load_dataset("nyu-visionx/VSI-Bench")

# Evaluate on full set
model_predictions = evaluate_model(vsi_bench_full)

# Score on both the full and debiased subsets
full_acc = compute_accuracy(model_predictions)
debiased_acc = compute_accuracy(model_predictions.filter(lambda x: not x["pruned"]))

Evaluation

[!TIP] TODO: link to the LMMS Eval Code

VSI-Bench evaluates performance using two metrics: for multiple-choice questions, we use Accuracy, calculated based on exact matches. For numerical-answer questions, we introduce a new metric, MRA (Mean Relative Accuracy), to assess how closely model predictions align with ground truth values.

We provide an out-of-the-box evaluation of VSI-Bench in our GitHub repository, including the metrics implementation used in our framework. For further detailes, users can refer to our paper and GitHub repository.

Files

• test-*.parquet: Parquet files containing dataset annotations (questions, answers, metadata).
  • test_debiased.parquet: Annotations retained by the v1 pilot (2,362 examples)
  • test_pruned.parquet: Annotations removed by the v1 pilot (2,768 examples)
• *.zip: Compressed video files for the dataset
  • arkitscenes.zip: Videos for the ARKitScenes dataset
  • scannet.zip: Videos for the ScanNet dataset
  • scannetpp.zip: Videos for the ScanNet++ dataset
• pruned_ids.txt: List of example IDs removed by the v1 manual pilot
• create_pq.py: Convenience script to regenerate parquet files from test.jsonl and pruned_ids.txt. Can be run with uv run create_pq.py.

Citation

If you use these datasets in your research, please cite the original VSI-Bench paper and our debiasing paper that produced VSI-Bench-Debiased:

@inproceedings{yang2025thinking,
    title={{Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces}},
    author={Yang, Jihan and Yang, Shusheng and Gupta, Anjali and Han, Rilyn and Fei-Fei, Li and Xie, Saining},
    booktitle={CVPR},
    year={2025},
}

@article{brown2025benchmark,
    title={{Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts}},
    author={Brown, Ellis and Yang, Jihan and Yang, Shusheng and Fergus, Rob and Xie, Saining},
    year={2025},
    journal={arXiv preprint arXiv:2511.04655},
}