VSTI-Bench

VSTI-Bench is a benchmark for visual-spatial temporal intelligence, evaluating video models on camera- and object-centric spatial reasoning over ScanNet videos. It accompanies the VLM-3R project.

• Videos: ScanNet.tar.gz (val-split ScanNet videos)
• Questions: test.json (5,736 QA items across 9 question types)

Erratum (2026-07-13): corrected camera-position ground truth

A bug in the ground-truth generation pipeline was reported by Jacob Yeung (CMU), whom we thank for the careful analysis. The camera center was extracted from camera-to-world poses using -R.T @ t (the world-to-camera translation) instead of the correct pose[:3, 3]. This affected every label that depends on the camera's world position.

Affected question types (labels corrected in this revision):

camera_movement_direction and obj_obj_relative_pos_lr/nf/ud (2,558 items) used the correct camera-center formula and are unchanged.

Removed items are those whose corrected geometry violates the generators' original validity filters (displacement/distance range, a 0.15 m ambiguity margin between multiple-choice options, and a 0.5 m minimum object distance). They are listed with reasons in erratum/dropped_items.json.

The pre-correction labels remain available in the git history of this repository (revisions before this erratum commit). Results computed on the old labels are not comparable to results on this revision. Note that models fine-tuned on training data generated with the buggy pipeline should also be retrained on corrected training data for a fair comparison.

The corrected generation code and the correction script (vlm_3r_data_process/scripts/vstibench/fix_vstibench_gt_labels.py) are available in the VLM-3R repository.