DSR Suite: Learning to Reason in 4D

Paper | Code

DSR Suite is a comprehensive resource designed to bridge the gap in dynamic spatial reasoning (DSR) for vision-language models (VLMs). It addresses the challenge of reasoning about the evolution of object geometry and relationships in 3D space over time (4D understanding).

Dataset Summary

The suite includes two primary data components constructed from in-the-wild videos using an automated pipeline:

• DSR-Train: A large-scale training dataset featuring 50,000 automated multiple-choice question-answer pairs (QAs).
• DSR-Bench: A high-quality benchmark consisting of 1,484 QAs further refined by human annotators to evaluate object- and scene-level 3D understanding, viewpoint transformations, and multi-object interactions.

The data generation pipeline leverages vision foundation models to extract geometric cues such as camera poses, local point clouds, object masks, orientations, and 3D trajectories to create fine-grained, procedural answers.

Sample Usage

The benchmark is designed for evaluation using a modified version of VLMEvalKit. To evaluate a model on the Spatial-Reasoning task, you can use the following command structure provided in the official repository:

cd VLMEvalKit_mine
CUDA_VISIBLE_DEVICES=0 python run.py --data Spatial-Reasoning --model YOUR_MODEL --work-dir spatial_reasoning

For detailed instructions on setting up the environment for data generation or model training, please refer to the official GitHub repository.

Citation

If you find our work useful, please consider citing:

@misc{zhou2025learning,
      title={Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models}, 
      author={Shengchao Zhou, Yuxin Chen, Yuying Ge, Wei Huang, Jiehong Lin, Ying Shan, Xiaojuan Qi},
      year={2025},
      eprint={2512.20557},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2512.20557}, 
}

Acknowledgement

• Qwen2.5-VL: the model codebase we built upon.
• VLMEvalKit: the evaluation framework we built upon.
• Grounded SAM2, Orient Anything, π^3: the models used in our data generation pipeline to extract 3D cues.
• Koala-36M: the video database we build QAs upon.