This repository contains the official PyTorch implementation of SpatialScore: https://arxiv.org/abs/2505.17012/.
Our new version paper has been accepted by CVPR 2026, and we have updated our up-to-date code and data! Feel free to reach out for discussions!
Current Leaderboard (You are welcome to test your models on SpatialScore!):
Project Page · Paper · SpatialScore_Benchmark · SpatialCorpus · Model
The aforementioned dependencies are necessary for conducting evaluations on SpatialScore. If you intend to utilize SpatialAgent; since it requires invoking various spatial perception tools, you may need to consult the following repositories to install the corresponding tool dependencies, and download their corresponding pre-trained checkpoints, including Rex-Omni, Map-Anything, GroundingDINO and DetAny3D.
A suitable conda environment named SpatialScore can be created and activated with:
conda env create -f environment.yaml
conda activate SpatialScore
Please check out SpaitalScore to download our proposed benchmark (SpatialScore).
If you cannot access Huggingface, you can use hf-mirror to download models.
export HF_ENDPOINT=https://hf-mirror.com # Add this before huggingface-cli download
You can follow the commands below to prepare the data:
huggingface-cli download --resume-download --repo-type dataset haoningwu/SpatialScore --local-dir ./ --local-dir-use-symlinks False
unzip SpatialScore_benchmark.zip
Considering the current mainstream model architectures, we have prioritized support for the Qwen2.5-VL and Qwen3-VL series models. You can evaluate them on SpatialScore using the following commands:
CUDA_VISIBLE_DEVICES=0,1 python test_qwen.py --model_name qwen3vl-4b --model_path ./huggingface/Qwen3-VL-4B-Instruct --dataset_json_path ./SpatialScore_benchmark/SpatialScore_benchmark.ndjson --output_dir ./eval_results
Now, the All-in-one script supporting all other models is also available. You can evaluate other models on SpatialScore using the following commands:
CUDA_VISIBLE_DEVICES=0,1 python test_all_in_one.py --model_name llava-ov-7b --model_path ../huggingface/LLaVA-OneVision-7B --dataset_json_path ./SpatialScore_benchmark/SpatialScore_benchmark.ndjson --output_dir ./eval_results
Our final evaluation encompassed rule-based evaluation and LLM-based answer extraction, which are combined to calculate the final accuracy. Therefore, you need to configure GPT-OSS and download the corresponding GPT-OSS-20B checkpoint before running the following script to compute the final score:
MKL_THREADING_LAYER=GNU CUDA_VISIBLE_DEVICES=0 python ./evaluate_results.py --input ./eval_results/qwen3vl-4b
If you use this code and data for your research or project, please cite:
@inproceedings{wu2026spatialscore,
author = {Wu, Haoning and Huang, Xiao and Chen, Yaohui and Zhang, Ya and Wang, Yanfeng and Xie, Weidi},
title = {SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2026},
}
Many thanks to the code bases from transformers, Qwen3-VL, and TACO.
If you have any questions, please feel free to contact haoningwu3639@gmail.com.
26 commits
This repository contains the official PyTorch implementation of SpatialScore: https://arxiv.org/abs/2505.17012/.
Our new version paper has been accepted by CVPR 2026, and we have updated our up-to-date code and data! Feel free to reach out for discussions!
Current Leaderboard (You are welcome to test your models on SpatialScore!):
Project Page · Paper · SpatialScore_Benchmark · SpatialCorpus · Model
The aforementioned dependencies are necessary for conducting evaluations on SpatialScore. If you intend to utilize SpatialAgent; since it requires invoking various spatial perception tools, you may need to consult the following repositories to install the corresponding tool dependencies, and download their corresponding pre-trained checkpoints, including Rex-Omni, Map-Anything, GroundingDINO and DetAny3D.
A suitable conda environment named SpatialScore can be created and activated with:
conda env create -f environment.yaml
conda activate SpatialScore
Please check out SpaitalScore to download our proposed benchmark (SpatialScore).
If you cannot access Huggingface, you can use hf-mirror to download models.
export HF_ENDPOINT=https://hf-mirror.com # Add this before huggingface-cli download
You can follow the commands below to prepare the data:
huggingface-cli download --resume-download --repo-type dataset haoningwu/SpatialScore --local-dir ./ --local-dir-use-symlinks False
unzip SpatialScore_benchmark.zip
Considering the current mainstream model architectures, we have prioritized support for the Qwen2.5-VL and Qwen3-VL series models. You can evaluate them on SpatialScore using the following commands:
CUDA_VISIBLE_DEVICES=0,1 python test_qwen.py --model_name qwen3vl-4b --model_path ./huggingface/Qwen3-VL-4B-Instruct --dataset_json_path ./SpatialScore_benchmark/SpatialScore_benchmark.ndjson --output_dir ./eval_results
Now, the All-in-one script supporting all other models is also available. You can evaluate other models on SpatialScore using the following commands:
CUDA_VISIBLE_DEVICES=0,1 python test_all_in_one.py --model_name llava-ov-7b --model_path ../huggingface/LLaVA-OneVision-7B --dataset_json_path ./SpatialScore_benchmark/SpatialScore_benchmark.ndjson --output_dir ./eval_results
Our final evaluation encompassed rule-based evaluation and LLM-based answer extraction, which are combined to calculate the final accuracy. Therefore, you need to configure GPT-OSS and download the corresponding GPT-OSS-20B checkpoint before running the following script to compute the final score:
MKL_THREADING_LAYER=GNU CUDA_VISIBLE_DEVICES=0 python ./evaluate_results.py --input ./eval_results/qwen3vl-4b
If you use this code and data for your research or project, please cite:
@inproceedings{wu2026spatialscore,
author = {Wu, Haoning and Huang, Xiao and Chen, Yaohui and Zhang, Ya and Wang, Yanfeng and Xie, Weidi},
title = {SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2026},
}
Many thanks to the code bases from transformers, Qwen3-VL, and TACO.
If you have any questions, please feel free to contact haoningwu3639@gmail.com.
26 commits