README
MSMU (Massive Spatial Measuring and Understanding Dataset for Spatial Intelligence)
🌐 Homepage | 🤗 Dataset | 📖 arXiv | GitHub
Dataset Details
Dataset Description
We introduce MSMU and MSMU-Bench: a new benchmark designed to enhance and evaluate multimodal models on spatial measuring and understanding. MSMU is featured as metric-accurate spatial annotations which are sourced from high-precision 3D scenes. It contains , 25K images, 700K QA pairs, and 2.5M numerical values, covering a wide range of quantitative spatial tasks (Existence, Counting, Scale Estimation, Grounding, Relative Position, Absolute Distance, Scale Comparison, and Reference Object Estimation).
🎯 We have released a full set of MSMU and MSMU-Bench.
Dataset Creation
We categorize the spatial tasks in MSMU into 8 types, the distribution of which is illustrated in Figure below (left). The QA distribution of MSMU-Bench is also shown in Figure below (right) which provides a detailed breakdown of these eight categories.
🏆 Mini-Leaderboard
We show a mini-leaderboard here. It shows the results of each sub-category and the overall performance.
Results
| Model | Existence | Object Counting | Scale Est. | Grounding | Relative Position | Absolute Distance | Scale Comparison | Ref. Object Est. | Average |
|---|---|---|---|---|---|---|---|---|---|
| Large Language Models (LLMs): Text only | |||||||||
| GPT-4-Turbo | 12.76 | 5.21 | 13.51 | 12.64 | 24.84 | 7.50 | 36.79 | 12.04 | 15.66 |
| Qwen2.5 | 4.25 | 0.00 | 0.78 | 13.79 | 0.62 | 0.00 | 16.04 | 1.57 | 4.63 |
| DeepSeek-V3 | 0.00 | 5.24 | 1.54 | 6.90 | 10.56 | 0.00 | 25.47 | 5.24 | 7.39 |
| Vision-Language Models (VLMs): Image + Text | |||||||||
| GPT-4o | 44.68 | 41.67 | 3.86 | 27.59 | 67.08 | 20.00 | 54.72 | 2.09 | 32.28 |
| Gemini-2 | 38.30 | 43.75 | 23.94 | 19.54 | 54.66 | 12.50 | 69.81 | 18.85 | 35.17 |
| Qwen2.5-VL-72B | 59.57 | 35.42 | 1.54 | 13.79 | 57.76 | 2.50 | 66.04 | 9.95 | 30.82 |
| Qwen2.5-VL-32B | 29.79 | 41.67 | 10.81 | 18.39 | 60.25 | 2.50 | 46.23 | 10.99 | 27.59 |
| Qwen2.5-VL-7B | 12.76 | 4.17 | 0.00 | 1.15 | 1.24 | 0.00 | 5.66 | 0.52 | 3.19 |
| Intern-VL3-78B | 47.62 | 42.71 | 6.47 | 26.32 | 56.94 | 13.33 | 64.10 | 16.46 | 33.63 |
| Intern-VL3-8B | 36.17 | 41.67 | 4.63 | 18.39 | 60.25 | 2.50 | 49.06 | 8.38 | 28.54 |
| LLaVA-1.5-7B | 1.54 | 36.46 | 5.02 | 20.69 | 42.86 | 5.00 | 38.68 | 0.52 | 19.45 |
| Depth-encoded VLMs: Image + Depth + Text | |||||||||
| SpatialBot | 10.64 | 46.88 | 15.83 | 28.74 | 66.46 | 5.00 | 50.94 | 8.90 | 29.17 |
| SpatialRGPT | 10.64 | 36.46 | 20.08 | 17.24 | 60.25 | 15.00 | 62.26 | 9.95 | 28.98 |
| Ours | 87.23 | 47.92 | 51.35 | 42.53 | 75.16 | 40.00 | 55.66 | 46.07 | 56.31 |
Citation
BibTeX:
@inproceedings{chen2025sdvlm,
title={SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models},
author={Pingyi Chen and Yujing Lou and Shen Cao and Jinhui Guo and Lubin Fan and Yue Wu and Lin Yang and Lizhuang Ma and Jieping Ye},
booktitle={NeurIPS},
year={2025},
}
Contributors
18 commits
README
MSMU (Massive Spatial Measuring and Understanding Dataset for Spatial Intelligence)
🌐 Homepage | 🤗 Dataset | 📖 arXiv | GitHub
Dataset Details
Dataset Description
We introduce MSMU and MSMU-Bench: a new benchmark designed to enhance and evaluate multimodal models on spatial measuring and understanding. MSMU is featured as metric-accurate spatial annotations which are sourced from high-precision 3D scenes. It contains , 25K images, 700K QA pairs, and 2.5M numerical values, covering a wide range of quantitative spatial tasks (Existence, Counting, Scale Estimation, Grounding, Relative Position, Absolute Distance, Scale Comparison, and Reference Object Estimation).
🎯 We have released a full set of MSMU and MSMU-Bench.
Dataset Creation
We categorize the spatial tasks in MSMU into 8 types, the distribution of which is illustrated in Figure below (left). The QA distribution of MSMU-Bench is also shown in Figure below (right) which provides a detailed breakdown of these eight categories.
🏆 Mini-Leaderboard
We show a mini-leaderboard here. It shows the results of each sub-category and the overall performance.
Results
| Model | Existence | Object Counting | Scale Est. | Grounding | Relative Position | Absolute Distance | Scale Comparison | Ref. Object Est. | Average |
|---|---|---|---|---|---|---|---|---|---|
| Large Language Models (LLMs): Text only | |||||||||
| GPT-4-Turbo | 12.76 | 5.21 | 13.51 | 12.64 | 24.84 | 7.50 | 36.79 | 12.04 | 15.66 |
| Qwen2.5 | 4.25 | 0.00 | 0.78 | 13.79 | 0.62 | 0.00 | 16.04 | 1.57 | 4.63 |
| DeepSeek-V3 | 0.00 | 5.24 | 1.54 | 6.90 | 10.56 | 0.00 | 25.47 | 5.24 | 7.39 |
| Vision-Language Models (VLMs): Image + Text | |||||||||
| GPT-4o | 44.68 | 41.67 | 3.86 | 27.59 | 67.08 | 20.00 | 54.72 | 2.09 | 32.28 |
| Gemini-2 | 38.30 | 43.75 | 23.94 | 19.54 | 54.66 | 12.50 | 69.81 | 18.85 | 35.17 |
| Qwen2.5-VL-72B | 59.57 | 35.42 | 1.54 | 13.79 | 57.76 | 2.50 | 66.04 | 9.95 | 30.82 |
| Qwen2.5-VL-32B | 29.79 | 41.67 | 10.81 | 18.39 | 60.25 | 2.50 | 46.23 | 10.99 | 27.59 |
| Qwen2.5-VL-7B | 12.76 | 4.17 | 0.00 | 1.15 | 1.24 | 0.00 | 5.66 | 0.52 | 3.19 |
| Intern-VL3-78B | 47.62 | 42.71 | 6.47 | 26.32 | 56.94 | 13.33 | 64.10 | 16.46 | 33.63 |
| Intern-VL3-8B | 36.17 | 41.67 | 4.63 | 18.39 | 60.25 | 2.50 | 49.06 | 8.38 | 28.54 |
| LLaVA-1.5-7B | 1.54 | 36.46 | 5.02 | 20.69 | 42.86 | 5.00 | 38.68 | 0.52 | 19.45 |
| Depth-encoded VLMs: Image + Depth + Text | |||||||||
| SpatialBot | 10.64 | 46.88 | 15.83 | 28.74 | 66.46 | 5.00 | 50.94 | 8.90 | 29.17 |
| SpatialRGPT | 10.64 | 36.46 | 20.08 | 17.24 | 60.25 | 15.00 | 62.26 | 9.95 | 28.98 |
| Ours | 87.23 | 47.92 | 51.35 | 42.53 | 75.16 | 40.00 | 55.66 | 46.07 | 56.31 |
Citation
BibTeX:
@inproceedings{chen2025sdvlm,
title={SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models},
author={Pingyi Chen and Yujing Lou and Shen Cao and Jinhui Guo and Lubin Fan and Yue Wu and Lin Yang and Lizhuang Ma and Jieping Ye},
booktitle={NeurIPS},
year={2025},
}
Similar repos
Linked in READMEs
Contributors
18 commits