MMSI-Bench

This repo contains evaluation code for the paper "MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence"

🌐 Homepage | 🤗 Dataset | 📑 Paper | 💻 Code | 📖 arXiv

🔔News

🔥[2025-10-23]: We added the normalized human response time for each MMSI-Bench sample and its difficulty level to our dataset on Hugging Face.

🔥[2025-06-18]: MMSI-Bench has been supported in the LMMs-Eval repository.

✨[2025-06-11]: MMSI-Bench was used for evaluation in the experiments of VILASR.

🔥[2025-06-9]: MMSI-Bench has been supported in the VLMEvalKit repository.

🔥[2025-05-30]: We released the ArXiv paper.

Load Dataset

from datasets import load_dataset

mmsi_bench = load_dataset("RunsenXu/MMSI-Bench")
print(mmsi_bench)

After downloading the parquet file, read each record, decode images from binary, and save them as JPG files.

import pandas as pd
import os

df = pd.read_parquet('MMSI_Bench.parquet')

output_dir = './images'
os.makedirs(output_dir, exist_ok=True)

for idx, row in df.iterrows():
    id_val = row['id']
    images = row['images']  
    question_type = row['question_type']
    question = row['question']
    answer = row['answer']
    thought = row['thought']

    image_paths = []
    if images is not None:
        for n, img_data in enumerate(images):
            image_path = f"{output_dir}/{id_val}_{n}.jpg"
            with open(image_path, "wb") as f:
                f.write(img_data)
            image_paths.append(image_path)
    else:
        image_paths = []

    print(f"id: {id_val}")
    print(f"images: {image_paths}")
    print(f"question_type: {question_type}")
    print(f"question: {question}")
    print(f"answer: {answer}")
    print(f"thought: {thought}")
    print("-" * 50)

Evaluation

Please refer to the evaluation guidelines of VLMEvalKit

🏆 MMSI-Bench Leaderboard

Acknowledgment

MMSI-Bench makes use of data from existing image datasets: ScanNet, nuScenes, Matterport3D, Ego4D, AgiBot-World, DTU, DAVIS-2017 ,and Waymo. We thank these teams for their open-source contributions.

Contact

• Sihan Yang: sihany077@gmail.com
• Runsen Xu: runsxu@gmail.com

Citation

@article{yang2025mmsi,
  title={MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence},
  author={Yang, Sihan and Xu, Runsen and Xie, Yiman and Yang, Sizhe and Li, Mo and Lin, Jingli and Zhu, Chenming and Chen, Xiaochen and Duan, Haodong and Yue, Xiangyu and Lin, Dahua and Wang, Tai and Pang, Jiangmiao},
  journal={arXiv preprint arXiv:2505.23764},
  year={2025}
}