Omni3D-Bench

This repository contains the Omni3D-Bench dataset introduced in the paper Visual Agentic AI for Spatial Reasoning with a Dynamic API Omni3D-Bench contains 500 challenging (image, question, answer) tuples of diverse, real-world scenes sourced from Omni3D for complex 3D spatial reasoning.

View samples from the dataset here.

The dataset is released under the Creative Commons Non-Commercial license.

Usage

The benchmark can be accessed with the following code:

from datasets import load_dataset
dataset = load_dataset("dmarsili/Omni3D-Bench")

We additionally provide a .zip file including all the images and annotations.

Annotations

Samples in Omni3D-Bench consist of images, questions, and ground-truth answers. Samples can be loaded as python dictonaries in the following format:

<!-- annotations.json -->
{
    "questions": [
        {
            "image_index"               : str, image ID
            "question_index"            : str, question ID
            "image"                     : PIL Image, image for query
            "question"                  : str, query
            "answer_type"               : str, expected answer type - {int, float, str}
            "answer"                    : str|int|float, ground truth response to the query
        },
        {
            ...
        },
        ...
    ]
}

Citation

If you use the Omni3D-Bench dataset in your research, please use the following BibTeX entry.

@inproceedings{marsili2025visual,
  title={Visual agentic ai for spatial reasoning with a dynamic api},
  author={Marsili, Damiano and Agrawal, Rohun and Yue, Yisong and Gkioxari, Georgia},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={19446--19455},
  year={2025}
}