IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering

This repository contains the dataset and evaluation protocols for IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering.

• Project Page: https://ir3d-bench.github.io/
• Code (GitHub): https://github.com/Piang/IR3D-bench

Abstract

Vision-language models (VLMs) excel at descriptive tasks, but whether they truly understand scenes from visual observations remains uncertain. We introduce IR3D-Bench, a benchmark challenging VLMs to demonstrate understanding through active creation rather than passive recognition. Grounded in the analysis-by-synthesis paradigm, IR3D-Bench tasks Vision-Language Agents (VLAs) with actively using programming and rendering tools to recreate the underlying 3D structure of an input image, achieving agentic inverse rendering through tool use. This "understanding-by-creating" approach probes the tool-using generative capacity of VLAs, moving beyond the descriptive or conversational capacity measured by traditional scene understanding benchmarks. We provide a comprehensive suite of metrics to evaluate geometric accuracy, spatial relations, appearance attributes, and overall plausibility. Initial experiments on agentic inverse rendering powered by various state-of-the-art VLMs highlight current limitations, particularly in visual precision rather than basic tool usage. IR3D-Bench, including data and evaluation protocols, is released to facilitate systematic study and development of tool-using VLAs towards genuine scene understanding by creating.

Motivation & Useful Findings

1. Inspired by Richard Feynman's aphorism ("What I cannot create, I do not understand."), we propose a new perspective to evaluate VLMs' spatial visual understanding via a pretext task: how well they "recreate this scene."
2. We found that the aim of scene reconstruction enables VLMs to spontaneously estimate key attributes (object ID, localization, color, material, object relations, etc.) via an inverse rendering fashion—critical for understanding what they see.
3. VLMs show surprising potential for human-like reflection during this "recreation" game: feeding VLMs their recreated scenes, they compare with originals and update their understanding of the scene (the key attributes they estimate). We expect this multi-round feedback iteration to unlock more possibilities for improving existing VLMs in both understanding and generation performance.

Pipeline Overview

Dataset Setup

This dataset contains the processed data for the IR3D-Bench benchmark. You can download the data directly from this Hugging Face dataset repository (Piang/IR3D-bench).

To download the processed data, ensure you have Git LFS installed and then clone the repository:

git lfs install
git clone https://huggingface.co/datasets/Piang/IR3D-bench

Sample Usage (Inverse Rendering & Evaluation)

To run inverse rendering tasks with state-of-the-art Vision-Language Models (VLMs) and evaluate their performance, refer to the following instructions derived from the project's GitHub repository.

Environment setup

1. Create Environment:

   conda create --name ir3d python=3.10
   conda activate ir3d
   

2. Install vllm:

   pip install vllm
   

3. Install Blender (on Linux):

   snap install blender --classic
   

4. Install SAM:

   pip install git+https://github.com/facebookresearch/segment-anything.git
   

Inverse Rendering

Task prompt for inverse rendering and gpt4o score is in prompts/gpt4o_as_evaluator.txt and prompts/vlm_estimate_params.txt within the code repository.

Latest Proprietary Models

Modified the model-name as defined in main_vllm.py to use the required model.

python main_vllm.py --model-type "model-name"

Open-source Models

Modified the model-name as you needed, such as "gpt-4o", "grok-3", etc.

python main_api.py \ 
    --image_dir /path/to/images \ 
    --result_dir /output/path \ 
    --prompt_path prompts/vlm_estimate_params.txt \ 
    --model_name "model-name"

Evaluation

To calculate metrics:

bash cal_metric.sh "/output/path" "/path/to/images" "GPI_ID"

For more detailed setup and usage, please refer to the IR3D-Bench GitHub repository.

Citation

If you find our work helpful, please consider citing:

@article{liu2025ir3d,
  title={IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering},
  author={Liu, Parker and Li, Chenxin and Li, Zhengxin and Wu, Yipeng and Li, Wuyang and Yang, Zhiqin and Zhang, Zhenyuan and Lin, Yunlong and Han, Sirui and Feng, Brandon Y},
  journal={arXiv preprint arXiv:2506.23329},
  year={2025}
}