POVBench

Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models
EMNLP 2026 Findings

Project page · Code

Given a sentence in which an observer says where they last saw an object — from their own point of view — a model must recover that perspective and localize the target in image space.

Crucially, the observer's right is not necessarily aligned with the camera's right.

Three conditions progressively reduce the amount of reasoning required:

The steps are operational reasoning, observer grounding, and relational reasoning.

See the paper for the task definition, construction pipeline, and results. The real-world evaluation data used in the paper (Tables 5–6) is not included in this release; refer to the paper for details.

Quick start

from huggingface_hub import snapshot_download

path = snapshot_download(repo_id="owl-owl/POVBench", repo_type="dataset")

Pass that path to the evaluation code as --dataset-dir.

This is a file-based dataset with a nested per-house layout, so datasets.load_dataset() and the Hub's dataset viewer do not apply. Download the folder and read the JSON directly.

Layout

POVBench/
├── meta.json                     # statistics and provenance
├── LICENSE, NOTICE               # licensing and upstream attribution
├── assets/overview.png           # the figure used by this card
└── artifacts/
    └── train_house_000NN/        # one directory per house, 47 in total
        ├── train_house_000NN.json          # ← the benchmark instances
        ├── exploration_images/*.png        # walkthrough frames
        ├── img_from_observer/*.png         # observer-viewpoint image (condition C)
        ├── exploration_log.json            # camera pose + visible objects per frame
        ├── per_image_visible_objects.json
        ├── gt_images/*.png                 # ground-truth visualizations
        ├── top_down_view.png               # floor-plan overview
        └── stage1_candidates.json, stage2..6_pairs.json   # generation records

Schema

Each scenario in train_house_000NN.json pairs an observer context (represented by the furniture associated with the speaker's activity) with an anchor (the landmark relative to which the target is described).

dir is one of front, up, left, right; a scenario holds up to four.

Enumerate instances from stage5 / stage6. placements also contains directions that were filtered out of the benchmark.

Inputs per condition

Ground truth

The scored ground truth is not stored in the dataset. eval_results.py in the code repository recomputes it from the anchor geometry and the stage3 surface verdict, then projects it into the image.

Contribution

Original contributions of POVBench: the task formulation and three-condition design, the capture procedure, observer–anchor pairing and target selection, ground-truth generation and filtering, and all natural-language sentences and annotations.

Built on ProcTHOR / AI2-THOR

The 3D environments are not ours. The houses were procedurally generated with ProcTHOR (Deitke et al., 2022), built on AI2-THOR (Kolve et al., 2017).

Both are from the Allen Institute for AI under the Apache License 2.0. Please cite them alongside this work. Required attribution is in NOTICE.

License

See LICENSE and NOTICE for the full terms, and the paper's Limitations section for known caveats.

Citation

@inproceedings{shirasaka2026povbench,
  title     = {Contextual Observer Grounding: Evaluating Situated Spatial
               Reasoning in Vision-Language Models},
  author    = {Shirasaka, Mimo and Zhang, Haochen and Bisk, Yonatan},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026}
}