CapNav is a benchmark for Vision Language Models evaluating on capability-conditioned navigation reasoning in indoor environments. The benchmark focuses on determining whether an embodied agent with specific physical constraints and abilities can navigate from a start area to a target area within a complex indoor scene.
This repository contains two complementary datasets:
from datasets import load_dataset
capnav = load_dataset("RichardC0216/CapNav", "capnav_v0", split="train")
agents = load_dataset("RichardC0216/CapNav", "agent_profiles", split="train")
print(len(capnav), capnav.column_names)
print(len(agents), agents.column_names)
capnav_v0)capnav_v0_with_answer.parquettrainBelow one can find the description of each field in the dataset.
question_id (str)question (str)scene_id (str)HM3D00000, MP3D00027).scene_type (str)home).scene_nodes (list[dict])node_id (str): Unique identifier of the nodename (str): Human-readable node name (e.g., room or area label)agent_name (str)answer ((bool))True / False) for the (question, scene, agent) triple.
Note:
answeronly provides a binary feasibility signal. For scene graphs, route-level traversability, and detailed ground-truth rationale, please refer to the full annotations inground_truth/(see below).
agent_profiles.parquetagent_name (str)HUMAN, WHEELCHAIR, SWEEPER).body_shape (str)cylinder, box).body_height_m (float)body_width_m (float)body_depth_m (float, optional)null for rotationally symmetric agents.max_vertical_cross_height_m (float)can_go_up_or_down_stairs (bool)can_operate_elevator (bool)can_open_the_door (bool)description (str)This repository also includes a ground_truth/ directory that provides the
complete ground-truth annotations used to derive the binary answer labels
in the main CapNav benchmark.
While the benchmark dataset exposes only a binary navigability outcome for each
(question, scene, agent) triple, the annotations in ground_truth/ contain
the underlying structural and traversability information that supports
more detailed inspection and analysis.
Specifically, this directory includes:
ground_truth/graphs/)ground_truth/traverse/)The binary answer field in the benchmark dataset is a distilled signal
derived from these annotations. Researchers interested in path validity,
failure cases, or the reasons behind infeasible navigation decisions should
refer to the files in ground_truth/.
Detailed descriptions of file formats and annotation semantics can be found in:
ground_truth/README.mdThe annotation pipeline, tooling, and quality control procedures are documented in the CapNav GitHub repository:
The CapNav benchmark is intended for:
If you find it useful for your research and applications, please cite our paper using this BibTeX:
@article{su2026capnav,
title={CapNav: Benchmarking Vision Language Models on Capability-conditioned Indoor Navigation},
author={Su, Xia and Chen, Ruiqi and Liu, Benlin and Ma, Jingwei and Di, Zonglin and Krishna, Ranjay and Froehlich, Jon},
journal={arXiv preprint arXiv:2602.18424},
year={2026}
}
23 commits
CapNav is a benchmark for Vision Language Models evaluating on capability-conditioned navigation reasoning in indoor environments. The benchmark focuses on determining whether an embodied agent with specific physical constraints and abilities can navigate from a start area to a target area within a complex indoor scene.
This repository contains two complementary datasets:
from datasets import load_dataset
capnav = load_dataset("RichardC0216/CapNav", "capnav_v0", split="train")
agents = load_dataset("RichardC0216/CapNav", "agent_profiles", split="train")
print(len(capnav), capnav.column_names)
print(len(agents), agents.column_names)
capnav_v0)capnav_v0_with_answer.parquettrainBelow one can find the description of each field in the dataset.
question_id (str)question (str)scene_id (str)HM3D00000, MP3D00027).scene_type (str)home).scene_nodes (list[dict])node_id (str): Unique identifier of the nodename (str): Human-readable node name (e.g., room or area label)agent_name (str)answer ((bool))True / False) for the (question, scene, agent) triple.
Note:
answeronly provides a binary feasibility signal. For scene graphs, route-level traversability, and detailed ground-truth rationale, please refer to the full annotations inground_truth/(see below).
agent_profiles.parquetagent_name (str)HUMAN, WHEELCHAIR, SWEEPER).body_shape (str)cylinder, box).body_height_m (float)body_width_m (float)body_depth_m (float, optional)null for rotationally symmetric agents.max_vertical_cross_height_m (float)can_go_up_or_down_stairs (bool)can_operate_elevator (bool)can_open_the_door (bool)description (str)This repository also includes a ground_truth/ directory that provides the
complete ground-truth annotations used to derive the binary answer labels
in the main CapNav benchmark.
While the benchmark dataset exposes only a binary navigability outcome for each
(question, scene, agent) triple, the annotations in ground_truth/ contain
the underlying structural and traversability information that supports
more detailed inspection and analysis.
Specifically, this directory includes:
ground_truth/graphs/)ground_truth/traverse/)The binary answer field in the benchmark dataset is a distilled signal
derived from these annotations. Researchers interested in path validity,
failure cases, or the reasons behind infeasible navigation decisions should
refer to the files in ground_truth/.
Detailed descriptions of file formats and annotation semantics can be found in:
ground_truth/README.mdThe annotation pipeline, tooling, and quality control procedures are documented in the CapNav GitHub repository:
The CapNav benchmark is intended for:
If you find it useful for your research and applications, please cite our paper using this BibTeX:
@article{su2026capnav,
title={CapNav: Benchmarking Vision Language Models on Capability-conditioned Indoor Navigation},
author={Su, Xia and Chen, Ruiqi and Liu, Benlin and Ma, Jingwei and Di, Zonglin and Krishna, Ranjay and Froehlich, Jon},
journal={arXiv preprint arXiv:2602.18424},
year={2026}
}
23 commits