RoboBench Official Evaluation Results

This dataset contains the official score tables and released model-output JSON files for RoboBench.

Use this repository to inspect the reported results, compare model outputs, and reproduce the released tables together with the RoboBench codebase.

Project links:

• Paper: https://arxiv.org/abs/2510.17801
• Project website: https://robo-bench.github.io/
• Code: https://github.com/yulin-luo/RoboBench
• RoboBench benchmark data: https://huggingface.co/datasets/LeoFan01/RoboBench

Contents

scores/
  multiple_choice_scores.csv           # Multiple-choice scores for Perception / Affordance / Failure
  multiple_choice_scores.md            # Markdown rendering of the same table
  multiple_choice_coverage.csv         # Answer coverage for the multiple-choice score table
  mc_final_scores.csv                  # Legacy alias; mc means multiple-choice
  mc_final_scores.md                   # Legacy alias; mc means multiple-choice
  mc_final_coverage.csv                # Legacy alias; mc means multiple-choice
  planning_q1_scores.csv               # Instruction + Generalized Planning Q1 scores
  planning_q1_scores.md                # Markdown rendering of the same table
  planning_q2q3_scores.csv             # Planning Q2/Q3 scores
  planning_q2q3_scores.md              # Markdown rendering of the same table
  coverage.json                        # Machine-readable coverage audit
  coverage.md                          # Human-readable coverage audit
  paper_table3_perception.csv          # CSV export of paper Table 3
  paper_table4_instruction_q1.csv      # CSV export of paper Table 4
  paper_table5_q2q3.csv                # CSV export of paper Table 5
  paper_table6_affordance_failure.csv  # CSV export of paper Table 6

outputs/
  1_instruction_comprehension/
  2_perception_reasoning/
  3_generalized_planning/
  4_affordance_reasoning/
  5_error_analysis/

The outputs/ tree is organized as:

outputs/<dimension>/<subtask>/<model>/
  final.json

Each JSON item follows this format. response is a string when a usable model output is available, and null when the model did not return a usable output for that sample.

{
  "id": "...",
  "response": "...",
  "model": "..."
}

Scoring Protocol

The released scores/ directory contains two groups of results: multiple-choice scores and planning scores. Multiple-choice scores cover tasks evaluated by selecting an answer from predefined options, while planning scores cover open-ended instruction comprehension and generalized planning evaluations.

The primary score files are:

• multiple_choice_scores.csv: Scores for tasks evaluated by choosing from predefined options: Perception Reasoning, Affordance Prediction, and Failure Analysis.
• multiple_choice_coverage.csv: Answer coverage for the multiple-choice score table.
• planning_q1_scores.csv: Scores for Instruction Comprehension plus Generalized Planning Q1.
• planning_q2q3_scores.csv: Scores for Generalized Planning Q2 and Q3.

The older mc_* filenames are retained only as compatibility aliases. In those filenames, mc means multiple-choice.

The mc_final_* files are duplicate aliases of the multiple-choice score files kept for existing links.

Instruction Comprehension and Generalized Planning use the planning evaluation protocol described in the paper and code release.

The scores/paper_table*.csv files provide convenient CSV exports of the main paper tables.

How to Use

Download the benchmark data and this results repository separately:

huggingface-cli download --repo-type dataset LeoFan01/RoboBench --local-dir data/RoboBench-hf
huggingface-cli download --repo-type dataset lyl010221-pku/RoboBench-Results --local-dir results/RoboBench-Results

For code usage and evaluation scripts, see:

https://github.com/yulin-luo/RoboBench

Verification

Last refreshed: 2026-07-04.

The released score tables were checked against the active paper tables after the latest response-gap refill:

• scores/multiple_choice_scores.csv is synchronized with the paper Perception/Affordance/Failure tables.
• scores/planning_q1_scores.csv and scores/planning_q2q3_scores.csv are synchronized with the paper Instruction/Planning tables.
• The released gemini-2.5-pro outputs for temporal grounding, cross-task navigation, and high-level planning error now contain valid responses for every released item.

Citation

@misc{luo2026robobenchcomprehensiveevaluationbenchmark,
      title={Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain}, 
      author={Yulin Luo and Chun-Kai Fan and Menghang Dong and Jiayu Shi and Xiangju Mi and Mengdi Zhao and Bo-Wen Zhang and Cheng Chi and Jiaming Liu and Gaole Dai and Rongyu Zhang and Ruichuan An and Kun Wu and Zhengping Che and Shaoxuan Xie and Guocai Yao and Zhongxia Zhao and Pengwei Wang and Guang Liu and Zhongyuan Wang and Tiejun Huang and Shanghang Zhang},
      year={2026},
      eprint={2510.17801},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2510.17801}, 
}