🤖 RoboBench

A comprehensive evaluation benchmark for Multimodal Large Language Models as embodied robot brains.

---

📌 Featured Resources

• 📄 Paper: RoboBench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
• 🌐 Project website: robo-bench.github.io
• 🤗 Dataset: LeoFan01/RoboBench
• 📊 Official results: lyl010221-pku/RoboBench-Results
• 💬 Prompts and pipeline: docs/PROMPTS_AND_PIPELINE.md
• 🧪 Multiple-Choice Question (MCQ) and Planning evaluation: docs/HY_EMBODIED_EVAL.md
• 💻 Code: github.com/yulin-luo/RoboBench
• 🖼️ ECCV 2026 poster: PDF · ECCV virtual page
• 🎬 Intro video (5 min): YouTube

Accepted to ECCV 2026.

📰 News

• 🖼️ 2026.09 - The official ECCV 2026 poster is released (PDF). Meet us at ECCV 2026 in Malmö, Sept 8–12 — see the ECCV virtual poster page.
• 📊 2026.07 - Congratulations to HY-Embodied! HY-Embodied-0.5 MoT-2B and Hy-Embodied-VLM-1.0 A3B include RoboBench-MCQ and RoboBench-Planning as part of their official evaluation suite, highlighting RoboBench as a recognized benchmark for embodied foundation models.
• 🎉 2026.07 - RoboBench is accepted to ECCV 2026. The official release reports results for 18 state-of-the-art MLLMs with the MLLM-as-world-simulator planning framework.
• 📰 2026.07 - RoboBench is featured by 具身智能之心, introducing our benchmark for evaluating MLLMs as embodied brains.

🔍 Overview

RoboBench evaluates MLLMs on robotic manipulation tasks by decomposing embodied intelligence into diagnostic abilities rather than reporting only end-to-end task success. It covers the full execution pipeline from instruction comprehension and perception to generalized planning, affordance reasoning, and failure analysis. The ECCV 2026 release contains 5 dimensions, 14 capability groups, 25 tasks (released as 32 subtask manifests), 6,092 QA pairs, and results for 18 state-of-the-art MLLMs.

✨ What RoboBench Provides

• Fine-grained embodied evaluation: 5 cognitive dimensions, 14 capability groups, 25 tasks, and 6,092 QA pairs for diagnosing where MLLMs succeed or fail as robot brains.
• 18-model leaderboard: closed-source, open-source, and embodied MLLMs are evaluated in the official results release, plus a GPT-5.4 text-only ablation for visual-grounding analysis.
• Cross-domain planning tests: Robot morphology, object type, viewpoint, attributes, and world-knowledge generalization.
• MLLM-as-world-simulator evaluation: Planning outputs are judged with a simulator-style MLLM evaluator for physically grounded task completion.
• Reproducible package: YAML-driven configuration, API inference, task-specific evaluators, checkpoint/resume, and multi-run aggregation.
• Reusable prompt pipeline: Open prompt construction utilities for robotic video and image-based question answering, with prompt coverage documented in docs/PROMPTS_AND_PIPELINE.md.

🧭 Benchmark Dimensions

🧪 Multiple-Choice Question (MCQ) and Planning

For comparison with embodied foundation models such as HY-Embodied, RoboBench provides two ready-to-run evaluation settings. Here, MCQ means Multiple-Choice Question:

After completing the dataset, config, and model-server setup described below, run either setting with one command:

# Validate the released metadata without running inference.
ROBOBENCH_CONFIG=config/benchmark.example.yaml \
  bash scripts/run_hy_embodied_eval.sh dry-run

# Run one setting, or replace mcq with planning/all.
ROBOBENCH_MODEL=hy_a3b \
  bash scripts/run_hy_embodied_eval.sh mcq

The helper calls the model through an OpenAI-compatible /v1/chat/completions endpoint; it does not load model weights directly. See Evaluating HY-Embodied on RoboBench for the vLLM serving example, configuration, smoke test, and output locations.

The mapping above is the current RoboBench-side interpretation of the two reported HY-Embodied settings. We are tracking confirmation of the exact reported protocol in Tencent-Hunyuan/HY-Embodied#14 and will synchronize the helper if their protocol differs.

⚙️ Installation

git clone https://github.com/yulin-luo/RoboBench.git
cd RoboBench

# Core package for API-based inference and evaluation.
pip install -e .

# Optional dependencies for local HuggingFace vision-language models.
pip install -e ".[local]"

📦 Requirements

• Python >= 3.10
• API-compatible endpoint supported by the OpenAI Python client
• OpenCV, PyYAML, Pydantic, NumPy, tqdm
• Optional local-model stack: torch, transformers, pillow

📚 Dataset

Download the released RoboBench dataset from Hugging Face and point the config to the local copy.

hf download LeoFan01/RoboBench \
  --repo-type dataset \
  --local-dir data/RoboBench-hf

The downloaded directory is the complete runtime data source. RoboBench reads the 32 released questions.json files and system_prompt.json, constructs the prompts deterministically, and resolves every image under paths.data_root. No separately generated prompt files or path-prefix rewrite configuration is required.

Validate the released question metadata before inference:

robobench --config config/benchmark.yaml inspect-data --metadata-only

After all images are downloaded, run the stricter file check:

robobench --config config/benchmark.yaml inspect-data

The complete release contains 32 subtask manifests, 6,092 questions, and 37,126 image references. The 32 manifests correspond to the paper's 25 tasks: two planning tasks are sharded by category at release time — material_affordance into 5 material buckets (articulated / deformable / rigid / special / multi-object) and physical_attribute into 4 attribute buckets (color / number / shape / size) — while all other tasks map one-to-one (25 + 4 + 3 = 32). RoboBench-MCQ covers 13 subtasks and 1,895 questions; RoboBench-Planning covers 19 subtasks and 4,197 questions.

Official score tables and model-output JSON files are hosted separately to keep this repository lightweight:

• lyl010221-pku/RoboBench-Results

The official leaderboard reports 18 evaluated MLLMs:

GPT-5.4, GPT-5.2, GPT-5, GPT-4.1, GPT-4o,
Claude-Opus-4.7, Claude-Sonnet-4.6, Claude-Sonnet-4.5, Claude-Haiku-4.5,
Gemini-3.1-Pro, Gemini-2.5-Pro, Gemini-2.5-Flash,
Qwen3-VL-8B, Qwen2.5-VL-7B-Instruct, LLaVA-OneVision-7B,
RoboBrain-2.0-7B, RoboBrain-2.5-4B, MiMo-Embodied-7B

🚀 Quick Start

1. 🛠️ Prepare a Config

cp config/benchmark.example.yaml config/benchmark.yaml

export DUBRIFY_API_KEY="your-api-key"
export ROBOBENCH_API_BASE_URL="https://your-endpoint/v1"
export ROBOBENCH_DATA_ROOT="$PWD/data/RoboBench-hf"
export ROBOBENCH_RESULTS_ROOT="$PWD/results"
export ROBOBENCH_CACHE_DIR="$PWD/cache"
export ROBOBENCH_JUDGE_API_BASE_URL="https://your-judge-endpoint/v1"
export ROBOBENCH_JUDGE_API_KEY="your-judge-api-key"

Edit config/benchmark.yaml to choose models, dimensions, and concurrency settings. Keep config/benchmark.yaml local; it is intentionally ignored by git.

2. 🧠 Run Inference

After pip install -e ., the package exposes a robobench command-line entrypoint. The CLI uses a top-level --config argument before the subcommand.

robobench --config config/benchmark.yaml inference \
  --model gpt-5.4 \
  --dimension perception_reasoning \
  --subtask static_attribute \
  --max-samples 1 \
  --run-id smoke_test

If you prefer not to install the package, use the equivalent Python module form:

PYTHONPATH=src python -m robobench.cli --config config/benchmark.yaml inference \
  --model gpt-5.4 \
  --dimension perception_reasoning \
  --subtask static_attribute \
  --max-samples 1 \
  --run-id smoke_test

For text-only ablation:

robobench --config config/benchmark.yaml inference \
  --model gpt-5.4 \
  --dimension perception_reasoning \
  --subtask static_attribute \
  --max-samples 1 \
  --run-id run_0_text_only \
  --text-only

HY-Embodied users who want to run the reported RoboBench-MCQ and RoboBench-Planning style settings can use the helper script in docs/HY_EMBODIED_EVAL.md. The helper expects a HY-Embodied model served through an OpenAI-compatible endpoint, such as the official Hy-Embodied-VLM-1.0 vLLM server; it does not load HY weights directly.

3. 📊 Evaluate Existing Results

robobench --config config/benchmark.yaml evaluate \
  --dimension perception_reasoning

Remove --max-samples when running the full selected dimension.

4. 🔁 Run an End-to-End Pipeline

robobench --config config/benchmark.yaml pipeline --repeats 3

The full pipeline runs inference, evaluation, and aggregation across repeated runs configured by runs.num_repeats.

🧩 Configuration

Most behavior is controlled from config/benchmark.yaml.

🔐 API Settings

🧠 Model Selection

models:
  - name: "gpt-5.4"
    provider: "openai"
    vision: true

text_only_variants:
  - name: "gpt-5.4"
    suffix: "text_only"

🧪 Dimension Selection

dimensions:
  perception_reasoning:
    enabled: true
    eval_type: "multi_choice"
    subtasks:
      - static_attribute
      - spatial_relation

📁 Paths

🗂️ Project Structure

RoboBench/
├── config/
│   └── benchmark.example.yaml
├── src/robobench/
│   ├── analysis/          # Dataset and correlation analysis utilities
│   ├── data/              # Dataset loading from released questions.json files
│   ├── evaluation/        # Multiple-choice, planning, point, IoU, trajectory evaluators
│   ├── generation/        # Generation-stage nodes
│   ├── inference/         # Async API client, checkpoints, image handling, local HF client
│   ├── pipeline/          # Dataflow nodes and executor
│   ├── prompts/           # Prompt builders and robot-task templates
│   ├── scoring/           # Multi-run aggregation and statistics
│   └── utils/             # Path utilities
├── pyproject.toml
└── README.md

📏 Evaluation Metrics

Planning evaluation can call an evaluator model, configured by evaluation.planning.eval_model, to judge action feasibility and task completion.

💬 Prompt Builder Example

from robobench.data.dataset import RoboBenchDataset
from robobench.prompts.builder import PromptBuilder

dataset = RoboBenchDataset("data/RoboBench-hf")
questions = dataset.load_questions(
    "perception_reasoning",
    "static_attribute",
    max_samples=1,
)
builder = PromptBuilder()
prompts = builder.build(questions, mode="base64")
builder.save(prompts, "prompts.jsonl")

🛠️ Development

pip install -e ".[dev]"
python -m compileall -q src
black src/
ruff check src/

📝 Citation

If you use RoboBench in your research, please cite:

@misc{luo2026robobenchcomprehensiveevaluationbenchmark,
      title={Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain},
      author={Yulin Luo and Chun-Kai Fan and Menghang Dong and Jiayu Shi and Xiangju Mi and Mengdi Zhao and Bo-Wen Zhang and Cheng Chi and Jiaming Liu and Gaole Dai and Rongyu Zhang and Ruichuan An and Kun Wu and Zhengping Che and Shaoxuan Xie and Guocai Yao and Zhongxia Zhao and Pengwei Wang and Guang Liu and Zhongyuan Wang and Tiejun Huang and Shanghang Zhang},
      year={2026},
      eprint={2510.17801},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2510.17801},
}

📜 License

This repository is released under the MIT License.