KE7/roboeval

A reproducible evaluation harness for Vision-Language-Action models across simulator backends

stars

commits

Python

primary language

Aug 26, 2026

updated

README

roboeval

roboeval is a CLI-driven evaluation harness for running VLAs against simulator backends through isolated HTTP services. It provides an ActionObsSpec compatibility gate before episode execution, per-component virtual environments for dependency isolation, sharded result collection, and built-in support for LITEN-style hierarchical evaluation in which a VLM planner issues subtask instructions to a low-level VLA.

Method / Contracts

roboeval treats each VLA and simulator as an independently launched component. The orchestrator communicates with a VLA policy server and a simulator worker over HTTP/JSON, validates their declared contracts, and records episode-level results from a reproducible YAML run config.

The main contract surfaces are:

Surface	Role
`ActionObsSpec` gate	VLA and simulator components declare action format, dimensionality, value range, camera roles, image format, state layout, and language inputs. Under the default strict mode, incompatible declarations stop the run before episode 1.
Host-process isolation	VLA servers, simulator workers, and optional VLM proxy processes run in separate `.venvs/` environments. This allows different Python and CUDA dependency stacks to coexist without a monolithic runtime.
Dependency isolation	Each VLA and simulator keeps its upstream package pins, Python version, CUDA assumptions, and optional micromamba/uv environment separate. This is a design choice: adding a new backend should not force the orchestrator or other backends onto the same dependency closure.
LITEN-style hierarchical evaluation	The hierarchical mode integrates the VLM-planner method introduced by Shah et al. (Learning Affordances at Inference-Time for Vision-Language-Action Models). The planner emits subtask calls that are executed by the same VLA server interface used for direct evaluation. roboeval is, to our knowledge, the first public VLA evaluation harness to ship a working LITEN integration.
Result records	`roboeval run` writes JSON with harness version, config snapshot, per-episode metadata, success flags, and optional shard metadata.

Documentation map

For a compact system overview, design rationale, supported-pair notes, tuning guidance, related systems, and decision records, see architecture, design, supported pairs, tuning, related work, and the RFC index.

Installation

For full prerequisites, platform notes, and per-component dependency details, see docs/install.md.

git clone https://github.com/KE7/roboeval.git
cd roboeval
roboeval setup pi05 libero

The setup script provisions the orchestrator plus the requested VLA and simulator environments under .venvs/.

Quickstart

roboeval setup pi05 libero
roboeval serve --vla pi05 --sim libero --headless
roboeval test --validate -c configs/libero_spatial_pi05_smoke.yaml
roboeval run -c configs/libero_spatial_pi05_smoke.yaml

serve launches the selected VLA and simulator workers. run executes the YAML configuration, including the declared VLA/simulator pair, task suite, episode count, server URLs, output directory, and optional LITEN endpoint. Additional examples are in docs/quickstart.md.

Supported VLAs and Simulators

The table describes shipped coverage. It is a support matrix, not a benchmark table; supported pairs are tested end-to-end.

VLA	Simulator	Coverage	Example config
Pi0.5	LIBERO	direct, LITEN	`configs/libero_spatial_pi05_smoke.yaml`, `configs/libero_spatial_pi05_liten_smoke.yaml`
Pi0.5	LIBERO-Pro	direct, LITEN	`configs/libero_pro_pi05_smoke.yaml`, `configs/libero_pro_pi05_liten_smoke.yaml`
Pi0.5	LIBERO-Infinity	direct, LITEN	`configs/libero_infinity_pi05_smoke.yaml`, `configs/libero_infinity_pi05_liten_smoke.yaml`
SmolVLA	LIBERO	direct, LITEN	`configs/libero_object_smolvla_smoke.yaml`, `configs/libero_object_smolvla_liten_smoke.yaml`
OpenVLA	LIBERO	direct, LITEN	`configs/libero_spatial_openvla_smoke.yaml`, `configs/libero_spatial_openvla_liten_smoke.yaml`
GR00T	LIBERO	direct, LITEN	`configs/libero_spatial_groot_smoke.yaml`, `configs/libero_spatial_groot_liten_smoke.yaml`
InternVLA	RoboTwin	direct, LITEN	`configs/robotwin_internvla_smoke.yaml`, `configs/robotwin_internvla_liten_smoke.yaml`
ACT	ALOHA Gym	direct, LITEN	`configs/aloha_gym_act_smoke.yaml`, `configs/aloha_gym_act_liten_smoke.yaml`
Diffusion Policy	gym-pusht	direct	`configs/gym_pusht_diffusion_policy_smoke.yaml`
VQ-BeT	gym-pusht	direct	`configs/gym_pusht_vqbet_smoke.yaml`
TDMPC2	Meta-World	direct	`configs/metaworld_tdmpc2_smoke.yaml`
InternVLA	ALOHA Gym	CI smoke	`configs/ci/aloha_gym_internvla_smoke.yaml`
ManiSkill2	ManiSkill2 backend	backend scaffold; x86_64 execution path	setup target `maniskill2`
RoboCasa	RoboCasa backend	simulator backend and registry support	setup target `robocasa`

Supported VLA launch names are pi05, vqbet, tdmpc2, smolvla, openvla, cosmos, groot, and internvla. Supported simulator launch names are libero, libero_pro, libero_infinity, robocasa, robotwin, aloha_gym, gym_pusht, maniskill2, and metaworld.

Current limitations

ManiSkill2 is platform-blocked on aarch64 because the required SAPIEN 2.x wheels are x86_64-only.
bridge_octo is platform-blocked on aarch64 by its current TensorFlow/dlimp dependency chain and does not ship in the v0.1.0 support matrix.
Some technically expressible pairs remain capability boundaries and do not ship root configs, including RoboCasa x GR00T.

Planned features

Multi-architecture CI matrix. aarch64 is currently the primary CI path; x86_64 execution paths exist but are not in the CI matrix.
Additional VLAs as their checkpoints become available.
More simulators. Community contributions are welcome; see docs/extending.md.

Extending

Extension cost. Adding a new VLA averages ~200 SLOC; adding a new simulator backend averages ~230 SLOC (across the v0.1.0 release; excludes blank lines, comments, and docstrings).

Add a VLA by implementing a policy server with /health, /info, /reset, and /predict, then registering it with roboeval serve.
Add a simulator by implementing a SimBackendBase backend with /init, /reset, /step, /obs, /success, and /info support through the sim worker.
Add a new compatibility path by declaring ActionObsSpec records on both sides and adding a smoke config under configs/.

See docs/extending.md for the extension architecture and step-by-step entry points.

Citations

If you use roboeval in your research, please cite us.

@software{elmaaroufi2026roboeval,
  title   = {roboeval: A reproducible evaluation harness for Vision-Language-Action models},
  author  = {Elmaaroufi, Karim and OMAR and Seshia, Sanjit A. and Zaharia, Matei},
  version = {0.1.1},
  date    = {2026-04-29},
  url     = {https://github.com/KE7/roboeval},
  license = {BSD-3-Clause}
}

License

roboeval is released under the BSD-3-Clause License.

Contributors

KE7

3 commits

Similar repos

KE7/libero-infinity

Scenic 3-based open-ended evaluation for robotic manipulation — infinite, statistically-diverse…

same org

allenai/vla-evaluation-harness

One framework to evaluate any VLA model on any robot simulation benchmark.

592

JAVAdcc/wam-lab

WAM-Lab: FastWAM + LIBERO evaluation bridge based on vla-evaluation-harness

NVlabs/RoboLab

A simulation benchmarking platform for evaluating generalist robot policies

483

robocurve/inspect-robots

Open source evals for physical AI. Run any LLM/VLA on any arm/humanoid against any real/sim…

385

EmbodiedArena/embodied-arena

OpenMOSS/VLABench

A large-scale benchmark for evaluating vision-language-action models, embodied agents, and…

470

WwZzz/VLAStudio

Imitation Learning; Robotics; Policy; VLA;

Languages

Python

90.2%

Shell

9.8%

KE7/roboeval

A reproducible evaluation harness for Vision-Language-Action models across simulator backends

stars

commits

Python

primary language

Aug 26, 2026

updated

README

roboeval

Method / Contracts

The main contract surfaces are:

Surface	Role
`ActionObsSpec` gate	VLA and simulator components declare action format, dimensionality, value range, camera roles, image format, state layout, and language inputs. Under the default strict mode, incompatible declarations stop the run before episode 1.
Host-process isolation	VLA servers, simulator workers, and optional VLM proxy processes run in separate `.venvs/` environments. This allows different Python and CUDA dependency stacks to coexist without a monolithic runtime.
Dependency isolation	Each VLA and simulator keeps its upstream package pins, Python version, CUDA assumptions, and optional micromamba/uv environment separate. This is a design choice: adding a new backend should not force the orchestrator or other backends onto the same dependency closure.
LITEN-style hierarchical evaluation	The hierarchical mode integrates the VLM-planner method introduced by Shah et al. (Learning Affordances at Inference-Time for Vision-Language-Action Models). The planner emits subtask calls that are executed by the same VLA server interface used for direct evaluation. roboeval is, to our knowledge, the first public VLA evaluation harness to ship a working LITEN integration.
Result records	`roboeval run` writes JSON with harness version, config snapshot, per-episode metadata, success flags, and optional shard metadata.

Documentation map

Installation

For full prerequisites, platform notes, and per-component dependency details, see docs/install.md.

git clone https://github.com/KE7/roboeval.git
cd roboeval
roboeval setup pi05 libero

The setup script provisions the orchestrator plus the requested VLA and simulator environments under .venvs/.

Quickstart

roboeval setup pi05 libero
roboeval serve --vla pi05 --sim libero --headless
roboeval test --validate -c configs/libero_spatial_pi05_smoke.yaml
roboeval run -c configs/libero_spatial_pi05_smoke.yaml

Supported VLAs and Simulators

The table describes shipped coverage. It is a support matrix, not a benchmark table; supported pairs are tested end-to-end.

VLA	Simulator	Coverage	Example config
Pi0.5	LIBERO	direct, LITEN	`configs/libero_spatial_pi05_smoke.yaml`, `configs/libero_spatial_pi05_liten_smoke.yaml`
Pi0.5	LIBERO-Pro	direct, LITEN	`configs/libero_pro_pi05_smoke.yaml`, `configs/libero_pro_pi05_liten_smoke.yaml`
Pi0.5	LIBERO-Infinity	direct, LITEN	`configs/libero_infinity_pi05_smoke.yaml`, `configs/libero_infinity_pi05_liten_smoke.yaml`
SmolVLA	LIBERO	direct, LITEN	`configs/libero_object_smolvla_smoke.yaml`, `configs/libero_object_smolvla_liten_smoke.yaml`
OpenVLA	LIBERO	direct, LITEN	`configs/libero_spatial_openvla_smoke.yaml`, `configs/libero_spatial_openvla_liten_smoke.yaml`
GR00T	LIBERO	direct, LITEN	`configs/libero_spatial_groot_smoke.yaml`, `configs/libero_spatial_groot_liten_smoke.yaml`
InternVLA	RoboTwin	direct, LITEN	`configs/robotwin_internvla_smoke.yaml`, `configs/robotwin_internvla_liten_smoke.yaml`
ACT	ALOHA Gym	direct, LITEN	`configs/aloha_gym_act_smoke.yaml`, `configs/aloha_gym_act_liten_smoke.yaml`
Diffusion Policy	gym-pusht	direct	`configs/gym_pusht_diffusion_policy_smoke.yaml`
VQ-BeT	gym-pusht	direct	`configs/gym_pusht_vqbet_smoke.yaml`
TDMPC2	Meta-World	direct	`configs/metaworld_tdmpc2_smoke.yaml`
InternVLA	ALOHA Gym	CI smoke	`configs/ci/aloha_gym_internvla_smoke.yaml`
ManiSkill2	ManiSkill2 backend	backend scaffold; x86_64 execution path	setup target `maniskill2`
RoboCasa	RoboCasa backend	simulator backend and registry support	setup target `robocasa`

Current limitations

ManiSkill2 is platform-blocked on aarch64 because the required SAPIEN 2.x wheels are x86_64-only.
bridge_octo is platform-blocked on aarch64 by its current TensorFlow/dlimp dependency chain and does not ship in the v0.1.0 support matrix.
Some technically expressible pairs remain capability boundaries and do not ship root configs, including RoboCasa x GR00T.

Planned features

Multi-architecture CI matrix. aarch64 is currently the primary CI path; x86_64 execution paths exist but are not in the CI matrix.
Additional VLAs as their checkpoints become available.
More simulators. Community contributions are welcome; see docs/extending.md.

Extending

Extension cost. Adding a new VLA averages ~200 SLOC; adding a new simulator backend averages ~230 SLOC (across the v0.1.0 release; excludes blank lines, comments, and docstrings).

Add a VLA by implementing a policy server with /health, /info, /reset, and /predict, then registering it with roboeval serve.
Add a simulator by implementing a SimBackendBase backend with /init, /reset, /step, /obs, /success, and /info support through the sim worker.
Add a new compatibility path by declaring ActionObsSpec records on both sides and adding a smoke config under configs/.

See docs/extending.md for the extension architecture and step-by-step entry points.

Citations

If you use roboeval in your research, please cite us.

@software{elmaaroufi2026roboeval,
  title   = {roboeval: A reproducible evaluation harness for Vision-Language-Action models},
  author  = {Elmaaroufi, Karim and OMAR and Seshia, Sanjit A. and Zaharia, Matei},
  version = {0.1.1},
  date    = {2026-04-29},
  url     = {https://github.com/KE7/roboeval},
  license = {BSD-3-Clause}
}

License

roboeval is released under the BSD-3-Clause License.

Similar repos

KE7/libero-infinity

Scenic 3-based open-ended evaluation for robotic manipulation — infinite, statistically-diverse…

same org

allenai/vla-evaluation-harness

One framework to evaluate any VLA model on any robot simulation benchmark.

592

JAVAdcc/wam-lab

WAM-Lab: FastWAM + LIBERO evaluation bridge based on vla-evaluation-harness

NVlabs/RoboLab

A simulation benchmarking platform for evaluating generalist robot policies

483

robocurve/inspect-robots

Open source evals for physical AI. Run any LLM/VLA on any arm/humanoid against any real/sim…

385

EmbodiedArena/embodied-arena

OpenMOSS/VLABench

A large-scale benchmark for evaluating vision-language-action models, embodied agents, and…

470

WwZzz/VLAStudio

Imitation Learning; Robotics; Policy; VLA;

Contributors

KE7

3 commits

Languages

Python

90.2%

Shell

9.8%