Same task, selected perturbation axes and presets — including robot joint-reset perturbation. Every reset() draws a new scene.

Highlights

• Open-Ended Evaluation -- Sample unlimited i.i.d. test scenes from constraint-checked probability distributions; success rate converges to a true population statistic
• 10 Composable Perturbation Axes -- Position, object identity, robot joint reset, camera, lighting, texture, distractor objects, background, articulation, sensor noise -- mix any subset
• Unlimited Tasks, Zero Hand-Writing -- Point at any LIBERO BDDL task file; Scenic programs are synthesized automatically
• Adversarial Search -- Cross-entropy Bayesian optimization finds worst-case scenes via VerifAI
• Task Reversal -- Flip any forward task backward for novel evaluation from goal-state initial configs
• Gym API -- Standard gym.Env wrapper with make_vec_env for RL/VLA training with built-in domain randomization

Contents

• Quick Start
• Perturbation Axes
• Usage
• Architecture
• Documentation
• Citation
• License
• Acknowledgments

Quick Start

Prerequisites

No make? See the Without make section below — all steps can be run with plain shell + uv.

Installation

# 1. Install uv (fast Python package manager) — https://docs.astral.sh/uv/
curl -LsSf https://astral.sh/uv/install.sh | sh
# Fallback: pip install uv

# 2. Clone and install
git clone https://github.com/KE7/libero-infinity.git && cd libero-infinity
make install        # creates venv, installs deps, installs pinned LIBERO source
make setup-assets   # downloads LIBERO assets from HF (only needed once)
make test           # runs test suite (headless)

git clone https://github.com/KE7/libero-infinity.git && cd libero-infinity

# Equivalent to: make install
uv venv --python 3.11
uv sync --extra dev
uv run libero-inf-bootstrap

# Equivalent to: make setup-assets
PYTHONPATH=src uv run python -c "from libero_infinity.runtime import ensure_runtime; ensure_runtime()"

# Equivalent to: make test
MUJOCO_GL=egl PYTHONPATH=src .venv/bin/python -m pytest tests/ -v

# Equivalent to: make test-fast  (no GPU required)
MUJOCO_GL=egl PYTHONPATH=src .venv/bin/python -m pytest tests/test_scenic.py tests/test_perturbation_policy.py tests/test_planner.py -v

# Equivalent to: make verify
MUJOCO_GL=egl uv run python scripts/verify_build.py

git clone https://github.com/KE7/libero-infinity.git
cd libero-infinity

# Create venv and install
uv venv --python 3.11
uv sync --extra dev
uv run libero-inf-bootstrap

# Download and validate HF assets, configure LIBERO runtime
PYTHONPATH=src uv run python -c "from libero_infinity.runtime import ensure_runtime; ensure_runtime()"

# Verify
MUJOCO_GL=egl uv run python -m pytest tests/test_e2e.py -v

Run Your First Evaluation

# Connect a VLA via robo-eval, then run:
MUJOCO_GL=egl libero-eval \
  --bddl src/libero_infinity/data/libero_runtime/bddl_files/libero_goal/put_the_bowl_on_the_plate.bddl \
  --perturbation combined --n-scenes 5 --verbose

Note: libero-eval requires a running robo-eval server — there is no built-in default policy. See examples/05_robo_eval_cli.sh to connect a VLA via the server, or use the Python API directly as shown below.

Evaluate a Real Policy (Python API)

The Python API is the primary way to evaluate VLA policies. Here is a complete, copy-paste-ready example using a HuggingFace VLA model:

import numpy as np
from transformers import AutoModelForVision2Seq, AutoProcessor
from libero_infinity.eval import evaluate
from libero_infinity.task_config import TaskConfig
from libero_infinity.compiler import generate_scenic_file

# --- 1. Load your VLA model ---
model_id = "openvla/openvla-7b-finetuned-libero-spatial"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForVision2Seq.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto", trust_remote_code=True
)

# --- 2. Define a policy callable: obs_dict -> action (7,) ---
def make_policy(instruction: str):
    """Create a policy closure that captures the task instruction."""
    def policy(obs: dict) -> np.ndarray:
        # Get the third-person camera image and flip to standard orientation
        image = obs["agentview_image"][::-1]  # OpenGL origin is bottom-left

        # Run VLA inference (model-specific — adapt to your model)
        inputs = processor(image, instruction).to(model.device)
        action = model.predict_action(**inputs)  # shape: (7,)

        # action format: [dx, dy, dz, dax, day, daz, gripper]
        # range: [-1, 1] per dimension (OSC_POSE controller)
        return np.array(action, dtype=np.float64)
    return policy

# --- 3. Iterate over all tasks in a benchmark suite ---
import glob

bddl_dir = "src/libero_infinity/data/libero_runtime/bddl_files/libero_spatial"
bddl_files = sorted(glob.glob(f"{bddl_dir}/*.bddl"))

for bddl_path in bddl_files:
    # Parse the task to get the language instruction
    cfg = TaskConfig.from_bddl(bddl_path)
    print(f"Task: {cfg.language}")

    # Create the policy with the task instruction
    policy = make_policy(cfg.language)

    # Auto-generate a Scenic perturbation program from the BDDL
    scenic_path = generate_scenic_file(cfg, perturbation="combined")

    # Evaluate on 50 perturbed scenes
    results = evaluate(
        scenic_path=scenic_path,
        bddl_path=bddl_path,
        policy=policy,
        n_scenes=50,
        max_steps=300,
        env_kwargs={"camera_heights": 224, "camera_widths": 224},  # match model resolution
        verbose=True,
    )
    print(results.summary())
    # -> "Success rate: 73.5% +/- 6.1% (147/200 scenes)"

generate_scenic_file(...) writes transient programs to scenic/generated/ by default. That directory is gitignored. Remove leftovers with make clean-generated.

scenic_path = generate_scenic_file(cfg, perturbation="combined")

results_lang    = evaluate(scenic_path, bddl_path, make_policy(cfg.language), n_scenes=50)
results_no_lang = evaluate(scenic_path, bddl_path, make_policy(""),           n_scenes=50)

lang_contribution = results_lang.success_rate - results_no_lang.success_rate
print(f"With language:    {results_lang.success_rate:.1%}")
print(f"Vision-only:      {results_no_lang.success_rate:.1%}")
print(f"Language contrib: {lang_contribution:+.1%}")
# A small gap means the policy is ignoring your instructions.
# cf. VLA-VA (arXiv 2602.17659) Table I: OpenVLA-OFT scores 83% vision-only vs. 97% full.

def make_chunked_policy(instruction: str, chunk_size: int = 50):
    """Policy wrapper for models that output action chunks."""
    action_queue = []

    def policy(obs: dict) -> np.ndarray:
        nonlocal action_queue
        if not action_queue:
            # Query the model for a new chunk of actions
            actions = my_vla_model.predict(obs, instruction)  # shape: (chunk_size, 7)
            action_queue = list(actions)
        return action_queue.pop(0)
    return policy

See docs/observations-actions.md for the full policy interface details (obs keys, action format, task instructions).

See docs/installation.md for detailed setup instructions and docs/evaluation_pipeline.md for the full CLI reference.

Perturbation Axes

LIBERO-Infinity exposes two complementary perturbation surfaces:

1. Online Scenic axes (10) — distributions over scene parameters that Scenic samples at every scenario.generate() call. The simulator reads the sampled params and applies them after env construction. These are the axes you toggle via --perturbation.
2. Offline BDDL-rewrite generators (3) — pre-generate alternative BDDL files (different goals, languages, or arenas) by editing the source BDDL text. These can't be Scenic-driven because LIBERO's BDDL parser reads goal/language/fixtures before Scenic-sampled params reach the simulator. They're invoked once per task to emit a set of variant BDDLs which the gym wrapper or eval harness selects from per reset.

Online Scenic axes

Offline BDDL-rewrite generators

All Scenic axes are arbitrarily composable -- specify any combination:

--perturbation position                    # single axis
--perturbation position,camera             # two axes
--perturbation object,lighting,distractor  # three axes
--perturbation combined                    # preset: position + object + robot + camera + lighting + distractor + background
--perturbation full                        # all online axes (combined + texture + articulation + sensor_noise)

See docs/scenic_perturbations.md for full parameter details, Scenic code examples, and more screenshots.

How It Works

LIBERO-Infinity is organized as a three-layer pipeline:

BDDL task file
     │  task_config.py parses objects, fixtures, regions
     ▼
Scenic 3 program  (auto-generated or hand-written)
     │  Scenic rejection sampler draws a scene
     ▼
MuJoCo simulation  (via LIBERO + robosuite)
     │  Policy rolls out; success/failure recorded
     ▼
Aggregate statistics with 95% Wilson confidence intervals

Usage

CLI Evaluation

# Standard evaluation (i.i.d. sampling with confidence intervals)
libero-eval --bddl path/to/task.bddl --perturbation full --n-scenes 200 --verbose

# Adversarial search for worst-case scenes
libero-eval --bddl path/to/task.bddl --mode adversarial --n-scenes 200

# Reversed task (goal becomes initial state)
libero-eval --bddl path/to/task.bddl --reverse --perturbation position --n-scenes 100

# Live visualization (requires DISPLAY)
libero-eval --bddl path/to/task.bddl --n-scenes 5 --watch cv2 --verbose

Python API

from libero_infinity.eval import evaluate

results = evaluate(
    scenic_path="scenic/combined_perturbation.scenic",
    bddl_path="path/to/task.bddl",
    policy=my_policy_fn,     # (obs_dict) -> action_array (7,)
    n_scenes=200,
    verbose=True,
)
print(results.summary())
# -> "Success rate: 73.5% +/- 6.1% (147/200 scenes)"

Gym Environment

from libero_infinity.gym_env import LIBEROScenicEnv, make_vec_env

# Single environment -- each reset() samples a new perturbed scene
env = LIBEROScenicEnv(
    bddl_path="path/to/task.bddl",
    perturbation="combined",
    resolution=256,
)
obs = env.reset()
for _ in range(300):
    action = my_policy(obs)
    obs, reward, done, info = env.step(action)
    if done:
        break
env.close()

# Parallel rollouts (subprocess-based)
vec_env = make_vec_env("path/to/task.bddl", n_envs=8, perturbation="full")

See docs/evaluation_pipeline.md for the complete CLI reference.

Architecture

Layered Scenic Design

See docs/architecture.md for detailed diagrams and design decisions.

Observation and Action Spaces

See docs/observations-actions.md for the full schema and policy interface.

Documentation

Links

Glossary

Citation

If you find LIBERO-Infinity useful in your research, please consider citing:

@article{libero_infinity2026,
  title     = {LIBERO-Infinity: Open-Ended Robotic Manipulation Evaluation
               via Scenic 3 Probabilistic Programs},
  author    = {Karim Elmaaroufi and OMAR and Matei Zaharia and Sanjit A. Seshia},
  year      = {2026},
}

LIBERO-Infinity is introduced in the OMAR blog post: Introducing OMAR by OMAR (One Man Army).

If you use the LIBERO benchmark tasks, please also cite:

@article{liu2023libero,
  title   = {LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning},
  author  = {Liu, Bo and Zhu, Yifeng and Gao, Chongkai and Feng, Yihao
             and Liu, Qiang and Zhu, Yuke and Stone, Peter},
  journal = {arXiv preprint arXiv:2306.03310},
  year    = {2023}
}

License

Acknowledgments

LIBERO-Infinity is built directly on top of the LIBERO codebase (installed from a pinned upstream commit, MIT License © 2023 Lifelong Robot Learning) and inherits its full 3D asset pack, including all assets originally attributed by LIBERO to their respective sources. We thank Bo Liu, Yifeng Zhu, and the LIBERO team (Yang et al., 2023) for building and open-sourcing the benchmark that makes this work possible.

The probabilistic programming layer of LIBERO-Infinity is built on Scenic 3, the scenario specification language developed at UC Berkeley by Daniel Fremont, Tommaso Dreossi, Shromona Ghosh, Xiangyu Yue, Alberto Sangiovanni-Vincentelli, and Sanjit A. Seshia (Fremont et al., 2023). Scenic's constraint-checking rejection sampler is what makes open-ended, physically-plausible scene distributions possible.

We acknowledge the LIBERO-PRO team for their prior work on VLA robustness evaluation, which helped motivate this work. LIBERO-Infinity also uses the MuJoCo physics engine.

Some 3D assets are sourced from the Google Scanned Objects dataset, licensed under CC BY 4.0.