Guardian FailCoT — Out-of-Distribution Real-Robot Benchmarks

This repository bundles the three real-world failure-detection benchmarks used to evaluate the Guardian vision-language model in the paper Scaling Cross-Environment Failure Reasoning Data for Vision-Language Robotic Manipulation (Pacaud et al., 2026):

• UR5-Fail — our newly collected three-view real-robot benchmark.
• RoboFail — single-view real-robot manipulation failure benchmark from Liu et al. (CoRL 2023).
• RoboVQA — single-view real-robot VQA test set from Sermanet et al. (ICRA 2024), restricted here to execution-success prediction questions.

All three are held out of training and used to measure cross-environment generalization of failure detectors. They are released here under a unified schema (metadata + InternVL-format conversations + raw multi-view images) so they can be evaluated with a single pipeline.

The training data (FailCoT: RLBench-Fail + BridgeDataV2-Fail) and the Guardian model are released separately on the Guardian collection.

Splits and statistics

robovqa contains only execution-verification questions (the original dataset's "planning" questions concern next-action prediction rather than full-plan verification, so they are not included here). The provided metadata_execution.jsonl for RoboVQA is restricted to the success/failure verification subset described in the paper.

Repository layout

Guardian-FailCoT-OOD-datasets/
├── README.md
├── ur5fail_test/
│   ├── metadata_execution.jsonl
│   ├── metadata_planning.jsonl
│   ├── internVL_dataset_execution_vanilla.jsonl
│   ├── internVL_dataset_execution_thinking.jsonl
│   ├── internVL_dataset_planning_vanilla.jsonl
│   ├── internVL_dataset_planning_thinking.jsonl
│   └── records.tar.gz                # extract -> records/<taskvar>/<failure_mode>/ep_<id>/<subtask_id>/*.png
├── robofail/
│   ├── metadata_execution.jsonl
│   ├── metadata_planning.jsonl
│   ├── internVL_dataset_execution_vanilla.jsonl
│   ├── internVL_dataset_execution_thinking.jsonl
│   ├── internVL_dataset_planning_vanilla.jsonl
│   ├── internVL_dataset_planning_thinking.jsonl
│   └── records.tar.gz
└── robovqa/
    ├── metadata_execution.jsonl
    ├── internVL_dataset_execution_vanilla.jsonl
    ├── internVL_dataset_execution_thinking.jsonl
    └── records.tar.gz

After downloading a split, extract its images with:

tar -xzf <split>/records.tar.gz -C <split>/

This creates <split>/records/..., the path prefix that all image fields in the JSONL files resolve against.

File formats

metadata_execution.jsonl / metadata_planning.jsonl

One JSON object per line. Each sample carries the full task context, the multi-view images, and a binary success/failure label, plus (for planning) the original instruction/plan and the perturbation that produced the failure.

Common fields:

• taskvar, episode_id, task_instruction, plan, images
• reward — overall sample label (1 = success, 0 = failure)
• planning_reward, execution_reward — which stage failed
• failure_mode — fine-grained category
• failure_reason — natural-language explanation (when available)

Planning-specific fields:

• old_task_instruction, old_plan, old_visible_objects — pre-perturbation reference
• correct_plan — ground-truth plan

internVL_dataset_*.jsonl

InternVL fine-tuning / evaluation format. One conversation per line:

{
  "id": 0,
  "image": ["records/.../start_img_viewpoint_0.png", "..."],
  "height_list": [256, 256],
  "width_list": [256, 256],
  "conversations": [
    {"from": "human", "value": "..."},
    {"from": "gpt",   "value": "<answer> True </answer> <category> success </category>"}
  ]
}

• vanilla variants emit a direct answer: <answer> bool </answer> <category> ... </category>.
• thinking variants emit chain-of-thought first: <think> reasoning </think> <answer> bool </answer> <category> ... </category>.

Image details

• Resolution: 256 × 256 PNG.
• Execution sample: up to 6 images (start + end of subtask × number of available viewpoints — 3 for UR5-Fail, 1 for RoboFail / RoboVQA).
• Planning sample: 1 image (front view of the initial scene).

How these benchmarks were built

• UR5-Fail. Collected on a tabletop UR5 with three Intel RealSense D435 cameras, running the 3D-LOTUS++ policy autonomously across 16 unique tasks (34 task-variations). Subtasks were manually labelled success / failure to obtain execution data; planning failures were generated via the automated perturbation pipeline of FailCoT.
• RoboFail. Re-exported from the original RoboFail benchmark into the Guardian schema (single-view teleoperated trajectories, hand-curated failure annotations).
• RoboVQA. Re-exported from the RoboVQA test split, restricted to execution success/failure verification (357 questions, single front view). Spans three embodiments (Everyday Robots mobile manipulator, a human arm, and a human with a grasping tool).

See Section V (Experiments) and Table II of the paper for the full evaluation protocol.

Quick start

from datasets import load_dataset

# Execution-verification metadata across all three benchmarks
ds = load_dataset("paulpacaud/Guardian-FailCoT-OOD-datasets", "metadata_execution")
print(ds)        # DatasetDict with splits: ur5fail_test, robofail, robovqa

# Or a single benchmark
ur5 = load_dataset("paulpacaud/Guardian-FailCoT-OOD-datasets", "metadata_execution", split="ur5fail_test")

Image paths in the records files are relative to the split directory, so make sure records.tar.gz has been extracted alongside the JSONL files before opening any sample.

Citation

@misc{pacaud2026guardian_failcot,
  title  = {Scaling Cross-Environment Failure Reasoning Data for Vision-Language Robotic Manipulation},
  author = {Paul Pacaud and Ricardo Garcia and Shizhe Chen and Cordelia Schmid},
  year   = {2026},
  eprint = {2512.01946},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO}
}

If you specifically use the RoboFail or RoboVQA splits, please also cite the original benchmarks:

@inproceedings{liu2023reflect,
  title     = {REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction},
  author    = {Liu, Zeyi and Bahety, Arpit and Song, Shuran},
  booktitle = {CoRL},
  year      = {2023}
}
@inproceedings{sermanet2024robovqa,
  title     = {RoboVQA: Multimodal Long-Horizon Reasoning for Robotics},
  author    = {Sermanet, Pierre and others},
  booktitle = {ICRA},
  year      = {2024}
}

License

Released under the Apache 2.0 license. The RoboFail and RoboVQA splits are redistributed in this repository for benchmark convenience; please refer to the original projects for their respective terms of use.