PRIMO Bench

Benchmark annotations for PRIMO R1 (paper): task-progress estimation and failure detection for long-horizon robotic manipulation, across four environments with in-domain and out-of-domain splits.

23,704 samples, 7 splits. This repo holds annotations only (71 MB); videos are in primo-video-media.

Splits

real-humanoid-ood is the cross-environment generalization split — real hardware, never seen in training. It is where SFT-only models fall off most sharply.

Start with robotwin. At 12.5 GB of video it is by far the cheapest complete split, and it covers both ID and OOD. behavior-1k needs ~6 TB of video; do not reach for it first.

Record format

{
  "problem": "Task info:\nSort in the warehouse\n\nInit Scene:\nA robot stands in front of a table ...",
  "problem_type": "numerical",          # numerical | regression | multiple choice | boolean | free-form
  "data_type": "video",
  "path": "./agibot/observation/359/687696/20.0/head_color.mp4",
  "solution": "<answer>20.0</answer>",  # ground truth, same tag format the model must emit
  "options": [],                        # non-empty only for multiple choice
  "done_actions": ["Retrieve wet wipes from left material frame with the left arm."],
  "meta_data": {"episode_id": 687696, "task_id": 359, "task_name": "Sort in the warehouse"},
  "init_frame_path": "./primo-video/agibot/frames/observation/359/687696/20.0/head_color_init.jpg",
  "current_frame_path": "./primo-video/agibot/frames/observation/359/687696/20.0/head_color_current.jpg"
}

Progress targets are percentages on a 0–100 scale. All media paths are relative and resolve against your local data root.

Browsing

from datasets import load_dataset

ds = load_dataset("LeonOverload/primo-bench-json", "robotwin-ood", split="train")
print(ds[0]["problem"], ds[0]["solution"])

Good for inspecting the data. To actually run an evaluation, use raw_json/ as below — the eval harness reads those files, not the JSONL shards.

Running the benchmark

1. Environment

git clone https://github.com/10-OASIS-01/PRIMO-R1 && cd PRIMO-R1
conda create -n primo-r1 python=3.11 && conda activate primo-r1
bash setup.sh

export VIDEO_DATA_ROOT=/path/to/PRIMO-Data
export MODEL_ROOT=/path/to/models

The vendored transformers-main/ tree that setup.sh installs last is required — installing a PyPI transformers over it is the usual cause of shape and processor errors.

2. Annotations

Download raw_json/ straight into $VIDEO_DATA_ROOT, which reproduces the layout the dataset registry expects (primo-bench/<source>/{id,ood}.json):

hf download LeonOverload/primo-bench-json --repo-type dataset \
    --include "raw_json/*" --local-dir /tmp/primo-bench

mkdir -p "$VIDEO_DATA_ROOT"
cp -r /tmp/primo-bench/raw_json/primo-bench "$VIDEO_DATA_ROOT/"

3. Videos

Pull only the group your split needs — see the table above. For robotwin:

hf download LeonOverload/primo-video-media --repo-type dataset \
    --include "robotwin.z*" --local-dir /tmp/primo-video-zips

That one pattern matches all parts including the final .zip. Reassemble and extract into $VIDEO_DATA_ROOT/primo-video/:

cd /tmp/primo-video-zips
7z x robotwin.zip -o"$VIDEO_DATA_ROOT/primo-video/"

# without 7-Zip:
#   zip -FF robotwin.zip --out robotwin_fixed.zip
#   unzip robotwin_fixed.zip -d "$VIDEO_DATA_ROOT/primo-video/"

The archives already contain the pre-extracted anchor frames, so there is no preprocessing step. The resulting tree:

$VIDEO_DATA_ROOT/
├── primo-bench/robotwin/{id,ood}.json
└── primo-video/robotwin/
    ├── videos/...
    └── frames/...

4. Model

hf download LeonOverload/PRIMO-R1-7B --local-dir "$MODEL_ROOT/PRIMO-R1-7B"

5. Evaluate

Models and splits are selected by editing a comment-toggled heredoc in the launcher — there are no CLI flags for them:

# src/eval/src/eval_interleave_local.sh
model_paths=$(cat <<EOF | grep -v '^#' | grep -v '^$'
$MODEL_ROOT/PRIMO-R1-7B
# $MODEL_ROOT/PRIMO-COT-SFT-7B
EOF
)

file_names=$(cat <<EOF | grep -v '^#' | grep -v '^$'
primo-bench-ood-robotwin
# primo-bench-id-robotwin
# primo-bench-ood-real-humanoid
EOF
)

bash src/eval/src/eval_interleave_local.sh

Or call the harness directly:

python src/eval/eval_interleave.py \
    --model_path "$MODEL_ROOT/PRIMO-R1-7B" \
    --file_name primo-bench-ood-robotwin \
    --output_path src/r1-v/eval_outputs/interleave/PRIMO-R1-7B/primo-bench-ood-robotwin.json \
    --sample_size 50          # smoke test; drop or set 0 for the full split

Results land in src/r1-v/eval_outputs/interleave/<model>/<split>.json, with per-sample records under results and a summary under final_acc. Runs resume from an existing output file, so an interrupted eval continues where it stopped.

Use eval_interleave_local.sh for PRIMO R1 models. eval_baseline_local.sh is for video-only baselines (Qwen2.5-VL, Cosmos-Reason1, RoboBrain, ProgressLM, InternVL 3.5), and eval_api.sh for OpenAI-compatible endpoints.

Metrics, and one thing to watch

Answers are scored per problem_type: multiple choice and boolean by exact match, free-form by mean ROUGE-1/2/L F-measure, numerical and regression by relative accuracy.

The harnesses score regression with three different formulas, and they are not comparable. Each output file records which one it used in its regression_metric field:

The published PRIMO R1 numbers come from linear_relative_accuracy. The threshold variant is stricter and takes only 10 discrete values per sample. Check the field before putting two numbers in the same table.

Decoding follows the Qwen2.5-VL demo: top_p=0.001, temperature=0.01. Larger top_p produces garbled output. The published results used an effective cap of 22 frames per video.

Reference results

Progress estimation averaged over four environments (paper Table 1):

Resources

Repo contents

• raw_json/ — the files the eval harness reads, in release layout
• jsonl/ — flattened shards for the Dataset Viewer
• jsonl_subsets/ — per-split shards backing the viewer's subset selector
• summary.json — row and shard counts generated at build time

The authoritative split registry is the cfg dict in src/r1-v/src/open_r1/DatasetLoader.py; add new splits there rather than passing raw paths through scripts.

Citations

If you find our work helpful for your research, please consider citing our work.

@misc{liu2026passiveobserveractivecritic,
      title={From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation}, 
      author={Yibin Liu and Yaxing Lyu and Daqi Gao and Zhixuan Liang and Weiliang Tang and Shilong Mu and Xiaokang Yang and Yao Mu},
      year={2026},
      eprint={2603.15600},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2603.15600}, 
}