PRIMO SFT Data

Stage-1 (SFT cold start) training annotations for PRIMO R1 (paper). Each record carries a chain-of-thought trace with planning / observation / reasoning subsections, which is what the model imitates before RL.

116,755 records across 10 subsets. Annotations only (856 MB); videos are in primo-video-media.

Subsets

Download by subset — start here

The annotations are small; the videos are not. behavior-1k is 19,991 records (17% of this repo) but its video archive is 5,981 GB — 91% of the entire 6.58 TB media release. Pulling the full mixture without meaning to is the single easiest mistake to make with this dataset.

Recommendation: skip behavior-1k unless you specifically need the long-horizon task-progress results. Its episodes are the longest in the release, so it is what to add if long-horizon progress estimation is your target — and it is safe to leave out otherwise. Everything except behavior-1k, agibot, and seed-bench-r1 comes to about 71 GB of video.

The training launchers read their mixture from an environment variable, so dropping a subset needs no code change:

# Full mixture minus behavior-1k
export SFT_DATASET=primo-sft-agibot,primo-sft-robovqa,primo-sft-robotwin-clean,primo-sft-robotwin-randomized,primo-sft-nextqa,primo-sft-perceptiontest,primo-sft-seed-bench-r1,primo-sft-star,primo-sft-sharerobot

# Or something small to start with: ~23 GB of video total
export SFT_DATASET=primo-sft-robotwin-clean,primo-sft-robotwin-randomized,primo-sft-sharerobot,primo-sft-robovqa

One subset

hf download LeonOverload/primo-sft-json --repo-type dataset \
    --include "jsonl_subsets/robotwin-clean/*" --local-dir ./primo-sft

Training-ready layout

The training entry points read raw_json/, not the JSONL shards. Download it into $VIDEO_DATA_ROOT so the paths the dataset registry expects (primo-sft/<source>/train_cot.json) line up:

export VIDEO_DATA_ROOT=/path/to/PRIMO-Data

# All 10 subsets, 856 MB
hf download LeonOverload/primo-sft-json --repo-type dataset \
    --include "raw_json/*" --local-dir /tmp/primo-sft
cp -r /tmp/primo-sft/raw_json/primo-sft "$VIDEO_DATA_ROOT/"

# Or just the subsets you plan to train on
hf download LeonOverload/primo-sft-json --repo-type dataset \
    --include "raw_json/primo-sft/robotwin-clean/*" \
    --include "raw_json/primo-sft/sharerobot/*" \
    --local-dir /tmp/primo-sft

Use --include on its own — passing filenames positionally makes the client ignore it.

Browsing

from datasets import load_dataset

ds = load_dataset("LeonOverload/primo-sft-json", "robotwin-clean", split="train")
print(ds[0]["planning"], ds[0]["observation"], ds[0]["reasoning"])

Convenient for inspection, but use raw_json/ for training.

Videos

Match each subset to its group and pull only those:

hf download LeonOverload/primo-video-media --repo-type dataset \
    --include "robotwin.z*" --include "sharerobot.z*" \
    --local-dir /tmp/primo-video-zips

Extract into $VIDEO_DATA_ROOT/primo-video/. See primo-video-media for reassembly — the archives are multipart, and they already include the pre-extracted anchor frames.

Record format

{
  "problem": "Task info:\nSort in the warehouse\n\nInit Scene:\n...",
  "problem_type": "numerical",
  "data_type": "video",
  "path": "./agibot/observation/414/687577/42.9/head_color.mp4",
  "solution": "<answer>42.9</answer>",
  "options": [],
  "done_actions": [...],
  "meta_data": {...},

  # CoT supervision — what stage 1 trains on
  "select": True,               # only select == true records are used
  "process": "...",
  "planning": "...",
  "observation": "...",
  "reasoning": "...",
  "answer": "...",
  "reward": ...,

  # interleaved-input anchors, read directly during training
  "init_frame_path": "./primo-video/agibot/frames/observation/414/687577/42.9/head_color_init.jpg",
  "current_frame_path": "./primo-video/agibot/frames/observation/414/687577/42.9/head_color_current.jpg",
  "video_hash": "..."
}

Two filters are applied at load time: select == true, and all four of process / planning / observation / reasoning non-empty. Records failing either are dropped, so the count reaching the trainer is lower than the raw row count.

Unlike evaluation — which re-derives anchor frames with OpenCV at run time — training reads init_frame_path and current_frame_path from the JSON. Those files ship inside the video archives, so no preprocessing step is needed. src/preprocess_video_frames.py exists for datasets you add yourself.

Training

git clone https://github.com/10-OASIS-01/PRIMO-R1 && cd PRIMO-R1
conda create -n primo-r1 python=3.11 && conda activate primo-r1
bash setup.sh

export VIDEO_DATA_ROOT=/path/to/PRIMO-Data
export MODEL_ROOT=/path/to/models
export SFT_DATASET=primo-sft-robotwin-clean,primo-sft-sharerobot   # see above

bash src/scripts/run_sft_7b.sh      # or run_sft_3b.sh

Stage-1 defaults follow the paper: learning rate 1.0e-6, per-device batch size 1, gradient accumulation 8, DeepSpeed ZeRO-3. Keep per_device_train_batch_size=1 — that is an invariant inherited from R1-V. The 3B launcher is scaled from the 7B recipe and has not been validated; the released checkpoint is 7B.

Stage 2 continues from the resulting checkpoint using primo-rl-json. If you would rather skip stage 1, the published cold start is PRIMO-COT-SFT-7B.

Resources

Repo contents

• raw_json/ — the files the training entry points read, in release layout
• jsonl/ — flattened shards for the Dataset Viewer
• jsonl_subsets/ — per-subset shards backing the viewer's subset selector
• summary.json — row and shard counts generated at build time

The authoritative dataset registry is the cfg dict in src/r1-v/src/open_r1/DatasetLoader.py.

Citations

If you find our work helpful for your research, please consider citing our work.

@misc{liu2026passiveobserveractivecritic,
      title={From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation}, 
      author={Yibin Liu and Yaxing Lyu and Daqi Gao and Zhixuan Liang and Weiliang Tang and Shilong Mu and Xiaokang Yang and Yao Mu},
      year={2026},
      eprint={2603.15600},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2603.15600}, 
}