Show-Harness Data

The demonstrations behind Show-Harness VLMs: two real embodiments (7-DoF Franka, 6-DoF AgileX) and two simulators (RoboLab, ManiSkill). One observation, one action unit — every unit a 2 cm translation on every rig, so the subsets mix without rescaling.

Paper · Code · Models · Project page

Each split is subdivided by fields in episodes.jsonl, not by directory. Within sim/, rollout_000..129 are RoboLab and rollout_130..229 are ManiSkill — but read the simulator field rather than the numbering, which is an implementation detail.

Layout

Show-Harness-Data/
├── real/
│   ├── rollouts.json          training samples
│   ├── episodes.jsonl         one line per episode
│   └── metadata/rollout_000/  actions.jsonl + agentview/*.png + wrist/*.png
└── sim/                       same three pieces

rollouts.json — Alpaca multimodal format, one sample per action step. Image paths are relative to the file itself, always agentview first and wrist second, matching the two leading <image> markers. Nine units: six MV_* directions (FWD BACK LEFT RIGHT UP DOWN), GRASP, RELEASE, and the terminal DONE.

{
  "instruction": "<image><image>" + rendered prompt,
  "input": "",
  "output": "MV_DOWN",
  "images": ["metadata/rollout_000/agentview/0000.png",
             "metadata/rollout_000/wrist/0000.png"]
}

episodes.jsonl — one line per episode, indexing by embodiment, task and environment. embodiment is franka (overhead exocentric) or piper (AgileX, first-person). Simulation episodes also carry simulator, env_id and the seed that reproduces them; ManiSkill episodes add shard and scheme.

actions.jsonl — the per-episode capture log (action, gripper state, end-effector pose, frame names), useful for deriving alternative supervision targets such as continuous EE deltas. It is the raw capture, not the supervision target: train on rollouts.json.

Image transform

Both views go through one transform — rotate/flip, centre-crop to 4:3, letterbox into 256×256 — so a wrist frame carries 32 black rows top and bottom on every subset, real and simulated alike. A policy trained on one source sees the same framing on any other.

⚠️ Direction convention on the AgileX arm

Directional units follow the Franka rig's overhead exocentric view throughout, a single convention shared by both embodiments. The AgileX rig observes from a first-person view that mirrors the forward–backward axis, so deployment to an AgileX arm requires exchanging MV_FWD and MV_BACK — otherwise the arm moves the wrong way, with no error raised. Franka and simulation require no conversion.

Usage

To train with the pipeline in the Show-Harness repository:

git clone https://github.com/showlab/Show-Harness && cd Show-Harness

bash train/scripts/download_dataset.sh            # fetch + register both splits
bash train/scripts/setup_llamafactory.sh          # one-time: clone upstream, build the venv

cp train/configs/qwen3_5_2b_lora.yaml train/configs/my_run.yaml   # set dataset: showharness_sim
CONFIG=train/configs/my_run.yaml GPU=0,1 bash train/scripts/train.sh

media_dir resolves automatically from the registered path, so the relative image paths work with no further configuration.

Models

The adapters trained on these splits are at Show-Harness-VLMs — five real-robot policies from real/ (40 epochs, one per backbone) and qwen3_5_2b_sim from sim/ (30 epochs, one policy for both simulators).

Citation

@misc{chen2026showharnessjustvlmagent,
      title={Show-Harness: Just a VLM Agent Can Play Robots}, 
      author={Yanzhe Chen and Zechen Bai and Zhijun Cao and Wenzheng Zeng and Kevin Qinghong Lin and Yiqi Lin and Guoqiang Liang and Kevin Yuchen Ma and Qiming Huang and Mike Zheng Shou},
      year={2026},
      eprint={2609.10522},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2609.10522}, 
}