Show-Harness VLMs

LoRA adapters that turn a vision-language model into a robot controller: two camera views in, one semantic action token out. Every unit is a 2 cm translation on every rig, so real and simulated data mix without rescaling.

Paper · Code · Dataset · Project page

Adapters

Two corpora. The five real-robot adapters differ only in backbone — same data, same hyper-parameters — so they read as a clean model-scale comparison. The simulation adapter is a single policy covering both simulators, not one per simulator.

The output vocabulary is only nine tokens, so every backbone fits the training set — compare these on closed-loop success rate, not on training curves.

Recipe, identical for every adapter: LoRA r=64, alpha=128, dropout 0.05, target=all, vision tower frozen; LR 1e-4 cosine with warmup ratio 0.1; bf16, DeepSpeed ZeRO-2; effective batch 32. The real corpus is 7,933 samples (5,070 Franka + 2,863 AgileX) × 40 epochs = 9,920 steps; the sim corpus is 13,753 samples (7,813 RoboLab + 5,940 ManiSkill) × 30 epochs = 12,900 steps. Trained with LLaMA-Factory.

Two contracts that fail silently

Chat template. Each folder's chat_template.jinja reproduces what training rendered — not the base model's own. Qwen3.5's official template emits an empty think block after the assistant turn even with enable_thinking=false, while the training template emits nothing. Serve the wrong one and the model still answers, just off distribution.

Direction convention. All directions follow the Franka rig's overhead exocentric view. The AgileX rig observes first-person, so deploying there requires swapping MV_FWD and MV_BACK. Franka and simulation need no conversion.

Usage

Serving and closed-loop deployment go through the Show-Harness repository, which owns the vLLM launch path, prompt assembly, and token-to-motion mapping.

# real-robot policy
MODEL=Qwen/Qwen3.5-2B FAMILY=qwen3_5 \
  LORA=qwen3_5_2b_showharness_ft=<path>/qwen3_5_2b bash scripts/serve_vlm.sh

# simulation policy — RoboLab and ManiSkill both select this one
MODEL=Qwen/Qwen3.5-2B FAMILY=qwen3_5 \
  LORA=qwen3_5_2b_showharness_sim=<path>/qwen3_5_2b_sim bash scripts/serve_vlm.sh

Adapter names follow <folder>_showharness_<split> and are what clients request with --model. All adapters here were trained against the v3 unified prompt, so run them with --version v3.

Data

The demonstrations these were trained on are at Show-Harness-Data — 164 real episodes (Franka + AgileX, 17 tasks) and 230 simulated (RoboLab + ManiSkill), every observation paired with exactly one action token.

License

The Qwen3.5 adapters are Apache-2.0. gemma4_e4b derives from google/gemma-4-E4B-it and is governed by the Gemma Terms of Use — the repo-level license tag cannot express both.

Citation

@misc{chen2026showharnessjustvlmagent,
      title={Show-Harness: Just a VLM Agent Can Play Robots}, 
      author={Yanzhe Chen and Zechen Bai and Zhijun Cao and Wenzheng Zeng and Kevin Qinghong Lin and Yiqi Lin and Guoqiang Liang and Kevin Yuchen Ma and Qiming Huang and Mike Zheng Shou},
      year={2026},
      eprint={2609.10522},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2609.10522}, 
}