τ₀-VLA: a Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation

This repo is the official implementation of τ₀-VLA: a Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation.

News

• [2026.09.21] We release the high-level proposal and world model, with weights, inference and fine-tuning code. See the high-level guide for setup and usage.

• [2026.09.20] We release LIBERO post-training and simulation evaluation. See the LIBERO guide for checkpoints, setup, and evaluation results.

• [2026.08.19] 📢 We plan to progressively release components of the high-level policy. Please stay tuned for updates.

• [2026.07.27] 🚀 We release the τ₀-VLA model Paper, Project Website, and Hugging Face.

Overview

τ₀-VLA is a hierarchical robot foundation model for long-horizon manipulation. A memory-augmented high-level policy generates the next subtask and uses world-model-guided test-time computation to search over alternatives when additional reasoning is needed. A generalist low-level policy then executes the selected subtask across robot embodiments.

The low-level policy combines a Qwen3.5 vision-language backbone with a Mixture-of-Transformers action expert trained through conditional flow matching. It uses a unified 40-dimensional state/action space and was trained on 40,115 hours of heterogeneous real-world robot data with multimodal co-training.

Model checkpoints

Installation

The reference environment uses Python 3.11, CUDA 12.8, and PyTorch 2.7.1.

git clone git@github.com:sii-research/tau-0-vla.git
cd tau-0-vla
bash scripts/setup.sh

High-level components

Proposal predicts the next subtask and updates memory from three camera views. World model generates a goal image from the current observation and a subtask. Both include inference and fine-tuning examples; use their separate environments.

See the high-level quickstart and example predictions.

Example data and post-training

example_data/ contains a small AgiBot World subset in LeRobot v3.0 format. The matching post-training recipe is under configs/example_agibot_world_gong/.

bash scripts/train.sh configs/example_agibot_world_gong/train.yaml \
    --model_name_or_path /path/to/tau-0-vla-checkpoint

For another dataset or robot, start from configs/_template/ and src/tau0_vla/adapters/_template/. The repository also includes a complete LIBERO simulation recipe under configs/libero/.

Serving and evaluation

Hardware serving uses joint-control checkpoints. LIBERO simulation uses a dedicated end-effector (EEF) policy server.

Serve a post-trained joint-control checkpoint:

python -m deploy.server --model outputs/<run_name>

Run open-loop evaluation:

python deploy/openloop.py --ckpt outputs/<run_name> --no-plot

LIBERO simulation evaluation

The LIBERO checkpoint fine-tunes the pretrained low-level policy for end-effector control across Spatial, Goal, Object, and Long tasks.

See the LIBERO guide for checkpoint downloads, model and simulator environments, evaluation commands, and benchmark results.

See deploy/ for the payload and action-order contracts.

Repository layout

src/tau0_vla/
├── adapters/    embodiment-specific data layouts and deployment I/O
├── data/        LeRobot loading, prompting, masking, and normalization
├── high_level/  proposal inference, serving, and fine-tuning
├── models/      Qwen3.5 backbone and flow-matching action expert
├── trainer/     post-training entry point
├── vlm/         multimodal collation and tokenization
└── utils/       logging and run specifications
configs/         reusable template, AgiBot World example, and LIBERO recipe
deploy/          policy server and open-loop evaluation
example_data/    bundled AgiBot World subset
high_level/      component guides, high-level examples, and world-model package
scripts/         setup, training, and normalization utilities

Additional documentation:

• Dataset format
• Data pipeline
• Robot adapters

Citation

If you find our work useful, please cite our paper:

@misc{cai2026tau0vlahierarchicalrobotfoundation,
  title={$\tau_0$-VLA: a Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation},
  author={Xiaowei Cai and Yunuo Cai and Bingao Chen and Jingxiao Chen and Zhi Chen
          and Siyuan Feng and Tengyu Hou and Jingshun Huang and Han Jiang and Runkun Ju
          and Dong Li and Mingxiang Li and Shaowei Li and Xinchen Li and Yifan Li
          and Yi Liu and Zhongyuan Liu and Jianlan Luo and Junwen Miao and Ruiqi Ni
          and Buqing Nie and Mingjie Pan and Xinlin Ren and Jianheng Song and Jiaxu Wang
          and Peiqi Wang and Sen Wang and Xiaoyan Wang and Dafeng Wei and Dongming Wu
          and Pengwei Xie and Pu Yang and Hangjian Ye and Xiangyu Yue and Jinyu Zhang
          and Qinglin Zhang and Xueyong Zhao and Pengfei Zhou and Yue Zhou},
  year={2026},
  eprint={2608.16885},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2608.16885},
}

License

Code and model weights are released under the Apache License 2.0. The bundled example data follows the license described in example_data/README.md.