Arxiv | Webpage

https://github.com/user-attachments/assets/49036e84-c397-4589-9024-efb05b14efa0

:books: Table of Contents:

1. Highlights
2. Getting Started
   • Simulation
   • Real-World
3. Checkpoints
4. TODO List
5. License
6. Citation.
7. Acknowledgment

:fire: Highlights

• :trophy: SOTA simulation performance Seer achieves state-of-the-art performance on simulation benchmarks CALVIN ABC-D and LIBERO-LONG.
• :muscle: Impressive Real-World performance Seer demonstrates strong effectiveness and generalization across diverse real-world downstream tasks.

:door: Getting Started

We provide step-by-step guidance for running Seer in simulations and real-world experiments. Follow the specific instructions for a seamless setup.

Simulation

CALVIN ABC-D

• Installation
• Running Code

LIBERO LONG

• Installation
• Running Code

Real-World

Real-World (Quick Training w & w/o pre-training)

For users aiming to train Seer from scratch or fine-tune it, we provide comprehensive instructions for environment setup, downstream task data preparation, training, and deployment.

• Installation
• Post-processing
• Fine-tuning & Scratch
• Inference

Real-World (Pre-training)

This section details the pre-training process of Seer in real-world experiments, including environment setup, dataset preparation, and training procedures. Downstream task processing and fine-tuning are covered in Real-World (Quick Training w & w/o pre-training).

• Installation
• Pre-processing
• Pre-training

:pencil2: Checkpoints

Relevant checkpoints are available on the website.

📆 TODO

• Release real-world expriment code.
• Release CALVIN ABC-D experiment code (Seer).
• Release the evaluation code of Seer-Large on CALVIN ABC-D experiment.
• Release the training code of Seer-Large on CALVIN ABC-D experiment.
• Release LIBERO-LONG experiment code.
• Release simpleseer, a quick scratch training & deploying code.

License

All assets and code are under the Apache 2.0 license unless specified otherwise.

Citation

If you find the project helpful for your research, please consider citing our paper:

@article{tian2024predictive,
  title={Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation},
  author={Tian, Yang and Yang, Sizhe and Zeng, Jia and Wang, Ping and Lin, Dahua and Dong, Hao and Pang, Jiangmiao},
  journal={arXiv preprint arXiv:2412.15109},
  year={2024}
}

Acknowledgment

This project builds upon GR-1 and Roboflamingo. We thank these teams for their open-source contributions.