LingBot-VA: Causal World Modeling for Robot Control

https://github.com/user-attachments/assets/cec7b7a6-953b-4fa4-8f1a-47efc1fce547

Table of Contents

• News
• Model Download
• Quick Start
  • Installation
  • attn_mode Configuration
  • Deploying LingBot-VA for Inference
    • Evaluation on RoboTwin-2.0
    • Evaluation on LIBERO
    • Run Image to Video-Action Generation
  • Post-Training LingBot-VA
    • Data Preparation
    • Custom Dataset Preparation
    • Training
• Performance
  • Simulation Evaluation
  • Real-world Deployment
• License
• Citation
• Acknowledgments

---

💫 Meet LingBot-VA! We've built an AR diffusion framework for simultaneous world modeling and action! 🤖✨

LingBot-VA has focused on:

• Autoregressive Video-Action World Modeling: Architecturally unifies visual dynamics prediction and action inference within a single interleaved sequence while maintaining their conceptual distinction.
• High-efficiency Execution: A dual-stream mixture-of-transformers(MoT) architecture with Asynchronous Execution and KV Cache.
• Long-Horizon Performance and Generalization: High improvements in sample efficiency, long-horizon success rates, and generalization to novel scenes.

🚀 News

• [2026-04-24] Weights for post-train on LIBERO-LONG released! (IMPORTANT: Ensure that va_libero_cfg.action_snr_shift, va_libero_cfg.used_action_channel_ids and va_libero_cfg.norm_stat in wan_va/configs/va_libero_cfg.py are synchronized with the latest version of the repository.)
• [2026-04-08] Post-training and inference code for the LIBERO dataset is now available!
• [2026-02-17] Post-training code and dataset released! Support fine-tuning LingBot-VA on custom robotic manipulation datasets.
• [2026-01-29] Weights and code for shared backbone released! Please stay tuned for our separated version!

---

📦 Model Download

• Pretrained Checkpoints for Post-Training

• Post-Training Dataset

---

🛠️ Quick Start

Installation

Requirements • Python == 3.10.16 • Pytorch == 2.9.0 • CUDA 12.6

pip install torch==2.9.0 torchvision==0.24.0 torchaudio==2.9.0 --index-url https://download.pytorch.org/whl/cu126
pip install websockets einops diffusers==0.36.0 transformers==4.55.2 accelerate msgpack opencv-python matplotlib ftfy easydict
pip install flash-attn --no-build-isolation

⚠️ Important: attn_mode Configuration

You MUST change the attn_mode setting depending on whether you are training or running inference. Since LingBot-VA is loaded via from_pretrained, this parameter is read from the model folder's transformer/config.json. You need to manually edit this file before launching.

Mode	attn_mode value	Notes
Training	"flex"	Required for training. Will not work for inference.
Inference / Evaluation	"torch" or "flashattn"	Required for inference. "flex" will cause errors at eval time.

How to change: Open <your-model-path>/transformer/config.json, find the "attn_mode" field, and set it to the appropriate value.

---

Deploying LingBot-VA for Inference

LingBot-VA supports both standalone execution and Server-Client architecture which separates the model environment from simulation. By isolating dependencies, the design avoids package clashes and supports distributed inference on GPUs, clusters, and other devices.

Evaluation on RoboTwin-2.0

Preparing the Environment

You can follow the official instructions from the original RoboTwin-2.0 repository:
https://robotwin-platform.github.io/doc/usage/robotwin-install.html

In summary:

1. Install Vulkan dependencies:

   sudo apt install libvulkan1 mesa-vulkan-drivers vulkan-tools
   

2. Clone the RoboTwin repository:

   git clone https://github.com/RoboTwin-Platform/RoboTwin.git && cd RoboTwin && git checkout 2eeec322
   

3. Modify script/requirements.txt with the following content:

   transforms3d==0.4.2
   sapien==3.0.0b1
   scipy==1.10.1
   mplib==0.2.1
   gymnasium==0.29.1
   trimesh==4.4.3
   open3d==0.18.0
   imageio==2.34.2
   pydantic
   zarr
   openai
   huggingface_hub==0.36.2
   h5py
   # For Description Generation
   azure==4.0.0
   azure-ai-inference
   pyglet<2
   wandb
   moviepy
   imageio
   termcolor
   av
   matplotlib
   ffmpeg
   

4. Modify line 8 of script/_install.sh:

   pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable" --no-build-isolation
   

5. Install dependencies:

   bash script/_install.sh
   

6. Download assets:

   bash script/_download_assets.sh
   

Deploying the Inference Server

# single GPU
bash evaluation/robotwin/launch_server.sh

# multi-GPU
bash evaluation/robotwin/launch_server_multigpus.sh

Executing the Inference Client

# single GPU
task_name="adjust_bottle";
save_root="results/";
bash evaluation/robotwin/launch_client.sh ${save_root} ${task_name}

# multi-GPU
save_root="results/"
task_group_id=0;
bash evaluation/robotwin/launch_client_multigpus.sh ${save_root} ${task_group_id}

Related experiments results will be save in /path/to/your/RoboTwin/${save_root}. Please note that an eval_result folder is also generated. This is a native output from RoboTwin and is identical to the contents in the results folder; it can be safely ignored. It is important to note that the inference server and client must be deployed on the same machine. For launching multi-GPU client, we padded the original 50 tasks to 56 via duplication and partitioned them into 7 groups to align with the 8-GPU configuration of our inference node. You can specify the task_group_id (0-6) to select a particular group for inference. For detailed grouping configurations, please refer to evaluation/robotwin/launch_client_multigpus.sh.

GPU Memory Requirements: Approximately 24GB VRAM for single-GPU RoboTwin evaluation with offload mode enabled (VAE and text_encoder offloaded to CPU).

Evaluation on LIBERO

Follow the official instructions to install LIBERO, then launch the server and client:

# server
bash evaluation/libero/launch_server.sh

# client
bash evaluation/libero/launch_client.sh

Run Image to Video-Action Generation

We also provide a script for image to video-action generation:

NGPU=1 CONFIG_NAME='robotwin_i2av' bash script/run_launch_va_server_sync.sh

GPU Memory Requirements: Approximately 18GB VRAM for single-GPU i2av inference with offload mode enabled (VAE and text_encoder offloaded to CPU).

Post-Training LingBot-VA

We support post-training (fine-tuning) LingBot-VA on custom robotic manipulation datasets. The training pipeline uses FSDP for distributed training and integrates with LeRobot dataset format.

Additional Dependencies

On top of the base installation, post-training requires:

pip install lerobot==0.3.3 scipy wandb --no-deps

Data Preparation

Download the post-training dataset from HuggingFace:

huggingface-cli download --repo-type dataset robbyant/robotwin-clean-and-aug-lerobot --local-dir /path/to/your/dataset

Custom Dataset Preparation

If you want to fine-tune LingBot-VA on your own robotic manipulation data, follow these steps:

Example Dataset

We provide a converted example dataset based on data from Issue #29. This dataset has been converted into the expected format and is fully supported for training. You can download it to understand the required data structure:

• Download: Example Dataset

This example can serve as a reference for converting your own robotic manipulation data into the proper format.

Data Pipeline Overview

When preparing your custom dataset, the data goes through the following processing pipeline:

1. Raw Data → Convert to LeRobot format (with metadata and video files)
2. Add Action Segmentation → Add action_config to episodes.jsonl
3. Extract Latents → Process videos through VAE according to video specifications
4. Dataset Loading → Load processed data with proper action dimensions for training

The final data should conform to these specifications:

Action Format:

• Output dimension: 30 dimensions, structured as follows:
  • Left arm EEF (end-effector): 7 dimensions
  • Right arm EEF (end-effector): 7 dimensions
  • Left arm joints: 7 dimensions
  • Right arm joints: 7 dimensions
  • Left arm gripper: 1 dimension
  • Right arm gripper: 1 dimension
• In your dataset class loader, map your robot's action dimensions to this standard 30-dimensional format. Missing dimensions are padded with 0.

Video Format:

• During VAE latent extraction, resize videos to ~256 × 256 pixels and downsample to 5-15 fps as a reference (adjust based on your task requirements).

Implementation Steps

Step 1: Convert your data to LeRobot format

Follow the official LeRobot dataset documentation to convert your raw data (e.g., HDF5, video files, etc.) into the standard LeRobot dataset format. Ensure that each episode contains the required observation videos, actions, and metadata.

Step 2: Add action_config field to episodes.jsonl

After converting to LeRobot format, you need to modify the meta/episodes.jsonl file to add an action_config field to each line. This field describes the temporal segmentation and natural language description of the robot's actions within each episode.

Each line in episodes.jsonl should follow this format:

{
  "episode_index": 0,
  "tasks": ["task description"],
  "length": 450,
  "action_config": [
    {
      "start_frame": 0,
      "end_frame": 450,
      "action_text": "Natural language description of the robot action in this segment.",
    }
  ]
}

• start_frame / end_frame: The frame range (0-indexed) of the action segment within the episode.
• action_text: A natural language description of what the robot does in this segment.

For episodes with a single continuous action, start_frame should be 0 and end_frame should equal the episode length. You can also define multiple segments per episode if your data contains sequential sub-tasks.

Step 3: Extract video latents with Wan2.2 VAE

LingBot-VA operates on video latent representations rather than raw pixels. You need to extract the latent features using the Wan2.2 VAE encoder and place them under the converted LeRobot dataset directory. Please refer to the Wan-Video documentation for instructions on how to run the VAE encoder.

The extracted latent files should be placed under latents/ in your dataset directory, mirroring the structure of videos/:

your_dataset/
├── videos/
│   └── chunk-000/
│       └── observation.images.cam_high/
│           ├── episode_000000.mp4
│           └── ...
├── latents/
│   └── chunk-000/
│       └── observation.images.cam_high/
│           ├── episode_000000_0_450.pth    # named as episode_{index}_{start_frame}_{end_frame}.pth
│           └── ...
└── meta/
    └── episodes.jsonl

Each .pth file is a dictionary containing the following fields:

The latent file naming convention episode_{index}_{start_frame}_{end_frame}.pth corresponds to the action_config segments defined in episodes.jsonl. For example, an episode with "start_frame": 0, "end_frame": 450 produces a latent file named episode_000000_0_450.pth.

Training

# RoboTwin
NGPU=8 CONFIG_NAME='robotwin_train' bash script/run_va_posttrain.sh

# LIBERO
NGPU=8 CONFIG_NAME='libero_train' bash script/run_va_posttrain.sh

For better training performance, use a larger global batch size (e.g., 32, 64). If you have limited GPU resources, you can increase gradient_accumulation_steps to achieve a larger effective batch size.

---

📊 Performance

We evaluate our model on both simulation benchmarks and real-world scenarios, and achieve state-of-the-art performance.

Simulation Evaluation

• RoboTwin 2.0

We are the first to propel RoboTwin 2.0 metrics performance past the 90+ threshold！

* All metrics are reported in percentage (%). Higher values are bolded.

• LIBERO

* All metrics are reported in percentage (%). Higher values are bolded.

 

Real-world Deployment

Six manipulation tasks across three categories: longhorizon tasks (Make Breakfast, Pick Screws), precision tasks (Insert Tube, Unpack Delivery), and deformable & articulated object manipulation (Fold Clothes, Fold Pants). Our method achieves state-of-the-art performance on both metrics (Progress Rate and Success Rate) with only 50 trials per task, substantially outperforming strong baseline π_0.5.

🪪 License

This project is released under the Apache License 2.0. See LICENSE file for details.

📚Citation

@article{lingbot-va2026,
  title={Causal World Modeling for Robot Control},
  author={Li, Lin and Zhang, Qihang and Luo, Yiming and Yang, Shuai and Wang, Ruilin and Han, Fei and Yu, Mingrui and Gao, Zelin and Xue, Nan and Zhu, Xing and Shen, Yujun and Xu, Yinghao},
  journal={arXiv preprint arXiv:2601.21998},
  year={2026}
}

🧩 Acknowledgments

This work builds upon several excellent open-source projects:

• Wan-Video - Vision transformer backbone
• MoT - Mixture-of-Transformers architecture
• The broader open-source computer vision and robotics communities

---

For questions, discussions, or collaborations:

• Issues: Open an issue on GitHub
• Email: Contact Dr. Qihang Zhang (liuhuan.zqh@antgroup.com) or Dr. Lin Li (fengchang.ll@antgroup.com)