[NeurIPS 2026] Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models

TABLE OF CONTENTS

1. News
2. Highlights
3. Method
4. Results
5. Installation
6. Quick Start
7. Model Zoo
8. Repository Layout
9. Citation
10. Acknowledgements
11. License

News

• [2026.09] 🎉 VLAct has been accepted to NeurIPS 2026 with two strong accepts!
• [2026.08] The paper, code, project site, continued-pretraining backbone, and selected downstream checkpoints are public.
• [2026.08] VLAct entered the RoboDojo leaderboard, ranking 6th of 35 policies by success rate and ahead of every explicitly designated world-action model in the August 24 snapshot.

Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models [Paper]
Senqiao Yang^†, Chengyao Wang^†, Yuxin Chen, Zixuan Wang, Longxiang Tang, Haokun Gui, Jinhui Ye, Changsheng Lu, Xiaoyang Wu, Mingkang Zhu
Advisors: Pengguang Chen, Shu Liu^✉, Zhuotao Tian, Hengshuang Zhao, Bei Yu, Jiaya Jia
_{^† Project leaders  ·  ^✉ Correspondence}

[!IMPORTANT] TL;DR. VLAct builds a reusable Qwen3-VL-4B action backbone from open data, reaching 82.6% on LIBERO-Plus, 54.8% on VLA-Arena, and 92.5% on RoboTwin 2.0. It also transfers to unseen robots: 49.5% with 20% of RoboCasa-GR1 data and 6th of 35 by success on RoboDojo's ARX X5.

💡 Why VLAct

Robot trajectories cannot be scraped from the web. They must be produced through embodied execution, and the space a policy must generalize over — scenes, objects, goals, embodiments, contact dynamics — is combinatorial and continuous. Even the largest robot datasets are sparse samples of that space.

Data scaling remains essential, but it is not the only axis. Under a fixed robot-data budget, downstream performance also depends on how effectively trajectories are distilled into reusable visual-action knowledge inside the backbone. VLAct treats continued pre-training as representation learning rather than only action fitting, and the VLM backbone as a first-order design variable for VLA.

The paper isolates three failure modes of naive VLA continued pre-training through controlled pilot experiments:

[!IMPORTANT] VLAct addresses all three during continued pre-training. Downstream, you discard the pre-training heads, attach a freshly initialized head of your choice, unfreeze the full backbone, and fine-tune normally. Across every comparison against the matched Qwen3-VL-OFT baseline, the VLM backbone weights are the only thing that changes: the downstream head, its initialization, data, optimizer, and budget are identical. The resulting gains are therefore attributable to the learned backbone representation.

🧩 Method

1 · Preserve the VLM prior

Freeze the vision encoder and the lower half of the LLM layers (shallow-layer protection), and mix image-caption data into every minibatch: L_total = L_action + 0.5 · L_VLM-CE. Lower layers carry broad visual and spatial processing; captions supply dense supervision over objects, attributes, relations, and scene context, keeping the trainable layers near their original operating regime.

2 · Diversify the action supervision

Attach three continuous heads — OFT, PI, and GR00T — to one shared latent, all predicting the same ground-truth chunk: L_action = L_OFT + L_PI + L_GR00T. No new head, no alignment module; head diversity is the supervision. Because the heads impose different decoder biases on the same problem, the backbone cannot lean on features only one of them can read. All heads share a single backbone forward pass, so the cost is a few lightweight decoders, not repeated backbone compute.

3 · Unify action semantics across embodiments

One shared head over a partially unified 20-D action layout: dims 1–12 are the two 6-DoF arms of bimanual embodiments (absolute joint angles), dims 13–18 the single-arm 6-DoF delta end-effector pose, dim 19 the shared gripper coordinate (Franka's gripper and AgileX's left gripper), dim 20 the right gripper. Each sample contributes loss only on its active dimensions; the rest are masked. Physically comparable dimensions share supervision, incompatible kinematics are never force-aligned, and no embodiment adapter, router, or conditioned decoder is introduced.

For periodic joints we add a wrap-aware loss so that 179° and −179° are 2° apart rather than 358°: δ_wrap = ((â − a) + π) mod 2π − π, applied to absolute joint dimensions only.

📊 Results

Comparisons against the matched Qwen3-VL-OFT baseline are controlled: the downstream action head and its initialization, data, optimizer, and fine-tuning budget are fixed, while only the backbone weights change. Published systems provide broader context but may use different training recipes. RoboDojo is an external leaderboard result, so metrics should not be compared across benchmarks.

The backbone also transfers to robots absent from continued pre-training. VLAct reaches 49.5% on RoboCasa-GR1 using only 20% of the downstream trajectories and 54.0% with the full set. On the official RoboDojo evaluation for ARX X5, it records a 10.66 average score and 7.60% success rate, ranking 6th of 35 policies by success in the August 24, 2026 snapshot.

🛠 Installation

Tested workflows assume Linux, Python 3.10, NVIDIA GPUs, and a CUDA-compatible PyTorch build.

git clone https://github.com/starVLA/VLAct.git
cd VLAct

conda create -n vlact python=3.10 -y
conda activate vlact

# Install a CUDA-compatible PyTorch build first, following pytorch.org.
python -m pip install -r requirements.txt
python -m pip install flash-attn==2.7.4.post1 --no-build-isolation
python -m pip install -e .

nvcc -V
python -m pip list | grep -E 'torch|transformers|flash-attn'

🚀 Run VLAct

1 · Continued pre-training

Follow the continued pre-training guide for the complete pipeline: base-model downloads, VLM and robot-data preparation, LeRobot v2.1 layout, cache and statistics generation, path configuration, and single- or multi-node training.

bash   scripts/run_scripts/Pretrain/pretrain_qwen3_single_node.sh   # 8 GPUs, one node
sbatch scripts/run_scripts/Pretrain/pretrain_qwen3_slurm.sh         # multi-node Slurm

[!NOTE] The launchers reference a machine-specific Accelerate/DeepSpeed configuration. Review the configuration notes in the pre-training guide before launching.

2 · Downstream fine-tuning and evaluation

RoboTwin ships OFT, PI, and GR00T variants; the current LIBERO and VLA-Arena launchers use PI, and the DOMINO launcher uses OFT:

bash scripts/run_scripts/RoboTwin/train_robotwin_qwen3oft.sh
bash scripts/run_scripts/RoboTwin/eval_robotwin_qwen3oft.sh    # also *_qwen3pi.sh, *_qwen3gr00t.sh

bash scripts/run_scripts/LIBERO/train_libero_qwen3pi.sh
bash scripts/run_scripts/VLA-Arena/train_vla_arena_qwen3pi.sh
bash scripts/run_scripts/DOMINO/train_domino_qwen3oft.sh

[!TIP] Every launcher opens with a marked configuration block. Review base_vlm, the benchmark data root, run_root_dir, and pretrained_ckpt before launching. The downstream launchers set --trainer.random_init_action_model True, so the transferred object is the backbone rather than the continued-pretraining action heads.

Benchmark environment setup and evaluation protocols live under examples/: LIBERO-plus · VLA-Arena · RoboTwin · DOMINO · RoboCasa · eval protocol.

3 · Recipe → implementation

All of these are set in pretrain_qwen3_single_node.sh; the multi-head framework itself is QwenHybrid_xrobot_padding.py.

[!NOTE] Paper setting vs. released artifact. The paper defines the auxiliary caption objective with weight 0.5 and reports a 16-GPU setup. The checked-in launcher and downloadable 100K-step artifact record --trainer.loss_scale.vlm 0.2; the artifact card records 4 nodes × 8 GPUs. Use the paper for the reported experimental setting and the artifact's training_config.original.yaml to reproduce that specific checkpoint.

📈 Model Zoo

The VLAct collection tracks public releases. For a new embodiment, dataset, or decoder, use the raw continued-pretraining backbone as the default starting point rather than a benchmark-specific policy.

Download the reusable backbone together with its resolved config and normalization statistics:

huggingface-cli download JasonYang66/VLAct-Qwen3VL4B-Pretrained \
  --local-dir playground/Pretrained_models/VLAct-Qwen3VL4B-Pretrained

Then set the downstream launcher's pretrained_ckpt to:

playground/Pretrained_models/VLAct-Qwen3VL4B-Pretrained/checkpoints/steps_100000_pytorch_model.pt

[!IMPORTANT] The continued-pretraining checkpoint is not a directly deployable policy. Keep config.yaml and dataset_statistics.json at the downloaded run root, match the target camera and action contracts, and initialize an incompatible downstream action head from scratch.

Checkpoint heads and headline paper results are not interchangeable: RoboTwin's 92.5% result uses OFT, while the released RoboTwin checkpoint uses GR00T; the VLA-Arena and LIBERO-Plus model pages use PI, while Tables 1 and 4 of the paper report OFT comparisons. The paper's RoboDojo result is the official 50-episode-per-task leaderboard snapshot, not a scaled local evaluation.

🗂 Repository Layout

starVLA/
├── model/framework/QwenHybrid_xrobot_padding.py   # VLAct: shared latent → OFT + PI + GR00T
├── model/framework/{QwenOFT,QwenPI_v4,QwenGR00T}.py
├── model/modules/action_model/                    # action heads + wrap-aware losses
├── dataloader/gr00t_lerobot/                      # mixtures, embodiment tags, action transforms
└── training/train_starvla{,_cotrain}.py           # VLA / VLA+VLM co-training entry points

examples/{DROID,InternA1,MolmoAct,RoboCoin}/       # pre-training data cleaning & cache building
examples/{LIBERO,LIBERO-plus,VLA-Arena,Robotwin,DOMINO,Robocasa_tabletop}/
scripts/run_scripts/{Pretrain,LIBERO,VLA-Arena,RoboTwin,DOMINO}/
deployment/                                        # real-robot policy server

✍️ Citation

Please cite the official arXiv paper:

@article{yang2026beyond,
  title={Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models},
  author={Yang, Senqiao and Wang, Chengyao and Chen, Yuxin and Wang, Zixuan and Tang, Longxiang and Gui, Haokun and Ye, Jinhui and Lu, Changsheng and Wu, Xiaoyang and Zhu, Mingkang and others},
  journal={arXiv preprint arXiv:2608.27550},
  year={2026}
}

@article{ye2026starvla,
  title={StarVLA-$$\backslash$alpha $: Reducing Complexity in Vision-Language-Action Systems},
  author={Ye, Jinhui and Gao, Ning and Yang, Senqiao and Zheng, Jinliang and Wang, Zixuan and Chen, Yuxin and Chen, Pengguang and Chen, Yilun and Liu, Shu and Jia, Jiaya},
  journal={arXiv preprint arXiv:2604.11757},
  year={2026}
}

@misc{community2026starvlalegolikecodebasevisionlanguageaction,
      title={StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing}, 
      author={StarVLA Community},
      year={2026},
      eprint={2604.05014},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2604.05014}, 
}

🙏 Acknowledgements

This work builds on StarVLA, LeRobot, GR00T, DeepSpeed, Qwen-VL, and InternVL; on the DROID, InternData-A1, RoboCoin, and MolmoAct datasets; and on the LIBERO-Plus, VLA-Arena, RoboTwin 2.0, DOMINO, RoboCasa, and RoboDojo benchmarks.

License

This repository is released under the MIT License.

Questions or suggestions: email the authors or open an issue.