🦾 Awesome VLA Study

Getting started with VLA? This guide takes you from the foundations to the frontier — diffusion and flow matching, state-of-the-art robot foundation model architectures, data scaling, RL fine-tuning, and world models. Papers in reading order.

📋 Prerequisites

• Basic probability & optimization (enough to follow ELBO, score matching derivations)
• Deep learning fundamentals (Transformers, attention, tokenization)
  • 💡 Starting from scratch? MIT 6.S191 — Intro to Deep Learning covers CNNs, Transformers, and generative models in a 1-week intensive bootcamp. More courses below.

💬 Weekly Format (Recommended)

• Paper presentation: 1–2 participants per week, 30 min/paper — architecture, training, key results
• Discussion: Compare design choices across the week's papers, discuss limitations and open questions (15–20 min)

---

Phase 1: Generative Model Foundations (Weeks 1–3)

📚 Core Material: MIT 6.S184 — Introduction to Flow Matching and Diffusion Models (Holderrieth & Erives, MIT CSAIL, 2025) | Course notes paper

Week 1: ODE/SDE Foundations & Diffusion Models

Week 2: Flow Matching, Score Matching & Training

Week 3: Generative Robotics & Review

---

Phase 2: Early Foundation Robot Models & Robot Policy (Weeks 4–5)

Week 4: Early Foundation Robot Models — RT-1, RT-2, Octo, OpenVLA

📎 Supplementary video: Stanford CS25 V3 — Low-level Embodied Intelligence

Key points: RT-1 (35M, no VLM) → RT-2 (55B VLM, action as text tokens) establishes the VLA concept. Octo (27M–93M, diffusion head, no VLM) and OpenVLA (7B, VLM + 256-bin discretization) are the first open-source generalist robot policies enabling community iteration.

Week 5: Core Robot Policies — Diffusion Policy, ACT, BeT

Key points: Three approaches to the multimodal action problem. Action chunking (predicting K future actions at once) is foundational for later VLA work.

---

Phase 3: Current RFM Architectures (Weeks 6–7)

Week 6: VLM + Action Head — CogACT, GR00T N1, X-VLA

Key points: All three use only the VLM's last hidden state to drive a separate action head.

Week 7: VLM + Action Expert — π0, InternVLA-M1

📎 Background: Transfusion — Zhou et al. (2024) | 2408.11039 — AR + diffusion in one transformer; π0's architectural basis

Key points: Unlike Week 6's action heads that only see the VLM's last hidden state, these action experts access VLM internal hidden states.

---

Phase 4: Data Scaling (Weeks 8–9)

Week 8: Large-Scale Robot Datasets — OXE, AgiBot World

📎 Data formats — Recording-oriented: rosbag (ROS 1), mcap (vendor-neutral, ROS 2 default). Training-oriented: RLDS (TensorFlow/OXE standard), LeRobotDataset (HuggingFace, Parquet + video).
📎 From the Evolution of Rosbag to the Future of AI Tooling — by the original rosbag author; covers rosbag V1→V2 → rosbag2 (sqlite3) → MCAP evolution

Key points: Large-scale multi-embodiment datasets that enable generalist robot policy pretraining. OXE standardized the data format across 22 robot embodiments via RLDS; AgiBot World provides high-quality data at scale.

Week 9: Data Collection Methods — UMI, VITRA, Human to Robot Transfer

Key points: Three data sources beyond robot teleoperation — UMI (embodiment-agnostic physical demos, <$200 hardware), egocentric video, and exocentric video.

---

Phase 5: Efficient Inference & Dual-System (Weeks 10–11)

Week 10: Fast-Acting VLA — SmolVLA & RTC

Key points: Two complementary approaches — SmolVLA compresses the model itself, RTC optimizes the inference pipeline. Can be combined.

Week 11: Dual-System VLA — Helix & Fast-in-Slow

Key points: Dual-System separates slow reasoning (VLM) from fast execution (lightweight policy) at different frequencies. Helix (separately trained) vs Fast-in-Slow (end-to-end trainable).

---

Phase 6: RL Fine-tuning, Reasoning & World Model (Weeks 12–14)

Week 12: RL Fine-tuning & Human-in-the-Loop — HIL-SERL, SimpleVLA-RL, π*0.6

Key points: Three RL approaches — HIL-SERL (human-in-the-loop, sample-efficient), SimpleVLA-RL (outcome rewards), π*0.6 (advantage-conditioned, learns from suboptimal data).

Week 13: Reasoning VLA — CoT-VLA, ThinkAct, Fast-ThinkAct

📎 Fast-ThinkAct's reasoning compression is orthogonal to Week 10's model compression (SmolVLA, RTC) — the two can stack.

Key points: Reasoning representation — image tokens (CoT-VLA) vs. visual latent (ThinkAct) vs. compressed latent tokens (Fast-ThinkAct). ThinkAct grounds reasoning in task-outcome RL instead of language supervision. Fast-ThinkAct shows planning structure, not verbosity, carries the signal (~10× faster, performance preserved).

Week 14: World Model — UniVLA, Cosmos Policy, DreamZero

Key points: Three ways to leverage world knowledge — training regularizer (UniVLA, no world prediction at inference), pretrained video FM as policy backbone (Cosmos Policy), joint world+action generation in latent space (DreamZero).

---

Contributing

Suggestions for papers, resources, or structural improvements are welcome — please open an issue or PR.

See Also

• 🔥 vla0-trl — A complete VLA in ~1,200 lines of Python. Fine-tunes Qwen2.5-VL with TRL's SFTTrainer to predict actions as text, scoring ~90% on LIBERO. Read the entire codebase in an afternoon.
• 🔥 vla-eval — One framework to evaluate any VLA model on any robot simulation benchmark.
• Awesome-RL-VLA — RL for VLA models
• Awesome-VLA-Robotics — Large-scale VLA paper collection
• awesome-physical-ai — A curated list of academic papers and resources on Physical AI

---

📚 Recommended Courses

Courses covering the prerequisites for this study guide — only those with recent (2023+) video lectures freely available on YouTube. Pick what you need.