Embodied-Aki

A Hierarchical-End-to-End Hybrid VLA Framework for General-Purpose Embodied AI - Towards Top-Tier Conference Performance (ICLR/ICML/CoRL)

Embodied-Aki is a hierarchical-end-to-end hybrid Vision-Language-Action (VLA) framework for general-purpose embodied AI. It combines the interpretability of hierarchical planning with the efficiency of end-to-end learning, targeting top-tier conference performance (ICLR/ICML/CoRL 2026).

Key Features

• 🏗️ Hierarchical-End-to-End Hybrid Architecture: Combines System 2 (slow, deliberate planning) with System 1 (fast, reactive control)
• 🔍 Cross-Model Adversarial Review: Executor-Reviewer mechanism prevents single-model blind spots
• 🛡️ Safety-Aware Training: Integrates reviewer feedback into training loss for safer deployment
• 🎯 Multi-Granularity Action Representation: Supports discrete tokens (planning) and continuous actions (control)
• 🚀 State-of-the-Art Performance: Targets 92%+ on LIBERO-Spatial, 90%+ on LIBERO-Object

📋 Table of Contents

• Installation
• Quick Start
• Model Architecture
• Training
• Evaluation
• Pre-trained Models
• Contributing
• Citation
• License

🔧 Installation

Prerequisites

• Python 3.10+
• CUDA 11.8+ (for GPU acceleration)
• Git

Step 1: Clone the Repository

git clone https://github.com/Akiii707/Embodied-Aki.git
cd Embodied-Aki

Step 2: Create Virtual Environment

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Step 3: Install Dependencies

pip install -r requirements.txt

Step 4: Install as Package (Optional)

pip install -e .

Quick Start

Basic Inference Example

from embodied_aki import EmbodiedAkiAgent

# Initialize agent
agent = EmbodiedAkiAgent(
    vla_checkpoint="checkpoints/vla_backbone.pth",
    policy_checkpoint="checkpoints/policy_head.pth",
    planner_model="qwen2.5-7b",
    device="cuda"
)

# Run inference
observation = {
    "image": load_image("kitchen_scene.png"),
    "proprioception": [0.1, 0.2, 0.3, 0.0, 0.0, 0.0],
    "task_description": "Pick up the red apple and place it in the bowl"
}

action = agent.predict(observation)
print(f"Predicted action: {action}")

Training Example

# Single GPU training
python scripts/train.py --config configs/train_config.yaml

# Multi-GPU training (DDP)
torchrun --nproc_per_node=4 scripts/train.py --config configs/train_config.yaml

# With custom dataset
python scripts/train.py \
    --config configs/train_config.yaml \
    --data.path /path/to/custom/dataset \
    --training.epochs 100

🏛️ Model Architecture

Embodied-Aki consists of five core components:

┌─────────────────────────────────────────────────────────────┐
│                    Task Description                          │
│            "Pick up the cup and pour water"                  │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                   LLM Planner (Qwen2.5)                      │
│  ┌──────────────────────────────────────────────────────┐   │
│  │ Subgoal 1: Navigate to sink                          │   │
│  │ Subgoal 2: Grasp cup                                 │   │
│  │ Subgoal 3: Position over container                   │   │
│  │ Subgoal 4: Tilt and pour                             │   │
│  └──────────────────────────────────────────────────────┘   │
─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│               VLA Backbone (Qwen2.5-VL)                      │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐   │
│  │ Visual Encoder│    │ LLM Backbone│    │ Action Tokens│   │
│  │   (ViT-L)    │───▶│  (Qwen2.5)  │───▶│   Embedding  │   │
│  └──────────────┘    └──────────────┘    ──────────────┘   │
─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│              Policy Head (Diffusion / ACT)                   │
│  ┌──────────────────────────────────────────────────────┐   │
│  │ Diffusion Policy: Iterative denoising                │   │
│  │ ACT Policy: Autoregressive transformer decoding      │   │
│  ──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
─────────────────────────────────────────────────────────────┐
│           Cross-Model Reviewer (Safety Check)                │
│  ┌──────────────────────────────────────────────────────┐   │
│  │ ✓ Collision risk assessment                          │   │
│  │ ✓ Force limit verification                           │   │
│  │ ✓ Joint constraint check                             │   │
│  │ ✓ Physical feasibility estimation                    │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
                            │
              ┌─────────────┴─────────────
              │ APPROVED                  │ REJECTED
              ▼                           ▼
    ┌──────────────────┐        ┌──────────────────┐
    │ Execute Action   │        │ Revise & Retry   │
    │ [joint_angles]   │        │ (Max 3 attempts) │
    └──────────────────┘        └──────────────────┘

Component Details

📊 Training

Dataset Preparation

Embodied-Aki supports multiple dataset formats:

# Download LIBERO dataset
python scripts/download_libero.py --output data/libero

# Download CALVIN dataset
python scripts/download_calvin.py --output data/calvin

# Custom dataset (LeRobot format)
python scripts/convert_to_lerobot.py --input /path/to/raw/data --output data/custom

Configuration

Edit configs/train_config.yaml to customize training:

training:
  epochs: 100
  batch_size: 32
  learning_rate: 1e-4
  optimizer: adamw
  gradient_clip: 1.0
  
vla_backbone:
  model_name: qwen2.5-vl-7b
  freeze_vision_encoder: true
  freeze_llm: false
  lora_rank: 64
  
policy_head:
  type: diffusion  # or 'act'
  diffusion_steps: 100
  action_chunk_size: 8

Running Training

# Start training
python scripts/train.py --config configs/train_config.yaml

# Resume from checkpoint
python scripts/train.py --config configs/train_config.yaml --resume checkpoints/epoch_50.pth

# Monitor with TensorBoard
tensorboard --logdir runs/

Evaluation

Benchmark Results

Running Evaluation

# Evaluate on LIBERO
python scripts/evaluate.py --benchmark libero --checkpoint checkpoints/best.pth

# Evaluate on CALVIN
python scripts/evaluate.py --benchmark calvin --checkpoint checkpoints/best.pth

# Sim2Real evaluation
python scripts/sim2real_eval.py --sim IsaacLab --real UnitreeG1

🤖 Pre-trained Models

️ Development

Project Structure

Embodied-Aki/
├── src/embodied_aki/          # Core package
│   ├── models/                 # Model implementations
│   │   ├──2605.xxxxx},
  year={2026}
}

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgements

• Qwen2.5 for the base language models
• UnifoLM-VLA for VLA architecture inspiration
• HY-Embodied for multi-modal embodied AI insights
• ARIS for cross-model collaboration methodology
• LeRobot for dataset utilities

📬 Contact

• Project Lead: Akiii707
• Email: your-email@example.com
• Issues: GitHub Issues

---