G-ACT: Generalized Action Chunking Transformer

G-ACT is a generalized extension of ACT (Action Chunking Transformer) that replaces the environment-specific ResNet-18 vision backbone with pluggable perception modules designed to bridge the sim-to-real gap.

The core idea: instead of feeding raw RGB pixels to the transformer, G-ACT optionally uses semantic segmentation masks and monocular depth maps as its visual representation. These are far more consistent between simulation and the real world than raw pixels, enabling policies trained entirely in simulation to transfer directly to real hardware.

---

The Problem: ACT is Environment-Specific

Standard ACT uses a ResNet-18 backbone to extract visual features directly from camera images. This works well when training and deployment environments look similar — but it means:

• A policy trained in simulation fails on real hardware (different textures, lighting, backgrounds)
• A policy trained in one lab setup fails in another
• Small changes to the physical setup (different lighting, table color) can break a policy

The root cause is that ResNet features are tightly coupled to low-level visual statistics (colors, textures, edges) that differ significantly between domains.

---

The Solution: Domain-Invariant Perception

G-ACT replaces (or augments) the ResNet backbone with two custom perception modules:

1. SO-101 Segmentation Model

Model: riversnow/so101-segmentation-model

A fine-tuned YOLO11-S instance segmentation model trained to segment different parts of the SO-101 robot arm (base, shoulder, elbow, wrist, gripper, etc.).

Instead of seeing a photo of a robot, the policy sees which pixels belong to which part of the robot — as a set of binary masks. This representation looks almost identical in simulation and in the real world, because segment boundaries are determined by robot geometry, not by texture or lighting.

2. Depth Anything V2

Model: depth-anything/Depth-Anything-V2-Small-hf

A state-of-the-art monocular depth estimation model that produces metric-scale depth maps from a single RGB camera. Depth information captures the 3D structure of the scene without requiring stereo cameras or depth sensors.

3. Combined Seg+Depth (Recommended)

The default and most powerful mode: per-segment depth statistics.

For each segment class (e.g., "elbow"), G-ACT computes:

• Mean depth within the segment
• Depth standard deviation
• Min/max depth
• Segment area ratio

This yields a compact feature vector (e.g., 6 classes × 5 stats = 30 values per camera) that encodes where each part of the robot is in 3D space — information that is highly consistent between simulation and reality, regardless of visual appearance.

---

Architecture

Camera Images (B, 3, H, W)
         │
         ▼
┌─────────────────────────────┐
│   Perception Backbone       │
│                             │
│  ┌──────────────────────┐   │
│  │ SO-101 Segmentation  │   │   riversnow/so101-segmentation-model
│  │  (YOLO11-S fine-tune)│   │   → segment masks (B, num_classes, H, W)
│  └──────────┬───────────┘   │
│             │               │
│  ┌──────────▼───────────┐   │
│  │  Depth Anything V2   │   │   depth-anything/Depth-Anything-V2-Small-hf
│  │  (monocular depth)   │   │   → depth map (B, 1, H, W)
│  └──────────┬───────────┘   │
│             │               │
│  ┌──────────▼───────────┐   │
│  │ Per-segment depth    │   │
│  │ statistics extractor │   │   → feature vector (B, num_classes × 5)
│  └──────────────────────┘   │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│   ACT Transformer           │   (identical architecture to original ACT)
│                             │
│  Encoder: [latent, robot    │
│   state, env state, *cams]  │
│        ↓                    │
│  Decoder: action queries    │
│        ↓                    │
│  Action head → chunk        │
└─────────────────────────────┘
              │
              ▼
     Action chunk (B, chunk_size, action_dim)

The transformer architecture is identical to ACT — only the vision frontend is changed. This means you can initialize from ACT checkpoints and fine-tune with the new backbone, or train from scratch.

---

Backbone Modes

G-ACT supports four backbone modes, configured via GACTConfig.backbone_type:

---

Installation

# Clone this repo
git clone https://github.com/your-username/G-ACT.git
cd G-ACT

# Install lerobot (provides ACT base, training infrastructure)
pip install -e lerobot/

# Install perception dependencies
pip install ultralytics transformers huggingface_hub

# The segmentation model weights are already included:
# models/so101-segmentation-model/weights/best.pt

---

Usage

Quick Start

from gact import GACTConfig, GACTPolicy, BackboneType

# Default config: seg+depth backbone, SO-101 segmentation model
config = GACTConfig(
    backbone_type=BackboneType.SEG_DEPTH,
    segmentation_model_id="riversnow/so101-segmentation-model",
    depth_model_id="depth-anything/Depth-Anything-V2-Small-hf",
    input_features={
        "observation.images.top": ...,   # your camera feature spec
        "observation.state": ...,
    },
    output_features={
        "action": ...,
    },
)

policy = GACTPolicy(config)

Use as a Drop-in ACT Replacement

# Use the original ResNet-18 backbone (identical to ACT)
config = GACTConfig(
    backbone_type=BackboneType.RESNET,
    vision_backbone="resnet18",
    pretrained_backbone_weights="ResNet18_Weights.IMAGENET1K_V1",
)

With Object Detection (DINOv2 + SAM)

config = GACTConfig(
    backbone_type=BackboneType.SEG_DEPTH,
    object_classes=["cube", "target_zone"],  # objects to track per frame
    target_class="cube",                      # which one the robot should manipulate
    dino_model_id="facebook/dinov2-small",
    sam_model_id="facebook/sam-vit-base",
    dino_feature_dim=384,
)

---

Simulation Training with SO101-Nexus

The recommended simulation environment for G-ACT is SO101-Nexus — a Gymnasium-compatible library with SO-101 specific environments across MuJoCo, ManiSkill, and Genesis backends.

The bundled so101-nexus/ directory at the root of this repo is a cloned copy.

Available Environments

Installation

# From the bundled clone:
pip install so101-nexus/packages/so101-nexus-mujoco      # MuJoCo backend
pip install so101-nexus/packages/so101-nexus-maniskill   # ManiSkill backend (GPU recommended)

# Or directly from PyPI:
pip install so101-nexus-mujoco

Quick Start (MuJoCo)

import gymnasium as gym
import so101_nexus_mujoco  # registers environments

env = gym.make("MuJoCoPickCubeGoal-v1", render_mode="rgb_array")
obs, info = env.reset()

for _ in range(256):
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        obs, info = env.reset()

env.close()

Training G-ACT on SO101-Nexus

The cube is the ideal target object — hard edges, high contrast, already the task object in all SO101-Nexus environments, and segments cleanly with Grounded SAM 2.

import gymnasium as gym
import so101_nexus_mujoco
from lerobot.configs.types import FeatureType, PolicyFeature
from lerobot.utils.constants import ACTION, OBS_STATE, OBS_IMAGES
from gact import GACTConfig, GACTPolicy, BackboneType

# SO-101 has 6 joints
config = GACTConfig(
    backbone_type=BackboneType.SEG_DEPTH,
    object_classes=["cube"],
    target_class="cube",
    input_features={
        f"{OBS_IMAGES}.top": PolicyFeature(type=FeatureType.VISUAL, shape=(480, 640, 3)),
        OBS_STATE: PolicyFeature(type=FeatureType.STATE, shape=(6,)),
    },
    output_features={
        ACTION: PolicyFeature(type=FeatureType.ACTION, shape=(6,)),
    },
    chunk_size=100,
    n_action_steps=100,
)

policy = GACTPolicy(config)
env = gym.make("MuJoCoPickCubeGoal-v1", render_mode="rgb_array")

ManiSkill (massively parallel, GPU)

ManiSkill supports up to 512 parallel environments on a single GPU — much faster data collection.

import gymnasium as gym
import so101_nexus_maniskill

env = gym.make(
    "PickCubeGoalSO101-v1",
    obs_mode="rgb",           # RGB frames + state
    control_mode="pd_joint_delta_pos",
    num_envs=512,
    render_mode="rgb_array",
)

With Debug Visualization

Enable the debug writer to record what G-ACT actually sees (segmentation overlays + depth maps) during training:

config = GACTConfig(
    backbone_type=BackboneType.SEG_DEPTH,
    object_classes=["cube"],
    target_class="cube",
    debug_output_dir="debug_out/run1",  # set to None to disable
    debug_save_raw=True,                # also save .npy arrays for offline analysis
    ...
)

Output per step in debug_out/run1/:

• step_000000_seg.png — coloured segmentation overlay (each robot part a distinct colour)
• step_000000_depth.png — depth map (magma colormap)
• step_000000_seg.npy / step_000000_depth.npy — raw arrays for offline analysis

Collecting a Dataset and Training

lerobot's training script handles dataset collection, normalization, and the training loop. Once you have a dataset (either collected via teleoperation or from a pre-existing lerobot dataset), training is:

# Collect demonstrations first (teleoperation or scripted policy)
python lerobot/scripts/control_robot.py \
  --robot-path lerobot/configs/robot/so101.yaml \
  --fps 30 \
  --repo-id your-hf-username/so101-pick-and-place

# Train G-ACT on your dataset
python lerobot/scripts/train.py \
  --policy.type gact \
  --dataset.repo_id your-hf-username/so101-pick-and-place \
  --output_dir outputs/gact_so101

The Sim2Real Training Loop

The recommended workflow for using G-ACT's sim2real capability:

1. Build a MuJoCo scene that roughly matches your real setup
   (same robot geometry, approximate object positions)
         ↓
2. Train G-ACT in simulation — collect N episodes via scripted
   or teleop policy, then train with lerobot
         ↓
3. Transfer to real hardware directly — no fine-tuning needed
   (the seg+depth representation is domain-invariant)
         ↓
4. If performance is poor, collect a small number of real
   demonstrations (~10-50) and fine-tune for a few epochs

The key advantage over standard ACT: step 3 should work without step 4, because G-ACT never sees raw pixels — only robot geometry (from YOLO) and scene structure (from depth), both of which are consistent between sim and real.

---

Repository Structure

G-ACT/
├── README.md
├── gact/
│   ├── __init__.py
│   ├── configuration_gact.py   # GACTConfig (backbone, object classes, target, transformer params)
│   ├── modeling_gact.py        # GACTPolicy and GACT nn.Module
│   ├── backbones.py            # SegmentationBackbone, DepthBackbone, ObjectSAMBackbone, SegDepthBackbone
│   └── processor_gact.py       # Pre/post processing pipeline (normalization, device placement)
├── models/
│   └── so101-segmentation-model/   # Downloaded from HuggingFace
│       ├── weights/best.pt          # YOLO11-S fine-tuned weights
│       ├── segment.py
│       └── README.md
└── lerobot/                    # Cloned from huggingface/lerobot
    └── src/lerobot/policies/act/   # Original ACT for reference

---

How Sim2Real Transfer Works

The pipeline below shows how G-ACT closes the sim2real gap:

Simulation                          Real World
──────────────────────              ──────────────────────
Rendered RGB frame          →       Real camera frame
     ↓                                   ↓
YOLO11-S segmentation               YOLO11-S segmentation
(masks look ~identical               (same model, similar
 to real due to geometry)             geometry → similar masks)
     ↓                                   ↓
Depth Anything V2                   Depth Anything V2
(relative depth ~consistent         (real depth may differ
 with real geometry)                 in scale, but structure
     ↓                               is consistent)
Per-segment depth stats         →   Per-segment depth stats
[elbow: mean=0.4, ...]              [elbow: mean=0.42, ...]
     ↓                                   ↓
         G-ACT Transformer (same weights)
                    ↓
              Action chunk

The key invariant: segment boundaries and relative depth between robot parts are determined by robot geometry, not by the environment. A simulation with perfect geometry will produce segment+depth features that are very close to those from a real camera, even if the textures and backgrounds are completely different.

---

Credits

• ACT — Tony Z. Zhao et al., Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
• lerobot — HuggingFace lerobot
• SO-101 Segmentation Model — riversnow/so101-segmentation-model (YOLO11-S fine-tuned for SO-101 robot arm segmentation)
• Depth Anything V2 — depth-anything/Depth-Anything-V2
• YOLO11 — Ultralytics