vla-hub

VLA (Vision-Language-Action) model hub for realtime inference on AWS ECS (EC2 GPU).

Six OSS VLA policies — GR00T N1.6, GR00T N1.7, π0.5, OpenVLA-7B, SmolVLA-450M, LAP-3B — packaged as independent gRPC endpoints in a single CDK project. Pick the model that fits your robot, deploy in minutes, swap models without re-architecting clients. Each stack ships its own VPC, internal NLB, ECS cluster, ASG, and ECR repo.

Adding a seventh model = one entry in vla-hub.json + a docker/<model>/ context

• matching stacks. The AzSelectorConstruct probes EC2 GPU capacity at deploy time and pins the ASG to a confirmed AZ — no manual capacity hunting.

Model Selection Policy

The hub does not restrict itself to fully-generalist VLAs. In practice every public VLA sits somewhere on a generalist↔specialist spectrum, and the threshold is fuzzy. The policy is therefore:

A model is a candidate as long as it is OSS, subject to two guardrails:

1. gRPC serving must be feasible. The model must either ship gRPC serving code, or be wrappable by a thin adapter under docker/<model>/serve.py. Models that only ship custom training code are excluded.
2. Stack compatibility. The model must run on a standard PyTorch or JAX container. Specialized CUDA kernels or non-standard runtimes are excluded on operational-cost grounds.

Models passing both guardrails are included even if they lean toward the specialist end. The capability matrix below lets operators and clients pick a fit on a case-by-case basis.

Capability Matrix

Note: Zero-shot feasibility depends on how close the customer robot is to the model's pretraining distribution. The table reflects embodiments that are explicitly part of pretraining; out-of-distribution embodiments require fine-tuning regardless of model.

Scenario fit

Where each model sits on the spectrum

Generalist  ─────────────────────────────────────  Specialist
       LAP-3B   π0.5    GR00T   SmolVLA   OpenVLA   Fast-WAM
              (co-train) (MLP)             (Bridge/  (LIBERO/
                                           Franka)   RoboTwin sim)
                                                      ❌ outside hub

Single-sim / single-embodiment specialists such as Fast-WAM are better operated ad-hoc outside the hub: they only run inside one sim, so they do not generalize across customer scenarios.

Architecture

Robot / Sim Client
  │
  ├─ gRPC (TCP:50050) → GR00T N1.6 NLB  → ECS Task (g6/g5 GPU) — GR00TInference
  ├─ gRPC (TCP:50051) → GR00T N1.7 NLB  → ECS Task (g6/g5 GPU) — GR00TInference
  ├─ gRPC (TCP:50052) → π0.5 NLB        → ECS Task (g5/g6 GPU) — PIInference
  ├─ gRPC (TCP:50053) → OpenVLA NLB     → ECS Task (g6/g5 GPU) — OpenVLAInference
  ├─ gRPC (TCP:50054) → SmolVLA NLB     → ECS Task (g6/g5 GPU) — SmolVLAInference
  └─ gRPC (TCP:50055) → LAP-3B NLB      → ECS Task (g6/g5 GPU) — LAPInference

Each stack is isolated:

• Independent VPC, NLB, ECS Cluster, ASG, ECR repo
• Internal NLB (not internet-facing) — gRPC clients must reside in the same VPC
• gRPC inference port: model-specific (50050–50055), one port per model
• Port 8080: HTTP health server (NLB health check target)

Stacks

Each model id × version combination from vla-hub.json produces an independent build stack and ECS stack. Adding a new model = entry in vla-hub.json + docker/<model>/ context + matching stacks emitted by the CDK app.

Current model entries (see vla-hub.json):

Quick Start

Prerequisites

npm install
npm run build
cdk bootstrap

Phase 1: Build container images

Each model has its own build stack. Deploy the build stacks for the models you want, then trigger their CodeBuild projects:

# GR00T N1.6 / N1.7: require HuggingFace token in Secrets Manager
# (default secret name: gr00t/hf-token; override with -c hfTokenSecretName=<name>)
cdk deploy Gr00tN16BuildStack
cdk deploy Gr00tN17BuildStack

# π0.5: no token needed (public GCS checkpoint)
cdk deploy PiBuildStack

# OpenVLA-7B: HuggingFace public model — token only needed for rate-limit
cdk deploy OpenvlaBuildStack

# SmolVLA-450M: HuggingFace public model
cdk deploy Smolvla450mBuildStack

# LAP-3B: HuggingFace public model (checkpoint ~12.4 GB baked in at build; PaliGemma
# tokenizer from gs://big_vision pre-baked too). No token needed.
cdk deploy Lap3BBuildStack

After CodeBuild completes, note the ECR image URIs from each stack's outputs.

Phase 2: Deploy inference stacks

# GR00T N1.6
cdk deploy Gr00tN16EcsStack \
  -c gr00tN16EcrImageUri=<account>.dkr.ecr.<region>.amazonaws.com/gr00t-n16-realtime:latest

# GR00T N1.7
cdk deploy Gr00tN17EcsStack \
  -c gr00tN17EcrImageUri=<account>.dkr.ecr.<region>.amazonaws.com/gr00t-n17-realtime:latest

# π0.5
cdk deploy PiEcsStack \
  -c piEcrImageUri=<account>.dkr.ecr.<region>.amazonaws.com/vla-pi-realtime:latest

# OpenVLA-7B
cdk deploy OpenvlaEcsStack \
  -c openvlaEcrImageUri=<account>.dkr.ecr.<region>.amazonaws.com/openvla-realtime:latest

# SmolVLA-450M
cdk deploy Smolvla450mEcsStack \
  -c smolvla450mEcrImageUri=<account>.dkr.ecr.<region>.amazonaws.com/smolvla-450m-realtime:latest

# LAP-3B (served from VlaHubStack — override the lap image URI via context)
cdk deploy VlaHubStack \
  -c lapEcrImageUri=<account>.dkr.ecr.<region>.amazonaws.com/vla-lap-realtime:3B-latest

Optional CDK context overrides

-c instanceTypes=g6.2xlarge,g5.2xlarge   # comma-separated fallback list
-c desiredCount=1

Stack outputs

gRPC Interfaces

Each model ships its own .proto. All five expose Infer + Health RPCs with the per-model schema appropriate to that model's input/output conventions.

Common shape for Infer:

• Input: image_data (JPEG bytes), instruction (string), optional joint states
• Output: action_chunks — map<string, bytes> (float32 tobytes, reshape to (H, DOF) per the model's action-chunk convention)

Health(HealthRequest) → HealthResponse is provided on every service for NLB / client probes.

Instance Type Selection

AzSelectorConstruct probes EC2 capacity at deploy time and pins the ASG to a confirmed AZ.

GR00T (and OpenVLA's FlashAttention path) require Ampere GPU (SM80+). π0.5 and LAP-3B use JAX and do not require FlashAttention.

Docker Contexts

docker/
├── gr00t-n16/   # GR00T N1.6 container (PyTorch + Isaac Lab gRPC server)
├── gr00t-n17/   # GR00T N1.7 container (PyTorch + Isaac Lab gRPC server)
├── pi/          # π0.5 container (JAX + pi0.5 gRPC server)
├── openvla/     # OpenVLA-7B container (PyTorch + HuggingFace gRPC server)
├── smolvla/     # SmolVLA-450M container (PyTorch + LeRobot gRPC server)
└── lap/         # LAP-3B container (JAX + openpi/lap gRPC server)

Each context is packaged as an S3 Asset and passed to CodeBuild. When docker/<model>/ changes, redeploy the corresponding BuildStack.

Design FAQ

Why ECS + NLB instead of Amazon SageMaker?

SageMaker supports Triton Inference Server internally, but it only exposes an HTTP REST endpoint externally. The Triton gRPC port (:8001) is inaccessible from outside the SageMaker container. This makes SageMaker unsuitable for two requirements:

1. Direct gRPC connection from robots — robots and simulators connect over gRPC directly; SageMaker cannot expose a raw gRPC port.
2. mTLS end-to-end — SageMaker terminates TLS at its own layer, so E2E mTLS between robot and inference container is not achievable.

ECS + NLB (L4 passthrough) hands TLS termination to the ECS Task itself, enabling full gRPC + mTLS control.

For simple PoC/demo with REST, SageMaker is fine. For production-style robot connectivity patterns, ECS + NLB is correct.

Why serve.py directly instead of Triton Inference Server?

Current verdict: Triton adds operational complexity without reducing code complexity. The blocking issue is the custom .proto requirement — migrating to Triton's standard generate.proto would require rewriting all client SDKs and the ZMQ-gRPC bridge. Additionally, π0.5 (JAX) and GR00T (PyTorch) cannot safely share a GPU due to JAX's pre-emptive memory allocation (XLA_PYTHON_CLIENT_MEM_FRACTION).

Re-evaluate when NVIDIA ships an official TensorRT-optimized GR00T package — that would eliminate the monkey-patch requirement and make adopting the standard Triton proto natural. See Triton Inference Server for upstream reference.

Why gRPC instead of WebSocket?

Key reasons for Physical AI inference workloads:

• Protobuf compression — 200 KB JSON observation → ~20 KB (~10× reduction)
• gRPC metadata — robot ID, model ID carried at schema level (no app-layer workarounds)
• HTTP/2 multiplexing — multiple concurrent robot streams over a single connection
• Strong typed interface — .proto auto-generates both server and client code

NLB sticky routing (TCP 5-tuple hash) works identically for both gRPC and WebSocket — it is not a differentiator. The advantages above are protocol-level, not routing-level.

Why NLB instead of ALB?

NLB operates at L4 (TCP passthrough) — TLS is terminated inside the ECS Task container, enabling E2E mTLS. ALB terminates TLS at the load balancer layer, breaking the mTLS chain. NLB also handles long-lived gRPC streaming connections without HTTP-layer timeouts.

Production Considerations

This repository is a sample for demonstration and PoC use. Before adapting it for production or customer-facing workloads, review the following — none are hard blockers, but each is a deliberate scoping choice in the sample:

• Exposure: gRPC endpoints are bound to an internal NLB (not internet-facing). Clients must reside in the same VPC. For external clients, add VPC peering, Transit Gateway, or PrivateLink — do not flip the NLB to internet-facing without an auth layer in front.
• Auth: There is no auth layer in front of gRPC. mTLS is supported at the protocol level (NLB is L4 passthrough, TLS terminates in the task), but certificate provisioning and client identity are out of scope here. Add before exposing beyond the trusted VPC.
• Availability: Each ECS service runs single-AZ by design (the AzSelectorConstruct pins the ASG to one confirmed-capacity AZ). For HA, switch to multi-AZ ASG with capacity rebalancing, or run two stacks in different AZs behind a regional endpoint.
• Data: The sample processes no customer data. Model weights are downloaded from HuggingFace at container runtime (not baked into images) using a short-lived HuggingFace token stored in AWS Secrets Manager. For air-gapped or offline deployments, mirror weights to an internal ECR or S3 artifact store and bake them into the image.
• Support: Provided as-is. No SLA, no backward compatibility guarantee across model versions or CDK revisions. Pin a tag and treat upgrades as PoC-grade migrations.

Operators are expected to evaluate the architecture against their own security, compliance, and availability requirements before any customer-facing use.

Edge / Jetson roadmap

The cloud serving path above (x86 + NVIDIA g5/g6) is the supported, validated track and does not change. This section adds a separate Jetson (L4T) edge track: the same SmolVLA-450M policy, repackaged to run on-device (Orin/Thor). It is a roadmap — the code lands incrementally, and on-device validation depends on hardware we do not currently own (see the FAQ).

What goes in, what comes out

In

• The same SmolVLA-450M weights as the cloud (lerobot/smolvla_base on HuggingFace) — no retraining, no new policy.
• A Jetson device you provide — Orin or Thor.

Out

• A gRPC inference server running on the device (localhost:50054) that exposes the same Infer / Health contract as the cloud — robot/sim clients connect identically; only host:port changes.
• A latency/throughput measurement (p50/p99 + Hz) so you know whether it meets your control-loop budget.

"Runs" and "runs in realtime" are deliberately separate outcomes: the eager PyTorch path gets it running (functional, Hz not guaranteed); the TensorRT path (Phase 4) makes it run in realtime.

How it runs (on a Jetson)

flowchart LR
  W["SmolVLA-450M weights<br/>(same as cloud)"] --> BUILD
  DEV(["Jetson<br/>Orin / Thor"]) --> PRE["preflight.sh<br/>detect → BASE_IMAGE"]
  PRE --> BUILD["Dockerfile.jetson<br/>build L4T image"]
  BUILD --> RUN["run.sh<br/>serve over gRPC"]
  RUN --> OUT(["gRPC endpoint<br/>localhost:50054"])
  RUN --> SMOKE["smoke_test.sh<br/>p50 / p99 + Hz"]
  RUN -. "realtime (Phase 4)" .-> TRT["build-trt.sh<br/>TensorRT engine"]
  TRT -. "BACKEND=tensorrt" .-> RUN

The L4T image must be built on the Jetson itself (aarch64 + Tegra) — x86 / Graviton-sbsa cannot build a Tegra image. The full day-1 runbook is edge/GETTING_STARTED_ON_HARDWARE.md; the short version (CDK-free, lives in edge/):

cd edge

# 1. Detect the device and print the exact base image to use:
./preflight.sh                                   # → Orin: …-py3-igpu | Thor: …-py3

# 2. Build the L4T image and serve, in one command:
export BASE_IMAGE=nvcr.io/nvidia/pytorch:25.08-py3-igpu   # (Orin; preflight prints yours)
BUILD=1 BASE_IMAGE=$BASE_IMAGE ./run.sh smolvla
# → gRPC: localhost:50054   health: http://localhost:8080/health

# 3. Measure latency + Hz:
./smoke_test.sh smolvla

# Or deploy a prebuilt image from a registry (1-click):
./deploy-edge.sh smolvla <account>.dkr.ecr.<region>.amazonaws.com/vla-smolvla-realtime:edge-latest

No Jetson on hand? You can still exercise the run-target logic on a plain x86 GPU host — build the cloud SmolVLA image and drive it through the same run.sh (see edge/README.md).

Status

SmolVLA-450M is the first (and currently only) edge target — already tagged "edge / small robot" in the capability matrix above. The software is staged so the whole day-1 path is scripted, with no setup guesswork.

[same SmolVLA-450M weights] ──┬─→ Dockerfile         → x86 image  → ECS/NLB     (cloud, validated, unchanged)
  (HF: lerobot/smolvla_base)  └─→ Dockerfile.jetson  → L4T image  → compose/run (edge, new)
       ↑ identical                ↑ base differs only   ↑ arch differs only
   serve.py / .proto = shared (architecture-neutral)