The local inference server for coding agents.
Pure Rust, one binary, Apple Silicon and NVIDIA. Anthropic and OpenAI APIs. The KV cache survives across turns, so turn 20 starts as fast as turn 2.

Quick Start · Why turns stay fast · Performance · HTTP API · Support Matrix · Architecture · Changelog

English · 简体中文

---

Quick Start

1. Install

# Apple Silicon (Homebrew)
brew install cklxx/tap/arle

# Apple Silicon or Linux x86_64 (one-line installer)
curl -fsSL https://github.com/acupof-ai/arle/releases/latest/download/install.sh | sh

# Linux + NVIDIA (Docker, no compile needed)
docker run --rm --gpus all -p 8000:8000 -v $PWD/models:/models:ro \
 ghcr.io/acupof-ai/arle:latest serve --backend cuda --model-path /models/Qwen3.6-27B

2. Serve a model

# MacBook: a 35B mixture-of-experts model in 4-bit (~19 GB), fetched from Hugging Face on first run
arle serve --backend metal --model-path mlx-community/Qwen3.6-35B-A3B-4bit --port 8000

# NVIDIA
arle serve --backend cuda --model-path /path/to/Qwen3.6-27B --port 8000

3. Point your agent at it

# Claude Code (Anthropic Messages API, streaming, tool use)
ANTHROPIC_BASE_URL=http://localhost:8000 ANTHROPIC_API_KEY=local claude

# Anything that speaks the OpenAI API (opencode, aider, the openai SDK, ...)
export OPENAI_BASE_URL=http://localhost:8000/v1 OPENAI_API_KEY=local

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")
print(client.chat.completions.create(
 model="default",
 messages=[{"role": "user", "content": "Hello from ARLE"}],
).choices[0].message.content)

Source builds need a backend. cargo build --release alone produces a CLI-only binary. Add --features cuda (NVIDIA) or --no-default-features --features metal,no-cuda,cli (Apple Silicon). See docs/install.md.

One binary, four modes

Full install matrix, uninstall, and build from source: docs/install.md · Examples: examples/.

---

Why turns stay fast

A coding agent re-sends the whole conversation every turn: system prompt, every prior tool result, every prior reply. Most local servers re-run prefill over all of it. ARLE keeps the prior turn's KV on the accelerator, shares prefix pages across requests through a radix cache, and re-prefills only the tokens the new turn added.

Same machine, same weights, 12-turn agent-shaped conversation (a 4.8K-token system prompt, then one ~350-token tool result per turn, 8.6K tokens by turn 12). Time to first token per turn:

_{Greedy, identical request bytes for both servers, 2026-09-02 · script: scripts/bench_multiturn_ttft.py · method and raw rows: wins entry. ARLE's cold prefill is slower on this model; the per-turn number is what a 20-turn session feels. The same table on Qwen3.6-35B-A3B is pending a machine without swap pressure.}

Restored turns are greedy-identical to cold prefill (needle ladder 115–8000 tokens ×3, every length deterministic).

On CUDA the same cache demotes prefix pages to host RAM under memory pressure and promotes them back on the next hit. INT8/FP8 paged KV is available behind --kv-cache-dtype (Qwen3.5/3.6 family, opt-in).

---

Performance

Measured on real hardware. Headline rows only; every number resolves to a snapshot in benchmarks/ or a dated entry in docs/experience/wins/.

Apple Silicon (M4 Pro, 48 GB, single stream)

The 35B mixture-of-experts model decodes as fast as the 4B dense model: only ~3B parameters activate per token.

Speculative decoding on Qwen3.6-27B: the model's own multi-token-prediction head drafts, the base model verifies. Output is bit-identical to greedy, decode goes 12.3 → 17.75 tok/s (+44%), past the 15.2 tok/s memory-bandwidth ceiling a single-token decoder cannot cross.

NVIDIA (one H20, 32K-token multi-turn agent prompts)

Against SGLang 0.5.13 on the same GPU and the same quantized kernel (Qwen3.6-27B, 33K prompt, one request): decode 16.69 ms per token vs 17.16 ms (2.8% faster); prefill 25.0 s vs 21.0 s (19% slower, being worked on).

Also served on CUDA: DeepSeek-V4-Flash (2×, 4×, 8×H20; FP8 and 4-bit expert weights) and Qwen3.8-27B in NVFP4 (24% fewer bytes than FP8, +5 to +21% decode at c=1–16). Full rows, configs, and the CUDA-graph and quantization details: docs/baselines.md.

On-Policy Distillation

The teacher is this server. The student trains on its own rollouts:

• Qwen3.5-4B: MATH-500 +27pp (0.518 → 0.792)
• Qwen3.5-27B: Terminal-Bench pass@1 +5.1pp (20.5 → 25.6%)

Method and raw data: benchmarks/README.md · docs/experience/wins/.

---

Architecture

One runtime, three surfaces, two backends. Serving, the local agent, and OPD training run the same Rust and model code; the OPD teacher is the production server.

flowchart TB
 Serve["arle serve<br/><sub>Anthropic + OpenAI APIs</sub>"]
 Agent["arle<br/><sub>local agent</sub>"]
 Train["arle train opd<br/><sub>on-policy distillation</sub>"]
 Core["infer-core<br/><sub>device-neutral engine · scheduler · KV cache</sub>"]
 Seam["infer-seam<br/><sub>two traits: BackendExecutor · KvPool</sub>"]
 CUDA["infer-cuda<br/><sub>FlashMLA · DeepGEMM · DeepEP</sub>"]
 Metal["infer-metal<br/><sub>MLX bridge</sub>"]
 Serve --> Core
 Agent --> Core
 Train --> Core
 Core --> Seam
 Seam --> CUDA
 Seam --> Metal

A new backend implements the two seam traits; the scheduler, cache, and server do not change.

Deep dive: docs/onboarding.md (30 min) · docs/architecture.md · docs/codebase-map.md.

---

Status

Full tiers: docs/support-matrix.md · docs/stability-policy.md.

---

Documentation

Design notes · HTTP API · Support Matrix · Architecture · Codebase Map · Environment · Troubleshooting · Contributing · All docs

---

License

MIT