RWKV-7 (Goose) × sglang

English · 简体中文

Production serving for RWKV-7 on sglang: token-exact against the reference implementation, quantized (int8/int4), and running on 11 platforms — 10 CUDA GPU models (2018's T4 through B200 and RTX 5090) plus Apple Silicon. Every number below has its raw log committed in bench/results/.

How this was built. Most of the code, kernels, measurements and documents in this project were produced by AI agents (Claude — Opus, Sonnet and Fable models) working under human direction; each finding under docs/findings/ names the model that produced it in its discovered_by field. This is stated here rather than left in the findings' front-matter because it should change how you read the claims: nothing here asks to be taken on the author's word. The benchmark scripts, the baselines they are quoted against, the raw logs and the numpy oracle are committed, and the correctness claims are checked against implementations this project did not write — BlinkDL's own runtime and reference above all.

Three numbers, then the map

Find your number by what you need:

Runs on sglang main, from sglang_mainline/ — the tree that produced the numbers below, committed as such. The model-support core is submitted upstream: sglang PR #30115.

Why RWKV-7 for serving

RWKV-7 is a recurrent model: its per-sequence state is a fixed size, no matter how long the context — a Transformer's KV cache grows with every token. Measured effect: going from 1 to 256 concurrent sequences, or growing the context 64×, each costs less than 0.2 GB of extra VRAM. High concurrency and long context are where this architecture wins.

What works

Quickstart

On sglang v0.5.17 (e.g. inside the lmsysorg/sglang:dev-cu12 container). RWKV-7 has no KV cache for sglang to size the token pool from, so a serving batch needs --max-total-tokens; scripts/serve.sh passes it for you.

cd /sgl-workspace/sglang
git apply <this-repo>/sglang_main_port/upstream_edits.patch   # 10 files of wiring edits
# then copy the RWKV-7 files (model, backend, kernels, config):
#   file list and destinations in sglang_main_port/README.md
python -m sglang.launch_server --model-path <rwkv7-model-dir> --trust-remote-code \
    --attention-backend triton --dtype float16 --disable-radix-cache \
    --max-total-tokens 1048576

The hand-written fast-path kernels are opt-in environment flags, all greedy-exact; the recommended production set is in scripts/serve.sh. Models: any fla-format RWKV-7 checkpoint (fla-hub/rwkv7-*), or our prequantized int8/int4 checkpoints on ModelScope (Hakureirm/rwkv7-g1-*).

On AMD ROCm: source scripts/rocm_env.sh, then use scripts/serve_rocm.sh. Contributor-owned gfx1100 evidence covers the correctness-first ROCm path across the complete public 0.1B / 0.4B / 1.5B / 2.9B / 7.2B / 13.3B matrix; exact revisions, gates, and raw logs are in docs/ROCM.md. A standalone HIP W8/W4 decode kernel now makes both quantized modes faster than dense at bsz1 and bsz8 across the measured size matrix. Fused ROCm quantized prefill covers M=9..256: W4 prefill improved 1.31-2.89x at bsz1 and 2.06-3.15x at bsz8 across 0.1B-13.3B; W8 uses a measured shape gate. RTN W4 model quality and physical CDNA validation remain open gates.

On a Mac: mlx_port/README.md.

Layout

sglang_mainline/   the implementation that runs: model, state backend, CUDA kernels, spec-decode worker
sglang_main_port/  the wiring edits to sglang's own files (upstream_edits.patch)
mlx_port/          native Apple Silicon implementation (MLX + Metal kernel)
bench/             every benchmark and correctness-gate script; raw outputs in bench/results/
docs/              USER.md · EVIDENCE.md · BENCHMARKS.md · FINDINGS.md + findings/ — the evidence chain
scripts/           serve.sh (recommended launch flags)
tools/             doc generators (gen_findings_index.py)

If you re-run a script in bench/ and get a different number, please open an issue — that is what the raw logs are committed for.