SGLang × AMD Instinct — Serving Datasheet

Verified launch commands and measured-vs-roofline performance for the latest Qwen, DeepSeek, GLM, and Moonshot models on AMD Instinct MI300X (gfx942) and MI355X (gfx950) with SGLang.

→ jhinpan.github.io/sglang-amd-cookbook

Every number on the site is traced to a source file. Un-measured model × hardware cells are shown explicitly as not benchmarked — we publish verified data only.

Models Covered

The site

Static, buildless, GitHub-Pages-ready. No framework, no bundler.

Open index.html directly (file://) or serve the folder — data loads via a <script> tag, so it works either way with no server.

Nothing else checks that models.js and app.js still agree, so there are tests. Run them all through one entry point:

bash verify.sh            # offline checks + regenerate published rows
bash verify.sh --render   # also the jsdom render test (needs npm)

verify.sh exists mainly to make sure the row-regeneration check actually runs. Where the raw records for a cell are on the box, it re-derives that cell's published benchmark rows and fails unless they are structurally equal to what models.js ships — the difference between "the numbers parse" and "the numbers are still the measured ones". It skips loudly when the raw records are absent rather than passing quietly.

The individual tests still work on their own:

npm i jsdom && node test_render.js

It renders the page, walks every verified cell, and asserts the recipe and the argument reference come out populated — including that every flag in a shipped launch command has a glossary entry rather than rendering as —.

On a box that cannot reach the npm registry, node verify-cookbook.js needs no dependencies and covers the failure modes that hand-editing models.js actually causes: that it still parses and exposes window.HW / window.MODELS, that every flag in every shipped launch command has a glossary entry (a port of app.js's parseFlags, so it agrees with the renderer), and that benchmark rows carry the fields the roofline gauge reads plus a source. It is a fallback, not a replacement — run test_render.js when you have network.

Recipes deep-link two ways: #m=<model> opens a model at its first verified cell, #m=<model>&c=<gfx>:<strategy> opens one specific cell. Use the second form whenever a model's cells disagree — Kimi-K3's DSpark cell and its plain cell reverse which one is faster.

The roofline gauge

Decode ceiling is a first-principles, memory-bound limit at batch size 1:

SOL_tok/s = (GPUs × HBM_bandwidth) ÷ (active_params × bytes_per_param)

with bytes_per_param = 1 (FP8), 2 (BF16), 0.5 (W4A16/INT4). The gauge shows measured single-stream decode ÷ this ceiling. The headroom is the optimization target this cookbook tracks — not a defect.

Mixed-precision checkpoints get an effective, weighted bytes_per_param over the active set rather than the label on the tin. Kimi-K3 is the worked example: its MXFP4 quantization covers only the routed-expert Linears, so a decode step streams 137.8 GB for 105.4 B active params — 1.31 bytes/param, not 0.5, because the bf16 attention and shared experts dominate what each token actually touches.

Tuning a new model's launch parameters

grid_k3/ is the reusable version of the search that produced the Kimi-K3 tuned recipes. It runs inside the serving container, unlike the older grid_search.sh, which builds a fresh container per config and so cannot be used from within one.

The method, in short: read the binding constraint off the scheduler's own decode lines (full token usage versus mamba usage says which pool is full), bisect the one monotone knob that sits on it, then coordinate-descend the discrete knobs at a fixed operating point. Section 7 of the Kimi-K3 playbook works through it with numbers.

Regenerating models.js

models.js is generated by an extraction + adversarial-verification pass over the playbooks, test scripts, index.html history, and grid_results/. Do not hand-edit the numbers — update the source playbooks and re-run extraction so every value stays source-traced.

One shape constraint the extraction has to respect: a model's summary is an array of topic-tagged paragraphs, one claim per entry, not a single string.

"summary": [
  { "text": "…one or two sentences: what this is." },     // no topic — renders as the lede
  { "topic": "accuracy", "text": "…" },
  { "topic": "tuning · kv pool", "text": "…" }
]

A 2,000-character string is still accepted and renders as one paragraph, which is exactly the wall this shape exists to prevent — the label rail is what makes a summary skimmable. Keep each entry to one claim and its evidence.

The renderer marks up identifiers in prose fields itself (summary, gotchas, env why, gap notes): flags, snake_case names, file names, org/repo paths and #12345 issue refs are set in mono, and ->, --, ~, 8x become →, —, ≈, 8×. Write those fields as plain ASCII prose; do not hand-wrap them in HTML.

Source playbooks

• Kimi-K3 on MI355X — Day-0 hybrid KDA/MLA MoE, MXFP4; plain + DSpark speculative decoding, GSM8K/AIME25 (test_kimi_k3.sh, eval_kimi_k3.sh) — plus a tuned recipe worth +27% throughput and +68% on DSpark, found by the search harness in grid_k3/. Re-measured 2026-08-08 on the upstream rocm/sgl-dev:v0.5.16 image (§2.0), which is ~10% faster and needs no fork or patch, and corrected: the long-context DSpark cliff was a draft-checkpoint RoPE bug (§5.4a), not a limit of speculative decoding. Extra harnesses from that pass: aime26_eval.py (AIME26 is not in run_eval; inherits SGLang's AIME25 scorer), degeneracy_probe.py (why --dataset-name random overstates accept length), prep_speedbench.py and prep_agentic_trace.py
• GLM-5.2-FP8 on MI300X — DSA tilelang, FP8; GSM8K/AIME25 + long-context (test_glm52_fp8.sh)
• GLM-5.2-FP8 on MI355X — the gfx950 re-measurement on SGLang 0.5.17 / ROCm 7.2.4, which retires the two mandatory bpreshuffle patches (the CK fix the old cell named as its own exit condition has landed), adds MTP/NEXTN speculative decoding — it works on ROCm now — and fp8_e4m3 KV, which is legal on the tilelang DSA path on ROCm and roughly doubles the pool. Fills the balanced and high-throughput cells; rows generated by gen_glm52_mi355x_rows.py
• GLM-5.3-Flash on MI355X — 320B/18B hybrid KDA+DSA model on the stacked SGLang model/ROCm PRs, with FP8 KV, AITER MoE/mHC, fused k-pool, GSM8K/AIME25, and an apples-to-apples GLM-5.2 comparison. Rows generated by gen_glm53_mi355x_rows.py. §11 is the pitfalls list — including a node where a broken rocminfo silently made tilelang target gfx900 and flydsl target gfx942, which brings the server up healthy and then kills it on the first real request (rocminfo_shim.sh). §12 is why aiter#5069's measured -25% kernel win moved serving throughput by 0.0%: the tables it retuned are read by two tiny GEMMs, half of it is unreachable on a block-quantised checkpoint, and our own pinned table covers only M=1 and M=32 while prefill drives M to 8192 (coverage_report.py is the census tool)
• GLM-5.3 on MI355X — the FULL GLM-5.3, not Flash: glm_moe_dsa at 756 GB, with the same architecture, quantization layout, shard count, and safetensors byte count as GLM-5.2-FP8 at the pinned revisions, so its recipes transfer. It now has a visible not_benchmarked site cell: serving and the long-cold-prefill fix were verified, while the single-run door-side throughput remains excluded from the datasheet. On the affected AITER c16d44b9 + Triton 3.7 stack, a single long prompt not already in the prefix cache aborts every TP rank on a Triton/LLVM iota_range(Begin <= End) assertion. The cause is AITER's fp8_mqa_logits switching, above 2 GiB of logits, to a gl.store path that does not compile for its BLOCK_M = 2 shape — so the single-request wall is min(L, chunked_prefill_size) * L * 4 >= 2**31 when num_q > 4096, putting it at 32,767 tokens for chunk size 16,384 and at 23,170 when a long prompt fits in one chunk. Older AITER and Triton 3.6 direct probes do not reproduce it, so this is version-specific rather than a property of every gfx950 DSA deployment. It is easy to miss in agentic traffic because a long agentic turn is usually a long cache hit. §1 has the one-unit boundary measurements, version matrix, wall table, and upstream fix, with which 1,001,869 cold tokens answer
• GLM-5.3-MXFP4 on MI355X — the same glm_moe_dsa weights as the cell above, requantised by a third party, and it wins on every shape measured at matched argv: +12.8% / +24.4% / +18.1% / +9.5% out tok/s at concurrency 1 / 8 / 32 / 64, 42.04% smaller on disk, 41.05% smaller per GPU, and a KV pool 27.5% larger — with GSM8K 1319 at 97.50% against fp8's 97.27% (McNemar z=0.55, n.s., 99.54% per-question agreement). "MXFP4" understates how conservative it is: read from the safetensors headers of all 282 shards, the routed experts save 307.6 GiB while MLA attention, the DSA indexer and the dense MLPs stay BF16 and give 12.9 GiB back — attention is 24.28 GiB here against 12.14 GiB in the fp8 release, so this checkpoint is less quantised than the baseline exactly where accuracy is most fragile (§1, which also corrects the model card twice). §2.2's 2 GiB MQA-logits cap applies unchanged — EAGLE, HiCache, fp8 KV and TP4 all leave the wall at 23,170 — and #36960 is now merged but is in neither of the two images here. §4.2 is the ablation that built the final recipe: fp8 KV is free money (+20.1% at c32, KV nearly doubles), EAGLE is worth 2.9× single-stream, --ep-size 8 loses to plain TP8, --enable-w4a4-mxfp4-megamoe is a no-op on ROCm (its whole effect is two DeepGEMM CUDA env vars), and HiCache is rejected on a 1923 s startup rather than on throughput. §3 is why sizing to the KV quotient kills the prefix cache. §7 is the reason this page has no site cell: under a heterogeneous long-context agent workload — not a fixed-shape sweep, which passed — the recipe takes a GPU memory access fault after ~24 minutes (#37648), and the nearest prior report #23784 is closed completed with no comment, PR or commit behind it
• DeepSeek-V4-Flash-0731 on MI355X — official 284B/13B checkpoint, FP4 experts + block-FP8 dense weights, target-only unified_kv_triton, three complete GSM8K runs and median-of-3 fixed-shape serving results (test_dsv4_flash.sh, rows generated by gen_dsv4_mi355x_rows.py)
• DeepSeek-V4-Pro-0813 on MI355X — official 1.6T/49B checkpoint on one 8x MI355X node, including the 20-minute startup characteristic, three complete GSM8K runs and median-of-3 serving results (test_dsv4_pro.sh, shared launcher test_dsv4.sh)
• Kimi-K2.6 on MI355X (test_kimi_k26.sh)

License

MIT