Qwen3.8-Flash-Next on a single DGX Spark (GB10)

Run Qwen3.8-Flash-Next — a ~176B-parameter model (125B main + 51B n-gram, 6B active) — on one NVIDIA DGX Spark / ASUS GX10 with vLLM, at full prefill speed, with MTP speculative decoding, working prefix caching, deterministic greedy decoding, and up to 500k tokens of context.

The catch this repo solves: the NVFP4 checkpoint is ~125 GiB, which does not fit next to a usable KV cache in the Spark's 128 GB unified pool. 48 GiB of that is the n-gram embedding ("PLE") table — a pure lookup that a token only touches 16 rows of. This repo patches the official vLLM image to serve that table from NVMe via mmap instead of keeping it resident. Weights drop to ~75 GiB, the rest of the pool goes to KV, and everything runs on stock GB10 kernels.

Along the way it also fixes two things that were broken for this model on GB10 — prefix caching (a vLLM block-size bug silently restored an all-zero Mamba state on every cache hit) and non-deterministic top-k in the sparse attention (a GB10 kernel that drops candidates) — and offers an optional hybrid checkpoint layout (NVFP4 experts + fp8 side layers) that decodes ~20% faster at the same quality.

TL;DR — run it on a DGX Spark

git clone https://github.com/blazux/qwen3.8-Flash-DGX.git && cd qwen3.8-Flash-DGX
./flash doctor      # docker, GPU, memory, disk, port, image, weights: tells you what is missing
./flash setup       # builds the image, downloads the checkpoint (NVIDIA's NVFP4, 124 GiB via Xet, resumable), prepares the hybrid layout
./flash serve       # the recommended recipe (profile "default"): hybrid, 500k context, deterministic
./flash wait        # first boot loads ~75 GiB of weights, 8-13 min; prints the KV pool when the API is up
./flash test        # health, coherence, prefix-cache hit, determinism, tok/s

Other recipes are one word away: ./flash profiles lists them (speed, context, context-1m, shared, published, native, v0.29), ./flash serve speed runs one, and any variable can still be overridden on the command line (./flash serve default MTP=3 PORT=18301). ./flash status, logs, stop, start, rm do what they say. Details in The flash command.

The same thing by hand, unchanged and still supported (everything flash does is these scripts):

docker build -t qwen38-flash-dgx .            # ~1 min: official vLLM image + the 12 patches below
scripts/download-weights.sh                   # nvidia/Qwen3.8-Flash-Next-NVFP4, ~124 GiB via Xet, resumable (one-time)
scripts/prepare-hybrid.sh                     # recommended: fp8 side layers, +20% decode, same quality (~10 min, one-time)
MODE=hybrid YARN=1 CTX=500000 scripts/serve.sh   # the recipe our own box runs; 500k context, ~13 min to load
docker logs -f qwen38-flash                   # ready at "Application startup complete"
scripts/smoke-test.sh                         # health, coherence, prefix-cache hit, determinism, tok/s

OpenAI-compatible API on http://localhost:18300/v1, model name qwen3.8-flash-next, tool calling and reasoning parsers on. Every default is the setting that scored best on our agentic tournament (see How the defaults are chosen); what you get on a GX10: ~34 tok/s single-stream decode, ~2,500–2,800 tok/s prefill, a ~680k-token KV pool, prefix caching, deterministic greedy output, 500k tokens of context. The checkpoint is NVIDIA's own NVFP4 quantization since 2026-09-14 (it replaced RadixArk's after a 5-pass head-to-head: same or better quality, +15–22% KV, −8% single-stream decode — the whole story is in Checkpoints); RadixArk's is one variable away, MODEL=RadixArk/Qwen3.8-Flash-Next-NVFP4, same recipe, same image. Want the checkpoint exactly as published? Drop prepare-hybrid.sh and MODE=hybrid. Want speed over the last percent of quality? MTP=3, and MODE=hybrid-mtp for more KV — both explained in the options table. Prefer the current vLLM release to the preview image? docker build -f Dockerfile.v0.29 -t qwen38-flash-dgx:v0.29 . and IMAGE=qwen38-flash-dgx:v0.29 — same recipe, same defaults, measured at parity (see vLLM v0.29.0 as the base image). Everything below is the long version: what was broken on GB10, what was fixed, and the numbers.

Independently reproduced on a DGX Spark by @jschmied — see issue #1 and their write-up.

Quoted tool markers

Both image recipes include a parser fix for literal or malformed <tool_call> markers in Qwen reasoning and ordinary text. Previously, quoting that marker could switch the parser into a tool preamble and discard subsequent text, including a final answer after </think>. The parser now buffers the marker and preserves it as text when ordinary prose follows, while recognizing a function-header prefix (<function=) as a tool call. Valid calls and existing empty-wrapper/end-of-stream handling are retained. The fix is enabled for the qwen3 parser; derived parser configurations retain their existing behavior.

That fix covers a marker followed by ordinary prose. It does not cover a marker followed by a well-formed function header — which is exactly what the model writes when it documents the format, inside a ```xml block or while reasoning about tool syntax. There the parser confirms the call, and everything after it is consumed as tool-call syntax: with tools in the request you get a tool call the model never meant to make, and without them the serving layer drops the call and returns content: null, so the whole answer disappears. The visible output stops at the fence opener, which is why this reads as "output dies on a backtick".

Patch 13 adds a second guard, also qwen3-only:

• Inside a fenced code block, <tool_call> and <function= stay text and open no call. Fences follow CommonMark: a run of three or more backticks or tildes at the start of a line (at most three spaces of indent) opens one, and only a run of the same character at least as long, with nothing but whitespace after it, closes it — so a ```xml block nested inside a ````md block does not close the outer one. Fence state is per channel and is re-synced on use, so a fence left open in reasoning never carries into the answer, even when the call follows </think> with nothing in between.
• A wrapper-less <function= header opens a call only at the start of a line. Mid-line it is prose naming the marker. This is the (CONTENT, FUNC_PREFIX) fallback, which patch 12 does not guard at all.

Not covered, both deliberate:

• Illustrative <tool_call> XML written in reasoning outside a fence still opens a call. Suppressing that would mean dropping the reasoning → tool-call transition, which this model does use.
• A fence the model opens and never closes keeps the guard active for the rest of the turn, so a real tool call after it is returned as text instead of being parsed. That is the cost of deciding from the text alone; test_unclosed_fence_suppresses_later_calls asserts it so a change is deliberate.

The regression patches extend vLLM's Qwen parser tests. To run them from a matching vLLM source checkout with its test dependencies installed (using absolute paths to this repository's patch files):

patch --batch --forward --fuzz=0 -p1 < /path/to/qwen3.8-Flash-DGX/src/patches/qwen-tool-preamble.patch
patch --batch --forward --fuzz=0 -p1 < /path/to/qwen3.8-Flash-DGX/src/patches/qwen-tool-preamble-tests.patch
patch --batch --forward --fuzz=0 -p1 < /path/to/qwen3.8-Flash-DGX/src/patches/qwen-tool-marker-guard.patch
patch --batch --forward --fuzz=0 -p1 < /path/to/qwen3.8-Flash-DGX/src/patches/qwen-tool-marker-guard-tests.patch
.venv/bin/python -m pytest tests/parser/engine -q

Patch 13's own module, tests/parser/engine/test_qwen3_literal_markers.py, is 30 cases over three chunk sizes: 21 fail and 9 pass on patch 12 alone, all 30 pass with patch 13. The 9 that pass either way are the no-regression guards — a real call still parses, a real call after a closed fence still parses, and patch 12's own inline-quoted-marker case is unchanged.

Update 2026-09-14 — NVIDIA's checkpoint is the default

• MODEL now defaults to nvidia/Qwen3.8-Flash-Next-NVFP4 in flash, serve.sh, download-weights.sh and prepare-hybrid.sh. RadixArk's checkpoint, the default until now, stays fully supported: MODEL=RadixArk/Qwen3.8-Flash-Next-NVFP4 on any of them (or ./flash setup default MODEL=…), nothing else changes. Already running RadixArk? Nothing breaks: the variable was always honoured, and the recipe, patches and profiles are identical for both.
• Why: a head-to-head on 2026-09-13/14 — same image (main), same recipe, only the checkpoint changed, 5 full passes of a 55-scenario agentic tournament per side plus speed, memory, long-context and determinism probes on the same boot. NVIDIA 88.8% ± 1.0 vs RadixArk 86.1% ± 1.9; behind on no scenario beyond one-run noise; reasoning-runaway rate identical; needle 6/6 to 413k and deterministic on both; KV pool +15% on that boot (679k vs 589k tokens, +22% on another); single-stream decode −7 to −8% (34.5 vs 37.1 tok/s), equal under load. The default rule of this repo is quality first: at parity or better, the checkpoint with more KV room and the vendor's own export wins. Numbers, protocol and the honest caveats: Checkpoints.
• What it costs you: ~3 tok/s of single-stream decode against RadixArk, and a 124 GiB download in 24 files, one of them 50 GiB, that only comes through Xet (the default of download-weights.sh since PR #19). On the v0.29 image the fp8 MTP drafter loads through the vllm#55513 backport; on the preview image through the stopgap shim — both validated.
• MODE=hybrid-mtp (the NVFP4 draft-experts graft, profile context) is RadixArk-only and now says so: NVIDIA's drafter is already fp8, which is exactly where its KV advantage comes from. The context profile pins MODEL=RadixArk/… for that reason.
• reasoning_effort: high no longer returns 400 (EFFORT_ALIAS=1, default). The chat template all these checkpoints share — NVIDIA's, RadixArk's and the abliterated copies — accepts only xhigh (its default), medium and low, and raises on anything else. vLLM passes the request's effort straight through (on /v1/messages, output_config.effort), and Claude Code sends high by default, so every such request failed with Unexpected reasoning effort high. serve.sh now serves a copy of the checkpoint's own template with its one effort-resolving line rewritten: high and max → xhigh, minimal → low. Every other value renders byte-identically, and a template without that check or that line is left alone. The copy is bind-mounted into the container from the first writable of the HF cache, ~/.cache/qwen38-flash-dgx/ and the checkout: an HF cache first created by a manual docker run belongs to root, and the alias must not silently degrade to the old 400 there (./flash doctor now warns about such a cache).
• FAST_ROWS=0: every PLE gather goes to the thread pool (new default). The mmap patch gathered decode-sized batches (≤ 512 unique rows) inline on one thread, so every row the page cache had dropped was its own serial page fault — and on a Spark the 48 GiB table never fits in cache. Measured within one boot (drowzeys' NVIDIA-based checkpoint, v0.29, hybrid, MTP=3, vm.swappiness=10), the path switched at runtime in ABBA-BAAB phases with fresh prompts each phase: 1 stream 35.1 → 37.9 tok/s (+8%), gather 11.4 → 5.0 ms; 4 streams 68.0 → 79.4 tok/s aggregate (+17%), gather 36.4 → 11.5 ms, every pool phase ahead of every inline phase. The rows gathered are the same either way, so outputs do not change. FAST_ROWS=512 restores the old path, which is ~0.8 ms faster per gather only when every row is already cached.

Update 2026-09-13 — what changed

Newest first. If you cloned this before, this is the short version; details in the linked sections.

2026-09-13 — NVIDIA's own NVFP4 checkpoint runs on the recipe, and three contributed options:

• nvidia/Qwen3.8-Flash-Next-NVFP4 is supported (issue #17, @PathosEthosLogos). Same recipe, MODEL=nvidia/Qwen3.8-Flash-Next-NVFP4; the hybrid layout works on it unchanged. It needed patch 11 (its MTP drafter's experts are blockwise fp8 under a ModelOpt mixed-precision config that vLLM 0.29 does not know how to load) and the hybrid shim extended to that config class. Patch 11 started as a stopgap shim; on the v0.29 image it is now a backport of vLLM's own fix (vllm#55513, by @techfury90), and only the preview image keeps the shim. Measured against RadixArk at equal recipe: quality at parity, needle 6/6 on both up to 413k, decode 34.0 vs 36.4 tok/s, KV pool +22–28% (721k tokens). The default stayed RadixArk that day; the 5-pass head-to-head of the next night made NVIDIA the default (see the 2026-09-14 update above). → Checkpoints
• Download knobs, PLE counters, KV_CACHE_MEM — @techfury90's PR #19: XET/EXCLUDE/MAX_WORKERS for download-weights.sh, five vllm:ple_mmap_* Prometheus counters (opt-in export with PROM_MULTIPROC=1), and KV_CACHE_MEM for an explicit KV budget. All verified live. We flipped Xet on by default right after: the Hub no longer serves files over 50 GB through the plain path, and NVIDIA's PLE table is one 50 GiB shard. → Watching the mmapped table
• COMPILE_CACHE — @AronRubin's PR #21 keeps vLLM's compiled graphs in docker volumes across boots: init engine 122 s → 41 s on our box (compilation 34 s → 0.5 s), outputs identical. Opt-in; worth it whenever something recreates the container for you. → Persistent compile cache
• ./flash doctor now reports a checkpoint as incomplete when a shard named by the index is missing (an interrupted download leaves dangling symlinks and everything looks present).

2026-09-12 — one command for newcomers, nothing removed for everyone else:

• ./flash — doctor, setup, serve <profile>, wait, test, status, logs, stop, start, rm. It is a thin front-end over the existing scripts: a profile is a plain env file in profiles/ holding the serve.sh variables for one recipe (default, speed, context, context-1m, shared, published, native, v0.29), setup runs the build / download / prepare steps only when they are not done yet, doctor checks the box before you spend an hour downloading. The scripts and every MODE=… scripts/serve.sh command in this README keep working exactly as before; if you already have a working setup there is nothing to change. → The flash command

2026-09-11 — the recipe runs on the vLLM v0.29.0 release too:

• Dockerfile.v0.29 builds the same recipe on vllm/vllm-openai:v0.29.0, the first official release that ships the model natively (as qwen4_exp), instead of the Qwen preview image. Prompted by @ChengYen-Tang (issue #14). Two of our patches are in that release and are dropped there (vllm#50729, the fp8 GEMM M%4 fix vllm#52775); the prefix-caching block-size fix, the GB10 FLA gate, the deterministic top-k kernel, the hybrid dispatch and the reduced draft vocabulary are still needed and were re-targeted; the PLE mmap patch was rewritten for the new layer. Measured at parity on the tournament (45/51 at 38.7 tok/s vs 45/51 at 38.5 on the preview image, same two scenarios failed; a first run gave 42.5/51 with two reasoning runaways, within the usual variance), decode 36.4 tok/s, prefill 2,529–3,026 tok/s, KV pool in the same range (~577k; the same recipe on the preview image boots anywhere between 565k and 630k depending on the page-cache state at profiling). scripts/serve.sh reads the base from an image label and adjusts the splitting ops. Not ported yet: the fp8 KV cache (patch 7). The preview Dockerfile stays the default until the v0.29 base has more field time. → vLLM v0.29.0 as the base image

2026-09-08 — the defaults are now decided by an agentic/coding benchmark (called tournament), and two new ones came out of it:

• Quality is the gate for defaults now. Every default in scripts/serve.sh is the setting that scored best on the 17-scenario agentic tournament (3 repeats); anything that only buys tok/s or TTFT is an option. MTP=3 (+7% decode, −1 point), fp8 KV, the M%4 padding and the exact top-k fallback are documented options, not defaults.
• The MTP drafter now scores a 65,536-token vocabulary instead of 248,320 (DRAFT_VOCAB=1, default): the target verifies every drafted token, so outputs are unchanged; the draft step reads 320 MiB of head instead of 1.27 GiB. Measured with the tournament, one variable at a time on the same day: 45/51, the best score of any configuration we ran, at 38.5 tok/s (+23%). Idea taken from MiaAI-Lab's recipe, reimplemented here (their code is AGPL). Same source for the MADV_RANDOM advice on the mmapped table, now on by default (no readahead: cold prefill −4–8%, cleaner page cache, a slightly larger KV pool). → Reduced draft vocabulary
• MODE=hybrid-mtp — @pfy's graft of Inferact's NVFP4 MTP draft experts onto the hybrid checkpoint (PR #11): −3.9 GiB of weights, +22% KV pool. On our box decode is unchanged (the cheaper draft is accepted less often) and the tournament is neutral (44/51), so it ships as an option for people who need context or concurrency more than the last percent of quality. → NVFP4 MTP draft experts
• Full same-day comparison behind those choices (all hybrid, MTP=2, prefix caching, YaRN 500k): exact torch.topk 43/51 @31.4 tok/s → deterministic kernel 44/51 @31.9 → MADV_RANDOM 44.5/51 @33.5 → reduced draft vocabulary 45/51 @38.5 (default) → same with MTP=3 44/51 @41.2 (option) → hybrid-mtp 44/51 @33.0, KV +22% (option).

2026-09-07 — kernel pin bump and multi-client guidance:

• Greedy decoding is deterministic now — at no prefill cost. The GB10 sparse-attention top-k kernel was non-deterministic and dropped candidates, diagnosed and reported upstream by @k3dani (issue #3, vllm#51782). First fixed with an exact torch.topk (deterministic but −20–40% on long prefill); now replaced by @jschmied's deterministic kernel (vllm#55122), compiled into the image: identical outputs at temperature 0 and full prefill speed back (32k: 1,794 → 2,996 tok/s). DET_TOPK=1 is the default; EXACT_TOPK=1 stays as a fallback. Kernel pin bumped 2026-09-07 (PR #10): signed-zero fix, low-shared-memory path, a launcher bug that would have crashed some long-context widths, and a faster kernel. → Deterministic top-k

2026-08-29 → 2026-09-07:

• Prefix caching works now — --enable-prefix-caching was crashing, then silently returning wrong answers on cache hits. Root cause was a vLLM block-size bug that made every prefix hit restore an all-zero Mamba state; two-line fix in the image. Getting there took @Saren-Arterius's pointer to vllm#50729 and their state-copy guard, and @0xBakeer's attempt to reproduce it, which sharpened the write-up. PREFIX_CACHE=1 is the new default. Repeated prefixes (system prompts, multi-turn, tool loops) skip the prefill: ~14 s → ~1.4 s TTFT on a 20k-token prefix. → Prefix caching now works
• Optional M%4 padding for the fp8 GEMM (PAD_M4=1, hybrid mode) — the image's blockwise-fp8 kernel is up to 10× slower on chunks whose row count is not a multiple of 4; the padding is @jschmied's. With prefix caching on (default) chunks are already aligned and it changes nothing, so it is off by default; with PREFIX_CACHE=0 it is worth about −40% TTFT at 8k. → M%4 padding
• Why decoding stalls when other clients prefill, and the slider for it — reported by @kutovoy (issue #9), reproduced and measured: it is vLLM's chunked prefill (one decode token per 3–7 s step while a new prompt is being prefilled), not a bug. EXTRA='--long-prefill-token-threshold 1024' trades single-stream TTFT for responsiveness; numbers and guidance in the concurrency section.
• Audit fixes — @sternnick audited the repo line by line against their own Spark (issue #8). Taken so far: the files fetched from @jschmied's repo are pinned by sha256 as well as by commit (and that repo is Apache-2.0 now), the fp8-KV guard only admits e4m3 (the kernel launch never handled e5m2), GPU_MEM defaults to 0.80 as the docs already recommended, and the undocumented VLLM_PLE_MMAP_CHUNK and the no-auth binding are in the docs. Their three fixes are merged with their authorship: the measured sizes (the checkpoint is 126 GiB, the table 48 GiB, plan 140 GB of disk), the PLE range guard (the last shard is partial), and the scripts (snapshot resolved from refs/main, a start check after docker run, the prefix-cache hit proven with vllm:prefix_cache_hits_total instead of a stopwatch).
• Two checkpoint modes — MODE=nvfp4 (as published) or MODE=hybrid (NVFP4 experts
  • fp8 side layers, one-time scripts/prepare-hybrid.sh): +20% decode, +8% KV, same quality. Our box runs the hybrid. The fp8 side-layer conversion and the original int4+fp8 dispatch it is ported from are @Saren-Arterius's. → Two checkpoint modes
• Also in the image: vllm#50729 (Mamba state-copy race, by @AndreasKaratzas) + a bounds guard, the GB10 FLA fixes and the faster PLE gather from @Saren-Arterius's fork.
• We benchmarked the int4 (Intel AutoRound) variant too with the same patches: fastest raw decode, but not deterministic and slowest cached-TTFT, so we did not adopt it. Numbers in docs/HOW-IT-WORKS.md.
• fp8 KV cache is available (KV_DTYPE=fp8_e4m3), contributed by @Nanetnounou: ×1.9 KV, 1M context on one box — at a speed and quality cost, so it is opt-in. → fp8 KV cache
• scripts/smoke-test.sh now also checks the prefix-cache hit and determinism, and measures decode on a real answer instead of ignore_eos (which produces meaningless numbers with this model). scripts/download-weights.sh now forwards HF_TOKEN (@wawimundo, PR #4).

Everything was measured on one ASUS GX10 with a 17-scenario agentic tournament (3 repeats each), single-request speed benches on real prompts, and state checksums for the prefix-caching work; nothing here is extrapolated.

Measured on an ASUS GX10 (GB10, 128 GB), single request, real prompts, greedy. Quality (a 17-scenario agentic tournament, 3 repeats) is identical across NVFP4 and hybrid: 45/51 both, same two scenarios failed by every quantization we tried. Details and the full comparison tables are in docs/HOW-IT-WORKS.md.

---

How the defaults are chosen: quality first, speed as an option

Every default in scripts/serve.sh is the setting that scored best on our agentic tournament (tool loops, long-context extraction, multi-step reasoning, hidden-test coding; 17 scenarios when the recipe was built, 55 for the checkpoint decision; 3–5 repeats, temperature 0.2), run on the GX10 with one variable changed at a time, on the same day. A change that only buys tok/s or TTFT and costs even a point there ships as an option, off by default, with its measured cost next to it. Two runs of the same configuration differ by up to 2 points day to day, so anything inside that band is treated as equal and the faster one wins; anything below it stays an option.

If your priority is raw throughput rather than the agent's reliability, the fast profile is MODE=hybrid MTP=3 (41 tok/s in the tournament against 38.5 for the default), and MODE=hybrid-mtp on top if you need the KV pool more than the last few percent of quality.

The flash command

./flash is the short path. It does not replace the scripts; it calls them with a named set of variables, checks what is already done, and tells you what is missing.

Profiles (profiles/*.env, each a handful of serve.sh variables; copy one to make your own):

Precedence: a variable already in your environment beats the profile (PORT=18301 ./flash serve works like the plain scripts), and KEY=VALUE arguments beat both. Container name and port default to qwen38-flash and 18300, like serve.sh.

Requirements

• An NVIDIA DGX Spark or compatible GB10 (sm_121) box, 128 GB unified memory, aarch64, recent NVIDIA driver, Docker with the NVIDIA container runtime.
• ~140 GB free disk for the checkpoint (+13 GB for the hybrid variant, +5 GB more for the NVFP4-MTP graft), on reasonably fast storage (the table is read from it at runtime — NVMe strongly recommended; the Spark's onboard NVMe is ideal).
• The base image is multi-arch, so docker build also works on x86 Blackwell (sm_120, e.g. RTX PRO 6000) for testing, though this is tuned for the Spark.

Download speed. scripts/download-weights.sh uses the Hugging Face Xet backend (XET=1, the default). It used to be off because it stalled on some Spark setups, but the Hub now refuses to serve files over 50 GB through the plain path at all — the NVIDIA checkpoint's PLE table is one 50 GiB shard, and the error it prints ("install hf_xet") is misleading, hf_xet is in the image. Xet is also much faster: --max-workers parallelises across files, so a checkpoint that is a dozen large shards leaves most of a gigabit idle over plain HTTPS. Measured by @techfury90 on a DGX Spark on gigabit fibre, pulling 81 GB (we saw the same ~105 MB/s on ours):

XET=0 falls back to plain HTTPS if Xet stalls for you. One caveat seen once: a Xet run ended in an httpx.ReadTimeout after the last file completed — every blob was intact, but the exit code was non-zero. Re-run to confirm; it is resumable, and a finished download re-checks in seconds. ./flash doctor tells you if an essential file (tokenizer, configs) or a shard named by the index is missing.

Do not run prepare-hybrid.sh on an unfinished download. The hybrid layout is a copy of the snapshot as it is at that moment; prepared too early it lacked tokenizer.json, and vLLM then died on "Couldn't instantiate the backend tokenizer… sentencepiece" (issue #17 — nothing is missing from the image). Since 2026-09-14 the script refuses an incomplete snapshot and, run again on an existing layout, repairs it by adding whatever the snapshot has gained; serve.sh and flash check the layout too and print that fix instead of vLLM's message.

Quickstart

The commands are in the TL;DR at the top. Once the log says Application startup complete, hit the OpenAI-compatible API:

curl http://localhost:18300/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen3.8-flash-next",
  "messages": [{"role":"user","content":"Write a haiku about a desktop supercomputer."}],
  "max_tokens": 512
}'

MODE=nvfp4 scripts/serve.sh (the default) serves the checkpoint as published at the native 262k context; YARN=1 CTX=500000 goes to 500k (validated with a needle-in-a-haystack at 414k tokens); GPU_MEM=0.80 is the long-running-service setting, see Tuning.

Two checkpoint modes: NVFP4 or hybrid

scripts/serve.sh serves one of two layouts of the same NVFP4 checkpoint (NVIDIA's by default, RadixArk's with MODEL=); pick with MODE=. The table below was measured on RadixArk's checkpoint; on NVIDIA's the same conversion takes decode from ~28 to ~34 tok/s and the KV pool from ~565k to ~680k.

Why it works: those side layers are dense and read in full on every decoded token, so they dominate decode bandwidth; the experts are sparse (10 of 512 active) and already 4-bit. Halving the dense part is where the tokens/s come from. The MoE path — where the quality lives — is untouched. Conversion uses @Saren-Arterius's fp8_convert.py (worst per-tensor max relative error 3.5%), and a small dispatch shim (src/vllm_fp8_hybrid_modelopt.py) that routes those layers to vLLM's blockwise-fp8 GEMM while the ModelOpt NVFP4 config keeps handling the experts.

scripts/prepare-hybrid.sh                 # builds <snapshot>-fp8hybrid/ next to the HF snapshot
MODE=hybrid YARN=1 CTX=500000 GPU_MEM=0.80 scripts/serve.sh

Our own box runs the hybrid. If you want the checkpoint exactly as published, stay on MODE=nvfp4 — you lose ~5 tok/s and nothing else.

NVFP4 MTP draft experts (MODE=hybrid-mtp, RadixArk only)

RadixArk only: NVIDIA's checkpoint already ships its MTP drafter in fp8 (that is where its KV advantage comes from), so there is nothing to graft and MODE=hybrid-mtp refuses it; the context profile pins MODEL=RadixArk/… for that reason. A graft on top of the RadixArk hybrid: the MTP draft head's routed experts (BF16 fused, ~4.7 GiB) are replaced by the NVFP4 draft experts from Inferact/Qwen3.8-Flash-Next-NVFP4 (1.4 GiB) — the combination neither parent ships: fp8 PLE table and a cheap draft. scripts/prepare-mtp-graft.sh builds it in ~5 min (~3.3 GB of real bytes: one shard rewritten without the 2 fused BF16 MTP tensors, a symlink to the donor shard, and the index + both quantization exclusion lists fixed — the blanket mtp.* globs are replaced by the donor's 29 explicit non-expert MTP module names in both config.json and hf_quant_config.json, or the draft loads unquantized and dies). Both parents are modelopt quantizations of the same base — verified by hashing embed_tokens.weight and all 29 shared draft tensors byte-for-byte before grafting. The graft reads through both parents (it is a directory of symlinks); don't delete either one.

Measured on the GX10 (hybrid, MTP=2, greedy, real answers):

Those are the author's numbers. Ours, measured the way defaults are decided here (same box, same day as the table in Reduced draft vocabulary, full-vocabulary draft on both arms):

So on this box the graft is a memory win, not a speed win: the NVFP4 drafter reads ~4× fewer bytes per draft step but its proposals are accepted less often, and the two cancel. It is quality-neutral, which is why it ships as an option rather than the default; take it when the KV pool or concurrency matters more than the last percent. Not yet measured in combination with the reduced draft vocabulary.

The draft swap is gated on greedy output equivalence: every emitted token is the target model's argmax (the draft only decides how many of its proposals get accepted per step), so scripts/greedy-probe.sh <label> run against both arms must produce byte-identical text. It does — 5/5 prompts, 400 tokens each, first-token logprobs identical to 4 decimals. Donor revision is pinned (103a7608…, sha256 0d44e6d7…); a different donor revision needs re-gating.

scripts/prepare-mtp-graft.sh              # one-time, after prepare-hybrid.sh
MODE=hybrid-mtp scripts/serve.sh

Prefix caching now works (and why it didn't)

--enable-prefix-caching used to crash this model on GB10 (CUDA illegal memory access) and, with a bounds guard in place, to silently return different answers on cache hits. We traced it (details in docs/HOW-IT-WORKS.md): vLLM's engine core overwrites cache_config.block_size with the smallest KV-group block size — 8 tokens here with MTP=2 (4 without), the QSA raw-key ring — while the Mamba state block is 1600 tokens. Two places used the former as the latter, so on a prefix hit the worker computed the state slot as (3200-1)//8 = 399 instead of 1, read past the block table row, and restored an all-zero Mamba state. The image carries a two-line fix; with it, cold and cache-hit outputs are bit-identical (state checksums and first-token logprobs match exactly) and the tournament score is unchanged.

What you get: multi-turn chats, agent/tool loops and shared system prompts skip the prefill of everything already seen. On a 20k-token prefix, TTFT goes from ~14 s to ~1.4 s; with 8 concurrent conversations, from ~80 s to ~4–6 s. PREFIX_CACHE=1 is the default. Mamba states are cached at 1600-token boundaries, so the tail of a prefix is recomputed — expect the benefit to start around a couple of thousand tokens.

Deterministic top-k (DET_TOPK=1, default)

The sparse attention (QSA) picks the top-k key blocks per query with a persistent_topk kernel. On GB10 that kernel is non-deterministic — identical greedy requests produce different outputs 2 times out of 4 — and can drop legitimate candidates (vllm#51782); reported against this repo by @k3dani in issue #3. The GB10 is more exposed than other GPUs: the cooperative kernel used elsewhere for decode is disabled on sm_12x, so this kernel runs for both prefill and decode.

Two fixes are in the image; the second is the default:

• DET_TOPK=1 (default) — deterministic kernel. @jschmied rewrote persistent_topk so that output slots are index-ordered and exact ties are resolved without candidate buffers (no truncation, exact pivot) — upstream as vllm#55122. The Dockerfile compiles it with the image's nvcc as a standalone extension (_C_det.so, ~15 s on a GX10, no vLLM rebuild) from his repo at a pinned commit, and an env-gated switch routes the QSA block selection to it. Measured on the GX10 (hybrid, MTP=2, prefix caching) with the 2026-09-07 pin (PR #10 by @jschmied: signed-zero canonicalisation, a deterministic low-shared-memory path, a launcher shared-memory bug fixed, and a faster kernel — 1.0–2.4× the stock kernel's time per call instead of 1.8–3.8×): 4/4 prompts stable, first-token logprobs identical to the 4th decimal, decode 31.8 tok/s, prefill 2,488 tok/s at 8k / 2,996 at 32k, needle 92k in 46 s — i.e. the same as the stock non-deterministic kernel (the previous pin measured 32.5 / 2,436 / 2,904 / 48 s).
• EXACT_TOPK=1 — exact torch.topk fallback. Our first fix: also deterministic and same tournament score, but −8% prefill at 8k and −20–40% at 32k+ (decode unchanged). Kept as a fallback (it wins over DET_TOPK when set), e.g. on a GPU where the kernel is not built.
• DET_TOPK=0 EXACT_TOPK=0 gives the stock kernel back.

Once vllm#55122 is merged into the release branch this image is built from, patch 8 becomes redundant. (Masking the never-written logits columns before the stock kernel does not restore determinism, so it is the kernel itself.)

Optional: M%4 padding for the fp8 GEMM (PAD_M4=1)

Hybrid mode runs the GDN/QSA side layers and shared experts through vLLM's blockwise-fp8 cutlass GEMM. On this image (sm_12x) that kernel routes any call whose row count M is not a multiple of 4 (or ≤ 64) to a swap_ab path that is much slower — upstream fixed it in C++ (vllm#52775) after the image was cut. @jschmied found it and wrote a drop-in that pads M to a multiple of 4 inside an opaque custom op (fp8_m4pad_patch.py, patch 9 in the Dockerfile, fetched at a pinned commit; issue #3).

Measured on the GX10 at the kernel level (K=4096, N=8192): ×1.7 below 2,048 rows (0.63 → 1.09 ms at M=1,601), ×10–11 above (0.87 → 9.6 ms at M=2,401); padding restores the aligned time in every case. At the server level it depends on how the scheduler cuts prefill chunks:

• PREFIX_CACHE=1 (default): no-op. The Mamba align mode clips every prefill chunk to the 1,600-token block boundary, so M % 4 == 0 on all large chunks. Same-session A/B on the hybrid (MTP=2): 8k 3.33 → 3.24 s, 32k 11.63 → 10.97 s, salted repeats within noise, and prompts built to leave a misaligned last chunk (8,801 / 8,803 tokens) showed no penalty either. Off by default.
• PREFIX_CACHE=0: use it. Chunks are then whatever the batch size leaves (an 8,001-token prompt is one 8,001-row chunk); @jschmied measured −40% TTFT at 8k and −10–15% at 30k on the stock image, and the unpatched kernel is bimodal (2.9–6.6 s at 8k depending on the cut).

PAD_M4=1 also sets VLLM_FP8_PAD_M4=1; scripts/serve.sh always passes the variable because the patch itself defaults to on when it is unset. NVFP4 mode does not use this GEMM.

Optional: persistent compile cache (COMPILE_CACHE)

scripts/serve.sh recreates the container on every run (docker rm -f, then docker run), so vLLM's compiled graphs — written to /root/.cache/vllm inside the container — are discarded and rebuilt on every boot. If you keep one container and cycle it with ./flash stop / ./flash start this costs nothing, which is why it went unnoticed. It costs you when something else recreates the container for you: a proxy that loads and evicts models on demand (llama-swap and friends), CI, or a tournament run that calls serve.sh between configurations.

COMPILE_CACHE=<name> mounts two docker volumes (<name>-vllm, <name>-flashinfer) over the two cache directories. COMPILE_CACHE=/some/path binds /some/path/vllm and /some/path/flashinfer instead, to put them on a chosen disk. Unset — the default — is exactly the behaviour above.

Measured on a GX10, hybrid + YaRN 500k + MTP=2, same recipe each time. Boot totals are not usable for this: weight loading varied between 464 s and 554 s on page-cache state alone, and CUDA-graph capture between 4 s and 13 s, both larger than the effect being measured. The signal is in init engine with capture excluded, one row per boot:

Reused (boots 3–5) is 35.9 s ± 2 s, against 116.2 s with the cache off: −80 s ± 2 s (−69%) per boot. Populating it costs nothing (boot 2 at 115.5 s against boot 1 at 116.2 s). Reused boots log Directly load AOT compilation from path …. Disk: 168 MB for the vLLM cache, 0.5 MB for FlashInfer. Startup only — no effect on outputs, so nothing for the tournament to say.

Triton's /root/.triton is deliberately not persisted. It looks like it should be the interesting one: jit_monitor warns that five kernels (_qsa_mqa_paged_kernel, _qsa_sparse_paged_gqa_splitk, _compute_local_logits_stats_, _rejection_kernel, _resample_kernel) JIT-compile during the first request. Boot 4 above tested it directly — vLLM cache warm, only the Triton volume emptied — and came out at 37.2 s against boot 3's 37.1 s: a 0.2 s difference, an order of magnitude below the capture noise. The five warnings appear in every boot either way, warm or cold. Mounting it would have been cargo cult.

Reduced draft vocabulary (DRAFT_VOCAB=1, default)

vLLM shares the target model's lm_head with the MTP draft, so every draft step scores all 248,320 vocabulary rows: a 1.27 GiB bf16 read per drafted token, on a decode step that is memory-bandwidth bound. With DRAFT_VOCAB=1 the drafter scores a private 65,536-row slice of the head (+320 MiB of memory) and every other token gets −∞, so the proposer's argmax/sampling code is untouched. The target still verifies every drafted token: outputs are identical to full-vocabulary drafting; only the acceptance rate can move, down, when the target wants a token outside the set (about 6% of the tokens of a French text, 75% → 68% acceptance here).

The 65,536 ids are the most frequent tokens of a small local corpus, then BPE merge order as a frequency proxy, plus every special and added token (chat template, tool-call and thinking markers, byte fallbacks). tools/build_draft_vocab.py rebuilds the set for another language mix; DRAFT_VOCAB=/path/ids.npy uses your own, DRAFT_VOCAB=0 disables.

Measured on the GX10 the way defaults are decided here — the 17-scenario agentic tournament, 3 repeats, temperature 0.2, one variable per run, all on the same day (hybrid, MTP=2, prefix caching, YaRN 500k):

Two runs of the same configuration on different days differ by up to 2 points, so the four first rows are equivalent in quality; the last one shows why MTP=3 stays an option. Single-stream bench numbers for the default: decode 36.6 tok/s, prefill 2,473 tok/s at 8k / 3,004 at 32k, needle 92k in 45.5 s, 4/4 deterministic. The KV pool loses the 320 MiB slice plus the full-width logits buffer the patch rebuilds per draft step (about 60k tokens at GPU_MEM=0.80).

vLLM v0.29.0 as the base image (Dockerfile.v0.29)

The default Dockerfile patches Qwen's preview image (qwenllm/qwen3.8-flash-next-vllm). vLLM v0.29.0 is the first official release that ships the model natively — the package moved to vllm/models/qwen4_exp, there is an arm64 image, and two of the fixes this repo carried are in the release. Dockerfile.v0.29 builds the same recipe on top of it:

docker build -f Dockerfile.v0.29 -t qwen38-flash-dgx:v0.29 .
IMAGE=qwen38-flash-dgx:v0.29 MODE=hybrid YARN=1 CTX=500000 scripts/serve.sh

Same weights, same snapshot, same env knobs and defaults (MODE, DET_TOPK, DRAFT_VOCAB, MADVISE, PREFIX_CACHE, MTP, …). Both images carry a qwen38.base label and scripts/serve.sh reads it to pick the right splitting-op names (BASE=preview|v0.29 overrides).

What changes in the patch set:

Two things got simpler on the release: the Inductor int64-indexing assert that forced torch.compile off on the preview image is gone (compile is on, graphs stay PIECEWISE because the gather still has to run between graph segments), and the FLA/short-conv kernels need no --enforce-eager workarounds.

Measured on our GX10, hybrid, the default recipe, YaRN 500k, same day as the preview numbers:

The two tournament runs fail exactly the same scenarios as the preview image (the two that every quantization we tried fails); the 42.5 of the first run is two length finishes, the reasoning runaways we see in about one run out of three on any configuration. So: parity, with a slightly lower draft acceptance that we have not chased yet (the KV pool is within the boot-to-boot range of the preview image). The preview Dockerfile remains the default until this base has more field time on our own box; if you want to be on the release line, it is ready and tested. Full port notes in docs/HOW-IT-WORKS.md.

Checkpoints: NVIDIA's NVFP4 (default) and RadixArk's

Two NVFP4 quantizations of Qwen3.8-Flash-Next fit this recipe, and the recipe does not care which one it serves: MODEL=<org/name> on download-weights.sh, prepare-hybrid.sh and serve.sh (or ./flash setup default MODEL=… then ./flash serve). Both have NVFP4 routed experts, the fp8 PLE table this whole repo is about, and bf16 side layers that prepare-hybrid.sh converts to blockwise fp8 the same way (the 300 side tensors are the same weights, down to the conversion error); the deterministic top-k, the reduced draft vocabulary, prefix caching and YaRN apply identically.

• nvidia/Qwen3.8-Flash-Next-NVFP4 — the default since 2026-09-14. NVIDIA's own ModelOpt export (issue #17, @PathosEthosLogos). 124 GiB in 24 files, one of them a 50 GiB PLE shard that only downloads through Xet (download-weights.sh uses Xet by default). Its MTP drafter keeps its experts in blockwise fp8 under a ModelOpt mixed-precision config; vLLM 0.29 could not load that (no method for those experts, and the layer index of the drafter not remapped), which is patch 11: on the v0.29 image a backport of vLLM's own fix (vllm#55513, src/patch_block_fp8_mtp.py, @techfury90), on the preview image the stopgap shim (src/vllm_modelopt_block_moe.py). Validated on both images.
• RadixArk/Qwen3.8-Flash-Next-NVFP4 — the default until 2026-09-14, the checkpoint every number in the sections above was measured on. 122 GiB in 418 files, plain ModelOpt NVFP4 config, bf16 MTP drafter (hence the hybrid-mtp graft, which only exists for it). Fully supported, one variable away:

MODEL=RadixArk/Qwen3.8-Flash-Next-NVFP4 scripts/download-weights.sh
MODEL=RadixArk/Qwen3.8-Flash-Next-NVFP4 scripts/prepare-hybrid.sh
MODEL=RadixArk/Qwen3.8-Flash-Next-NVFP4 MODE=hybrid YARN=1 CTX=500000 scripts/serve.sh
#   or: ./flash setup default MODEL=RadixArk/Qwen3.8-Flash-Next-NVFP4 && ./flash serve default MODEL=RadixArk/Qwen3.8-Flash-Next-NVFP4

Why the default changed: the head-to-head (2026-09-13/14)

The rule of this repo is quality first; a default only moves on measured data, never on a smoke test. So the two checkpoints were run through the same protocol, back to back, on the same box:

• same image (qwen38-flash-dgx, v0.29 base, the tree of main), same recipe (hybrid, deterministic top-k, draft vocab 65k, MTP=2, YaRN 500k, prefix caching), only MODEL differs;
• one boot per checkpoint; on that boot: speed, memory, long-context and determinism probes, then 5 full passes of the agentic tournament — 55 scenarios (tool loops with injected faults, long ledgers, logic grids, calendar and org-chart deduction, hidden-test coding), temperature 0.2, 32k reasoning tokens per turn, partial credit on multi-item scenarios — then a dedicated block on the four coding tasks where this model's reasoning most often runs away (10 samples each);
• 16 h of GPU time, 275 scored runs per side, every trajectory kept.

How to read it, honestly:

• The 2.7-point gap is above the noise floor of a 5-pass design (bootstrap CI [+0.4, +5.3], permutation p = 0.048) but about half of it is luck on the runaway-prone coding tasks, where both checkpoints derail at the same rate. The half that is not luck is a handful of slips only RadixArk made: a counting error on a paginated ledger (5 of 5 passes there, 8 of 18 across three images; NVIDIA 14 of 15), one fabricated tool trajectory, two mis-executed multi-step tool tasks. NVIDIA's only systematic weakness in the data is a reporting convention (it under-counts a total_moved line while executing the 30 operations perfectly).
• The claim this data supports is "parity or better on quality, +15–22% KV, −8% single-stream decode, vendor checkpoint whose loading fix is upstream" — not "NVIDIA is 2.7 points smarter".
• Under that reading the default rule is unambiguous: at quality parity or better, the checkpoint that leaves more room for context and concurrency, and that the next vLLM release will load with no patch at all, is the one to ship. If your use is a single chat stream where 37 vs 34 tok/s is what you feel, RadixArk keeps a small edge and is one variable away.

Derivatives of either checkpoint in the same ModelOpt layout (abliterated variants such as Jiunsong/SuperQwen3.8-Flash-Next-abliterated-NVFP4-DGX-Spark or drowzeys/keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47) run with the same commands; we do not ship or endorse them, we only note that the recipe does not care. Compressed-tensors derivatives (orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 and lychee888/…-FP8PLE, issue #23) are a different format: their quantization config lives in config.json (no hf_quant_config.json, which is fine, vLLM reads it from there) and their dense side layers are already fp8, so the hybrid step has nothing to convert — prepare-hybrid.sh and flash doctor say so and point to the published profile (MODE=nvfp4). The PLE table of the lychee888 build uses RadixArk's exact file and key layout, so the mmap patch should apply; we have not booted one ourselves.

Tuning (env vars for scripts/serve.sh)

Watching the mmapped table (vllm:ple_mmap_*)

The PLE table is the one component whose cost depends on runtime state rather than configuration: how much of its 47.7 GiB the page cache is holding decides your prefill speed, and that moves as the KV pool, the request mix and the OS all pull on the same unified memory. The module exports five counters so this is visible on a dashboard rather than only in a windowed log line that a container restart destroys:

vllm:ple_mmap_lookup_ops_total         lookups (hash + gather + H2D)
vllm:ple_mmap_op_seconds_total         cumulative seconds in the lookup op, GPU wait included
vllm:ple_mmap_gpu_wait_seconds_total   of which: waiting for GPU work queued ahead of the lookup (not PLE cost)
vllm:ple_mmap_dedup_seconds_total      of which: copying the row ids to the host and deduplicating them
vllm:ple_mmap_gather_seconds_total     of which: the row gather (disk reads)
vllm:ple_mmap_stage_seconds_total      of which: staging the rows for the GPU (pinned copy, H2D launch)
vllm:ple_mmap_rows_total               rows gathered
vllm:ple_mmap_bytes_total              bytes read from the table

The lookup starts with a blocking copy of the row ids to the host, and that copy waits for every GPU kernel queued ahead of it: the n-gram hashing and the layers before the PLE layer. op_seconds therefore mixes their compute with the lookup's own cost (one prefill window read 165 ms per lookup, of which 8 ms was the gather). gpu_wait_seconds is that wait on its own, so op − gpu_wait is what the lookup itself costs the step. The periodic PLE mmap stats log line shows the same split at its end: gpu-wait X ms/op, host Y ms/op (dedup, gather, stage).

They are registered in the EngineCore process, so they only reach /metrics when prometheus_client runs in multiprocess mode. vLLM turns that on only for api_server_count > 1; scripts/serve.sh does it for the single-server setup used here when you opt in with PROM_MULTIPROC=1, by pointing PROMETHEUS_MULTIPROC_DIR at a fresh tmpfs.

Switching to multiprocess mode was checked against a live server by diffing the complete /metrics before and after: vLLM's other 71 metric families are exported with identical label sets and no per-process pid label. The only loss is the 35 *_created families, which prometheus_client does not export in multiprocess mode; that is why exporting the counters is opt-in, so nothing changes for existing dashboards unless you ask for it.

The views worth graphing:

# host seconds per lookup: what the PLE lookup itself costs each step
(rate(vllm:ple_mmap_op_seconds_total[5m]) - rate(vllm:ple_mmap_gpu_wait_seconds_total[5m]))
  / rate(vllm:ple_mmap_lookup_ops_total[5m])
# page-cache health: the share of the lookup's own time spent on disk
rate(vllm:ple_mmap_gather_seconds_total[5m])
  / (rate(vllm:ple_mmap_op_seconds_total[5m]) - rate(vllm:ple_mmap_gpu_wait_seconds_total[5m]))
# NVMe read bandwidth from the table
rate(vllm:ple_mmap_bytes_total[5m])

The disk share is the page-cache health signal. It climbs as the cache is squeezed and falls as the hot region settles in. Divide by op_seconds alone and it also moves with GPU load, which says nothing about the cache. Pair it with Cached from a node exporter, since nothing in vLLM's own metrics exposes the quantity that actually governs it. VLLM_PLE_MMAP_PROMETHEUS=0 turns the counters off.

Throughput and concurrency

Single-stream numbers understate this model on a GB10. @jschmied traced one box under load (RadixArk NVFP4, 8k ctx, no speculative decoding, using vLLM's native PLE CPU offload rather than this repo's mmap — the table-serving cost behaves the same way) and found aggregate throughput scales far past single-stream:

Two things worth knowing (their words, lightly condensed):

• The paged table is an argument for concurrency, not against it. Page-fault cost per token falls 4.4× from c=1 to c=48: batched tokens share n-gram rows and the page cache keeps the hot set, so the marginal token is far cheaper than the first. The table gather itself never exceeded ~25% of one CPU core.
• A low --max-num-seqs is indistinguishable from saturation if you only look at tok/s. With --max-num-seqs 2 their sweep flatlined at ~33 tok/s while vllm:request_queue_time_seconds_sum climbed to 142 s. Check max-num-seqs before quoting an aggregate number — this repo's default is now 8 for that reason.

Method and harness: load-and-waits.md.

Decoding clients stall while other clients prefill (the long-prefill-token-threshold slider)

Reported by @kutovoy in issue #9: with two or more agents on the box, a client that is decoding drops from 30 tok/s to 0.1–0.5 tok/s for a minute or two, then recovers. Reproduced here in minutes and it is not a bug in the recipe: vLLM runs one step at a time, and with chunked prefill every step that carries a prefill chunk also carries exactly one token for each decoding request. On this model a chunk of 8,192 tokens (--max-num-batched-tokens 8192, the default) takes ~3.5 s of compute at ~2,400 tok/s, plus the n-gram lookups of a prompt the page cache has never seen (0.3–0.4 s per chunk here, more on a box that is short on cache or swapping). So while any new prompt is being prefilled, a decoding client gets one token per step, i.e. one every 4–7 s. Prefix caching does not help: the prompts are new.

The knob is --long-prefill-token-threshold N (per-request chunk cap; the total step budget stays at 8,192 for the decoders), passed through EXTRA=. Measured on the GX10 (hybrid, MTP=2, prefix caching): one client decoding, then two other clients sending cold ~72k-token prompts 10 s later.

Read it as a slider, not a fix: on one GPU, keeping a decoding client at X tok/s while others prefill means at most ~2,400 / X prefill tokens per step, and every step below 8,192 tokens costs single-stream prefill (a 1,024-token cap already takes 36% off an 8k TTFT). A step still holds one chunk of each running prefill, which is why 512 does not reach 5 tok/s. The smaller chunks do have one unambiguous benefit: peak swap-out during the prefills fell from 90–120 MB/s to 7–60 MB/s, because the activation peak per step shrinks.

• Single main user, occasional second client (our case): keep the default. Long prompts land fast; the rare overlap costs the other client a slow minute.
• Several interactive agents that must stay responsive: EXTRA='--long-prefill-token-threshold 1024' (or 512 if TTFT matters less than never stalling). Confirmed in the field by @techfury90 with 2–6 parallel agents. Keep swap small (vm.swappiness=10 — the Spark default is 60; a 134 GB swap file lets the kernel page vLLM itself out instead of dropping cache, and once it is swapped every step page-faults) and use PREWARM=1. Lower GPU_MEM (0.75) only if the box is actually swapping: with several long-context agents the KV pool matters more than page cache for the table. SEQS=4 limits how many prefills can interleave.
• The structural way out is a second Spark: the four ConnectX-7 ports exist for that, and vLLM's prefill/decode disaggregation puts the prefills on the other box.

How it fits — the one idea

A token's n-gram lookup reads 16 rows × 160 bytes ≈ 2.5 KB. Over a 20k-token prefill that's ~1.3 GB of small reads — under a second on NVMe, and the hot n-grams stay in the page cache. So the 48 GiB table never needs to be in the unified pool: we mmap the checkpoint's model-plefp8-*.safetensors shards and gather rows on demand. Nothing else about the model changes — the hashing, dequant, and the sparse attention all run stock.

Full details, including the GB10-specific bugs this works around and the long-context findings, are in docs/HOW-IT-WORKS.md.

GB10 kernel fixes and the faster gather (contributed)

From @Saren-Arterius's fork, merged here with thanks:

• FLA shared-memory gate — sm_121 reports 99 KiB of shared memory per block but the flash-linear-attention gate asked for 100 KiB, so all 36 GDN layers silently ran on small tiles. One sed in the Dockerfile lowers the gate to 99 KiB.
• chunk_delta_h num_warps pin — works around a tl.dot race on Blackwell (fla#953). Correctness, not speed.
• PLE gather hot path — CPU dedup of row ids, a persistent pinned staging buffer with an async H2D copy, GPU-side expansion through the inverse index, and an inline fast path for decode-sized batches (VLLM_PLE_MMAP_FAST_ROWS, module default 512; larger gathers are split into VLLM_PLE_MMAP_CHUNK=2048-row tasks across WORKERS threads). serve.sh now sets it to 0 (FAST_ROWS): on a Spark enough rows miss the page cache that overlapping their faults on the pool wins, by 8% at 1 stream and 17% at 4. Also: bf16/f16 tables, VLLM_PLE_MMAP_DIR to serve the table from another directory, and a periodic PLE mmap stats log line (VLLM_PLE_MMAP_STATS_SEC, default 30).
• Mamba state-copy guard — with vllm#50729 (the overlapping-copy race fix by @AndreasKaratzas), a bounds check that turns an out-of-range block id into a skipped copy plus a log counter instead of a dead CUDA context. With the block_size fix above the counter stays at 0; if you ever see mamba state-copy guard: N out-of-range, something upstream regressed — please report it.
• fp8_convert.py — the side-layer conversion behind MODE=hybrid.

Their fork goes further with an int4 (Intel AutoRound) + fp8 hybrid checkpoint: qwen3.8-Flash-DGX-AutoRound. We benchmarked it with the same patches: ~34 tok/s decode, 44/51 on the tournament, but it is not deterministic even with the exact top-k and Marlin's atomic adds off, and it has the slowest prefill of the three — we kept the NVFP4-based layouts.

Alternative: vLLM's native PLE CPU offload

vLLM ships its own path (VLLM_PLE_CPU_OFFLOAD=1) that keeps the table in pinned host RAM in a separate worker process. On a Spark that RAM is the same pool as the GPU, so it saves less than the mmap — but @jschmied got it running and documented two things you will need if you go that way (neither applies to the mmap patch, which is a single process):

1. _get_ple_embedding_quant_method() in ple_layer.py only accepts Fp8Config; with the NVFP4 checkpoint the quant config is modelopt_fp4, so the FP8 PLE shards are rejected and loading dies on ngram_embedding.weight_scale. Accepting modelopt/modelopt_fp4 there fixes it.
2. The worker hands CUDA tensors to the GPU process over IPC via pidfd_getfd, which kernel.yama.ptrace_scope=1 (the Ubuntu/DGX OS default) forbids between sibling processes. In Docker: --cap-add=SYS_PTRACE. Under systemd: AmbientCapabilities=CAP_SYS_PTRACE. It fails ~10 minutes in, after all shards have loaded, with an unhelpful Engine core initialization failed.

Details: results-radixark-vllm.md.

What's in here

flash                             one-command front-end: doctor / setup / serve <profile> / wait / test / status …
profiles/*.env                    named recipes for it (default, speed, context, context-1m, shared, published, native, v0.29)
Dockerfile                        official vLLM Flash-Next preview image + the patches below (default)
Dockerfile.v0.29                  same recipe on the vLLM v0.29.0 release (patches 3 and 9 dropped, 7 not ported, 11 is the vllm#55513 backport)
src/vllm_ple_mmap.py              1. mmap PLE table (opaque splitting op)            VLLM_PLE_MMAP=1
                                     handles both layouts (preview forward_impl hook / v0.29 embedding swap)
src/mamba_utils_guarded.py        3. vllm#50729 + bounds guard (drop-in mamba_utils.py)
src/patch_mamba_block_size.py     4. prefix-caching block_size fix
src/patch_qsa_exact_topk.py       5. exact, deterministic QSA top-k                  VLLM_QSA_EXACT_TOPK=1
(Dockerfile patch 8)              8. deterministic persistent_topk kernel, built at docker build  VLLM_QSA_DET_TOPK=1
                                     from @jschmied's repo (pinned commit) — vllm#55122
(Dockerfile patch 9)              9. M%4 padding for the blockwise-fp8 GEMM (@jschmied,      VLLM_FP8_PAD_M4=1
                                     pinned commit) — hybrid mode with prefix caching off
src/patch_mtp_draft_vocab.py     10. reduced draft vocabulary for the MTP drafter          VLLM_MTP_DRAFT_VOCAB=<ids.npy>
src/draft_vocab_65536.npy            the default 65,536-id set (tools/build_draft_vocab.py rebuilds it)
src/patch_block_fp8_mtp.py       11. vllm#55513 backport: block-FP8 MTP experts in ModelOpt mixed checkpoints (v0.29 base)
src/vllm_fp8_hybrid_modelopt.py   6. NVFP4 experts + fp8 side layers dispatch        VLLM_FP8_HYBRID=1
                                     (patches the NVFP4 and the mixed-precision ModelOpt config classes)
src/vllm_modelopt_block_moe.py   11. FP8_BLOCK_SCALES layers in ModelOpt mixed checkpoints (NVIDIA's MTP
                                     experts) -> vLLM's block-fp8 MoE method. Preview base only; the v0.29
                                     image uses the vllm#55513 backport instead           VLLM_MODELOPT_BLOCK_MOE=0 disables
src/patch_qsa_fp8_kv.py           7. fp8_e4m3 KV cache on the QSA path (by @Nanetnounou) --kv-cache-dtype fp8_e4m3
src/test_ple_mmap_cpu.py          CPU unit test for the gather (no GPU needed)
src/test_qsa_exact_topk_cpu.py    CPU unit test for the exact top-k (no GPU needed)
src/test_block_fp8_mtp_cpu.py     CPU unit test for the vllm#55513 backport (no GPU needed; v0.29 image)
tools/fp8_convert.py              side-layer bf16 -> blockwise fp8 (by @Saren-Arterius)
scripts/download-weights.sh       MODEL (default nvidia/Qwen3.8-Flash-Next-NVFP4), EXCLUDE, MAX_WORKERS, XET
scripts/prepare-hybrid.sh         one-time: build the -fp8hybrid snapshot
scripts/prepare-mtp-graft.sh      one-time: graft the NVFP4 MTP draft experts onto it (MODE=hybrid-mtp, RadixArk only)
tools/vllm_watch.py               live per-session view of prompts / reasoning / outputs / stats (needs LOG_REQUESTS=1; @0x3dlux)
scripts/serve.sh                  MODE=nvfp4|hybrid|hybrid-mtp, PREFIX_CACHE, DET_TOPK, DRAFT_VOCAB, MADVISE, EXACT_TOPK, PAD_M4, KV_DTYPE, YARN, ...
scripts/smoke-test.sh             health, coherence, prefix-cache hit, determinism, tok/s
scripts/greedy-probe.sh           greedy probe set; diff two arms to gate a draft/checkpoint swap
docs/HOW-IT-WORKS.md

Run the unit tests (no GPU):

docker run --rm -v "$PWD/src:/t" -w /t --entrypoint python3 qwen38-flash-dgx test_ple_mmap_cpu.py
docker run --rm -v "$PWD/src:/t" -w /t --entrypoint python3 qwen38-flash-dgx test_qsa_exact_topk_cpu.py
docker run --rm -v "$PWD/src:/t" -w /t --entrypoint python3 qwen38-flash-dgx:v0.29 test_block_fp8_mtp_cpu.py

Limitations & notes

• One big model at a time. At GPU_MEM=0.80 this uses most of the 128 GB pool; don't co-locate another large model (an 8B embedding model next to it already starves the KV cache — we moved ours to another machine).
• Full torch.compile is off on the preview image (an Inductor int64-indexing assert on sm_121; gone on the v0.29 base); on both, the serve script uses PIECEWISE CUDA graphs with the PLE lookup as a splitting op.
• 1M context needs the fp8 KV cache (KV_DTYPE=fp8_e4m3, see above; preview image only, not ported to the v0.29 base yet), which costs speed and some quality; in bf16 a single 1M request needs ~26 GiB of KV and 500k with YaRN is the validated ceiling (800k booted but got OOM-killed on a long prefill).
• Exact top-k costs prefill on long prompts (see above). The implementation is a plain torch.topk over the full visible width per chunk; a fused kernel would recover most of it — PRs welcome.
• No authentication, binds all interfaces. scripts/serve.sh publishes the API on 0.0.0.0:$PORT with no key, like the upstream image. On a shared network put it behind your gateway or pass EXTRA='--api-key <secret>' (vLLM then requires it as a Bearer token).
• Weights are not included and the checkpoint carries Qwen's license (with a MAU/revenue clause) — review it before production use.

Credits

• Download knobs (Xet, EXCLUDE, MAX_WORKERS), the vllm:ple_mmap_* Prometheus counters and KV_CACHE_MEM: @techfury90 (PR #19); also spotted the upstream fix for NVIDIA's fp8 MTP experts (vllm#55513).

• COMPILE_CACHE, the persistent compile cache: @AronRubin (PR #21).

• The pointer to NVIDIA's own NVFP4 checkpoint: @PathosEthosLogos (issue #17).

• tools/vllm_watch.py, the live session viewer: @0x3dlux (issue #12).

• The nudge to port the recipe to the vLLM v0.29.0 release: @ChengYen-Tang (issue #14).

• The two ideas behind the reduced draft vocabulary and the MADV_RANDOM table advice come from MiaAI-Lab's recipe for their own checkpoint; both are reimplemented here from scratch (their code is AGPL-3.0) and measured on this checkpoint.

• Model: Qwen team, Alibaba — Qwen3.8-Flash-Next.

• NVFP4 checkpoints: nvidia/Qwen3.8-Flash-Next-NVFP4 (the default since 2026-09-14) and RadixArk/Qwen3.8-Flash-Next-NVFP4 (the default before, still supported).

• NVFP4 MTP draft experts (the hybrid-mtp graft donor): Inferact/Qwen3.8-Flash-Next-NVFP4; the graft recipe follows thavoc's graft write-up and Peuqui/mtp-quant-transplant.

• Serving engine and base image: vLLM (vllm/vllm-openai:qwen38-flash-next, the release/qwen38next recipe / PR #53896); the Mamba state-copy race fix is vllm#50729 by @AndreasKaratzas.

• GB10 FLA fixes, the faster PLE gather, the state-copy guard and the fp8 side-layer conversion: @Saren-Arterius (qwen3.8-Flash-DGX-AutoRound).

• The fp8_e4m3 KV cache patch for the QSA path: @Nanetnounou (issue #6, vllm#54426).

• The non-deterministic persistent_topk diagnosis and upstream report: @k3dani (issue #3, vllm#51782).

• The deterministic persistent_topk kernel (vllm#55122, patch 8), the fp8 GEMM M % 4 finding and its padding drop-in (patch 9), the independent reproduction on a DGX Spark, the native-offload fixes and the concurrency measurements: @jschmied (issue #1, qwen38-flash-next-gb10).

• The mmap-PLE patch, the hybrid dispatch for ModelOpt-NVFP4, the prefix-caching root cause and fix, the exact top-k path and the GB10 serving recipe in this repo: see LICENSE (Apache-2.0).