Qwen3.8-Flash-Next on one DGX Spark

Serving a 126.0 GiB checkpoint on a machine with 121.63 GiB of memory — at 41.5 tok/s on code, across the full 262k context, scoring within noise of the checkpoint's published GSM8K.

Two small patches to SGLang, plus one flag most people won't find. Reproducible recipe, microbenchmarks, and measured numbers below.

Day-zero work, 2026-08-26 — the model, the SGLang support PR and this repo are all the same day old. Everything here is measured on one GB10, not projected. Where I'm extrapolating, I say so.

Correction (2026-08-29). Patch 2 as first published — widening the trtllm gate to sm_12x — was wrong: on GB10 that path is flashinfer XQA, and it silently corrupts long-context decode (runs of token id 0 in 1/4 requests at 120k prompt tokens, 4/4 at 210k; HTTP 200 throughout). Upstream measured it on two independent Spark setups and retired the widening in sglang#36806; the fix for GB10 is the Triton varlen fallback of sglang#36845, which this recipe now ships as patch 2 and which I validated here: exact needle retrieval 4/4 at each of 120k, 190k and 210k. If you cloned this before 2026-08-29, re-run ./scripts/prepare.sh. Details in the patch 2 section.

---

The problem

Qwen3.8-Flash-Next is 125B total / 6B active + 51B of n-gram embeddings (PLE) + 4B MTP. The only checkpoint that gets close to a single Spark is RadixArk/Qwen3.8-Flash-Next-NVFP4, and it quantizes only the routed experts:

A DGX Spark has 121.63 GiB. That's 4.4 GiB short before any KV cache, activations or CUDA context.

Why the built-in offload doesn't help here

Every engine ships the same escape hatch, and on unified memory none of them work:

On a B300, host RAM is different memory and you free 47.7 GiB of VRAM. On a Spark, host RAM and GPU memory are the same 121.63 GiB pool — moving the table between them frees nothing. vLLM's official recipe asks for TP2 minimum on GB300; the SGLang cookbook lists H200/B200/B300/GB300 and GB10 is not on it.

---

Patch 1 — the PLE table lives on NVMe

The idea is not mine — it is Qwen's, and it is in their tech report (§2.3.2):

Because embedding tables are sparsely accessed and deterministically addressed, they can be scaled with negligible additional per-token computation and stored in off-accelerator storage.

The whole layer is designed around it. §2.3.1: "We place it at Layer 2, allowing host-memory prefetching to overlap with the computation of the first layer." What follows is an implementation of that for SGLang on a unified-memory box, not a discovery.

The 47.7 GiB PLE table is not a weight matrix you multiply. It's a lookup table: 20M rows of 160 bytes, and a token reads 16 of them. There is exactly one PLE layer (ple_layer_ids: [2]), so one gather per forward pass.

GB10 reports cudaDevAttrPageableMemoryAccessUsesHostPageTables == 1 — the GPU resolves host virtual addresses through the host page tables, so a CUDA kernel can dereference a pointer into a file-backed mmap and the OS services the faults. I verified this against SGLang's own Triton gather kernel, with the page cache dropped by hand, before touching SGLang (bench/test_mmap_gather.py):

So the patch is three lines in Qwen4ExpPinnedHostEmbedding.__init__ — swap torch.empty(..., pin_memory=True) for a torch.from_file(shared=True):

 cpu_weight = nn.Parameter(
-    torch.empty(source_weight.shape, dtype=source_weight.dtype,
-                device="cpu", pin_memory=True),
+    _alloc_ple_table(source_weight.shape, source_weight.dtype),
     requires_grad=False,
 )

Nothing downstream changes. The Triton gather kernel, the prefetch stream and the CUDA graphs all still receive a host pointer — they don't care that it's now backed by a file. cuda graph: True in the scheduler confirms graph capture still works.

Measured cost in service: under 3% of wall clock. 138 KB/token of disk reads against the 2.5 KB actually used — that's 4 KB page granularity, not a bandwidth problem (20 MB/s on an NVMe). The patch also sets madvise(MADV_RANDOM), which matters more in long prefill than in decode.

The backing file is sparse and persists across restarts (deterministic name), so it only occupies what has been written.

Correctness is verified, not assumed — see the GSM8K number below. A loader that only reinterprets the fp8 bytes without applying the scale would serve wrong embeddings silently; bench/test_mmap_write.py checks the write path is bit-exact and the dequant matches.

Patch 2 — QSA has no decode kernel on sm_121 (and the obvious fix is wrong)

is_sm100_supported() requires major == 10. GB10 is (12, 1). Two independent sites gate on it and together they dead-end:

1. arg_groups/overrides.py::_qwen3_5_hybrid_overrides returns {}, so attention_backend never receives the family default (triton) and falls through to the global default — flashinfer.
2. qwen_sparse_attn_backend.py::_resolve_trtllm_sparse_decode returns None, so QSA falls back to its packed varlen path, which needs FA2 (not in the image) or the FA4 cute interface.

FA4 cute then fails to compile:

MLIRError: expects `coord` and shape of view are weakly congruent, but got
'!cute.layout<"(?,?):(?{i64 div=8},1)">', '!cute.coord<"(_,_,?)">'
  flash_attn/cute/flash_fwd.py:393, in epilogue

Net result: no working QSA decode path at all on sm_121. Setting --attention-backend triton fixes the first site only.

What I did first, and why it was wrong

The first version of this patch widened the gate (is_sm100_supported() or is_sm120_supported()), on the reasoning that flashinfer 0.6.17 ships trtllm_batch_decode_with_kv_cache for this device and would fail loudly if it didn't. It does not fail loudly. On sm_12x that call does not run trtllm-gen at all — there are no sm12x cubins — it routes to XQA, and XQA is numerically wrong on GB10 once the sparse selection has to prune, i.e. with more than indexer_budget (2048) tokens of history, and visibly so only from ~120k tokens up. Two independent measurements with real weights (dpolistwm, 2 Sparks TP=2; BBuf, 2 Sparks):

Every short benchmark in this README, GSM8K included, never enters that kernel. My own two long-context runs (128k and 240k, needle found) were n=1 each — consistent with "1 in 4", not evidence against it. That is the lesson: a widened gate has to be tested in the regime the gate was protecting, with n ≥ 4, not in the regime already being measured.

Updated 2026-08-30: the merged kernel (and this recipe ships it)

Upstream merged sglang#36845 with a different, faster kernel than its 2026-08-28 revision: an agent-optimized implementation (Codex + Kimi K3 via KDA-1.5) under sglang/kernels/kda_kernels/qwen38_qsa_sm121/ — BF16 tensor-core QK/PV with FP32 online softmax and in-launch split-KV merge, specialized to this model's exact decode contract. On this GB10 it measures 4-5x the 2026-08-28 Triton kernel at decode batch 1-4 (28 us vs 154 at batch 1) with identical numerics. The recipe now vendors that package verbatim (patches/kda_kernels/) and routes to it inside its contract, keeping the 2026-08-28 Triton kernel as the fallback outside it (patches/qsa_sm121_kda.py + patches/qsa_sm121_varlen.py). Validated here: needle retrieval 9/9 exact at 120k/190k/210k, decode after those prompts 46-92 tok/s (previously 30-48), short-context decode unchanged.

The 2026-08-28 patch (now the fallback)

Upstream retired the widening (sglang#36806: exact is_sm120() only, GB10 excluded) and BBuf added a narrow Triton online-softmax kernel as the sm_121 varlen fallback (sglang#36845: one query per sequence, device-side cu_seqlens, CUDA-graph safe). patches/qsa_sm121_triton.py inserts that hunk into the image's backend and patches/qsa_sm121_varlen.py is the kernel, verbatim; serve.sh mounts it.

Validated on this Spark (2026-08-29): the kernel against a PyTorch reference at the model's real shape (24 query heads / 2 KV heads / head dim 256, which the PR's differential tests did not cover) — max abs error 0.001, CUDA-graph replay with changed lengths ok, 0.10 ms per call; then end to end, needle retrieval 4/4 exact at 120k, 190k and 210k prompt tokens, decode 30–48 tok/s after those prompts, code suite unchanged. The short-context numbers below are unaffected: the dense path is the same.

---

Quickstart

Needs: a DGX Spark (or another GB10), Docker, ~140 GB of free disk, and membership in the docker group.

git clone https://github.com/hashd1ve/qwen38-flash-next-one-dgx-spark
cd qwen38-flash-next-one-dgx-spark

# 1) weights — 135 GB, ~20 min at 100 MB/s
./scripts/download.sh

# 2) pull the image, extract the two files from it, patch them, verify by AST
./scripts/prepare.sh

# 3) serve on :30000
./scripts/serve.sh

First boot is ~9 minutes: 8.5 min of weight loading (CPU-bound on dequant, ~182 MB/s from disk) plus pool allocation and graph capture. Roughly 2.5 of those minutes are the PLE table being written to its backing file; subsequent boots rewrite the same bytes.

Then:

python3 verify.py     # content -> thinking control -> GSM8K, in that order

---

The flags that matter, and why

--prefill-attention-backend triton      # trtllm_mha prefill is gated to SM100
--decode-attention-backend trtllm_mha   # decode explicitly allows SM120 — worth +32% on code
--quantization modelopt_fp4
--ple-offload-embedding                 # with patch 1, "offload" means NVMe, not pinned RAM
--language-only
--mamba-radix-cache-strategy extra_buffer   # required: page_size=64 is forced for compressed QSA
--mem-fraction-static 0.85              # 0.72 dies with total_rest_memory=-1.00 GB
--reasoning-parser qwen3 --tool-call-parser qwen3_coder
--speculative-algorithm NEXTN --speculative-num-steps 3
--speculative-eagle-topk 1              # PLE requires topk=1
--speculative-num-draft-tokens 4
--speculative-draft-model-quantization unquant   # the 31 MTP tensors are BF16 in an NVFP4 checkpoint

For long-context work, trade KV pool for headroom — see the stability warning below:

MEMFRAC=0.79 PREFILL=1024 CTX=262144 ./scripts/serve.sh

Results

One GB10, 121.63 GiB unified memory, tp1, ctx 32k unless stated.

On the GSM8K number, and how far to trust it. At n=200 the standard error is 1.39 pp, so 96.0% sits inside the noise band around the 97.27% reference (the gap is 0.9 SE — not significant).

But the two numbers come from different harnesses: 97.27% is RadixArk's, measured with sgl-eval; 96.0% is verify.py in this repo, with its own prompt and last-number extraction. Same benchmark, different protocol, so this is an indicative comparison and not a like-for-like one. For scale of how much protocol matters: Qwen's own tech report (Tab. 2) reports GSM8K 92.2 for this model. Three numbers, three harnesses, one model.

So what this measurement actually establishes is narrow and worth stating plainly: it rules out the silent failure mode of patch 1 — a PLE table served wrong from NVMe would not land here. It does not prove zero degradation. At this sample size a 2–3 pp regression would be invisible, and against a different harness the baseline itself moves by more than that.

An earlier run scored 59/60 = 98.3%, and that number was retired rather than kept: at n=60 one item is worth 1.7 pp. It was the flattering read of a noisy sample. Note also that the two runs differ in more than sample size — the n=200 run is on the final config (MTP + trtllm_mha decode) — so a clean A/B would need n=200 without MTP, which has not been run.

How the speed got there: 13.1 tok/s on code at first boot → 31.5 with MTP → 41.5 once decode moved to trtllm_mha. That's 3.2× with no change to the checkpoint. MTP helps code far more than prose, the same asymmetry other speculative decoders show on this hardware — the drafter is right about predictable text and wrong about prose.

---

Long context: it works, and it will take the box down if you are careless

The full 262,144-token context runs. --context-length 262144 needs no extra memory over 32k, because the KV pool is sized in tokens and already holds more than one full context (273,536), and because 36 of the 48 layers are Gated DeltaNet whose recurrent state is fixed size per request and does not grow with context. Measured at --mem-fraction-static 0.85, ctx 262144, one request at a time:

The needle is an invented fact ("the access code for the Argamasilla archive is …") inserted at the halfway point of a Don Quijote excerpt — real, varied prose, not repeated text a prefix cache would compress unfairly. QSA retrieves it at every length including a quarter of a million tokens.

Prefix caching is what makes this usable. Sending the same prompt a second time:

Three minutes of prefill becomes under two seconds. For agent workloads that resend a large context every turn, the corpus is paid for once per session rather than once per turn.

Decode at long context, measured over 300 generated tokens on a cache hit so the clock is decode and not prefill:

About 20% of degradation across the whole range.

The stability warning

Running a sequence of long prefills — four lengths up to 240k, flushing the prefix cache between each so the numbers would be cold — made the machine unresponsive. Ping still answered and TCP still accepted on every port, but no service completed a response and SSH died during banner exchange: classic userspace starvation. It did not recover on its own.

At --mem-fraction-static 0.85 SGLang takes ~103 GiB and leaves ~18 GiB for the OS, prefill activations, and page cache for the 47.7 GiB PLE table. A 240k prefill on top of that is enough to tip it. If you work at long context, give the box room:

• --mem-fraction-static 0.78–0.80. There is slack to give back: the KV pool holds 273,536 tokens and one full context needs 262,144.
• --chunked-prefill-size 1024 instead of 2048. Prefill activation memory scales with the chunk.
• --max-running-requests 1 or 2 for long-context work. Each request costs 110 MB of recurrent state plus its KV.

What I do not know is why the kernel could not reclaim its way out of it. A 47.7 GiB clean, file-backed mapping is exactly the kind of memory the page cache is supposed to drop under pressure. Two observations that should fit together and don't:

• mincore(2) reported the whole table resident even immediately after POSIX_FADV_DONTNEED on a range of it, while /proc/meminfo reported 26 GiB of Cached total at the same moment. Those two cannot both be right, and I have not found which is wrong. (This is why there is no residency tooling in this repo — see above.)
• My first guess was that the loader writes the whole table on every boot, leaving 47.7 GiB of dirty pages that cannot be evicted until written back. That guess does not survive its own data: Dirty was 154 MB when I looked, so writeback had already happened and the pages were clean. The mechanism is still open.

Populating the file once and mapping it read-only afterwards remains the most interesting open item — it would cut 2.5 minutes off every boot regardless, and if page state does turn out to matter it would settle that too. But it is a hypothesis to test, not a fix I can claim.

A methodology note on the prefill numbers

The time-to-first-token column above is trustworthy — those runs each carry a different needle, and the clock is dominated by a long measurable phase. The throughput figures derived from them (roughly 670–890 tok/s) are contaminated and are not published as a headline: every length is a prefix of the same corpus, so the longer runs reuse work the shorter ones cached. The tell is that one 240k run reported 1,227 tok/s where a cold one gives ~700 — the most flattering number of the session was the one that meant the least. A clean re-measurement with /flush_cache between lengths and disjoint corpus offsets is the right way to do it; that run is what took the machine down, so it is unfinished.

Gotchas the docs don't mention

• --attention-backend trtllm_mha is refused, but --decode-attention-backend trtllm_mha is not. The single flag sets both phases, and prefill is gated to SM100:

  ValueError: TRTLLM MHA backend for prefill is only supported on Blackwell GPUs (SM100).
  

  Read literally, that says the backend is unavailable. It isn't — the decode-side check in server_args.py lists is_sm120_supported() explicitly, so consumer Blackwell is a supported target for trtllm_mha decode. Splitting the phases gets you +32% on code (31.5 → 41.5 tok/s) and costs nothing. This one applies to any sm_120/121 box, model-independent.

• --mem-fraction-static 0.72 will not boot. total_rest_memory=-1.00 GB, dying in _handle_max_mamba_cache with mamba_cache_per_req=110.11 MB. Needs 0.85. Note the ceiling is artificial — the physical memory was free the whole time.

• reasoning_parser is auto-detected but not applied. Startup logs Auto-detected template features: ... reasoning_parser=qwen3 while server_args shows reasoning_parser=None. Without passing it explicitly, thinking text lands in content instead of reasoning_content.

• --load-format dummy is unusable with this model. Dummy init materializes the 47.7 GiB PLE table in RAM and the OOM killer takes the scheduler. It removes the obvious fast smoke-test path.

• nvidia-smi reports [N/A] for memory used on GB10. Use free. SGLang falls back to torch.cuda.mem_get_info() on its own and reports 124546 MiB.

• The MTP weights are BF16 inside an NVFP4 checkpoint — without unquant the draft inherits modelopt_fp4 from the body.

• PLE forbids two-batch overlap and NGRAM speculation, and requires topk=1.

Why single-stream stops at ~42 tok/s (and how to get to ~50)

Not bandwidth. The decisive measurement is concurrency scaling on the same server:

The box delivers 95 tok/s when given enough work. At C=1 it delivers 42.8. So single-stream is latency-bound, with roughly 2.2× of capacity sitting idle — not starved of memory bandwidth, which is what I assumed for most of a night before measuring it.

The way to use that headroom without concurrency is to verify more speculative tokens per forward. C=4 with 4 draft tokens is 16 token-rows per forward and yields 95 tok/s; C=1 is 4 rows and yields 42.8. Eight rows at C=1 would interpolate to roughly 60.

And that is exactly what is capped. From qwen_sparse_attn_backend.py:

NotImplementedError: Qwen QSA requires speculative_num_draft_tokens <= the QSA compress ratio (4):
the pending index-key ring holds one group; got 8

with the reason in the comment right above it: "The pending-group ring keys state by position % ratio". The indexer's pending compressed-key ring has exactly compress_ratio slots, so verifying more than 4 tokens at once makes two of them collide in the same slot and corrupts the keys the sparse attention selects blocks with. That is a real correctness constraint of the implementation, not a conservative guard — unlike the sm100 gate in patch 2, lifting this one would silently degrade attention rather than fail loudly.

It is liftable, and I lifted it. patches/qsa_ring_width.py separates the ring's width from compress_ratio and touches exactly three sites — the buffer allocation in qsa_kv_pool.py, the two slot builders in qsa/metadata.py, and the guard. compress_ratio keeps its meaning as the micro-block size everywhere else, so the block arithmetic (block_topk, compressed_page_size) is untouched by construction rather than by review. Set SGLANG_QSA_RING_WIDTH=8; without it, original behaviour.

Measured at --speculative-num-steps 7 --speculative-num-draft-tokens 8:

+18% on code with quality intact — and the 7.12 is the mechanical proof that more than four draft tokens per forward are being accepted, which the original ring made impossible.

Two things to know before using it. Prose gets 36% slower: seven draft steps cost 23 ms and prose accepts 0.42 of them, so you pay for drafts that get rejected. This is a code-tuned trade, and the flag that would adapt it per-request (--speculative-adaptive) is incompatible with the PLE. And the gain has a ceiling: measured at 3, 7 and 15 steps, acceptance grows logarithmically while draft cost grows linearly, so the optimum is a broad plateau at 7–9 steps and ~50 tok/s. Ring 16 at 15 steps measures 42.9 — worse than ring 8.

Acceptance is also not raisable by sampling. Swept on the wide ring at 7 steps, temperature 0.7 / 0.4 / 0.2 / 0.0 measures 50.2 / 47.7 / 50.0 / 50.7 tok/s with acceptance moving only 2.58 to 2.65 — even at greedy, where the target is deterministic and the drafter is predicting exactly its top-1. Acceptance here is bounded by the drafter's accuracy, not by sampling entropy, and the drafter is the MTP head that ships with the model. There is no DFlash2 drafter for this architecture.

It is not the default here for that reason, and because "verified" means two targeted tests and one benchmark, not a proof.

sm_121 is a second-class citizen, systematically

Patch 2 is not an isolated gap. Every one of these was hit on this box, with its exact error:

Of three flashinfer MoE backends, only flashinfer_cutlass runs here. Consumer Blackwell — RTX 50-series as much as GB10 — keeps landing on the generic path or on nothing at all.

Kernel flags that changed nothing

Measured, n=10, medians, code prompt. All ranges overlap the baseline's 41.1–43.2:

No available kernel flag moves single-stream throughput measurably. An earlier note in this repo claimed --speculative-attention-mode decode was worth +17% in forward rate; that came from dividing throughput by a noisy acceptance figure and does not survive direct comparison. Retracted.

Also already on, and worth knowing so nobody chases them: SGLANG_ENABLE_QWEN4_PLE_FUSION (default true), index_share_for_mtp_iteration (default true), and the fused HC mix/combine kernels (this model passes both of their conditions — batch ≤ 24 rows and hc_count × hidden % 2048 == 0). --enable-scattered-sconv is a TP-multi-rank optimization and does nothing at tp=1.

What an iteration is actually made of

Measured on the documented config, C=1, code prompt, n=10:

An iteration is 3 x draft + 1 x target, so 66 = 3d + 56 gives d = 3.3 ms per draft forward. The draft is nearly free: 10 ms of a 66 ms iteration. The target forward is 85% of it, and speculation is worth a clean 2.4x over not speculating.

Two things follow. Fewer speculative steps cannot help — you would save 3.3 ms and lose acceptance, so 3 steps is optimal, and the QSA cap at 4 draft tokens is not costing draft compute, it is costing width. And the same decomposition revises the estimate for widening the ring upward: seven steps would add 23 ms of draft to a 79 ms iteration, and at 4.5–5 accepted that lands at 57–63 tok/s, not the ~48 estimated earlier from the concurrency curve. Wide error bars in both directions, and still gated behind a correctness-risky rewrite of four files.

It also independently confirms the --decode-attention-backend trtllm_mha win on a measurement with no speculation noise in it: 13.1 tok/s with triton decode against 17.8 with trtllm_mha, +36%.

The ceiling does not move with acceptance either

Throughput here is iterations/s x accepted tokens, so the obvious remaining lever is acceptance — and copy-heavy work is where speculative drafting should shine. Five tasks at C=1, ordered from most to least "copy":

Acceptance did rise as predicted — median 3.33 against 2.77 on free generation, hitting the maximum of 4.00 — and the verbatim reproduction was correct. Throughput did not move. Which means iteration time grew in proportion: 66 ms per iteration at 2.77 accepted, 79 ms at 3.33.

A hypothesis for why, stated as one: something in the forward does not amortize over accepted tokens. Attention and MoE do — one weight read serves all of them. A recurrence cannot, because S_t depends on S_{t-1}, and 36 of the 48 layers are Gated DeltaNet. If the GDN state has to be rolled forward once per accepted token, accepting more tokens buys proportionally more sequential work and throughput pins.

That hypothesis did not survive its own tests. Two knobs aim at it from different directions and neither moves anything: --mamba-ssm-dtype bfloat16 against the model's declared float32 (halving the state's bytes) measures 41.3, and --enable-tf32-matmul (speeding the fp32 matmuls the recurrence runs on) measures 40.8. If the GDN recurrence were the bottleneck, one of those should have shown.

What is left is the plainer reading, and it is the one the concurrency scaling already pointed at: at batch 1 a forward is 48 layers of small kernels with the GPU underutilized, and no single component dominates. That is what "latency-bound" means here, and it is why concurrency fixes it and nothing else does.

Where the speed actually goes

13.5 tok/s without speculation is about 41% of the memory-bandwidth roofline, and the reason is counter-intuitive: it isn't the experts.

The "6B active" are the NVFP4 part — roughly 1.2 GB read per token. But the ~3.5B dense parameters (GDN, QSA, mHC, shared expert, embeddings) are still BF16 and are read in full on every token: ~7 GB. So the small half of the model dominates the clock. At the Spark's ~273 GB/s that puts the ceiling near 33 tok/s.

Two different kinds of headroom, then:

• Implementation. Moving decode to trtllm_mha already took code from 31.5 to 41.5 tok/s, which closed a good part of this gap. What's left untested: more MTP steps. Qwen's tech report (Tab. 4) measures a mean accepted length of 4.07 under four-step speculative decoding (4.20 on GSM8K, 4.26 on HumanEval). This recipe runs three steps and sees 2.25–3.10, so there is measurable room — and it's the paper's number, not a guess.
• Another knob from the report (§2.2, Inference Efficiency): Qwen keep the widened residual state in FP8, which "halves the bytes moved for the residual state relative to BF16, with almost no loss in quality." Since decode here is memory-bound, that goes straight to the clock — if SGLang exposes it for this architecture. Not checked.
• Structural (moves the roofline itself): quantizing the dense parameters to FP8 would take the ceiling from ~33 to ~55 tok/s. That's a requantization project, not a flag. Note that unsloth's GGUF conversions do quantize those parts, which is how UD-Q4_K_XL fits in 111.3 GB — at a measurable fidelity cost, and on a stack without prefix caching or MTP.

What I have not measured

• Clean cold prefill throughput. See the methodology note above: the numbers exist but are contaminated by prefix reuse, and the uncontaminated re-run is the thing that took the box down.
• Why the box could not reclaim. The stability failure is reproducible and the mitigation works, but the mechanism is unexplained and my first explanation for it was wrong. See above.
• A clean quality A/B. The n=200 GSM8K run is on the final config; there is no matched n=200 run without MTP, so sample size and configuration changed together between the two quality runs.
• Anything beyond GSM8K. One benchmark, arithmetic-flavoured. The checkpoint's own card also publishes AIME26 (pass@1 98.75%), which would be a much stronger check and has not been run here.
• Contexts beyond 32k, and concurrency beyond 4 in-flight requests.
• The other quantization path. unsloth publishes GGUF conversions of this model where UD-Q4_K_XL (111.3 GB) fits a Spark natively, no patches — at 93.5% of full-precision fidelity by their own metric. Untested here. Their docs independently reach the same conclusion about the PLE table, which is worth reading: "can be offloaded to SSD via mmap".

Layout

patches/ple_mmap.py           patch 1 — applied to models/qwen4_exp.py
patches/qsa_sm121_kda.py      patch 2 — applied to layers/attention/qwen_sparse_attn_backend.py (sglang#36845, merged revision)
patches/kda_kernels/          the merged KDA kernel package, vendored verbatim, mounted by serve.sh
patches/qsa_sm121_varlen.py   the 2026-08-28 Triton kernel, kept as the out-of-contract fallback
patches/qsa_sm121_triton.py   the 2026-08-28 patcher, superseded by qsa_sm121_kda.py
patches/qsa_trtllm_sm120.py   DEPRECATED — the first patch 2; corrupts long-context decode on GB10
patches/qsa_ring_width.py     EXPERIMENTAL — widens the QSA pending ring past 4 draft tokens
scripts/download.sh           fetch the NVFP4 checkpoint
scripts/prepare.sh            extract from the image, patch, verify by AST
scripts/serve.sh              launch
verify.py                     content -> thinking control -> GSM8K
bench/test_mmap_gather.py     can a Triton kernel read an mmap'd file on this hardware?
bench/test_mmap_write.py      is the write path bit-exact and the fp8 dequant right?
bench/test_qsa_kernels.py     the two QSA decode paths, in isolation (needs a free GPU)
bench/test_long_prefill.py    prefill timing + needle retrieval at long context
bench/test_decode.py          reproducible decode benchmark (n samples, medians)
bench/test_concurrency.py     concurrency scaling -- is decode bandwidth-bound or latency-bound?
bench/test_copy_heavy.py      does higher draft acceptance raise throughput? (no)

Page-cache residency, and a measurement I could not trust

0xBakeer make a point about this table that applies here too:

the n-gram table is 320M rows addressed by a 3-gram hash, so a workload almost never touches the same row twice early on — it never warms naturally, even when the table would fit in cache entirely.

Their fix is a tool that warms the table with one sequential read; their A/B measured +6% throughput for it. I wrote the equivalent for this recipe and withdrew it: mincore(2) reported the full 47.7 GiB as resident even after explicitly evicting a range of it, which cannot be squared with the 26 GiB of Cached that /proc/meminfo reports at the same moment. The same tool behaves correctly on a small file I control, so the bug is specific to this mapping and I have not found it. Rather than ship a number I cannot reconcile, there is no residency tooling here yet.

One asymmetry worth flagging while that stays open: this recipe's loader writes the whole table at startup, so every page passes through the cache on the way in — unlike an mmap'd read-only GGUF, which only faults in what it touches. That may mean the table starts warm here for free. It is a plausible story, not a measurement, which is exactly why the tool mattered.

A note on method

The two mmap microbenchmarks were written before touching SGLang, which is why patch 1 worked on the first boot. The attention microbenchmark was written after three nine-minute boots had learned the same thing more slowly.

Kernel microbenchmark first, real boot second. --load-format dummy would be the obvious middle step and it does not work here, for the reason above.

Credits

• Qwen for the model, and for the tech report — which says outright that these tables belong in off-accelerator storage (§2.3.2). Everything here follows from that sentence.
• 0xBakeer for the llama.cpp recipe that lands on the same trick from a different stack, and for measuring what this repo had left open: full 262k context, prefill throughput, major-fault counts, and a table-warming A/B. their warm_table.py is the tool this repo still owes you.
• RadixArk for the NVFP4 checkpoint and, importantly, for publishing a GSM8K number for that exact checkpoint — without a reference score there is no way to tell "the model is like this" from "we broke it".
• SGLang for shipping qwen4_exp support on day zero.
• MiaAI-Lab and AEON-7 for the format — single-Spark recipes documented well enough to reproduce.

MIT.