Qwen3.8-Flash-Next on a single DGX Spark (GB10) — int4 + int8 + fp8 hybrid

Run Qwen3.8-Flash-Next — a ~176B-parameter model (125B main + 51B n-gram, 6B active) — on one NVIDIA DGX Spark / ASUS GX10 with vLLM: ~50–60 tok/s single-stream decode with MTP=3 (~2,100 tok/s prefill), working prefix caching, and a never-evict pin that keeps your system prompt's KV resident through arbitrary traffic.

Forked from blazux/qwen3.8-Flash-DGX, which established the foundation this fork stands on: the ~49 GiB fp8 n-gram ("PLE") table is a pure lookup that a token only touches 16 rows of, so it is served from NVMe via mmap instead of living in the 128 GB unified pool (full story: docs/HOW-IT-WORKS.md). Upstream serves the official NVFP4 checkpoint at 25–28 tok/s; for that path and its tuning guide, use upstream. This fork replaces the checkpoint with Intel's W4A16 AutoRound int4 plus an int8 GPTQ lm_head and blockwise-fp8 side layers (all prepared by CPU-only tools in tools/), an fp8 PLE table, and a set of GB10/vLLM patches — roughly 1.8× faster decode than the NVFP4 recipe on the same box.

Upstream's NVFP4 recipe independently reproduced on a DGX Spark by @jschmied — see blazux#1 and their write-up, which also contributed the concurrency findings below.

Throughput and concurrency

Measured on this stack with the default serve.sh config (DGX Spark, default GPU clocks; int4 MTP draft experts, DRAFT_VOCAB=1, MTP=3, prefix caching on, SEQS=16, 8192-token prefill chunks). One caveat: on this box the PLE table is served from another machine's RAM over RDMA (the magi branch), the fastest row source there is — with the table on a regular local NVMe, as in the Quickstart, expect decode about 2–3% slower than the numbers below. Single-stream decode by workload, thinking enabled — reproduce with bench_qwen35.sh (from albond's 122B recipe) pointed at your endpoint, on a freshly started container:

Note: arithmetic mean ± sample standard deviation over 4 runs (2 scripts × 2 runs, 2026-09-09). Draft acceptance with thinking on is ~70% (2.1 of 3); with enable_thinking: false it is ~87% and decode runs 5–10% faster than the table.

Under concurrency, TTFT grows with MTP's prefill cost (see the next section), not with the paged table. On upstream's NVFP4 path @jschmied measured aggregate throughput scaling to ~267 tok/s at 48 streams (load-and-waits.md); two portable takeaways: per-token page-fault cost falls with concurrency (batched tokens share n-gram rows), and a low --max-num-seqs silently queues requests — check vllm:request_queue_time_seconds_sum before quoting an aggregate number.

Requirements

• An NVIDIA DGX Spark or compatible GB10 (sm_121) box, 128 GB unified memory, aarch64, recent NVIDIA driver, Docker with the NVIDIA container runtime.
• ~130 GB free disk for the checkpoint + fp8 PLE table, on reasonably fast storage (the table is read at runtime — NVMe strongly recommended).
• The base image is multi-arch, so docker build also works on x86 Blackwell (sm_120) for testing, though this is tuned for the Spark.

Quickstart

git clone https://github.com/Saren-Arterius/qwen3.8-Flash-DGX-AutoRound.git
cd qwen3.8-Flash-DGX-AutoRound

docker build -t qwen38-flash-dgx .   # official image + this fork's patches

# The prepared checkpoint + PLE table (one-time, ~116 GiB):
hf download Saren/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid-MTP_int4RTN --local-dir /models/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid-MTP_int4RTN
hf download Saren/Qwen3.8-Flash-Next-ple-table-fp8 --local-dir /models/ple-table-fp8
# option: the same checkpoint with the MTP draft experts left in bf16 (+3.5 GiB, ~3% slower decode):
#   hf download Saren/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid --local-dir /models/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid
# (or build them yourself from Intel's release: ./prepare.sh — see below)

# Point serve.sh at your checkpoint + table dirs, then:
./serve.sh                           # boots on :8000 (~5 min with fastsafetensors)
docker logs -f qwen38-flash          # wait for "Application startup complete"

Then hit the OpenAI-compatible API:

curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen",
  "messages": [{"role":"user","content":"Write a haiku about a desktop supercomputer."}],
  "max_tokens": 512
}'

serve.sh is a thin example config over scripts/serve-intel-ar.sh — every knob is an env var. The defaults below are serve.sh's (the recommended entry point); where the bare scripts/serve-intel-ar.sh falls back to something else, the note says so. Keep your machine's real settings as an edited copy or a local-only commit on top.

Modify the weights yourself

The quickstart's two hf download repos are the finished artifacts — hashes verified against the local originals. If you'd rather build (or audit) them yourself from Intel/Qwen3.8-Flash-Next-W4A16-AutoRound, one script runs the whole pipeline (CPU-only — a NAS box is fine):

./prepare.sh /models/Qwen3.8-Flash-Next-W4A16-AutoRound /models/ple-table-fp8

Each step is explained in prepare.sh's header comments: int8 lm_head repack, fp8 side-layer conversion, n-gram index strip, fp8 table fetch, the quantization_config rewrite, and the int4 MTP-draft-experts variant. On that last one: this vLLM build has no auto-round loader, but its GPTQ config (AutoGPTQConfig → Marlin kernels) reads the same packed tensors — the GPTQModel-style dynamic rules exclude the families that are not int4-packed and flip the head to 8-bit. The original AutoRound config is kept as config.json.autoround.

Serving

docker build -t qwen38-flash-dgx .
MODEL_DIR=/models/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid-MTP_int4RTN \
TABLE_DIR=/models/ple-table-fp8 \
PREFIX_CACHE=1 PIN_PROMPT="You are HomeBot, the household assistant." \
scripts/serve-intel-ar.sh

or edit the paths in serve.sh (the example config used above) and run it.

Limitations & notes

• One big model at a time — and on a Spark the OS and GPU share the pool; prefer the deterministic GPU_MEM=0.01 + KV_BYTES sizing over a large fraction (an OOM inside the unified pool can freeze the box).
• 1M context is out of reach on one box: the QSA layers refuse an fp8 KV cache, and in bf16 a single 1M request needs ~30 GiB of KV. 500k with YaRN was upstream's validated ceiling.
• Weights are not included and the checkpoint carries Qwen's license (with a MAU/revenue clause) — review it before production use.

What runs in what precision

The patches

Everything is applied at image build time (see the Dockerfile); each patch is independent and gated by an env var where it changes behavior.

1. PLE mmap upgrades (src/vllm_ple_mmap.py, extends upstream's patch)

• Any table dtype: bf16/f16 tables and fp8 (with weight_scale) are all accepted; row size is derived from the safetensors headers. The fp8 table halves the bytes read per token vs bf16.
• VLLM_PLE_MMAP_DIR: the table no longer has to live inside the checkpoint dir — point it at any directory of safetensors shards (NFS, local NVMe, a RAM-backed device...). The backend matters: the ~49 GiB table outgrows what the page cache can keep warm next to the model, so gathers cost ~1.3 ms/op from a RAM-backed source vs 5–9 ms from local NVMe vs 30–50 ms over NFS — decode impact in the appendix below.
• Hot path: per-step dedup of row ids on CPU (np.unique), gather of unique rows only, staging through a persistent pinned buffer with an async H2D copy, and GPU-side expansion via the inverse index. A decode fast path (VLLM_PLE_MMAP_FAST_ROWS, default 512) skips the thread pool entirely for small gathers. Net effect: ~7.2 → ~2.5–3.8 ms per lookup op on a RAM-backed table.
• VLLM_PLE_MMAP_MADV_RANDOM=1: madvise(MADV_RANDOM) the mmap so faults stay single-page — for tables on remote RAM or boxes with no page-cache headroom.
• Stats: VLLM_PLE_MMAP_STATS_SEC (default 30) logs PLE mmap stats (last Ns): calls, op ms, gather ms, rows, MB and resets the counters each period.

2. FLA shared-memory gate (Dockerfile sed)

sm_121 reports 99 KiB of shared memory per block — the same as ADA, where the flash-linear-attention Triton kernels use their big GDN tiles — but the gate in vllm/third_party/flash_linear_attention/ops/utils.py demands 100 KiB, so all 36 GDN layers silently fell back to small tiles. Lowering the gate to 99 KiB (101376) lets GB10 take the big-tile path. (Found the hard way in the Qwen3.5-122B Spark recipe — ported from Entrpi/qwen3.5-122B-A10B-on-spark's patch_fla_shmem.py.)

3. int8 lm_head enablement (Dockerfile sed on model.py / mtp.py)

Upstream constructs ParallelLMHead without quant_config, forcing a bf16 head (1.27 GiB, and a bf16 GEMV per token over a 248320 vocab). One added kwarg in both the main model and the MTP draft lets the head pick up the checkpoint's int8 GPTQ packing. Without the mtp.py half, MTP ≥ 3 crashes at load ("no module or parameter named 'lm_head.qweight'").

4. int4+fp8 hybrid dispatch (src/vllm_fp8_hybrid.py, VLLM_FP8_HYBRID=1)

vLLM's GPTQ config quantizes listed layers and leaves the rest to UnquantizedLinearMethod — it has no notion of "this excluded layer is actually fp8 in the checkpoint". This shim wraps AutoGPTQConfig: it scans the checkpoint metadata for F8_E4M3 weights with a weight_scale_inv sibling and routes exactly those layers to vLLM's blockwise-Fp8Config (weight_block_size=[128,128], dynamic activation scheme) while everything else keeps the GPTQ path. VLLM_USE_DEEP_GEMM=0 is required on sm_121 (DeepGEMM hits CUDA_ERROR_LAUNCH_FAILED); the triton fallback is fine.

5. Prefix caching: on, and fixed (src/patch_mamba_align_split.py)

Upstream runs --no-enable-prefix-caching because of a CUBLAS error in the GDN in_proj GEMM on the cached-block path. With the fp8 side layers that GEMM runs a different kernel, and prefix caching is stable in our serving (PREFIX_CACHE=1).

It also works properly now. On this hybrid model the reconciled cache hit is the minimum across all KV cache groups (full attention + four mamba/GDN state groups), and mamba "align" mode can only cache a state at a prefill chunk end on a 1600-token boundary. The image's scheduler aligned those chunk ends to cache_config.block_size — which the engine rewrites to the minimum group block size (8, the MTP draft granularity) — so chunks ended where no mamba state was cacheable, a cold request published zero reusable mamba states, and a repeated prompt only got fast on the 3rd try (the miss triggers junction machinery that rebuilds the boundary one request late). Long prompts effectively never hit. The patch makes the split use cache_config.mamba_block_size (1600). Verified: an 8k-token repeat goes 10.1 s → 0.90 s on the 2nd request.

The same rewritten block_size also poisoned the worker side: the align-mode state-slot seed (mamba_hybrid.py) divided by it too, so a prefix hit at 6400 tokens seeded state column 799 instead of 3, read past the block-table row and restored a wrong (often all-zero / stale) mamba state — greedy outputs visibly changed on cache hits. Root-caused upstream by blazux (his fix: 8347e7c); the same one-line seed fix is folded into this patch. Verified: cold-vs-hit first-token logprobs now agree within the stack's normal run-to-run jitter (Marlin atomic-add nondeterminism), where before the fix greedy outputs diverged within the first few tokens.

Notes: the prefix-cache granularity is large (1600 tokens; shorter prefixes get no reuse), and a repeat hit tops out at round_down(P,1600) − 1600 — MTP (eagle-style) always recomputes the last matched block.

6. Never-evict prompt pinning (src/patch_never_evict.py)

--never-evict-kv-cache-prompt-includes "<substring of your system prompt>" pins the KV blocks of any prompt containing that marker: they are held in a side queue on BlockPool, excluded from the free count, and thus never handed out for eviction — your assistant's system prompt stays cached no matter what other traffic does. --never-evict-kv-cache-max-fraction (default 0.25) caps the pin. The pin set is replaced on each matching request, so an updated system prompt releases the old blocks automatically.

Verified end-to-end: a pinned 8k prompt still answers in 0.94 s after 2M tokens of unique traffic (3.1× full KV-pool turnover).

Implementation notes: the marker is tokenized once and matched as a token-id subsequence (first/last token dropped — BPE merges at the boundaries); the pin is keyed on block hashes, not block ids, because this hybrid model frees mamba/GDN state blocks mid-request — each freed block is re-claimed by the pin the moment free_blocks() sees it. Only prefix-cacheable KV groups participate (the MTP draft layer's group is not, and must be skipped). This is a pin-only port of our arc_pin2 patch from the Qwen3.5-122B Spark stack, which built the pin on top of the ARC GPU-eviction work in vllm#40270; the ARC/2Q policies themselves were deliberately dropped — the stock free queue is C-speed on a path this model hits every step.

Self-check (no GPU): docker run --rm -v "$PWD/src:/t" --entrypoint python3 qwen38-flash-dgx /t/test_never_evict_pin.py.

7. Mamba state-copy guard (src/mamba_utils_guarded.py)

Hardens the align-mode state-copy kernels against the "CUDA illegal memory access / Xid 31 under load" crash class (also blazux#2): backports vllm#50729 (overlapping state-copy race) and bounds-checks every block id against its state pool before dereferencing — an out-of-range id skips the copy and bumps a counter (logged as mamba state-copy guard) instead of taking down the CUDA context.

With the block-size seed fix (patch 5) the out-of-range ids the guard was absorbing are gone at the root: the counter is expected to stay at 0, and a nonzero count is logged as an error — it now indicates a new bug worth reporting, not a known quirk.

8. Prefix-cache tracing (src/patch_hit_debug.py, HIT_DEBUG=1)

Set HIT_DEBUG=1 (→ VLLM_HIT_DEBUG=1 in the container) to log, per request: the per-KV-group hit reconciliation (which group truncated the hit), mamba boundary-state publication (which slots were real/null/hashed), cached-block evictions, and prefill chunk-stop decisions. This is what found the bug in patch 5; costs nothing when off.

9. Deterministic QSA top-k (src/patch_qsa_exact_topk.py + Dockerfile patch 10)

Taken from upstream blazux/qwen3.8-Flash-DGX (8347e7c, 4b723de, 0022e36). The sparse attention (QSA) picks its top-k key blocks with vLLM's persistent_topk kernel, which is non-deterministic on GB10 and can drop real candidates (vllm#51782, diagnosed by @k3dani in blazux#3). Two fixes, both opt-in env vars, DET_TOPK=1 the default:

• DET_TOPK=1 — deterministic kernel by @jschmied (vllm#55122): index-ordered output slots and bit-exact tie resolution (signed zero canonicalised), plus a deterministic low-shared-memory fallback. The sources are fetched at a pinned commit (sha256-checked) and compiled with the image's nvcc at build time as a standalone _C_det.so — no vLLM rebuild; qsadet_patch.py (also @jschmied's) wires the QSA block selection to it. Upstream measured the whole prefill penalty of the exact path gone (GX10, 32k: 1,794 → 2,996 tok/s) with decode unchanged.
• EXACT_TOPK=1 — exact torch.topk over the visible columns (blazux's first fix): also deterministic, −20–40% on long prefill; kept as the fallback and wins over DET_TOPK when set. Self-check (no GPU): docker run --rm -v "$PWD/src:/t" -w /t --entrypoint python3 qwen38-flash-dgx test_qsa_exact_topk_cpu.py.

10. MTP drafter options: int4 draft experts + reduced draft vocabulary

Both on by default (the Quickstart checkpoint + DRAFT_VOCAB=1); the plain -hybrid checkpoint and DRAFT_VOCAB=0 remain options. Both leave outputs unchanged — the target verifies every drafted token, only the draft's cost and acceptance move. Measured on a DGX Spark (bench_qwen35.sh with enable_thinking: false, T=0, second run):

With thinking on (albond's original script) draft acceptance is ~70% in all three configs — only 1.0% of reasoning tokens fall outside the 65k set — and the default config is still the fastest (Code 56.5, JSON 62.1, LongCode 48.5 tok/s, see the Throughput table).

• int4 draft experts — Intel's checkpoint leaves the MTP layer's 512 routed experts in bf16 (~4.7 GiB on the unquantized MoE path). tools/quantize_mtp_experts_int4.py writes a variant checkpoint (hardlinks for the untouched shards) with those experts as int4 g128 RTN in Intel's exact GPTQ layout and drops the layers.48 exclusion, so the draft runs on GPTQ-Marlin like the main layers. Prebuilt: Saren/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid-MTP_int4RTN — point MODEL_DIR at it. Idea from upstream's hybrid-mtp mode (blazux#11 by @pfy, after thavoc's graft write-up), done here as RTN int4 instead of an NVFP4 graft so no new kernel path is needed.

• DRAFT_VOCAB=1 (src/patch_mtp_draft_vocab.py, from upstream blazux 0c6df7e, idea from MiaAI-Lab's recipe reimplemented there) — vLLM shares the target's lm_head with the drafter, so every draft pass scores all 248,320 rows. With the flag the draft scores a private 65,536-row slice (src/draft_vocab_65536.npy: corpus frequency + BPE order + all special tokens; tools/build_draft_vocab.py rebuilds it) and every other id is −inf. This fork's head is int8 GPTQ-Marlin, so the slice is dequantized from the checkpoint's GPTQ tensors at first use (616 → 320 MiB read per draft step).

• DRAFT_HEAD=int4 (same patch) — the other way to halve the head read without touching the vocabulary: a private int4 g128 RTN copy of the full head, quantized with vLLM's own marlin_quantize at first use (~23 s) from the checkpoint's int8 GPTQ tensors, run through the same GPTQ-Marlin kernel (616 → 313 MiB per draft step; +313 MB resident). Outputs cannot change (the target verifies every token), only acceptance can, and it does: int4 RTN puts ~11% relative error on the head (int8: ~0.5%). Measured 2026-09-10, thinking off, 600-token answers, second run of each prompt:

  	int8 shared head (default)	DRAFT_HEAD=int4
  Traditional Chinese prose	34.6 tok/s, 31% accept	35.5 tok/s, 30%
  Cantonese prose	32.8 tok/s, 28%	32.3 tok/s, 25%
  English prose	40.4 tok/s, 44%	39.6 tok/s, 39%
  Python + tests	62.5 tok/s, 86%	60.7 tok/s, 78%

  The bandwidth saved is handed back in rejected drafts, so it stays off. A calibrated int4 (AutoRound/GPTQ on real MTP hidden states) might keep more acceptance; not tried. Context: krisbailey.com's "shortlist MTP" write-up found the same ~4% ceiling for head-shrinking tricks once the head kernel is bandwidth-bound, which Marlin already is here.

Speculative decoding and TTFT

MTP raises decode substantially but puts a floor (~0.8 s) under time-to-first-token: vLLM's v1 engine only emits the first token after the drafter has run, and MTP's draft layer is a stateful autoregressive transformer — on every prefill chunk it must run a full-chunk-width forward (always eager: above the cudagraph capture sizes) to sync its own KV/GDN state, plus k−1 sequential single-token passes. Cross-attention drafters like DFlash don't pay this, but Flash-Next has no such drafter — MTP is what ships in the checkpoint. If your workload is TTFT-sensitive, weigh MTP depth against MTP=0; only 0 removes the floor.

What's in here

Dockerfile                    official vLLM Flash-Next image + the patches above
serve.sh                      example launcher config (edit paths, run)
prepare.sh                    build the checkpoint + table from Intel's release
src/vllm_ple_mmap.py          mmap PLE table (any dtype, relocatable dir, fast gather)
src/vllm_fp8_hybrid.py        int4+fp8 hybrid dispatch on the GPTQ config
src/patch_never_evict.py      never-evict system-prompt KV pinning
src/patch_mamba_align_split.py  prefix-cache chunk-alignment fix
src/patch_hit_debug.py        prefix-cache tracing (VLLM_HIT_DEBUG)
src/patch_qsa_exact_topk.py   exact, deterministic QSA top-k (VLLM_QSA_EXACT_TOPK=1; from blazux)
(Dockerfile patch 10)         @jschmied's deterministic persistent_topk kernel, built at docker build
src/test_qsa_exact_topk_cpu.py  CPU unit test for the exact top-k (no GPU needed)
src/patch_mtp_draft_vocab.py  private MTP draft head: reduced vocabulary (from blazux) / int4 (VLLM_MTP_DRAFT_VOCAB, VLLM_MTP_DRAFT_HEAD)
src/draft_vocab_65536.npy     the default 65,536-token draft id set (from blazux)
tools/build_draft_vocab.py    rebuild the draft id set (from blazux)
tools/quantize_mtp_experts_int4.py  int4 RTN the MTP draft experts -> -MTP_int4RTN checkpoint
src/mamba_utils_guarded.py    hardened align-mode state copy (vllm#50729 + guard)
src/test_ple_mmap_cpu.py      CPU unit test for the gather (no GPU needed)
src/test_never_evict_pin.py   CPU unit test for the pin (no GPU needed)
scripts/serve-intel-ar.sh     the docker run behind serve.sh
scripts/smoke-test.sh         health + coherence + prefill/decode numbers
src/patch_prefill_metrics.py  per-step prefill counters for Prometheus (ITER_DETAILS=1)
bench/ppwatch.sh              live prefill tok/s from those counters
bench/decode_bench.py         batch-1 decode / TTFT / spec-acceptance bench
tools/                        CPU-only checkpoint preparation
docs/HOW-IT-WORKS.md          upstream's mmap-PLE story
docs/OPTIMIZATIONS.md         this fork's patches in depth

Credits

• Model: Qwen team, Alibaba — Qwen3.8-Flash-Next.
• This is a fork of blazux/qwen3.8-Flash-DGX — the original mmap-PLE idea, the GB10 serving recipe, the NVFP4 path, and docs/HOW-IT-WORKS.md are theirs.
• int4 checkpoint this fork builds on: Intel/Qwen3.8-Flash-Next-W4A16-AutoRound (AutoRound); fp8 PLE table from Qwen/Qwen3.8-Flash-Next-FP8.
• Several pieces originate in the Qwen3.5-122B-A10B Spark recipes: the int4 AutoRound + int8 lm_head serving approach from albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4; the FLA shared-memory gate fix and the int4+fp8 hybrid idea from Entrpi/qwen3.5-122B-A10B-on-spark. The never-evict pin was built for that 122B stack on top of the ARC GPU-eviction work in vllm#40270 and re-ported here.
• Serving engine and base image: vLLM (vllm/vllm-openai:qwen38-flash-next, the release/qwen38next recipe / PR #53896).
• MTP drafter options: the reduced draft vocabulary (patch, id set, build tool) from upstream blazux/qwen3.8-Flash-DGX (0c6df7e; idea from MiaAI-Lab's recipe, reimplemented there); the quantized draft experts follow upstream's hybrid-mtp mode (blazux#11 by @pfy).
• Deterministic QSA top-k: the exact torch.topk fix and the kernel wiring/pins from upstream blazux/qwen3.8-Flash-DGX; the deterministic persistent_topk kernel itself by @jschmied (vllm#55122); the bug diagnosed by @k3dani (blazux#3).
• Independent reproduction of the upstream recipe, the native-offload fixes and the concurrency measurements: @jschmied (blazux#1, qwen38-flash-next-gb10).
• License: Apache-2.0.

Appendix: where to put the PLE table (and why not to bother with PLE_PREFETCH)

Batch-1 decode bench (bench/decode_bench.py, medians of 3; W1 = fresh-prompt 1000-token decode, W2 = pinned ~8k prefix hit + 256 tokens; DGX Spark GB10, MTP=3). Only the table location and the prefetch flag change between rows (all local NVMe is Gen 4):

Takeaways:

• Put the table on local NVMe. It costs ~15% decode vs an exotic fast row source, and it is the simple recipe. Serving straight off a NAS works but costs ~40% (and that was NFS-over-RDMA to a btrfs box — a plain GbE NAS will be worse).
• Don't bother with PLE_PREFETCH=1 (VLLM_PLE_MMAP_PREFETCH). The batch-assembly prefetch only wins where the row source is very slow (NFS, +2–3 tok/s, within run-to-run variance), is a wash on fast sources, and is net-negative on local NVMe: the handoff/wait in consume() costs more than the inline gather it replaces (confirmed on a warm page cache). It stays experimental and default-off.
• TTFT is unaffected by any of this (the MTP drafter floor dominates), and between-restart variance on this bench is >10% — treat small deltas above accordingly.

About those "RDMA row daemon" rows: that is a custom one-sided RDMA READ server that pins the whole table in another box's RAM — the fastest row source measured, at the price of a second machine and an ibverbs science project. If you have a NAS with ≥64 GB of RAM and a ≥100 Gbit RDMA link to your Spark — and no second DGX Spark to put to better use — the magi branch ships the tool (src/ple_rdma/) and setup notes ("PLE table over RDMA"). Everyone else: local NVMe is the recipe.