Qwen3.8 Flash-Next · GB10 UltraFast

A vLLM serving recipe for Qwen3.8-Flash-Next on a single NVIDIA DGX Spark or other GB10 system.

The promoted configuration, v19a-w5 (MTP=4 window-5 on the deterministic floor), decodes a copy-heavy workload at up to 76.8 tok/s on one stream and 265.9 tok/s aggregate across eight concurrent streams — +9.2% (+6.4 tok/s) over the same floor at MTP=3, measured head-to-head on the same pinned image with two boots per arm (medians and per-boot spreads in docs/benchmarks). The speed still comes from MTP speculative decoding with block rejection, not lower-bit target weights.

What this fork adds on top of the upstream recipes:

• A verified-deterministic floor: the MoE moe_align_block_size arrival-order shim plus the drafter-Marlin atomicAdd epilogue fix and the pinned triton config, with runtime behavioral fingerprints (AWFP) and a 37/37 kernel-level stochastic-correctness gate over the real GPU block verifier.
• A fixed suite-position tool-loop failure is recorded, not hidden: copy-agentic-1 fails at rep 2 on three independent W5 boots (attribution unresolved, reproduction queued) — see docs/benchmarks.
• Owner-shaped benchmarks (multi-turn tool loops and real file edits) alongside the copy/prose/code suite.
• +31.3% Chinese prose throughput (34.8 → 45.7 tok/s at equal verified quality — MGSM 16/16, needle tests intact) from a rebalanced MTP draft vocabulary: 39.5% Han tokens in the 65,536-id draft vocab, built from a 48.7M-token mixed corpus. ~+20% sustained on the promoted configuration (41.7 tok/s; the W5 window trades ~3% back for the copy/code gains).

Results

This fork — v19a-w5 (promoted) vs the MTP=3 reference vs the stock baseline, all on this machine, this bench

v19a-w5/v19a-w4: same pinned image (iter7g), same floor (shim + MARLIN epilogue fix + pinned triton config + compile-cache mount), two boots per arm. Stock anchor: v16b exactly as the upstream recipe ships it (iter6d image, stock vocabulary, no floor fixes), booted and measured for this table. Window tok/s medians; per-round JSON in docs/benchmarks.

The promoted configuration beats the stock baseline at every concurrency point (+4.1% to +6.7%); the W4 reference is a wash against stock on the copy workload — its gains are in Chinese prose (below) — which is the adaptive-width case in one table.

Quality on the same boots: code-edit 61.8 vs 59.4 tok/s (25/25 verified both arms), MGSM 16/16 both arms, prose-zhen 41.7 vs 43.2 (the known W5 CJK deep-acceptance trade-off), ctx128k within the noise band. Chinese context: prose-zhen on this floor runs +20% over the stock-vocabulary baseline (34.8 tok/s) — the full +31.3% lands on the W4-style window — with reasoning quality verified (MGSM zh 16/16).

Owner-shaped workloads (first live run on the promoted configuration)

Multi-turn tool loops (10 scenarios × 5) and real file-edit tasks (5 tasks × 5, verified by executing the repo's tests) — the agentic shape the recipe is tuned for: structured tool-argument emission dominates the generated tokens.

Baseline reference (v16b)

v16b on one GB10 with the promoted launch settings. Each throughput figure is the fastest of three rounds on a copy-heavy, low-effort workload with a shared 9,600-token cached prefix, counted as completion tokens per second over the window in which all streams are decoding. Method and per-round data.

Cold prefill, with nothing cached, is 2× to 3.4× faster than the original recipe on the same GB10. Each rate is prompt tokens divided by cold time to first token, three repeats per cell, so it slightly understates pure prefill speed. Method and data.

On the agent-shaped workload, v16b spends 23% less time per decode step than the upstream base, or 1.30× as many steps per second. Benchmark method · Evaluation.

Requirements

• One NVIDIA GB10 with about 121 GB of unified memory, ARM64 Linux, Docker and the NVIDIA container runtime.
• A driver compatible with the pinned CUDA 13.0 image. Check nvidia-smi and a GPU container before building.
• Local storage, preferably NVMe, for the checkpoint, FP8 PLE table, downloads and Docker layers (about 130 GB free for the upstream checkpoint and table), plus about 4.8 GiB for the T80 drafter shard data.
• Python 3, the Hugging Face hf CLI, jq, GNU md5sum and sha256sum.

Quick start

With a coding agent

Run a coding agent that can use a terminal on your everyday computer, and paste this prompt. The agent connects to your GB10 over SSH and does the setup there. Set up key-based SSH login to the GB10 first.

Set up the Qwen3.8 Flash-Next GB10 UltraFast recipe on my GB10 over SSH.
Read AGENTS.md in this repository and follow it exactly.
Ask me how to reach the GB10, run the read-only checks, show me what you found, and get my approval before you install, download, build or launch anything.

The agent first checks the GB10's hardware, memory, disk, Docker and port 8000 without changing anything. It then asks for your approval before each of five steps: getting the repository and tools, downloading about 130 GB of public model files, building the image, building the drafter directory, and launching the server. Long steps run in the background on the GB10, so a dropped connection doesn't interrupt them. The agent never stops or removes anything that was already running. AGENTS.md has the full instructions.

Manual

Download the pinned public checkpoint and PLE table, build the image and the T80 drafter directory, then launch v16b and send a test request. BUILD.md has the reference hashes, build report and what each step checks.

# Clone and install the Hugging Face CLI
git clone https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast.git
cd qwen3.8-flash-next-gb10-ultrafast
python3 -m venv recipe/.venv
. recipe/.venv/bin/activate
pip install --upgrade huggingface_hub

# Download the pinned checkpoint and FP8 PLE table
mkdir -p "$HOME/models/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid" "$HOME/models/ple-table-fp8"
hf download Saren/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid \
  --revision 8b82f0b7abe3d1150a7827d298c75e86267636ae \
  --local-dir "$HOME/models/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid"
hf download Saren/Qwen3.8-Flash-Next-ple-table-fp8 \
  --revision 50511b0a41aa1d34b8beb7e5d4bb06a0b650dc14 \
  --local-dir "$HOME/models/ple-table-fp8"

# Build the base image, the verification-gate image (iter7g), then the
# T80 dense-MTP drafter directory
bash recipe/build/image/build.sh --run
bash recipe/build/image/build-iter7g.sh --run
bash recipe/build/model/build.sh --run

# Show the full docker command, then start the promoted server (runs detached)
bash recipe/config/v19a-w5/launch.sh --print
bash recipe/config/v19a-w5/launch.sh --run

# Wait for the server to load, then send a test request
until curl -fsS http://localhost:8000/v1/models >/dev/null 2>&1; do sleep 10; done
curl -fsS http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen","messages":[{"role":"user","content":"Reply with OK."}],"max_tokens":512}'

Follow startup with docker logs -f qwen38-flash. The server exposes an OpenAI-compatible API on port 8000 under the model name qwen. It accepts text, images (image_url) and video (video_url), since the recipe keeps the model's vision encoder. Audio input is not supported. The T80 builder converts nine drafter modules at group size 32; its reference conversion took 4.626 s after download and wrote 5,118,048,680 new bytes.

Running on your own GB10? If the recipe helped, a ⭐ helps other Spark owners find it, and your numbers are welcome as a benchmark submission.

Configuration

The promoted launch, v19a-w5, pins GPU_MEM=0.01, KV_BYTES=16g, SEQS=8, CTX=262144, MTP depth 4 (the W5 window), block rejection with probabilistic draft sampling, the mixed 39.5%-Han draft vocabulary, and prefix caching — on the deterministic floor: the MoE arrival-order shim, the drafter-Marlin epilogue fix, the pinned triton config and the compile-cache mount (all set in recipe/config/v19a-w5/env). launch.sh sources that env file under a cleared environment, so edit the file rather than exporting variables. Changing any pinned value produces a new variant whose speed and quality must be measured separately; the MTP=3 reference (v19a-w4) and the upstream-stock configuration (v16b) ship alongside it. Configuration reference.

How it works

AutoRound INT4 experts + FP8 side layers + INT8 output head
             │
             ├── FP8 PLE table, read by mmap from storage
             ├── MTP depth 4 (adaptive 3-4), 65,536-id mixed draft vocabulary
             │   (39.5% Han — +31.3% Chinese prose throughput)
             └── fast PLE gather + piecewise CUDA graphs + prefix cache
                         ↓
                vLLM verifier and sampler → response

Target precision. AutoRound W4A16 packs the routed-expert weights to four bits with 16-bit activations. FP8 side layers and an INT8 output head keep the nonexpert paths at eight bits, where 3-bit GGUF and NVFP4 recipes compress target weights further. The FP8 PLE table is memory-mapped from storage rather than held entirely in GPU memory, which leaves room for the 16 GB KV pool.

Drafting. The dense T80 MTP drafter proposes three to four tokens per step (the promoted v19a-w5 runs the W5 window; the verification width adapts per step on the armed configuration). A 65,536-id draft vocabulary — rebalanced to 39.5% Han tokens for the Chinese gains above — and FP8 drafter experts make each proposal cheaper. The target verifies every drafted token with block rejection, so the drafter changes speed, not the output distribution. Unlike n-gram copy speculation, which needs a matching earlier span, the MTP head drafts from the model itself and also covers original code, reasoning and tool output.

Per-step overhead. A low-latency SM12x GEMM, a sort-free verify-path top-k and piecewise CUDA graphs trim compute, selection and launch overhead around each target pass. The image runs vLLM's V2 model runner, where async scheduling is on by default with MTP, and adds an asynchronous short-convolution host-to-device transfer, guarded recurrent-state alignment and retained draft prefix-cache blocks.

Architecture and build provenance.

Benchmarks and evaluation

• Stream curve. Three rounds at each of 1 to 8 streams on the copy-heavy workload, reported as peak and median, with tokens per step and time to first token. The five-stream point was measured in two sessions. Runner · raw per-round records in docs/benchmarks.
• Agent-shaped decode. Cached long system prefix, tool schemas, six coding tasks, three repeats each, 1,500 generated tokens, thinking on. Step-time comparison.
• Quality. A fixed 492-item suite (code with execution checks, math, knowledge, instruction following, tool calls, long-context needles), a 24-item long-generation arm and a teacher-forced prefill comparison. Evaluation.

Repository layout

Acknowledgements

This project's design took shape against Restech Technology Services, UNSW's self-hosted high-performance inference infrastructure, and this project owes that service a direct debt: the GB10 hardware and the compute access that made the work possible at all came from there. Qwen3.8-flash-next-gb10-ultrafast is an independent project and is not affiliated with UNSW; the acknowledgement is where the lineage ends.

Credits and licenses

The Qwen model authors and the vLLM, Intel AutoRound and FlashInfer contributors. Model weights and downloaded datasets keep their own terms.

Everything in this repository, code, docs, data and images, is licensed under Apache-2.0. You can use, modify and redistribute it, including commercially. See LICENSE.

GitHub's "Cite this repository" menu reads CITATION.cff.

Draft vocabulary note

The promoted configuration ships a mixed 65,536-id draft vocabulary — 25,862 Han entries (39.5%) built from a cleaned 48.7M-token corpus (Chinese Wikipedia, Chinese technical docs, code) — which is what delivers the +31.3% Chinese prose throughput. The upstream English/code-weighted set ships alongside it (draft-vocab-ids-K65536.txt in the stock config) for pure-English sessions. Output quality is unaffected either way, because the target model verifies every drafted token with block rejection; the vocabulary changes which tokens get drafted, not what the target can say. See Configuration for details.