Qwen3.8-Flash-Next on 2× CMP 170HX

Qwen3.8-Flash-Next (W4A16) on two NVIDIA CMP 170HX cards, pipeline-parallel with vLLM: 226 tok/s for one user (code), 840 tok/s across eight, 6,300–7,100 tok/s prefill from 4k to 64k prompts, the full 262,144-token window, and HumanEval 157/164, GSM8K 1,271/1,319. Measured on PCIe Gen2 x4 links.

Serves Intel/Qwen3.8-Flash-Next-W4A16-AutoRound with a vLLM v0.30.0 fork in a container: an OpenAI-compatible API with native MTP speculative decoding, reasoning and tool calls. The ~102 GB N-gram (PLE) table stays on the NVMe drive and is read through an mmap; everything else is on the two GPUs.

git clone https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next
cd cmp170hx-qwen38-flash-next
./download.sh     # checkpoint at the pinned revision (~170 GB) + the MTP draft head
./start.sh        # tunes the host (sudo), starts the container, waits for /health

The API listens on 127.0.0.1:8000 as model qwen3.8-flash-next-w4a16. ./stop.sh stops it. Settings go in .env (see .env.example).

Requirements

• 2× NVIDIA CMP 170HX, each exposing 64 GiB. No P2P or link modification needed: pipeline parallelism sends one activation per step between the cards.
• Linux, NVIDIA driver 580 or newer, Docker with the NVIDIA Container Toolkit.
• ~180 GB free on an NVMe SSD for the checkpoint. The PLE table is read from it during every forward pass, so keep other heavy writers off that drive.
• Enough RAM for a useful page cache over the PLE table (measured with 93 GB).
• sudo for host-tune.sh (or HOST_TUNE=0; see below).

Benchmarks

Measured 2026-10-17 with the files in this repository (image ghcr.io/alexbrandes/cmp170hx-qwen38-flash-next@sha256:d6fc872d943b3e9e5cc9eb583b51d72af200e039fbbb7401700bb38d665f0957, started by start.sh), using the sparkDash benchmark protocol at commit 3e6d4ef: greedy, thinking off, output forced to 400 tokens, sent to the engine endpoint. Each cell is the median of three runs, with the range in brackets; sparkDash itself makes one pass per level.

Decode, code

Decode, json

Decode, prose

Decode, structured

Prefill (one cold request)

Decode at depth (our addition, not part of sparkDash: one request, the prose prompt after filler, 400 tokens)

Quality

Same container and session, greedy, thinking off.

• GSM8K copies vLLM's tests/evals/gsm8k/gsm8k_eval.py in chat mode: 5-shot, 1,024 tokens, last integer in the reply.
• HumanEval uses EvalPlus's instruct prompt and OpenAI's check_correctness program (3 s limit).
• Chat mode only. This is a chat-only model: on a raw /v1/completions prompt it ends its turn at once (<|im_end|>, p ≈ 0.99), so completion-mode scores (GSM8K 22, HumanEval 134) measure the prompt format, not the model.
• For scale: Intel's model card reports GSM8K 0.9682 for this checkpoint (0.9673 for BF16), measured with its own harness.

What runs

The fork's commits, oldest first:

1. Pipeline parallelism and an NVMe-mmapped PLE table. Builds on the community CMP 170HX work credited in NOTICE.
2. Mamba state under pipeline parallelism. Deferred recurrent-state copies use request-slot block tables; batch rows had changed owners by then.
3. W4A16 draft-only lm_head for MTP.
4. CUDA-graph-safe PLE lookup and host-side N-gram ids. Includes the stale-tail fix that made graph-mode outputs depend on request history.
5. Bounded prefill beside running decodes. A new long prompt keeps advancing while other requests decode.
6. Fused QSA prepare. A backport of vllm-project/vllm#57097.

Host settings (host-tune.sh)

start.sh runs it through sudo; HOST_TUNE=0 skips it.

• 170 W power limit on both cards.
• 16 KB readahead on the model's drive. PLE rows are small random reads.
• Drop the page cache and compact memory before start. The engine's pinned host buffers then land on contiguous pages; without it decode often starts in a mode about 10% slower. It empties the whole page cache, so other workloads on the host lose theirs, and the weights load from disk again.

Reasoning

The checkpoint's chat template is used unmodified. Requests that enable thinking get the template's default effort, xhigh; send reasoning_effort (low, medium, xhigh) to change it. All numbers above have thinking off.

Caveats

• One host: 2× CMP 170HX at PCIe Gen2 x4, Intel Core Ultra 5 225, 93 GB RAM, a consumer QLC NVMe (Kingston NV3 2 TB), Ubuntu 24.04, driver 610.43.02. Other hosts have not been measured.
• The PLE table is read from the NVMe while serving. Another heavy writer on the same drive stalls it: on our host a KV-cache disk tier on that drive cut long-prompt prefill by two thirds.
• Greedy decoding at 8 requests in flight with MTP is not bitwise repeatable; HumanEval scored 157 in three sessions with the failing set varying by a task.
• The decode-at-depth table and the three-run medians are additions to the sparkDash protocol; everything else follows it.

Licence

MIT for this repository (LICENSE). The engine is Apache-2.0, the model has its own licence; see NOTICE for every component and credit.