RTX5090 Blackwell GPU Optimized Qwen-3.8-27B-NVFP4

Runs on SparkInfer, SGLang and vLLM unmodified — configs for all three are below. Serving many users at once? See concurrency.

420 tok/s writing code. One consumer GPU.

SparkInfer × this NVFP4 build × the DSpark v2 drafter — an engine, a checkpoint, and a speculative drafter optimized against each other, compounding to 4.3×. The drafter never changes what the model says: the target verifies every drafted token.

GeForce RTX 5090–specific NVFP4 checkpoint of Qwen/Qwen3.8-27B, quantized with NVIDIA Model Optimizer. Serves the full native 262,144-token context on 32 GB.

With the DSpark v2 drafter: 264.8 tok/s overall — up to 420 on code — on SparkInfer (its bench harness; the HTTP server is autoregressive-only today) and 161.7 tok/s on SGLang's OpenAI server. Without speculation: 92.9 tok/s SparkInfer · 85.8 SGLang.

This is the final build. Everything is quantized down to the lm_head, the unused MTP head is removed from the weights, and the file ships as two shards:

Blackwell tensor cores only. Hopper can load the files but cannot run NVFP4.

---

Quick start — one docker run

SparkInfer is a zero-dependency C++/CUDA engine for Blackwell. The image is prebuilt and published to GHCR — no compiler, no CUDA toolkit, no Python stack:

docker run --gpus all -p 8080:8080 -v qwen38:/models \
  ghcr.io/gittensor-ai-lab/sparkinfer-qwen38:latest

These weights download themselves on first run into the qwen38 volume (~18 GB, cached after), so that one command is the whole setup. The server is ready in about 10 s on later starts.

curl localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen38-nvfp4",
  "messages": [{"role": "user", "content": "What is the capital of Japan?"}],
  "chat_template_kwargs": {"enable_thinking": false}
}'

Serves text, images and video through the same OpenAI-compatible API. ~1 GB image, Blackwell (sm_120) only. One caveat if you wire this into an agent framework: /v1/models currently advertises input_modalities as text and image only, so a client that routes on that field will not offer video even though video_url parts work — send video explicitly for now (sparkinfer#983). Provenance is attested to the image digest:

gh attestation verify oci://ghcr.io/gittensor-ai-lab/sparkinfer-qwen38:latest \
  -R gittensor-ai-lab/sparkinfer

Tuning

Defaults are the values measured as correct on a 32 GB card, so the two easy mistakes are pre-solved. Override any of them with -e:

The output cap reserves context per request, so raising it without also raising CTX makes concurrent requests fail with server overloaded. The NVFP4 lm_head loads natively — nothing needs converting.

Speculative decode benchmark

The same image runs the DSpark benchmark, fetching the drafter on first use:

docker run --rm --gpus all --ipc=host -v qwen38:/models \
  ghcr.io/gittensor-ai-lab/sparkinfer-qwen38:latest bench code

Building from source instead

git clone https://github.com/gittensor-ai-lab/sparkinfer && cd sparkinfer
cmake -B build -DCMAKE_CUDA_ARCHITECTURES=120 -DBUILD_SERVER=ON && cmake --build build -j
./build/server/sparkinfer_server -m <weights> --tokenizer <weights>/tokenizer.json \
  --ctx 262144 --host 0.0.0.0 --port 8080

Needs CUDA 12.8+ and rustc ≥ 1.79.

---

Performance

Engines, same weights, identical prompts

24 prompts across four workloads (chat, code, math, JSON), 256 max tokens, temperature=0, thinking off, sequential, through each engine's OpenAI API, every row measured at --ctx 262144. SGLang 0.5.18 with FP8 KV; SparkInfer v0.5.5 with its default int8 KV.

SparkInfer is both the faster engine without speculation and the one that holds the most context: 262,144 tokens costs it 27.9 GB, and it still answers at --ctx 360000 (31.2 GB), though decode falls to ~74 tok/s at that extreme.

Serving more than one user

Every figure above is a single stream. Under concurrent load SparkInfer v0.5.5 decodes the whole batch in one packed forward — earlier versions time-sliced one sequence at a time and stayed flat at ~77 tok/s no matter how many clients arrived.

Distinct prompts per request so no cache flatters the result, 128 tokens each, --ctx 40960, measured client-side over the streaming API. 32/32 requests succeeded at every level. Aggregate throughput peaks around 8–16 concurrent and is flat to 32; per-request latency degrades from there, so 8 is the sweet spot on one card.

Where this loses. vLLM 0.28 on the same weights, same box, same harness: 134.5 / 266.6 / 545.8 / 546.6 tok/s at 2 / 4 / 8 / 16. We lead at 2 concurrent and trail by about 1.6× at 8–16. If you are serving many simultaneous users, vLLM is still the faster server; if you are serving one stream, or a few, SparkInfer is the faster engine and holds far more context.

Repeated long prefixes: ~19×

A configured prefix — a long system prompt, a document, a tool manifest — is kept across requests instead of being re-prefilled. With a 32,022-token shared prefix:

Change one word of the prefix and the next request costs 3.30 s again, which is what makes the speedup attributable to the cache rather than to warm-up. For "load a document, ask ten questions", or agentic traffic that repeats a system prompt and tool definitions on every call, this outweighs any decode difference on this page.

This is a configured prefix (SPARKINFER_SERVER_PREFIX_TOKEN_FILE, a file of token ids), not automatic caching over whatever prefix your requests happen to share, and it engages on sequential requests rather than under concurrency.

SparkInfer + DSpark: 2.73× in the bench harness

SparkInfer's DSpark path exists in its bench harness, not yet in the HTTP server — so it is not something you can curl today, but it does show what the engine and this drafter do together. Measured on the same checkpoint and the same DSpark v2 drafter, 256 tokens per workload, greedy:

Against SGLang's 1.89× on the same drafter, this is the larger speculation gain — the block drafter feeds a verifier that keeps up with it. Code accepts nearly the whole 8-token block (τ 6.78 of a possible 8).

Read these as their own harness, not as server numbers. The bench runs at a few hundred tokens of context, where its autoregressive baseline is 97.1 tok/s against the server's 92.9, and it uses one prompt per workload rather than the server benchmark's six. What transfers is the ratio, not the absolute tok/s.

Speculation is lossless by construction — the target verifies every drafted token — and 9 of 12 bench runs came back token-identical to their autoregressive baseline. The three that differed diverged at a single token and then followed a different but valid continuation, which is what a near-tie in the target's own logits looks like when the verify step uses a different GEMM shape than single-token decode.

If you want speculation over HTTP today, serve with SGLang.

Against the other NVFP4 builds on the same GPU

This build and RadixArk were measured here back to back on identical SGLang flags (FP8 KV, --mm-feature-transport cpu, 2 running requests), both at --ctx 262144. Unsloth's figures are the earlier vLLM measurement on the same GPU — a different engine, included for scale.

Two gaps stand out. RadixArk's drafter is BF16 (3.5 GB against our 1.41 GB), and its draft KV pool is sized to the full target context, so turning speculation on costs it most of its window: 37,900 tokens left, where ours keeps 165,169. And RadixArk does not load on SparkInfer at all — its Gated-DeltaNet projections (linear_attn.in_proj_qkv, in_proj_z, out_proj) are rejected by the compressed-tensors loader as malformed FP8. This checkpoint runs on all three engines unmodified.

---

Accuracy

Quantizing the lm_head and dropping the MTP head did not move quality. On a matched 60-item smoke (GPQA Diamond, AIME 2025, MMLU-Pro — 20 items each, thinking on, temperature=1.0, seed 20260815):

The two engines agree on the weights. The budgets differ because SparkInfer caps output per request, so read this as same weights, same accuracy — not as an engine ranking. Both runs carry 6–9 length truncations per task, where thinking filled the budget before an answer appeared.

For a like-for-like comparison against Unsloth NVFP4, the earlier vLLM measurement on the previous build of this checkpoint stands at 45/60 (75%) for both: GPQA 13/20 vs 14/20, AIME 15/20 vs 14/20, MMLU-Pro 17/20 tied.

These are 20-item smokes, not published benchmark scores. Do not cite them as GPQA / AIME / MMLU-Pro results.

---

Serving

SGLang — with speculation (fastest)

docker run -d --name sgl --gpus all --ipc=host --shm-size 32g -p 30000:30000 \
  -v hf_cache:/root/.cache/huggingface \
  lmsysorg/sglang:latest \
  sglang serve \
    --model-path gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \
    --trust-remote-code --tp-size 1 \
    --context-length 262144 \
    --kv-cache-dtype fp8_e4m3 \
    --attention-backend flashinfer \
    --chunked-prefill-size 2048 \
    --mamba-radix-cache-strategy extra_buffer_lazy \
    --mamba-ssm-dtype bfloat16 \
    --mem-fraction-static 0.90 \
    --max-running-requests 2 \
    --max-mamba-cache-size 12 \
    --speculative-algorithm DSPARK \
    --speculative-draft-model-path gittensor-model-hub/Qwen3.8-27B-DSpark-NVFP4 \
    --speculative-dspark-block-size 7 \
    --speculative-draft-model-quantization modelopt_fp4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --host 0.0.0.0 --port 30000

Drop the four --speculative-* lines for the maximum-context profile.

Two flags that are easy to get wrong:

• --speculative-draft-model-quantization modelopt_fp4 is required. Without it SGLang assumes BF16 and mis-loads the packed U8 tensors.
• --max-mamba-cache-size must leave headroom. Under DSpark each request takes 4 Gated-DeltaNet state slots, so size 4 provisions exactly one request with nothing spare and eviction can fail with AssertionError: Can not alloc mamba cache. Rule of thumb: ≥ 4 × --max-running-requests + 4.

vLLM

vllm serve gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \
  --quantization modelopt \
  --kv-cache-dtype fp8 \
  --trust-remote-code \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.97 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

vLLM 0.27.x. First boot JITs the FlashInfer SM120 FP4 GEMM (nvcc + CUDA 13 headers); limit parallel JIT on smaller host RAM with MAX_JOBS=2. Util 0.97 is required for native 256k on 32 GB — 0.90 only holds ~205k KV.

Qwen3.8 is a thinking model; for short answers pass "chat_template_kwargs": {"enable_thinking": false}.

---

Chat template

Qwen3.8's own template, with agentic fixes. It renders byte-identical to upstream Qwen/Qwen3.8-27B on every non-tool path, so ordinary chat, thinking and vision behave exactly as the base model does.

Reasoning effort

xhigh (default), medium, low — invalid values raise:

{"chat_template_kwargs": {"reasoning_effort": "medium"}}

On this checkpoint xhigh is the cheapest setting at equal accuracy. It reasons more directly rather than simply longer:

12 verifiable problems, temperature=0, 3k budget, no truncations. A harder 10-problem set held the same ordering (562 vs 939 tokens). Keep the default unless you have a specific reason not to.

Tool calling

XML by default, matching the qwen3_xml (vLLM) and qwen3_coder (SGLang) parsers:

<tool_call>
<function=get_weather>
<parameter=city>
Paris
</parameter>
</function>
</tool_call>

Assistant tool_calls replayed from an OpenAI-style response render whether function.arguments is a dict or a JSON string. The string form previously raised TypeError: Can only get item pairs from a mapping on the second turn of an agentic loop, which broke direct apply_chat_template, llama.cpp and LM Studio users. Server-side tool calling was never affected — SGLang and vLLM normalize arguments to a dict before rendering.

Template kwargs

system and developer roles are both accepted, and consecutive tool responses are grouped into a single turn.

Validation. Agent correctness 8/8 (tool selection, argument accuracy, parallel calls, no-tool restraint, synthesis, multi-step, failure recovery, tools with thinking off); 240 held-out conversations render clean across four configurations; multi-turn prompts are token-level prefix-stable, so KV cache reuse holds across turns; and a paired A/B against the previous template left DSpark acceptance unchanged to slightly better.

Template contributions from @TheChola.

---

Speculative decoding

A DSpark drafter trained and quantized specifically against this NVFP4 checkpoint ships alongside it:

• 🚀 Qwen3.8-27B-DSpark-NVFP4 — recommended, 1.41 GB
• Qwen3.8-27B-NVFP4-RTX5090-DSpark — BF16 source, 2.72 GB

Measured on 240 held-out prompts, identical across every profile

+31.7% throughput over the built-in MTP head while using a quarter of its memory, and +29.3% over the stock DSpark drafter it was adapted from — which is why the MTP head is now deleted from the weights entirely.

This table uses the 240-prompt held-out harness (longer outputs, its own settings). It is not comparable to the 92.9 / 85.8 / 161.7 figures at the top, which come from the 24-prompt cross-engine harness; each set is internally consistent.

The v2 drafter is retrained on a corpus rendered exactly as the model is served — XML tool calls with a real tools array, think blocks and the reasoning-effort preamble present — closing a train/serve mismatch that had cost the most on agentic traffic.

Per domain

Acceptance measured per request with the prefix cache flushed between requests, from cumulative verify counters rather than a sampled gauge.

Structured output sustains longer accepted blocks than open-ended prose: predictable token streams are easier to draft. Long-context is the one regression — the training corpus caps sequences at 2,048 tokens, so it was never well represented.

Agentic tool calling. The "JSON / structured output" row is schema-constrained generation — those prompts declare no tools and contain no <tool_call> blocks, so they do not measure agentic tool use. Measured separately on 60 purpose-built agentic scenarios across 10 tool schemas:

Tool-call syntax is predictable, so emitting several calls at once accepts longest; multi-step chains accept least, since each new tool boundary is a fresh decision point. Tool-call emission rate is unchanged (38/60 for both), so this is an acceptance gain, not a change in behaviour.

Output quality is unchanged. The target verifies every drafted token under strict acceptance, so speculation changes speed, not outputs.

Choosing a profile

On one 32 GB card you can have the full 262K window or ~1.9× decode. Speculation accelerates decode only, never prefill — cold TTFT at 250K is ~121 s either way.

---

Why lm_head is NVFP4, and why MTP is gone

At concurrency 1 this model is weight-bandwidth bound, not compute bound. An earlier build streamed 18.80 GiB of weights per token at 81.6 tok/s — a 1.65 TB/s read rate against the RTX 5090's 1.79 TB/s spec, about 92% of peak. Decode reads the whole model once per token, so bytes removed convert almost linearly into tokens per second.

lm_head is a full-vocabulary (248,320 × 5,120) GEMM evaluated on every token — 2.54 GB of that read in BF16, 0.72 GB in NVFP4. It was the only large BF16 block left on the per-token critical path. Embeddings are the same size but are a gather (~10 KB/token), so quantizing them saves capacity, not speed.

The bandwidth model predicted 89.7 tok/s for the NVFP4-lm_head build; measured 88.45 — within 1.4%.

The MTP head was ~0.85 GB of tensors that no supported serving path uses once the DSpark drafter is in play, and the drafter beats MTP by 31.7% on a quarter of the memory. Removing it shrinks the download and the load-time footprint without touching a single computed value.

The previous builds are preserved on branches: pre-final (NVFP4 lm_head, MTP still present) and pre-lmhead4 (BF16 lm_head).

The standalone -No-MTP and -LMHead4 variants are superseded — this repo now is both.

---

Recipe (NVIDIA ModelOpt)

NVFP4: MLP, lm_head, and the remaining Linear layers. Left in BF16 on purpose: vision tower, embeddings, and the Gated-DeltaNet conv1d / in_proj_a / in_proj_b. The MTP head is not quantized because it is not present.

Note for tooling authors: excluded modules must be listed in two places — hf_quant_config.json (exclude_modules) and config.json (quantization_config.ignore). Missing the second one is what produces Parameter lm_head.input_scale not found at load.

---

License

Apache 2.0, same as the Qwen3.8-27B base model.