Qwen3.8-Flash-Next — EXL3 3.05 bpw

Original model: Qwen/Qwen3.8-Flash-Next by the Qwen team, released under the Qwen Community License 1.0 — not an OSI license; see License below. This artifact contains modified weights (EXL3 trellis quantization); the original model is © Qwen.

Direct source: turboderp/Qwen3.8-Flash-Next-exl3, branch/revision 3.05bpw_h5_ng5 — 3.05 bpw experts, 5-bit head, n-gram table included.

An EXL3 (ExLlamaV3 trellis) 3.05 bpw quantization of Qwen3.8-Flash-Next: a 125B-parameter mixture-of-experts model with about 6B active parameters, plus a 51B hashed n-gram embedding and a 4B MTP layer, with a vision encoder. The pack is ~85 GB (~79 GiB) against ~360 GB for the original bf16 release.

This quantized artifact exists first and foremost to run with the veloGB10 inference engine (gb10_inference) on one, two or four NVIDIA DGX Spark / GB10 systems — it is the pack we load, validate and serve with veloGB10's EXL3 kernels. The weights are standard EXL3, so this artifact can be used for any purpose, with any framework that reads the format, and it is fit and proven to work with veloGB10.

The quantization is turboderp's 3.05bpw_h5_ng5 EXL3 pack (exllamav3 1.4.4), redistributed here with these differences from turboderp's repository: a vLLM-compatible config.json and a regenerated tensor index model.safetensors.index.json (the pack's originals are kept as the .native files), the pack_scan.json record behind the regenerated index, and the original full-precision (bf16) vision tower as a separate file, vision_tower_bf16.safetensors (see Notes).

The weights are the unmodified pack. No abliteration, fine-tuning or merging is applied — the weight files (shards, n-gram table, MTP patch) are byte-identical to the pack as distributed.

Serving

Serve it with the veloGB10 engine (gb10_inference), which reads EXL3 packs directly from --model-dir at TP=1, TP=2 and TP=4. The engine loads its PTX kernel artifacts from the working directory, so run it from wherever the binary and src/ptx/ live.

Single node (TP=1)

./gb10_inference --server \
  --model-dir /path/to/model/Qwen3.8-Flash-Next-exl3-3.05bpw \
  --port 9000 \
  --max-seq-len 262144 \
  --max-batch 1 \
  --max-tokens 65536 \
  --prefix-cache on \
  --default-presence-penalty 1.5 \
  --mtp=auto

Two nodes (TP=2)

./gb10_inference --node --port 29500                                    # on the peer

./gb10_inference --server \
  --model-dir /path/to/model/Qwen3.8-Flash-Next-exl3-3.05bpw \
  --tp 2 \
  --nodes <node ip address>:29500 \
  --port 9000 \
  --max-seq-len 262144 \
  --max-batch 1 \
  --max-tokens 65536 \
  --prefix-cache on \
  --default-presence-penalty 1.5 \
  --mtp=auto

Four nodes (TP=4)

Start ./gb10_inference --node --port 29500 on each of the three peer GB10s, then:

./gb10_inference --server \
  --model-dir /path/to/model/Qwen3.8-Flash-Next-exl3-3.05bpw \
  --tp 4 \
  --nodes <node 1 ip>:29500,<node 2 ip>:29500,<node 3 ip>:29500 \
  --port 9000 \
  --max-seq-len 262144 \
  --max-batch 1 \
  --max-tokens 65536 \
  --prefix-cache on \
  --default-presence-penalty 1.5 \
  --mtp=auto

The peers need no model copy. The head plans each rank's shard and ships only that, through the TP blob cache (~/.cache/gb10_tp): roughly 57.5 GiB per node at TP=2 and 46.8 GiB per node at TP=4, not the whole pack. Only missing blobs move, so a second start of the same model syncs nothing. The server is OpenAI-compatible on http://<head-ip>:9000/v1.

Performance

Measured with VeloBenchmark 0.1.0 (TP=1/TP=2 on 2026-09-30, TP=4 on 2026-10-02) against the served model, one request at a time, reasoning effort low. Decode includes MTP speculation.

Pure-code decode — ANSI C sorting, ~3.2K output tokens:

Prefill — one measurement per input size:

Time to first token (TP=4): 0.24 s at ~550 tokens, 0.64 s at ~2.1K, 1.92 s at ~6.2K, 3.19 s at ~10.3K, 5.69 s at ~18.5K and 10.90 s at ~34.9K.

Against a single GB10: TP=2 is ×1.36 on decode and ×1.5–1.6 on prefill; TP=4 is ×1.6 on decode and ×2.0–2.2 on prefill.

Concurrency — every figure above is --max-batch 1. With several busy requests the engine shares one batched step from about four requests onward (roughly ×1.45 aggregate at 8 requests and ×1.9 at 16 on one node, code workload; TP=2/TP=4 proportionally). Two and three requests still take turns, so their aggregate stays at the single-request rate. Greedy output is byte-identical to running each request alone, at every topology.

Specifications

Files

Notes

• Quantization detail: 3.05 bpw on the linear layers, with the lm_head, the n-gram table and the vision tower at 5 bits and the MTP layer at 3 bits. The n-gram embedding is sharded (128 shards, 2,500,012 rows each) and is the larger part of the on-disk size.
• Vision: images are served at TP=1, TP=2 and TP=4; they are resized so the longer side is at most --image-max-edge (default 1024), and they work past the 2,051-token dense window. veloGB10 runs the tower from the included full-precision bf16 file (vision_tower_bf16.safetensors) when it is in the model folder, and falls back to the 5-bit tower inside the shards when it is not. Video and audio parts return 400 for now.
• Quality: the pack ships turboderp's qbench_prompts set and a pack_scan.json record; use those to reproduce the pack-level check on your own hardware.
• Long context is 262,144 tokens per the config, not the 1M that the hosted Qwen3.8-Flash API offers.
• Engine defaults change output bytes versus veloGB10 v0.7.0: --prefill-chunk 4095 (was 2048) and --qsa-key-rope full. Both have documented escape hatches in the engine's documentation.

License

The Qwen Community License 1.0 applies (a copy is included as LICENSE). It is a custom license, not an OSI-approved one. In short, as of this writing:

• You may use, modify, publish and distribute the model and derivative works, provided the copyright notice and license are included.
• If you serve the model or a derivative as a Model-as-a-Service or AI Work Assistant business, you need a separate license from Qwen (internal use is exempt).
• Products with more than 100M monthly active users or US$20M monthly revenue must display the model name prominently.

Read LICENSE for the binding text. This repository redistributes Qwen's weights in quantized form and is not affiliated with or endorsed by Qwen.

Credits

• Model and license: Qwen/Qwen3.8-Flash-Next — © the Qwen team, Qwen Community License 1.0.
• EXL3 quantization: turboderp — the 3.05bpw_h5_ng5 pack and the accompanying qbench prompts, built with exllamav3.
• Engine: veloGB10.