Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS — HC-Q5_0 ("HCQ5_0")

Qwen3.8-Flash-Next (arch qwen4exp) on the GSQ-RCO IQ3_XXS quantization pack, with one measured model change: the hyper-connection (HC) trunk requantized BF16 -> Q5_0, plus MTP draft sidecars (a full-vocab one and a reduced 65,280-token LM-head one) for speculative decoding.

• Base pack: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF (IQ3_XXS)
• Model change: 194 tensors (all hc_{attn,ffn}_{up,down}.weight + output_hc_{up,down}.weight) BF16 -> Q5_0. Every other tensor is byte-identical to the base pack (verified by spot-check).
• Draft: mtp-draft-q3-v65k.gguf — the model's own MTP (nextn) layer, Q3 mix, with the draft LM head sliced to the 65,280 most frequent tokens (99.6% of vocab).

Measured on the target hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151, 60 GB LPDDR5X, Vulkan/RADV):

Decode at depth 0 is bandwidth-bound on this UMA machine (34.6 t/s x ~3.4 GB/token is at the measured ~115 GB/s DRAM read roofline); the HC remap removes 0.845 GB from every forward pass (the HC trunk was ~1.28 GB/pass of BF16, ~33% of all per-token weight bytes, while the experts were already at ~2.5 bpw).

What changed in the weights (and why)

Quality gates for the HCQ5_0 remap (run before shipping):

• 128-token greedy generation: identical for the first 47 tokens, then a benign word-choice branch (a requant of the residual-mixing path flips one logit tie; not corruption).
• 33K-token needle (marker at 16K depth): found.
• 1K-token coherence: clean.
• Perplexity was not measured (needs a calibration set + long CPU/GPU run); GSQHalo.cpp reports 3.9044 for the base IQ3_XXS pack on wikitext if you want a reference point.

Files

Model facts (from the GGUF): arch qwen4exp, block_count 48, embedding_length 2560, attention.head_count 24, expert_count 512 / expert_used_count 10 (MoE "A3B" class), context_length 262144, vocab 248,320 (ggml gpt2 tokenizer, qwen35 pre-tokenizer), PLE: n-gram size 3, 16 heads, 160-wide per-layer embeddings. All quantization types in the files are standard ggml types (F32, BF16, Q2_0, Q3_K, Q4_K, Q5_K, Q6_K, Q8_0, IQ2_XXS/XS/S, IQ3_XXS/S, IQ4_XS/NL, Q5_0, F16) — no custom/proprietary formats.

How to run

Engine note (verified 2026-10-08 against ggml-org/llama.cpp master 08246a2): the qwen4exp architecture itself is upstream llama.cpp (merged 2026-08), and this trunk loads and runs on stock llama.cpp — verified by a stock llama-quantize full-tensor dry-run of both shards (all 1,224 tensors accepted, no errors; only standard types, no fork-only quant formats). The caveats are about the sidecar features, not the weights:

feature	stock llama.cpp	recommended fork
trunk (2 shards)	loads, runs	loads, runs (fastest)
PLE table (shard 2)	-lzm on (mmap, on-demand rows) or -lm dio	--ngram-on-disk (O_DIRECT + 64 reader threads, 256 MiB row cache) or -lzm on-direct
MTP draft (both files here)	not supported: the files load, but stock's MTP graph requires
blk.48.nextn.hc_head_norm.weight, which these sidecars do not carry (they name the head-mixer
norm blk.48.nextn.shared_head_norm.weight), and qwen4exp.draft_vocab_size is a fork-only key
--spec-type draft-mtp works with both drafts

Use a fork for MTP. The fork this model was developed and measured on is sleepyeldrazi/llama.cpp-strix, branch feat/strix-prod (a fork of LaurentZuijdwijk/llama.cpp; Vulkan/Strix-Halo; carries the sidecar-MTP loader that accepts these drafts, the reduced-vocab draft support, --ngram-on-disk, QSA gather, drafter arbiter).

GSQHalo.cpp (Aristo94/GSQHalo.cpp, ROCm/HIP, fork of halo-box/strix-llama.cpp) runs the trunk fine (it has GSQ-RCO prefill kernels and --ngram-on-disk), but neither draft in this repo loads there: its sidecar loader requires the final head mixer as either nextn.hc_head_{norm,down,up} or the trunk's output_hc_{norm,down,up} set, while these sidecars ship nextn.shared_head_norm + output_hc_{down,up} only (load aborts with "MTP block has no final mixer"), and the v65k file's 65,280-wide head is a dimension mismatch anyway. A GSQHalo-compatible draft is regenerable from the model's nextn layer with the converter's expected names.

Stock llama.cpp (trunk only, no MTP)

llama-server -m Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-HCQ5_0-00001-of-00002.gguf \
  -c 262144 -ngl 999 -fa on --kv-unified \
  -ctk q4_0 -ctv q4_0 \
  -lzm on \            # PLE table streamed from disk (mmap); or -lm dio
  -b 512 -ub 512 -t 16 --parallel 1

On a 60 GB UMA machine, 256K context with q4_0 KV + full weights fits with ~3-7 GiB to spare (measured: 50.1 GiB in the driver + ~2 GiB process + ~1.3 GiB kernel).

Custom fork (feat/strix-prod) — full config with MTP (what the numbers above used)

llama-server -m Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-HCQ5_0-00001-of-00002.gguf \
  -c 262144 -ctk q4_0 -ctv q4_0 -ctkd q4_0 -ctvd q4_0 \
  -fa on --kv-unified --n-gpu-layers 999 --device vulkan0 \
  --ngram-on-disk --fit off -ctxcp 2 \
  -b 512 -ub 512 -t 16 --parallel 1 \
  -md mtp-draft-q3-v65k.gguf --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.8

Measured settings that mattered (see PORT-LOG.md of the development repo for the full A/Bs):

• MTP n_max 3: best at every depth. Deeper drafts cost an extra full-MoE draft pass without enough acceptance gain; adaptive n_max was slower than fixed on this drafter.
• q4_0 KV (main and draft): 256K steady state with ~5.7 GiB headroom; q8_0 is the alternative at 128K (+4% faster than f16).
• MTP is content-dependent: with high-acceptance text it is a big win (up to ~1.9x on the base pack at depth 0), but on low-acceptance prose it can be slower than spec-off — the drafter is a full MoE layer. The fork ships a drafter cost arbiter (--spec-draft-arbiter, off by default) that stops drafting when it stops paying.
• -ctxcp (context checkpoints) is ~112 MiB+ each and only needed for prefix caching.

GSQHalo.cpp (public alternative; trunk only)

# from https://github.com/Aristo94/GSQHalo.cpp (ROCm 7.14 / gfx1151) — trunk, no MTP
llama-server -m Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-HCQ5_0-00001-of-00002.gguf \
  -c 262144 -ngl 999 -fa on -lm dio -lzm on -b 8192 -ub 8192 \
  --cache-ram 2048 -ctxcp 8

(The trunk runs there; MTP with the drafts in this repo does not — see the engine table above.)

Provenance

• Weights: GSQ-RCO IQ3_XXS pack (ISTA DASLab) -> HC trunk BF16 -> Q5_0 remap with stock llama-quantize --tensor-type-file (1,030 tensors byte-copied, 194 converted).
• Drafts: the model's MTP head exported as a sidecar (mtp-draft-q3.gguf), and a reduced version of it with the head sliced to 65,280 columns (mtp-draft-q3-v65k.gguf, port-tools/slice-draft-vocab.py, FR-Spec style, cf. vLLM QWEN4EXP_DRAFT_VOCAB).
• Development + measurements: engine work in sleepyeldrazi/llama.cpp-strix branch feat/strix-prod (Vulkan/Strix-Halo fork of LaurentZuijdwijk/llama.cpp). The full port log, baselines, and per-feature A/B reports (PORT-LOG.md, port-notes/, port-tools/) live in the author's working directory; the serve script used for the numbers above is serve-qwen-hcq5-mtp.sh from the same tree.

License

Apache-2.0, like the base GSQ-RCO pack and Qwen3.8-Flash-Next. This repo contains a derived quantization (HC trunk BF16 -> Q5_0, everything else byte-identical) and a derived MTP draft sidecar.