Qwen3.8-Flash-Next GSQ-RCO Coder IQ1_M, NInfer v3 artifact
0
3 commits
2 linked in READMEs
updated Oct 9, 2026
A NInfer v3 artifact of ISTA-DASLab's
Qwen3.8-Flash-Next GSQ-RCO Coder
for NInfer-all, the master branch of iamwavecut/ninfer-all.
The Coder release removes half of Qwen3.8-Flash-Next's routed experts, keeping the 256 of 512 per
layer that code, agentic tool use, vision and spatial reasoning need (chosen with RCO against the
full model), and stores the rest at about 3.5 bits per weight; IQ1_M names the effective 1.89 bits
per parameter of the original model, not a block type. This artifact keeps each tensor's ggml
blocks byte for byte, so nothing is requantized, and the engine multiplies the blocks in place.
Stock NInfer builds refuse these files: the model family and the gguf_* formats exist only in that
line; build master from commit 82f120de on.
NInfer serves it with up to eight concurrent requests, prompt-prefix reuse, structured output and
its Vision tower; MTP drafting is not available (no GSQ-RCO release carries the MTP layer).
| component | representation |
|---|---|
| routed experts, 48 layers × 256 | the GGUF's own blocks, one type per layer and projection: gate and up IQ2_S (20 layers), IQ3_XXS (17), IQ3_S (10), IQ4_XS (1); down IQ4_NL (39), Q2_0 (9); expert-major, so one expert is one contiguous range of bytes |
| shared experts | the GGUF's own blocks per tensor: IQ4_NL and Q8_0 down, Q4_K, Q5_K, Q6_K and IQ4_XS gate and up |
| 36 Gated DeltaNet and 12 sparse-attention (QSA) layers | the GGUF's own blocks, Q4_K to Q6_K and IQ4_XS per tensor |
| output head, token table | Q6_K, IQ4_XS |
| router (256 rows), hyper-connection projections | BF16, as in the GGUF |
GDN controls, norms, convolution, A_log, dt_bias | BF16/FP32, restored from llama.cpp's exporter conventions |
| Vision tower | the release's mmproj-Qwen3.8-Flash-Next-BF16.gguf in BF16 (0.9 GB), the same file as the unpruned releases': the Qwen3.5/3.6 tower, 27 blocks of width 1152, merging 2×2 patches onto the text model's 2,560, with the checkpoint's image and video preprocessor configurations |
| n-gram table | not in this file: the model names its table by SHA-256 (dd55c289…6cc3), the same table as every Flash-Next release, published once as WaveCut/Qwen3.8-Flash-Next-ngram-table-NInfer-v3 |
| chat template | Qwen/Qwen3.8-Flash-Next's chat_template.jinja |
One file, Qwen3.8-Flash-Next-GSQ-RCO-Coder-IQ1_M-ninfer-v3.ninfer, of 30,511,861,248 bytes
(28.42 GiB), next to its conversion report, SHA256SUMS, NOTICE and LICENSE. Conversion
command, from the repository's tree:
python3 -m tools.convert --model Qwen3.8-Flash-Next \
--recipe qwen3_8_flash_next_gguf --components text,vision \
--source gguf=Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf \
--source ngram=Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf \
--source vision=mmproj-Qwen3.8-Flash-Next-BF16.gguf \
--device cpu --rows-per-chunk 65536 \
--name qwen3.8-flash-next-coder --out Qwen3.8-Flash-Next-GSQ-RCO-Coder-IQ1_M-ninfer-v3.ninfer
The converter reads the table shard to record its digest; the rows themselves are in the table
repository (this conversion read the Q2_0 release's shard, the same file). --model needs the
configuration, tokenizer and preprocessor files of
Qwen/Qwen3.8-Flash-Next; the converter takes the
256-expert count from the GGUF.
Download the artifact and the n-gram table, then serve them from a build of master
(its README; CMAKE_CUDA_ARCHITECTURES is
86, 89 or 120a, and 120a needs CUDA 13.1 or newer):
hf download WaveCut/Qwen3.8-Flash-Next-GSQ-RCO-Coder-IQ1_M-NInfer-v3 \
Qwen3.8-Flash-Next-GSQ-RCO-Coder-IQ1_M-ninfer-v3.ninfer --local-dir models
hf download WaveCut/Qwen3.8-Flash-Next-ngram-table-NInfer-v3 \
Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer --local-dir models
M=models/Qwen3.8-Flash-Next-GSQ-RCO-Coder-IQ1_M-ninfer-v3.ninfer
T=models/Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer
# One 24 GB GPU: the experts in page-locked host memory (25.1 GB), the most used of them cached on the GPU.
ninfer-serve $M --ngram-table $T --model-id qwen3.8-flash-next-coder --expert-residency host --max-context 32768
# One 24 GB GPU and little RAM: the experts stay in the file and stream into a GPU cache.
ninfer-serve $M --ngram-table $T --model-id qwen3.8-flash-next-coder --expert-residency disk --max-context 32768
The Docker image (an NVIDIA driver of the CUDA
13 branch, 580 or newer, and the NVIDIA Container Toolkit) runs the same commands as serve with
the files under /models, and answers on http://localhost:8080/v1:
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
-v "$PWD/models:/models" -v ninfer-cache:/cache \
ghcr.io/iamwavecut/ninfer-all serve /models/Qwen3.8-Flash-Next-GSQ-RCO-Coder-IQ1_M-ninfer-v3.ninfer \
--ngram-table /models/Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer \
--model-id qwen3.8-flash-next-coder --expert-residency host --max-context 32768
A model without its n-gram table is refused; --no-ngram-table runs it without, a non-standard
experimental mode with no practical use (see the table's card). --vision adds the Vision tower
(0.9 GB on the GPU) for images and video.
Qwen3.8-Flash-Next
describes the placements and the execution.
The Coder card reports SWE-bench Verified 75.60 and LiveCodeBench v6 86.28 at xhigh reasoning effort in llama.cpp (BF16 base: 82.80, 87.43). These benchmarks have not been run in NInfer.
October 2026, greedy decoding, CUDA 12.8 (an sm_86 build), the repository's generate test
(ninfer_qwen4_exp_generate_real) on one NVIDIA L40S (48 GB, PCIe) with the table read from its own
artifact. Decode is measured over the 70 tokens of a short answer to a 20-token prompt (the expert
cache still filling) and over the first five tokens after a 4,463-token prompt; prefill is that
prompt in 512-token chunks. On 48 GB the device expert cache (39.9 GB) holds every expert of this
build once it has seen them, so the long-prompt figures are an upper bound for a 24 GB card, whose
cache holds about half of them.
| placement | decode, short answer | decode after 4,463 tokens | prefill |
|---|---|---|---|
| experts in pinned host memory (25.1 GB) | 34.7 tok/s | 80.4 tok/s | 1,648 tok/s |
| experts on disk, the file in the page cache | 41.6 tok/s | 74.5 tok/s | 1,938 tok/s |
Quantized and pruned weights: ISTA-DASLab,
Qwen3.8-Flash-Next GSQ-RCO Coder GGUF,
produced with GSQ and RCO
from Qwen/Qwen3.8-Flash-Next. The model is
released under the Qwen Community License 1.0, reproduced in LICENSE, and its conditions
apply to these files; ISTA-DASLab publish their GGUFs under Apache-2.0. Attribution notices are
collected in NOTICE. This repository only re-packs those weights into NInfer's container.
Qwen3.8-Flash-Next GSQ-RCO Coder IQ1_M, NInfer v3 artifact
0
3 commits
2 linked in READMEs
updated Oct 9, 2026
A NInfer v3 artifact of ISTA-DASLab's
Qwen3.8-Flash-Next GSQ-RCO Coder
for NInfer-all, the master branch of iamwavecut/ninfer-all.
The Coder release removes half of Qwen3.8-Flash-Next's routed experts, keeping the 256 of 512 per
layer that code, agentic tool use, vision and spatial reasoning need (chosen with RCO against the
full model), and stores the rest at about 3.5 bits per weight; IQ1_M names the effective 1.89 bits
per parameter of the original model, not a block type. This artifact keeps each tensor's ggml
blocks byte for byte, so nothing is requantized, and the engine multiplies the blocks in place.
Stock NInfer builds refuse these files: the model family and the gguf_* formats exist only in that
line; build master from commit 82f120de on.
NInfer serves it with up to eight concurrent requests, prompt-prefix reuse, structured output and
its Vision tower; MTP drafting is not available (no GSQ-RCO release carries the MTP layer).
| component | representation |
|---|---|
| routed experts, 48 layers × 256 | the GGUF's own blocks, one type per layer and projection: gate and up IQ2_S (20 layers), IQ3_XXS (17), IQ3_S (10), IQ4_XS (1); down IQ4_NL (39), Q2_0 (9); expert-major, so one expert is one contiguous range of bytes |
| shared experts | the GGUF's own blocks per tensor: IQ4_NL and Q8_0 down, Q4_K, Q5_K, Q6_K and IQ4_XS gate and up |
| 36 Gated DeltaNet and 12 sparse-attention (QSA) layers | the GGUF's own blocks, Q4_K to Q6_K and IQ4_XS per tensor |
| output head, token table | Q6_K, IQ4_XS |
| router (256 rows), hyper-connection projections | BF16, as in the GGUF |
GDN controls, norms, convolution, A_log, dt_bias | BF16/FP32, restored from llama.cpp's exporter conventions |
| Vision tower | the release's mmproj-Qwen3.8-Flash-Next-BF16.gguf in BF16 (0.9 GB), the same file as the unpruned releases': the Qwen3.5/3.6 tower, 27 blocks of width 1152, merging 2×2 patches onto the text model's 2,560, with the checkpoint's image and video preprocessor configurations |
| n-gram table | not in this file: the model names its table by SHA-256 (dd55c289…6cc3), the same table as every Flash-Next release, published once as WaveCut/Qwen3.8-Flash-Next-ngram-table-NInfer-v3 |
| chat template | Qwen/Qwen3.8-Flash-Next's chat_template.jinja |
One file, Qwen3.8-Flash-Next-GSQ-RCO-Coder-IQ1_M-ninfer-v3.ninfer, of 30,511,861,248 bytes
(28.42 GiB), next to its conversion report, SHA256SUMS, NOTICE and LICENSE. Conversion
command, from the repository's tree:
python3 -m tools.convert --model Qwen3.8-Flash-Next \
--recipe qwen3_8_flash_next_gguf --components text,vision \
--source gguf=Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00001-of-00002.gguf \
--source ngram=Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf \
--source vision=mmproj-Qwen3.8-Flash-Next-BF16.gguf \
--device cpu --rows-per-chunk 65536 \
--name qwen3.8-flash-next-coder --out Qwen3.8-Flash-Next-GSQ-RCO-Coder-IQ1_M-ninfer-v3.ninfer
The converter reads the table shard to record its digest; the rows themselves are in the table
repository (this conversion read the Q2_0 release's shard, the same file). --model needs the
configuration, tokenizer and preprocessor files of
Qwen/Qwen3.8-Flash-Next; the converter takes the
256-expert count from the GGUF.
Download the artifact and the n-gram table, then serve them from a build of master
(its README; CMAKE_CUDA_ARCHITECTURES is
86, 89 or 120a, and 120a needs CUDA 13.1 or newer):
hf download WaveCut/Qwen3.8-Flash-Next-GSQ-RCO-Coder-IQ1_M-NInfer-v3 \
Qwen3.8-Flash-Next-GSQ-RCO-Coder-IQ1_M-ninfer-v3.ninfer --local-dir models
hf download WaveCut/Qwen3.8-Flash-Next-ngram-table-NInfer-v3 \
Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer --local-dir models
M=models/Qwen3.8-Flash-Next-GSQ-RCO-Coder-IQ1_M-ninfer-v3.ninfer
T=models/Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer
# One 24 GB GPU: the experts in page-locked host memory (25.1 GB), the most used of them cached on the GPU.
ninfer-serve $M --ngram-table $T --model-id qwen3.8-flash-next-coder --expert-residency host --max-context 32768
# One 24 GB GPU and little RAM: the experts stay in the file and stream into a GPU cache.
ninfer-serve $M --ngram-table $T --model-id qwen3.8-flash-next-coder --expert-residency disk --max-context 32768
The Docker image (an NVIDIA driver of the CUDA
13 branch, 580 or newer, and the NVIDIA Container Toolkit) runs the same commands as serve with
the files under /models, and answers on http://localhost:8080/v1:
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
-v "$PWD/models:/models" -v ninfer-cache:/cache \
ghcr.io/iamwavecut/ninfer-all serve /models/Qwen3.8-Flash-Next-GSQ-RCO-Coder-IQ1_M-ninfer-v3.ninfer \
--ngram-table /models/Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer \
--model-id qwen3.8-flash-next-coder --expert-residency host --max-context 32768
A model without its n-gram table is refused; --no-ngram-table runs it without, a non-standard
experimental mode with no practical use (see the table's card). --vision adds the Vision tower
(0.9 GB on the GPU) for images and video.
Qwen3.8-Flash-Next
describes the placements and the execution.
The Coder card reports SWE-bench Verified 75.60 and LiveCodeBench v6 86.28 at xhigh reasoning effort in llama.cpp (BF16 base: 82.80, 87.43). These benchmarks have not been run in NInfer.
October 2026, greedy decoding, CUDA 12.8 (an sm_86 build), the repository's generate test
(ninfer_qwen4_exp_generate_real) on one NVIDIA L40S (48 GB, PCIe) with the table read from its own
artifact. Decode is measured over the 70 tokens of a short answer to a 20-token prompt (the expert
cache still filling) and over the first five tokens after a 4,463-token prompt; prefill is that
prompt in 512-token chunks. On 48 GB the device expert cache (39.9 GB) holds every expert of this
build once it has seen them, so the long-prompt figures are an upper bound for a 24 GB card, whose
cache holds about half of them.
| placement | decode, short answer | decode after 4,463 tokens | prefill |
|---|---|---|---|
| experts in pinned host memory (25.1 GB) | 34.7 tok/s | 80.4 tok/s | 1,648 tok/s |
| experts on disk, the file in the page cache | 41.6 tok/s | 74.5 tok/s | 1,938 tok/s |
Quantized and pruned weights: ISTA-DASLab,
Qwen3.8-Flash-Next GSQ-RCO Coder GGUF,
produced with GSQ and RCO
from Qwen/Qwen3.8-Flash-Next. The model is
released under the Qwen Community License 1.0, reproduced in LICENSE, and its conditions
apply to these files; ISTA-DASLab publish their GGUFs under Apache-2.0. Attribution notices are
collected in NOTICE. This repository only re-packs those weights into NInfer's container.