Qwen3.8-Flash-Next GSQ-RCO IQ3_S, NInfer v3 artifact

A NInfer v3 artifact of ISTA-DASLab's Qwen3.8-Flash-Next GSQ-RCO IQ3_S (3.50 bits per weight over the transformer weights) for NInfer-all, the master branch of iamwavecut/ninfer-all. Qwen3.8-Flash-Next is a 125B-parameter mixture of experts: 48 layers of 512 routed experts, 10 of them active per token, about 6B active parameters. The GGUF assigns a ggml quantization type to every tensor; this artifact keeps each tensor's ggml blocks byte for byte, so nothing is requantized, and the engine multiplies the blocks in place. IQ3_S is the release the GSQ-RCO card recommends: the strongest of the four.

Stock NInfer builds refuse these files: the model family and the gguf_* formats exist only in that line; build master from commit 82f120de on. NInfer serves it with up to eight concurrent requests, prompt-prefix reuse, structured output and its Vision tower; MTP drafting is not available (no GSQ-RCO release carries the MTP layer).

What is inside

One file, Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-ninfer-v3.ninfer, of 55,729,335,808 bytes (51.90 GiB), next to its conversion report, SHA256SUMS, NOTICE and LICENSE. Conversion command, from the repository's tree:

python3 -m tools.convert --model Qwen3.8-Flash-Next \
  --recipe qwen3_8_flash_next_gguf --components text,vision \
  --source gguf=Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf \
  --source ngram=Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf \
  --source vision=mmproj-Qwen3.8-Flash-Next-BF16.gguf \
  --device cpu --rows-per-chunk 65536 \
  --name qwen3.8-flash-next --out Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-ninfer-v3.ninfer

The converter reads the table shard to record its digest; the rows themselves are in the table repository. The table shard of every release is the same file (SHA-256 316b46f3…e113); the digest above came from the Q2_0 release's. --model needs the configuration, tokenizer and preprocessor files of Qwen/Qwen3.8-Flash-Next (config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json, preprocessor_config.json, video_preprocessor_config.json).

Running

Download the artifact and the n-gram table, then serve them from a build of master (its README; CMAKE_CUDA_ARCHITECTURES is 86, 89 or 120a, and 120a needs CUDA 13.1 or newer):

hf download WaveCut/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-NInfer-v3 \
  Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-ninfer-v3.ninfer --local-dir models
hf download WaveCut/Qwen3.8-Flash-Next-ngram-table-NInfer-v3 \
  Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer --local-dir models
M=models/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-ninfer-v3.ninfer
T=models/Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer

# One RTX PRO 6000 (96 GB): every expert in device memory.
ninfer-serve $M --ngram-table $T --model-id qwen3.8-flash-next --max-context 32768

# Two 32 GB GPUs (Linux): every expert in device memory, one pipeline stage per GPU.
ninfer-serve $M --ngram-table $T --model-id qwen3.8-flash-next --devices 0,1 --max-context 32768

# One 24 GB GPU: the experts in page-locked host memory (50.3 GB), the most used of them cached on the GPU.
ninfer-serve $M --ngram-table $T --model-id qwen3.8-flash-next --expert-residency host --max-context 32768

# One 24 GB GPU and little RAM: the experts stay in the file and stream into a GPU cache.
ninfer-serve $M --ngram-table $T --model-id qwen3.8-flash-next --expert-residency disk --max-context 32768

The Docker image (an NVIDIA driver of the CUDA 13 branch, 580 or newer, and the NVIDIA Container Toolkit) runs the same commands as serve with the files under /models, and answers on http://localhost:8080/v1:

docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all serve /models/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-ninfer-v3.ninfer \
  --ngram-table /models/Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer \
  --model-id qwen3.8-flash-next --expert-residency host --max-context 32768

With host experts the GPU holds the 4.5 GB of dense weights, the request state and an expert cache that takes what is free after startup (--expert-cache-mib sets it). With disk experts the host keeps no copy of the banks: the cache reads the experts each layer routes to from the file, through the OS page cache. The n-gram rows are read from the table's file the same way (16 per token) unless --ngram-ram loads the 28.8 GB table into RAM. A model started without its table is refused; --no-ngram-table runs it without, a non-standard experimental mode with no practical use (see the table's card). --vision adds the Vision tower (0.9 GB on the GPU) for images and video. Every expert in device memory takes 53.7 GiB on one RTX PRO 6000 and 26.5 + 27.8 GiB on two RTX 5090s; three 24 GB cards (--devices 0,1,2) should hold it too, but were not measured. The KV cache of the 12 sparse-attention layers is BF16. Qwen3.8-Flash-Next describes the placements and the execution.

Quality

NInfer computes what llama.cpp computes from the same GGUF. Over the first 72 KB of the WikiText sample in the repository's eval/corpora/perplexity-1m, in 2,560-token windows that advance by 1,024 targets (llama.cpp's --ppl-stride 1024 -c 2048), the fourteen windows both programs score the same way (14,336 targets) give:

Window by window the two differ by -0.022 to +0.016 nats. NInfer applies the scales of the IQ2_S and IQ3_XXS experts in FP32 where llama.cpp's vector kernels round an integer rescale, so decode logits can differ in the last bits.

The GSQ-RCO card evaluates this quantization in llama.cpp at AIME25 100.00, GPQA-Diamond 92.93 and LiveCodeBench v6 86.86 (BF16: 100.00, 91.92, 87.43). These benchmarks have not been run in NInfer.

Speed

October 2026, greedy decoding, CUDA 12.8, the repository's generate test (ninfer_qwen4_exp_generate_real), on one RTX 3090 (310 W power limit, PCIe 4.0 x16) in a host with 62 GB of RAM, 23 cores of an AMD EPYC 7663 and an NVMe drive. Decode is measured over the 86 tokens of a short answer to a 20-token prompt and over the first five tokens after a 4,463-token prompt; prefill is that prompt in 512-token chunks. These are the weights of this file, then in a single file with the n-gram table (the table and the Vision tower are not read here); the Q2_0 artifact ran on the same card in the same sitting.

The RTX PRO 6000 and RTX 5090 rows ran this file with the separate table artifact, a 120a build with CUDA 13.1, 2026-10-06, single-GPU rows pinned to the GPU's NUMA node:

• RTX PRO 6000 Workstation Edition: 600 W, PCIe 4.0 x16; short answer 87 tokens.
• RTX 5090s: 600 W boards, PCIe 5.0 x16; short answer 81 tokens.

Device memory is what nvidia-smi reports: the expert cache takes what is free after startup. Without CUDA graphs the host-memory decode is 33.0 and 28.7 tok/s; disk experts decode eagerly. llama.cpp runs this GGUF on the same machine with the experts on the CPU (--n-cpu-moe 48 -t 23) at 29.6 tok/s decode (llama-bench tg128) and 312 tok/s prefill (pp512); its Q2_0 path reaches 12.7 and 368 tok/s there.

Credits and license

Quantized weights: ISTA-DASLab, Qwen3.8-Flash-Next GSQ-RCO GGUFs, produced with GSQ and RCO from Qwen/Qwen3.8-Flash-Next. The model is released under the Qwen Community License 1.0, reproduced in LICENSE, and its conditions apply to these files; ISTA-DASLab publish their GGUFs under Apache-2.0. Attribution notices are collected in NOTICE. This repository only re-packs those weights into NInfer's container.