Qwen3.8-27B GSQ-RCO IQ3_S, NInfer v3 artifact

A NInfer v3 artifact of ISTA-DASLab's Qwen3.8-27B GSQ-RCO IQ3_S (3.5 bits per weight) for NInfer-all, the master branch of iamwavecut/ninfer-all, which serves the RTX 3090, RTX 4090, RTX 5090 and RTX PRO 6000 Blackwell. The GGUF assigns a ggml quantization type to every tensor; this artifact keeps each tensor's ggml blocks byte for byte, so nothing is requantized, and the engine multiplies the blocks in place. Stock NInfer builds refuse this file: the gguf_* formats exist only in that line, from commit 8706bfc0 on.

What is inside

One .ninfer file of 15,017,456,896 bytes (13.99 GiB; 10.95 GiB of text weights are resident without Vision and drafters), next to its conversion report, SHA256SUMS and NOTICE.

Conversion command, from the repository's tree:

python3 -m tools.convert \
  --model Qwen3.8-27B \
  --recipe qwen3_8_27b_gguf \
  --source gguf=Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf \
  --source vision=mmproj-Qwen3.8-27B-BF16.gguf \
  --source dflash2=Qwen3.8-27B-DFlash2 \
  --components text,vision,mtp,dflash2 \
  --resource chat_template.jinja=tools/chat_templates/qwen3_8.jinja \
  --proposal \
  --name qwen3.8-27b \
  --out Qwen3.8-27B-GSQ-RCO-IQ3_S-ninfer-v3.ninfer

--model needs only the base model's configuration and tokenizer files.

Running

A single stream with MTP from the Docker image (an NVIDIA driver of the CUDA 13 branch, 580 or newer, and the NVIDIA Container Toolkit):

hf download WaveCut/Qwen3.8-27B-GSQ-RCO-IQ3_S-NInfer-v3 Qwen3.8-27B-GSQ-RCO-IQ3_S-ninfer-v3.ninfer --local-dir models
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all serve /models/Qwen3.8-27B-GSQ-RCO-IQ3_S-ninfer-v3.ninfer \
  --model-id qwen3.8-27b --max-context 176128 --kv-capacity 176128 --kv-dtype rk8v4 \
  --gdn-state-fp16 --spec mtp --draft-tokens 3

The server answers OpenAI and Anthropic requests on http://localhost:8080/v1. A build of master (its README; CMAKE_CUDA_ARCHITECTURES is 86, 89 or 120a, and 120a needs CUDA 13.1 or newer) takes the same flags:

ninfer-serve models/Qwen3.8-27B-GSQ-RCO-IQ3_S-ninfer-v3.ninfer --model-id qwen3.8-27b \
  --max-context 176128 --kv-capacity 176128 --kv-dtype rk8v4 --gdn-state-fp16 \
  --spec mtp --draft-tokens 3

176,128 tokens fit a 24 GB card; an RTX 5090 takes the full 262,144. --spec dflash2 --draft-tokens 5 drafts with the DFlash2 adapter instead; --vision adds images. Products of up to eight tokens (decode, verification) decode each weight once for every token and dot it with ggml's q8_1 activations; prompts run llama.cpp's integer tensor-core kernel, vendored into the engine. GGUF block formats describes both.

Quality

WikiText-2 test perplexity with the protocol of the GSQ-RCO model card (the test rows joined by blank lines, disjoint 2,048-token windows; ninfer-perplexity --context 2048 --disjoint, BF16 KV):

Reasoning, with NInfer's capability evaluation (EvalScope 1.10.0, temperature 1.0, one sampled run each, RTX 5090), the protocol the official artifact was scored with:

The GSQ-RCO card reports 100 on AIME 2025 and 89.39 on GPQA-Diamond for this quantization with its own protocol (BF16: 100 and 89.90). One run of GPQA-Diamond's 198 questions varies by about two points.

Speed

September 2026, rk8v4 KV, one request at a time, greedy, thinking off, measured with the repository's reference client (tools/bench/refbench.py). Every row pairs this artifact with the official Qwen3.8-27B NInfer artifact on the same card in the same sitting: an RTX 3090 at 420 W and an RTX 4090 at 450 W with a 176,128-token window (the official DFlash2 row on the RTX 4090 at 167,936), an RTX 5090 at 450 W with 262,144. Short chat is the mean decode rate over five 512-token answers.

(this artifact / the official one). The official artifact's published reference numbers, measured on other boards (RTX 3090 at 390 W, RTX 5090 at 575 W), are 48.2 / 97.2 / 117.1 tok/s on the RTX 3090, 54.8 / 109.0 / 138.1 on the RTX 4090 and 92.0 / 184.7 / 221.0 on the RTX 5090 for the three short-chat rows.

Credits and license

Quantized weights: ISTA-DASLab, Qwen3.8-27B GSQ-RCO GGUFs, produced with GSQ and RCO from Qwen/Qwen3.8-27B. DFlash2 adapter: z-lab. Tokenizer and frontend resources: Qwen/Qwen3.8-27B. All are Apache-2.0; their attribution notices are collected in NOTICE. This repository only re-packs those weights into NInfer's container.