Huihui-Qwen3.8-27B-abliterated, NInfer v3 artifact

A NInfer v3 artifact of huihui-ai/Huihui-Qwen3.8-27B-abliterated, the abliterated Qwen3.8-27B with the lowest reported divergence from the base checkpoint among the widely used releases (the first fifteen layers are untouched; independent KL divergence measurements sit around 0.008) and a working MTP head.

Built for NInfer-all, the master branch of iamwavecut/ninfer-all: ashalliants' NInfer-3090 consolidated with the fixes and features of the other NInfer forks. It serves the RTX 3090, RTX 4090, RTX 5090 and RTX PRO 6000 Blackwell, each with its measured kernel route profile built in (other GPUs calibrate once at first start).

What is inside

Conversion command (from the fork's tree, CUDA device):

python3 -m tools.convert \
  --model Huihui-Qwen3.8-27B-abliterated \
  --recipe qwen3_8_27b \
  --source dflash2=Qwen3.8-27B-DFlash2 \
  --components text,vision,mtp,dflash2 \
  --resource chat_template.jinja=tools/chat_templates/qwen3_8.jinja \
  --proposal \
  --name qwen3.8-27b-abliterated \
  --out Huihui-Qwen3.8-27B-abliterated-ninfer-v3.ninfer

SHA256SUMS lists the checksum of every file.

Running

A single RTX 3090 serves a 198,400-token window with Vision and MTP. From the Docker image (an NVIDIA driver of the CUDA 13 branch, 580 or newer, and the NVIDIA Container Toolkit):

hf download WaveCut/Huihui-Qwen3.8-27B-abliterated-NInfer-v3 \
  Huihui-Qwen3.8-27B-abliterated-ninfer-v3.ninfer --local-dir models
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all serve /models/Huihui-Qwen3.8-27B-abliterated-ninfer-v3.ninfer \
  --model-id qwen3.8-27b --max-context 198400 --kv-capacity 198400 --kv-dtype rk8v4 \
  --gdn-state-fp16 --spec mtp --draft-tokens 3 --lm-head-draft \
  --vision --vision-residency overlay --vision-max-merged 12288

The server answers OpenAI and Anthropic requests on http://localhost:8080/v1. A build of master (its README; CMAKE_CUDA_ARCHITECTURES is 86, 89 or 120a, and 120a needs CUDA 13.1 or newer) takes the same flags:

ninfer-serve models/Huihui-Qwen3.8-27B-abliterated-ninfer-v3.ninfer --model-id qwen3.8-27b \
  --max-context 198400 --kv-capacity 198400 --kv-dtype rk8v4 --gdn-state-fp16 \
  --spec mtp --draft-tokens 3 --lm-head-draft \
  --vision --vision-residency overlay --vision-max-merged 12288

The file uses only upstream's qwen3_8_27b recipe, so NInfer-3090 builds from v0.11.0 on read it as well.

The artifact also carries the DFlash2 adapter for --spec dflash2; only the MTP path was measured on this artifact.

On the current master

The September 2026 reference measurements of master (2172a598) cover the upstream neroued/Qwen3.8-27B-NInfer artifact, built with the same qwen3_8_27b recipe; this artifact was not measured again, and its drafts may be accepted at other rates than the upstream model's. One request, greedy, thinking off, text only:

Past 262,144 tokens the window runs with --rope-yarn. Filled to the largest rk4v4 window, the upstream model returned all three codes planted in the document on the RTX 3090, 4090 and 5090; at the RTX PRO 6000's 1,048,576 tokens it returned two of them. Every table is in the reference measurements.

Earlier measurements on an RTX 3090 (September 2026, franken-v0.11)

A Release build (sm_86, CUDA 13.1) of the fork's franken/v0.11 branch, now kept as the tag franken-v0.11, with the command line above plus --max-concurrency 4 --host-kv-mib 12288 --host-state-slots 8, greedy decoding, thinking off. The reference is WaveCut/Qwen3.8-27B-MXFP8-CRACK-NInfer-v3 served by the previous NInfer line of the same fork with the same flags and the same client.

Memory with one active request: 16.7 GiB of weights and 5.54 GiB of runtime state at the 198,400-token rk8v4 window, which leaves 1.12 GiB free on a 24 GiB card.

Decode speed is the median per request with MTP (three draft tokens, about 3.3 accepted per round on prose).

Quality over 1,179 items (GSM8K 200, MMLU-Pro 280, HumanEval 164, MGSM-ru 250, Global-MMLU-ru 285), the same items and client for every column:

Against the previous engine's CRACK run this artifact wins 46 items and loses 37 (exact McNemar p = 0.38); no suite differs significantly.

Perplexity on the quick corpus of ninfer-ppl-1m-v1 (261,167 scored tokens, 4,096-token window, 2,048-token stride, rk8v4 KV): 4.3750 for this artifact against 4.4685 for CRACK on the same engine.

Caveats

This is an abliterated model: refusals are removed at the checkpoint level, and the usual hesitation before destructive actions goes with them. Put the guardrails in the tool layer, not in the model. rk8v4 is a lossy KV format (about +0.08 % perplexity against bf16 on the vanilla checkpoint) chosen for the context window; --kv-dtype int8 or bf16 trade window for fidelity.