A single-file NInfer V3 (.ninfer) artifact of
QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4,
built for an NVIDIA RTX 5090 (sm_120a), for Infernix
(recommended) and NInfer. No conversion step at load time:
point infernix-serve (or ninfer-serve) at the .ninfer file and run.
This repository is the quantized/derivative NInfer form of the QUASAR checkpoint above: the QAT-trained NVFP4 weights are carried over bit-exactly, and two speculative-decoding components (DFlash2 draft model, indexed proposal head) are added so the artifact runs with the engines' fastest decode paths.
Recommended engine: Infernix, my engine grown from NInfer, which replaces my earlier ninfer-custom fork. In an agentic coding replay on the 27B it served 90 % of prompt tokens from its prefix cache and reached the first token in about half NInfer's average time, and it prefills long prompts 1.3-1.5× faster, at the same quality (benchmarks, measured on NInfer's official 27B artifact). It also runs Qwen3.8-Flash-Next. The artifact still works with NInfer itself and with ninfer-custom.
| Component | Contents |
|---|---|
text | Qwen3.5-family hybrid backbone: 64 layers (Gated Delta-Net linear attention + full attention every 4th layer), hidden 5120, vocab 248,320, native context 262,144 |
vision | qwen3_5_vision tower (depth 27) for image/video input |
mtp | Qwen3.5 MTP companion for MTP speculative decoding |
dflash2 | 5-layer sliding-attention draft model (window 2048, selector rank 256 / top-16) — the --spec dflash2 path |
proposal | Indexed 131,072-row proposal head gathered from the QUASAR output head — enables cheap copy/proposal drafting via --lm-head-draft |
Format census of the 1,006 prepared weight objects: 256 NVFP4, 646 BF16, 6 Q8 (grouped absolute-max, scale-32, FP16 scales), 96 FP32, 1 Q4, 1 INT32 index table. (The file holds 1,590 objects in total; the remainder is the grafted dflash2 draft, kept in the reference artifact's original storage formats, plus small resource tables.)
The artifact was built with
tools/convert/quasar_nvfp4.py
from the master branch
of Wallawalla47/ninfer-custom
(qwen3_8_27b_nvfp4-quasar-proposal.ninfer.conversion.json, included here, is the full
per-tensor conversion report). Sources and treatment:
nvfp4-pack format) — the dominant source. Every quantised linear projection (GDN
query/key/value/z, MLP gate/up, full-attention projections — all 256 NVFP4 objects) is
imported in its encoded NVFP4 form, bit-exact (import_encoded): the QAT-trained
packed FP4 weights and their block scales are carried over unchanged, with no
requantisation round-trip. At runtime these drive NInfer's W4A4 NVFP4 kernels, which
dequantise weights on the fly against the stored scales and quantise activations in-kernel.qwen3_8_27b_nvfp4-official.ninfer, artifact id
3aec71e9…) — its dflash2 component is grafted verbatim into this artifact so the
DFlash2 speculative path runs on the QUASAR backbone.q4_g64_fp16, grouped_absmax)
plus an INT32 token-id table; this is what --lm-head-draft serves copy-heavy
proposals from.lm_head are requantised to
grouped Q8 (q8_g32_fp16): the engine's fused MTP op hard-requires Q8 with n=1024,
and because the MTP tower shares the main model's text/output_head, the main model's
logits move to the fast n248320_k5120 Q8 kernel as well. The GDN a/b projections were
NVFP4 in the source, but the v3 NVFP4 storage layout requires 128-row tiles and these
are 48-row tensors, so the converter dequantises them to BF16 — an upcast (the same
values the trained FP4 codes represent, stored in a wider format), not a
re-quantisation. The token embedding, norms, state initialisers and the vision tower
stay in their source BF16/FP32 formats.No tensor is dequantised and then re-quantised: the 256 NVFP4 objects are a direct import
of the trained packed weights, so the base model's quality is preserved by construction.
The only true re-quantisation is the BF16 → Q8 move for the MTP layer and shared
lm_head above (engine op requirement), and the one format change that loses nothing is
the GDN a/b NVFP4 → BF16 upcast.
Requires an NVIDIA RTX 50-series GPU (built and tuned on an RTX 5090; the weights take ~20 GB of VRAM). Prebuilt Windows executables of Infernix are on its releases page and need an NVIDIA driver of 580 or later; on Linux, build from source (quick start).
infernix-serve.exe qwen3_8_27b_nvfp4-quasar-proposal.ninfer --host 127.0.0.1 --port 8080 --max-context 240000 --max-concurrency 2 --spec dflash2 --draft-tokens 7 --lm-head-draft --ngram-draft-tokens 15 --ngram-min-match 12 --kv-dtype int8 --preserve-thinking --host-context-mib 52000 --pending-timeout-ms 900000 --prefill-chunk 4096 --kv-capacity auto --vram-headroom-mib 0 --log-colours on --ngram-archive-mib 2048 --ngram-session-mib 256 --ngram-native-sessions --request-log-jsonl log.json --default-thinking-budget 16384 --thinking-budget-message "Considering the limited time available to the user, I must stop thinking now. Time to act:" --tolerant-tool-calls
It runs DFlash2 speculative decoding with 7 draft tokens, the indexed proposal head
(--lm-head-draft) and n-gram copy drafting, int8 KV, and the hybrid prefix cache with a
52,000 MiB tier in system RAM (--host-context-mib). --kv-capacity auto sizes the KV pool to the
free VRAM. Lower --host-context-mib on a machine with less system RAM, and on a single 32 GB GPU
stop any other resident model first. Add --prefix-cache-file PATH to keep the prefix cache across
restarts. infernix-serve.exe --help lists every option.
With NInfer: the artifact also runs on NInfer and on
ninfer-custom with ninfer-serve and the same
command; drop any option that engine's ninfer-serve --help does not list.
Tested on 2026-10-09 on an RTX 5090 (Windows) with the Infernix windows-2026.10.09 release:
the command above (on another port) started with this artifact and served a chat request, a
streamed request, a tool call, and Anthropic Messages and Responses requests.
--vision. --vision-offload on keeps the vision tower in system RAM
instead of VRAM, which frees about 0.8 GiB of VRAM for the KV cache.--rope-yarn-factor F (F from 1 to 4,
up to 1M tokens).On this artifact, DFlash2 with 7 draft tokens decoded 36.3% faster on average than MTP with 4 draft tokens across 1k–240k contexts (A/B of 2026-09-17, on ninfer-custom).
The following is the README of the source checkpoint
QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4, copied from its source repository for reference.
library_name: transformers pipeline_tag: image-text-to-text license: apache-2.0 base_model: Qwen/Qwen3.8-27B base_model_relation: quantized tags:
19.7 GB · 496/496 transformer linears in NVFP4 (W4A4) · 0.909 GPQA-Diamond vs. 0.914 BF16 · serves directly with vLLM.
Among the public NVFP4 builds compared below, QUASAR is both the smallest and the highest-scoring: 90.9 GPQA-Diamond and 100% AIME'26.
Independent evaluation: A third-party NVFP4 shootout on the official Qwen3.8-27B repo compares QUASAR against other public NVFP4 checkpoints under the same evaluation and serving setup.
Smaller sibling — Qwen3.5-4B: W4A16 / vLLM · W4A4 / Blackwell · Q4_0 GGUF / llama.cpp · collection · paper
More QUASAR checkpoints: Gemma 4 collection · Muse-Glimmer collection · paper.
QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 is a 4-bit NVFP4 version of
Qwen/Qwen3.8-27B, produced with
QUASAR, a quantization-aware training (QAT) method.
QUASAR trains the NVFP4 weights directly against the frozen BF16 model, then exports standard NVFP4 weights with no custom inference path. This lets us quantize all 496 transformer linears—including attention and gated delta-net—to NVFP4 while preserving near-BF16 quality: the smallest of the compared public NVFP4 builds, while scoring highest among them on both GPQA-Diamond and AIME'26.
📄 Paper: QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction
Compatible with vLLM, with no conversion step:
pip install "vllm>=0.27"
vllm serve QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 \
--max-model-len 262144 \
--gpu-memory-utilization 0.85 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'
On a 32 GB card such as an RTX 5090, lower the context to --max-model-len 65536.
Requires an NVIDIA GPU with FP4 support (Blackwell, compute capability 10.0+).
Two other public NVFP4 builds of this model, evaluated under the same setup (GPQA-Diamond: 2 runs, n=396; AIME'26: 3 repeats, n=90). Bold = best among the NVFP4 builds.
| Model | Size | NVFP4 linears | GPQA-D | AIME'26 |
|---|---|---|---|---|
| QUASAR (this model) | 19.7 GB | 496/496 | 90.91 | 100.0 |
| BF16 original | 55.6 GB | — | 91.41 | 100.0 |
| Unsloth NVFP4 | 23.4 GB | 168/496 | 89.39 | 97.78 |
| Inferact NVFP4 | 26.4 GB | 304/496 | 87.63 | 96.67 |
One epoch of loss-aware NVFP4 quantization-aware distillation against the frozen BF16 teacher: global batch size 32, learning rate 1e-6, 2446 steps.
@article{counathe2026quasar,
title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction},
author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi},
journal={arXiv preprint arXiv:2608.13966},
year={2026}
}
A single-file NInfer V3 (.ninfer) artifact of
QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4,
built for an NVIDIA RTX 5090 (sm_120a), for Infernix
(recommended) and NInfer. No conversion step at load time:
point infernix-serve (or ninfer-serve) at the .ninfer file and run.
This repository is the quantized/derivative NInfer form of the QUASAR checkpoint above: the QAT-trained NVFP4 weights are carried over bit-exactly, and two speculative-decoding components (DFlash2 draft model, indexed proposal head) are added so the artifact runs with the engines' fastest decode paths.
Recommended engine: Infernix, my engine grown from NInfer, which replaces my earlier ninfer-custom fork. In an agentic coding replay on the 27B it served 90 % of prompt tokens from its prefix cache and reached the first token in about half NInfer's average time, and it prefills long prompts 1.3-1.5× faster, at the same quality (benchmarks, measured on NInfer's official 27B artifact). It also runs Qwen3.8-Flash-Next. The artifact still works with NInfer itself and with ninfer-custom.
| Component | Contents |
|---|---|
text | Qwen3.5-family hybrid backbone: 64 layers (Gated Delta-Net linear attention + full attention every 4th layer), hidden 5120, vocab 248,320, native context 262,144 |
vision | qwen3_5_vision tower (depth 27) for image/video input |
mtp | Qwen3.5 MTP companion for MTP speculative decoding |
dflash2 | 5-layer sliding-attention draft model (window 2048, selector rank 256 / top-16) — the --spec dflash2 path |
proposal | Indexed 131,072-row proposal head gathered from the QUASAR output head — enables cheap copy/proposal drafting via --lm-head-draft |
Format census of the 1,006 prepared weight objects: 256 NVFP4, 646 BF16, 6 Q8 (grouped absolute-max, scale-32, FP16 scales), 96 FP32, 1 Q4, 1 INT32 index table. (The file holds 1,590 objects in total; the remainder is the grafted dflash2 draft, kept in the reference artifact's original storage formats, plus small resource tables.)
The artifact was built with
tools/convert/quasar_nvfp4.py
from the master branch
of Wallawalla47/ninfer-custom
(qwen3_8_27b_nvfp4-quasar-proposal.ninfer.conversion.json, included here, is the full
per-tensor conversion report). Sources and treatment:
nvfp4-pack format) — the dominant source. Every quantised linear projection (GDN
query/key/value/z, MLP gate/up, full-attention projections — all 256 NVFP4 objects) is
imported in its encoded NVFP4 form, bit-exact (import_encoded): the QAT-trained
packed FP4 weights and their block scales are carried over unchanged, with no
requantisation round-trip. At runtime these drive NInfer's W4A4 NVFP4 kernels, which
dequantise weights on the fly against the stored scales and quantise activations in-kernel.qwen3_8_27b_nvfp4-official.ninfer, artifact id
3aec71e9…) — its dflash2 component is grafted verbatim into this artifact so the
DFlash2 speculative path runs on the QUASAR backbone.q4_g64_fp16, grouped_absmax)
plus an INT32 token-id table; this is what --lm-head-draft serves copy-heavy
proposals from.lm_head are requantised to
grouped Q8 (q8_g32_fp16): the engine's fused MTP op hard-requires Q8 with n=1024,
and because the MTP tower shares the main model's text/output_head, the main model's
logits move to the fast n248320_k5120 Q8 kernel as well. The GDN a/b projections were
NVFP4 in the source, but the v3 NVFP4 storage layout requires 128-row tiles and these
are 48-row tensors, so the converter dequantises them to BF16 — an upcast (the same
values the trained FP4 codes represent, stored in a wider format), not a
re-quantisation. The token embedding, norms, state initialisers and the vision tower
stay in their source BF16/FP32 formats.No tensor is dequantised and then re-quantised: the 256 NVFP4 objects are a direct import
of the trained packed weights, so the base model's quality is preserved by construction.
The only true re-quantisation is the BF16 → Q8 move for the MTP layer and shared
lm_head above (engine op requirement), and the one format change that loses nothing is
the GDN a/b NVFP4 → BF16 upcast.
Requires an NVIDIA RTX 50-series GPU (built and tuned on an RTX 5090; the weights take ~20 GB of VRAM). Prebuilt Windows executables of Infernix are on its releases page and need an NVIDIA driver of 580 or later; on Linux, build from source (quick start).
infernix-serve.exe qwen3_8_27b_nvfp4-quasar-proposal.ninfer --host 127.0.0.1 --port 8080 --max-context 240000 --max-concurrency 2 --spec dflash2 --draft-tokens 7 --lm-head-draft --ngram-draft-tokens 15 --ngram-min-match 12 --kv-dtype int8 --preserve-thinking --host-context-mib 52000 --pending-timeout-ms 900000 --prefill-chunk 4096 --kv-capacity auto --vram-headroom-mib 0 --log-colours on --ngram-archive-mib 2048 --ngram-session-mib 256 --ngram-native-sessions --request-log-jsonl log.json --default-thinking-budget 16384 --thinking-budget-message "Considering the limited time available to the user, I must stop thinking now. Time to act:" --tolerant-tool-calls
It runs DFlash2 speculative decoding with 7 draft tokens, the indexed proposal head
(--lm-head-draft) and n-gram copy drafting, int8 KV, and the hybrid prefix cache with a
52,000 MiB tier in system RAM (--host-context-mib). --kv-capacity auto sizes the KV pool to the
free VRAM. Lower --host-context-mib on a machine with less system RAM, and on a single 32 GB GPU
stop any other resident model first. Add --prefix-cache-file PATH to keep the prefix cache across
restarts. infernix-serve.exe --help lists every option.
With NInfer: the artifact also runs on NInfer and on
ninfer-custom with ninfer-serve and the same
command; drop any option that engine's ninfer-serve --help does not list.
Tested on 2026-10-09 on an RTX 5090 (Windows) with the Infernix windows-2026.10.09 release:
the command above (on another port) started with this artifact and served a chat request, a
streamed request, a tool call, and Anthropic Messages and Responses requests.
--vision. --vision-offload on keeps the vision tower in system RAM
instead of VRAM, which frees about 0.8 GiB of VRAM for the KV cache.--rope-yarn-factor F (F from 1 to 4,
up to 1M tokens).On this artifact, DFlash2 with 7 draft tokens decoded 36.3% faster on average than MTP with 4 draft tokens across 1k–240k contexts (A/B of 2026-09-17, on ninfer-custom).
The following is the README of the source checkpoint
QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4, copied from its source repository for reference.
library_name: transformers pipeline_tag: image-text-to-text license: apache-2.0 base_model: Qwen/Qwen3.8-27B base_model_relation: quantized tags:
19.7 GB · 496/496 transformer linears in NVFP4 (W4A4) · 0.909 GPQA-Diamond vs. 0.914 BF16 · serves directly with vLLM.
Among the public NVFP4 builds compared below, QUASAR is both the smallest and the highest-scoring: 90.9 GPQA-Diamond and 100% AIME'26.
Independent evaluation: A third-party NVFP4 shootout on the official Qwen3.8-27B repo compares QUASAR against other public NVFP4 checkpoints under the same evaluation and serving setup.
Smaller sibling — Qwen3.5-4B: W4A16 / vLLM · W4A4 / Blackwell · Q4_0 GGUF / llama.cpp · collection · paper
More QUASAR checkpoints: Gemma 4 collection · Muse-Glimmer collection · paper.
QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 is a 4-bit NVFP4 version of
Qwen/Qwen3.8-27B, produced with
QUASAR, a quantization-aware training (QAT) method.
QUASAR trains the NVFP4 weights directly against the frozen BF16 model, then exports standard NVFP4 weights with no custom inference path. This lets us quantize all 496 transformer linears—including attention and gated delta-net—to NVFP4 while preserving near-BF16 quality: the smallest of the compared public NVFP4 builds, while scoring highest among them on both GPQA-Diamond and AIME'26.
📄 Paper: QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction
Compatible with vLLM, with no conversion step:
pip install "vllm>=0.27"
vllm serve QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 \
--max-model-len 262144 \
--gpu-memory-utilization 0.85 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'
On a 32 GB card such as an RTX 5090, lower the context to --max-model-len 65536.
Requires an NVIDIA GPU with FP4 support (Blackwell, compute capability 10.0+).
Two other public NVFP4 builds of this model, evaluated under the same setup (GPQA-Diamond: 2 runs, n=396; AIME'26: 3 repeats, n=90). Bold = best among the NVFP4 builds.
| Model | Size | NVFP4 linears | GPQA-D | AIME'26 |
|---|---|---|---|---|
| QUASAR (this model) | 19.7 GB | 496/496 | 90.91 | 100.0 |
| BF16 original | 55.6 GB | — | 91.41 | 100.0 |
| Unsloth NVFP4 | 23.4 GB | 168/496 | 89.39 | 97.78 |
| Inferact NVFP4 | 26.4 GB | 304/496 | 87.63 | 96.67 |
One epoch of loss-aware NVFP4 quantization-aware distillation against the frozen BF16 teacher: global batch size 32, learning rate 1e-6, 2446 steps.
@article{counathe2026quasar,
title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction},
author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi},
journal={arXiv preprint arXiv:2608.13966},
year={2026}
}