Qwen3.5-4B — native Q4_0 QAT for llama.cpp

Project page · Code

Native Q4_0 QAT for Qwen3.5-4B: 17–22% lower mean KL than the tested bartowski and Unsloth Q4_0 builds (91.3% top-1 agreement vs 90.4% / 90.2%), 35% lower than the other QAT Q4_0 tested (YoozLabs, 88.4%), and it beats a larger Q4_K_M of the BF16 original: 0.235 vs 0.319 KL.

2.70 GB · 200/200 decoder projections trained directly on Q4_0 · trained codes packed without re-rounding · vision projector · optional MTP · stock llama.cpp / Ollama / LM Studio.

QUASAR family: NVFP4 W4A16 / vLLM · NVFP4 W4A4 / Blackwell · Q4_0 GGUF / llama.cpp (this repo) · collection · QUASAR paper

Run it

ollama run hf.co/QUASAR-QAT/Qwen3.5-4B-QUASAR-Q4_0-GGUF:Q4_0

llama-cli -hf QUASAR-QAT/Qwen3.5-4B-QUASAR-Q4_0-GGUF:Q4_0

LM Studio: search QUASAR-QAT Qwen3.5-4B. For image input, load mmproj-Qwen3.5-4B-F16.gguf alongside (llama-server -m Qwen3.5-4B-QUASAR-Q4_0.gguf --mmproj mmproj-Qwen3.5-4B-F16.gguf).

Files

Same-harness Q4_0 comparison

llama-perplexity --kl-divergence against the BF16 original (F16 GGUF) on 774 held-out prompts rendered with the chat template, 4096-token chunks. Lower KL and higher top-1 agreement = closer to the original. Full evaluation and reproduction details → EVAL.md.

QUASAR has the lowest KL and highest top-1 agreement of the Q4_0 files tested — PTQ and QAT alike — and also beats the larger Q4_K_M reference while using 3% fewer bytes. Unlike the other QAT file tested, which was trained on a group-64 lattice and then re-quantized to Q4_0, QUASAR is trained directly on the Q4_0 lattice and packs the trained codes without a second rounding pass.

What is quantized

Q4_0 on all 200 decoder projections; Q8_0 embeddings in the default file; norms, conv1d kernels and small delta-net gate vectors remain F16. The GGUF stores QUASAR's trained INT4 codes and scales directly — no llama-quantize re-rounding — and the packed blocks were verified byte-for-byte against the training export.

Training

The 200 decoder projections were trained directly on the Q4_0 lattice with QUASAR using self-distillation from the frozen BF16 model. Training reduced teacher KL from .0383 with plain Q4_0 rounding to .0142.

Citation

@article{counathe2026quasar,
  title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction},
  author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi},
  journal={arXiv preprint arXiv:2608.13966},
  year={2026}
}