Gemma 4 E4B — ~3× lower KL than Google's official W4A16 QAT

Native QUASAR W4A16-G64 checkpoint for vLLM: 0.022 vs 0.064 KL to BF16, higher point estimates on all 8 downstream benchmarks, and +36% H100 output throughput in our vLLM configuration, versus Google's official W4A16 QAT.

Hard tokens are positions where BF16's top-1 probability is below 0.9. Same 512 held-out prompts and BF16 reference for both checkpoints.

vllm serve QUASAR-QAT/gemma-4-E4B-it-QUASAR-W4A16-G64 --dtype float16

Standard compressed-tensors W4A16. Text and image input; 128K context. For llama.cpp / Ollama / LM Studio, use the GGUF twin.

Faster serving

In our vLLM 0.28 test on one H100, group-64 dispatches to the Machete kernel while Google's group-32 goes to Marlin: 4,225 vs 3,111 output tokens/s at the same batch. Single-request latency is at parity.

Downstream benchmarks

Higher point estimates on all 8 downstream benchmarks than Google's QAT model.

Same harness and backend for both checkpoints; full configs, BF16 results and paired-bootstrap CIs are in EVAL.md.

Native QAT checkpoint

Native QAT checkpoint: the INT4 codes come directly from training, rather than a post-training re-quantization. export_receipt.json verifies the export per tensor. The GGUF twin carries the same codes and scales and ships a --verify script that checks the two files against each other.

Run it

• vLLM: vllm serve QUASAR-QAT/gemma-4-E4B-it-QUASAR-W4A16-G64 --dtype float16 (tested on vLLM 0.28.0 and 0.29.0; --dtype float16 keeps the stored fp16 scales exact), then any OpenAI-compatible client. Text and image inputs; 131,072-token context.
• Format: 258 decoder linears (q/k/v/o/gate/up/down) int4 symmetric, group_size 64, fp16 scales, compressed-tensors pack-quantized; embeddings, norms, per-layer-input tables and the vision/audio towers untouched (bf16). 10.2 GB.

QUASAR

QUASAR is loss-aware quantization-aware training. It improves the low-bit reconstruction during training, then exports to standard deployment formats with zero inference overhead.

This checkpoint was healed against the BF16 teacher for 626M tokens: one epoch over ~398k prompts self-distilled from BF16 Gemma 4 E4B-it with thinking on, fp32 master weights, INT4 group-64.

Related

• GGUF (llama.cpp / Ollama / LM Studio) twin of this checkpoint: QUASAR-QAT/gemma-4-E4B-it-QUASAR-Q4_0-GGUF
• Gemma 4 12B, same recipe: GGUF · W4A16 · collection · Gemma 4 E4B & 12B — QUASAR 4-bit QAT
• Base model: google/gemma-4-E4B-it · Paper: arXiv:2608.13966 · Org: huggingface.co/QUASAR-QAT

License

Based on Gemma 4 by Google DeepMind. Upstream LICENSE/NOTICE apply.

Citation

arxiv.org/abs/2608.13966

@article{counathe2026quasar,
  title={QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction},
  author={Counathe, Vincent and Athiwaratkun, Ben and De Sa, Christopher and Zhang, Tianyi},
  journal={arXiv preprint arXiv:2608.13966},
  year={2026}
}