EOQ: Entropy-Optimal Quantization for LLMs

PolarQuant -- optimal Gaussian quantization via Walsh-Hadamard rotation + Lloyd-Max centroids. Achieves near-lossless compression (PPL 6.39 vs 6.37 FP16) with 2.75x VRAM reduction.

arXiv preprint: arXiv:2603.7424577

---

PolarQuant v5: Final Results

Benchmark (Qwen3.5-9B, RTX PRO 6000 Blackwell)

PolarQuant Q5 + torchao INT4 is the recommended inference path: 43 tok/s at 6.5 GB VRAM with PPL 6.56 (+0.19 vs FP16). PolarQuant dequantized to FP16 achieves PPL 6.39 (+0.02), confirming near-lossless quantization quality.

Ablation Study (Q5, Qwen3.5-9B)

Hadamard rotation accounts for 98% of the improvement. The Walsh-Hadamard transform makes weight distributions approximately Gaussian, enabling near-optimal uniform quantization. Lloyd-Max centroids provide a small additional gain when combined with rotation, but are ineffective alone.

Apple Silicon (MLX)

PolarQuant runs natively on Apple Silicon via MLX with 4-bit quantization:

Model: caiovicentino1/Qwen3.5-9B-PolarQuant-MLX-4bit

---

How PolarQuant Works

FP16 weights --> Normalize (L2) --> Hadamard Rotate --> Lloyd-Max Quantize --> INT codes + norms
                                    (Gaussian dist)    (optimal centroids)

1. Block normalization: divide weight blocks by L2 norm
2. Walsh-Hadamard rotation: transforms weights to approximate N(0,1) distribution (this is 98% of the quality gain)
3. Lloyd-Max quantization: optimal centroids for Gaussian distribution (small additional gain)
4. Compact storage: INT codes (1 byte per weight) + FP16 per-block norms

For inference, two paths are supported:

• Dequant + torchao: Dequantize to FP16, apply torchao INT4 -- 43 tok/s, 6.5 GB VRAM, PPL 6.56 (recommended)
• PolarEngine Triton: Custom kernel keeps weights quantized -- 34 tok/s, 7.8 GB VRAM, PPL 6.89

---

EOQ Compression History

The project evolved through several generations:

EOQ v1-v3 Results (Qwen3.5-9B)

Multi-Model Results (EOQ Q5, RTX PRO 6000 Blackwell)

---

EOQ Dynamic (Mixed-Bit)

Inspired by Unsloth Dynamic 2.0, EOQ Dynamic assigns different bit widths per tensor based on quantization sensitivity:

Verified Results (Qwen3.5-9B)

---

PolarEngine v4: Quantized Inference

Custom Triton kernel that keeps weights quantized in GPU VRAM -- no dequantization to FP16 needed.

Benchmark (Qwen3.5-9B, RTX PRO 6000 Blackwell)

Key Optimizations

• Matmul FWHT: 25x faster Walsh-Hadamard via torch.matmul(x, H128) (1 kernel vs 29)
• FWHT cache: Q/K/V projections reuse same result (69x total speedup)
• Pre-scaled centroids: Lloyd-Max centroids x 1/sqrt(block_size) baked into lookup table
• INT4 nibble packing: Half-block order, halves VRAM for Q3/Q4 layers (36% savings)
• Triton tiled GEMV: Fused centroid lookup + dot product, autotuned per layer shape

vLLM Plugin

Install and use as a vLLM quantization method:

pip install git+https://github.com/caiovicentino/polarengine-vllm.git

# Quantize
python -m polarengine_vllm.quantize --model Qwen/Qwen3.5-9B --output ./polar-9b/

# Serve (when vLLM supports the plugin)
vllm serve ./polar-9b/ --quantization polarengine

See polarengine-vllm for full documentation.

---

Inference: Recommended Path

The recommended inference path is PolarQuant Q5 + torchao INT4:

Download PolarQuant Q5 (9.1 GB) --> Dequant to FP16 --> torchao INT4 --> 43 tok/s, 6.5 GB VRAM, PPL 6.56

GPU-Accelerated Loading

EOQ loads 5x faster than FP16 thanks to GPU-accelerated dequantization:

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from torchao.quantization import quantize_, Int4WeightOnlyConfig
import torch

# Load PolarQuant Q5 codes (auto-dequantizes to FP16)
model = AutoModelForCausalLM.from_pretrained(
    "caiovicentino1/Qwen3.5-9B-PolarQuant-Q5",
    dtype=torch.float16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("caiovicentino1/Qwen3.5-9B-PolarQuant-Q5")

# Apply torchao INT4 for fast inference
quantize_(model, Int4WeightOnlyConfig(group_size=128))

# Generate
inputs = tokenizer("Hello!", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))

---

Published Models

---

Experimental Findings

Eight experiments (A-H) were conducted to explore compression techniques:

---

Project Structure

dct-quantization/
├── core/              # EOQ pipeline: rANS, quantization, format
│   ├── rans.py        # rANS encoder/decoder (19/19 tests)
│   ├── rans_blocked.py # Blocked rANS with random access (8/8 tests)
│   ├── eoq.py         # EOQ compression pipeline
│   ├── eoq_format.py  # .eoq file format (23/23 tests)
│   ├── svd_hybrid.py  # W = Q + LR for sub-2.5 bpw
│   ├── quantized_linear.py  # INT4 packed nn.Module (47/47 tests)
│   └── weight_loader.py     # Universal HuggingFace loader
├── kernels/           # 16 CUDA kernel variants + PolarEngine
│   ├── k01-k16        # Shared memory, half2, uint4, multi-row, etc.
│   ├── k05_combined.py # Ultimate kernel (all optimizations)
│   └── polar_engine.py # PolarQuant inference kernel (Triton + torch)
├── llamacpp/          # C rANS decoder + GGUF converter
│   ├── eoq_rans.c     # C decoder (31/31 tests, 120 MB/s)
│   ├── eoq_ggml.h     # GGML integration header
│   └── eoq_convert.py # GGUF compression tool
├── llamacpp_integration/  # Full llama.cpp integration
│   ├── patches/       # 5 patches for llama.cpp
│   ├── eoq_rans_v2.c  # Optimized decoder (114 MB/s, 83/83 tests)
│   ├── eoq_cuda_decompress.cu  # GPU decompression
│   └── PR_DESCRIPTION.md
├── experiments/       # 8 validated experiments (A-H) + EOQ sweep
├── benchmarks/        # Perplexity, speed, memory profiling
├── tools/             # CLI: compress, decompress, chat server
├── notebooks/         # Colab notebooks
├── research/          # 19 literature review documents
└── tests/             # 200+ tests across all modules

llama.cpp Integration

EOQ can serve as a transport compression layer for GGUF files:

• Decode rANS at load time, use existing Q4_K kernels at runtime
• Zero runtime overhead
• 5 patches ready, C decoder (114 MB/s), CUDA decompressor
• See llamacpp_integration/PR_DESCRIPTION.md

Citation

@article{vicentino2026polarquant,
    title={PolarQuant: Near-Lossless LLM Quantization via Walsh-Hadamard Rotation
           and Entropy-Optimal Coding},
    author={Vicentino, Caio},
    journal={arXiv preprint arXiv:2603.7424577},
    year={2026}
}

Requirements

Python 3.10+, PyTorch 2.0+, transformers, numpy, scipy

pip install -r requirements.txt

License

Apache 2.0