[!IMPORTANT] Naming notice (2026-04-10). The "HLWQ" technique used in this model is being rebranded to HLWQ (Hadamard-Lloyd Weight Quantization). The change is only the name; the algorithm and the weights in this repository are unchanged.

The rebrand resolves a name collision with an unrelated, earlier KV cache quantization method also named HLWQ (Han et al., arXiv:2502.02617, 2025). HLWQ addresses weight quantization with a deterministic Walsh-Hadamard rotation and Lloyd-Max scalar codebook; Han et al.'s HLWQ addresses KV cache quantization with a random polar rotation. The two methods are technically distinct.

Existing loaders that load this repository by ID continue to work without changes. Future model uploads will use the HLWQ name.

Reference paper for this technique: arXiv:2603.29078 (v2 in preparation; v1 still uses the old name).

⚡ HLWQ-Engine v4 -- Qwen3.5-9B

Custom Triton kernel inference for HLWQ -- 34 tok/s with direct quantized computation (no dequantization).

Note: For production use, we recommend HLWQ Q5 + torchao instead (43 tok/s vs 34 tok/s). HLWQ-Engine v4 is a research artifact demonstrating direct quantized inference without dequantization.

---

🎯 Key Results

---

📊 Performance Evolution

vs Recommended Approach

---

🔬 Architecture

HLWQ-Engine v4 performs inference directly on quantized weights without dequantization, using custom Triton kernels:

Input Activations
       |
       v
  [FWHT Cache] --> Hadamard Transform (cached per forward pass)
       |
       v
  [Triton GEMV] --> Centroid lookup + accumulate (fused kernel)
       |
       v
  [Scale by Norms] --> Output Activations

Key Optimizations (v3 -> v4)

Matmul FWHT: Replaced butterfly-algorithm FWHT with torch.matmul(x, H128) -- cuBLAS is faster than custom code for 128-dim transforms.

FWHT Cache: Q, K, V projections in attention share the same input activation. Cache by data_ptr to avoid redundant transforms. Auto-cleared between model.forward() calls via pre-hook.

---

🚀 Usage

# HLWQ-Engine v4 requires the polarengine-vllm package
pip install polarengine-vllm

from polarengine_vllm import HLWQizer
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the quantized model
model = AutoModelForCausalLM.from_pretrained(
    "caiovicentino1/Qwen3.5-9B-HLWQ-Engine-v4",
    dtype="bfloat16", device_map="auto", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("caiovicentino1/Qwen3.5-9B-HLWQ-Engine-v4")

output = model.generate(
    **tokenizer("What is machine learning?", return_tensors="pt").to("cuda"),
    max_new_tokens=200
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

---

🔧 Technical Details

Known Limitations

• Slower than torchao: cuBLAS INT4 matmul is highly optimized; custom Triton GEMV cannot yet match it
• Higher VRAM: Stores quantized weights + lookup tables (12.2 GB vs 6.5 GB)
• No SplitK: Large matrices (2048x8192) are bottlenecked without split-K parallel reduction
• Research code: Not recommended for production deployment

---

🔗 Links

• \U0001f4c4 Paper (arXiv) -- HLWQ: Optimal Gaussian Weight Quantization
• 💻 Code (GitHub) -- Full research codebase
• \U0001f50c vLLM Plugin -- Production inference integration
• \U0001f4e6 Recommended: HLWQ Q5 -- Faster, smaller, better quality

---

📖 Citation

@article{vicentino2026polarquant,
  title={HLWQ: Optimal Gaussian Weight Quantization via Hadamard Rotation for LLM Compression},
  author={Vicentino, Caio},
  journal={arXiv preprint arXiv:2603.7424577},
  year={2026}
}

---

🙏 Acknowledgements

Built with PyTorch, Triton, and the Qwen team's open-weight models.