llm-qat

This repository contains the training code of LLM-QAT introduced in our work: "LLM-QAT: Data-Free Quantization Aware Training for Large Language Models"

In this work, we investigate quantization-aware training for LLMs (LLM-QAT). In addition to quantizing weights and activations, we also quantize the KV cache, which is critical for increasing throughput and support long sequence dependencies at current model sizes. We experiment with LLaMA models of sizes 7B, 13B, and 30B, at quantization levels down to 4-bits. We observe up to ~20 points improvement over training-free methods when quantizing weight, activations and kv cache to 4-bit, 8-bit and 4-bit, respectively.

News

• Mar 3, 2025: 🚀 We released a new QAT framework: ParetoQ, which achieves the state-of-the-art performance on binary/ternary/2-bit/3-bit/4-bit quantization! [paper] [code]

Citation

If you find our code useful for your research, please consider citing:

@article{liu2023llm,
    title={LLM-QAT: Data-Free Quantization Aware Training for Large Language Models},
    author={Liu, Zechun and Oguz, Barlas and Zhao, Changsheng and Chang, Ernie and Stock, Pierre and Mehdad, Yashar and Shi, Yangyang and Krishnamoorthi, Raghuraman and Chandra, Vikas},
    journal={arXiv preprint arXiv:2305.17888},
    year={2023}
}

Run

1. Requirements:

• python 3.9, pytorch >= 1.13
• pip install -r requirement.txt
• Install apex from source (https://github.com/NVIDIA/apex)

2. Steps to run:

(1) Synthesize data:

• Download the llama-7B model from huggingface. Find it in the huggingface cache and update the path in 'generate_data.py'
• Run python generate_data.py i Here i is the GPU id, ranging from 0 to 63, because we use 64 GPUs to synthesize data in parallel. You can also change 'n_vocab 'in your code to adjust the degree of parallelism.
• Run python merge_gen_data.py to merge all the generated data in one .jsonl file.

(2) Quantization-aware training:

• Specify the data path and the pre-trained model path in scrips/run.sh file.
• Run bash run_train.sh $w_bit $a_bit $kv_bit E.g. bash run_train.sh 4 8 4 for 4-bit weight 8-bit activation and 4-bit kv-cache.

Quantized LLaMA-7B Models

The results reported in the paper is run with the internal LLaMA codebase in Meta. We reproduced our experiments with huggingface codebase and released code here. The results are close to those in the paper. For clearity, we list the zero-shot common sense reasoning accuracy of the opensourced version in the following table.

Acknowledgement

This code is partially based on HuggingFace transformer repo.

Contact

Zechun Liu, Reality Labs, Meta Inc (zechunliu at meta dot com)

Barlas Oguz, Meta AI (barlaso at meta dot com)

Changsheng Zhao, Reality Labs, Meta Inc (cszhao at meta dot com)

Relevant Projects

MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases [Paper] [Code]

SpinQuant: LLM Quantization with Learned Rotations [Paper] [Code]

License

BiT is CC-BY-NC 4.0 licensed as of now.