This repository gathers work around different llm quantization methods. Study include the following framework: LLM.int8() (HF), QLoRA (HF), Quanto (HF), llama.cpp (via python bindings), AutoGPTQ (HF & pure), AutoAWQ (HF & pure), AQLM (HF) and LoftQ (HF)
Some results are available in results_ppl.txt
See benchmark: https://docs.google.com/spreadsheets/d/13YbklQfTJMaBNH2Oo9xorFc2Liil5CDITXYePiG1gvA/edit?usp=sharing For additional notes, see: https://docs.google.com/document/d/1pvBRsOkU0ml733xnc_1iEsry7HwysyeDImyY3ZAsZQY/edit?usp=sharing
This repository gathers work around different llm quantization methods. Study include the following framework: LLM.int8() (HF), QLoRA (HF), Quanto (HF), llama.cpp (via python bindings), AutoGPTQ (HF & pure), AutoAWQ (HF & pure), AQLM (HF) and LoftQ (HF)
Some results are available in results_ppl.txt
See benchmark: https://docs.google.com/spreadsheets/d/13YbklQfTJMaBNH2Oo9xorFc2Liil5CDITXYePiG1gvA/edit?usp=sharing For additional notes, see: https://docs.google.com/document/d/1pvBRsOkU0ml733xnc_1iEsry7HwysyeDImyY3ZAsZQY/edit?usp=sharing
