ISTA-DASLab/gemma-2b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of google/gemma-2b.

9 commits

1 linked in READMEs

updated Mar 10, 2024

See the code

README

Official AQLM quantization of google/gemma-2b.

For this quantization, we used 1 codebook of 16 bits.

Results (0-shot acc):

Model	Quantization	WinoGrande	PiQA	HellaSwag	ArcE	ArcC	Model size, Gb
gemma-2b	None	0.6472	0.7715	0.5279	0.7403	0.4053	5.0
	1x16	0.6275	0.7318	0.4582	0.6923	0.3259	1.7

To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the official GitHub repo.

aqlm

endpoints_compatible

gemma

safetensors

text-generation

text-generation-inference

transformers

Similar repos

ISTA-DASLab/gemma-2b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of google/gemma-2b.

ISTA-DASLab/Mistral-7B-v0.1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of Mistral-7B-v0.1.

ISTA-DASLab/Mixtral-8x7B-Instruct-v0_1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-Instruct-v0.1.

ISTA-DASLab/Mistral-7B-v0.1-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Llama-2-13b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-13b-hf.

ISTA-DASLab/Llama-2-70b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-70b-hf.

ISTA-DASLab/Llama-2-13b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-13b-hf.

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge

ISTA-DASLab/gemma-2b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of google/gemma-2b.

9 commits

1 linked in READMEs

updated Mar 10, 2024

See the code

README

Official AQLM quantization of google/gemma-2b.

For this quantization, we used 1 codebook of 16 bits.

Results (0-shot acc):

Model	Quantization	WinoGrande	PiQA	HellaSwag	ArcE	ArcC	Model size, Gb
gemma-2b	None	0.6472	0.7715	0.5279	0.7403	0.4053	5.0
	1x16	0.6275	0.7318	0.4582	0.6923	0.3259	1.7

To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the official GitHub repo.

aqlm

endpoints_compatible

gemma

safetensors

text-generation

text-generation-inference

transformers

Similar repos

ISTA-DASLab/gemma-2b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of google/gemma-2b.

ISTA-DASLab/Mistral-7B-v0.1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of Mistral-7B-v0.1.

ISTA-DASLab/Mixtral-8x7B-Instruct-v0_1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-Instruct-v0.1.

ISTA-DASLab/Mistral-7B-v0.1-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Llama-2-13b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-13b-hf.

ISTA-DASLab/Llama-2-70b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-70b-hf.

ISTA-DASLab/Llama-2-13b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-13b-hf.

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge