ISTA-DASLab/Mistral-7B-v0.1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of Mistral-7B-v0.1.

3 commits

1 linked in READMEs

updated Mar 11, 2024

See the code

README

Official AQLM quantization of Mistral-7B-v0.1.

For this quantization, we used 1 codebook of 16 bits.

Results (0-shot acc):

Model	Quantization	WinoGrande	PiQA	HellaSwag	ArcE	ArcC	Model size, Gb
Mistral-7B-v0.1	None	0.7364	0.8047	0.6115	0.7887	0.4923	14.5
	1x16	0.6914	0.7845	0.5745	0.7504	0.4420	2.51

To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the official GitHub repo.

aqlm

endpoints_compatible

mistral

safetensors

text-generation

text-generation-inference

transformers

Similar repos

ISTA-DASLab/Mistral-7B-v0.1-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

ISTA-DASLab/Mixtral-8x7B-Instruct-v0_1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-Instruct-v0.1.

ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-v0.1.

ISTA-DASLab/Mistral-7B-Instruct-v0.2-AQLM-2Bit-2x8

Model

Official AQLM quantization of [mistralai/Mistral-7B-Instruct-v0.2

ISTA-DASLab/gemma-2b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of google/gemma-2b.

ISTA-DASLab/Phi-3-mini-4k-instruct-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Phi-3-medium-4k-instruct-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge

ISTA-DASLab/Mistral-7B-v0.1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of Mistral-7B-v0.1.

3 commits

1 linked in READMEs

updated Mar 11, 2024

See the code

README

Official AQLM quantization of Mistral-7B-v0.1.

For this quantization, we used 1 codebook of 16 bits.

Results (0-shot acc):

Model	Quantization	WinoGrande	PiQA	HellaSwag	ArcE	ArcC	Model size, Gb
Mistral-7B-v0.1	None	0.7364	0.8047	0.6115	0.7887	0.4923	14.5
	1x16	0.6914	0.7845	0.5745	0.7504	0.4420	2.51

To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the official GitHub repo.

aqlm

endpoints_compatible

mistral

safetensors

text-generation

text-generation-inference

transformers

Similar repos

ISTA-DASLab/Mistral-7B-v0.1-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

ISTA-DASLab/Mixtral-8x7B-Instruct-v0_1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-Instruct-v0.1.

ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-v0.1.

ISTA-DASLab/Mistral-7B-Instruct-v0.2-AQLM-2Bit-2x8

Model

Official AQLM quantization of [mistralai/Mistral-7B-Instruct-v0.2

ISTA-DASLab/gemma-2b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of google/gemma-2b.

ISTA-DASLab/Phi-3-mini-4k-instruct-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Phi-3-medium-4k-instruct-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge