ISTA-DASLab/Mistral-7B-Instruct-v0.2-AQLM-2Bit-2x8

Model

Official AQLM quantization of [mistralai/Mistral-7B-Instruct-v0.2

6 commits

1 linked in READMEs

updated Apr 13, 2024

See the code

README

Official AQLM quantization of mistralai/Mistral-7B-Instruct-v0.2 .

For this quantization, we used 2 codebooks of 8 bits.

Results:

Model	Quantization	MMLU (5-shot)	Model size, Gb
mistralai/Mistral-7B-Instruct-v0.2	None	0.5912	14.5
	2x8	0.4384	2.3

8-bit

aqlm

conversational

endpoints_compatible

finetuned

mistral

safetensors

text-generation

text-generation-inference

transformers

Similar repos

ISTA-DASLab/Mixtral-8x7B-Instruct-v0_1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-Instruct-v0.1.

ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-v0.1.

ISTA-DASLab/Mistral-7B-v0.1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of Mistral-7B-v0.1.

ISTA-DASLab/Mistral-7B-v0.1-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-8x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Meta-Llama-3-8B-Instruct-AQLM-2Bit-1x16

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3-8B-Instruct

ISTA-DASLab/Phi-3-medium-4k-instruct-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge

ISTA-DASLab/Mistral-7B-Instruct-v0.2-AQLM-2Bit-2x8

Model

Official AQLM quantization of [mistralai/Mistral-7B-Instruct-v0.2

6 commits

1 linked in READMEs

updated Apr 13, 2024

See the code

README

Official AQLM quantization of mistralai/Mistral-7B-Instruct-v0.2 .

For this quantization, we used 2 codebooks of 8 bits.

Results:

Model	Quantization	MMLU (5-shot)	Model size, Gb
mistralai/Mistral-7B-Instruct-v0.2	None	0.5912	14.5
	2x8	0.4384	2.3

8-bit

aqlm

conversational

endpoints_compatible

finetuned

mistral

safetensors

text-generation

text-generation-inference

transformers

Similar repos

ISTA-DASLab/Mixtral-8x7B-Instruct-v0_1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-Instruct-v0.1.

ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-v0.1.

ISTA-DASLab/Mistral-7B-v0.1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of Mistral-7B-v0.1.

ISTA-DASLab/Mistral-7B-v0.1-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-8x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Meta-Llama-3-8B-Instruct-AQLM-2Bit-1x16

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3-8B-Instruct

ISTA-DASLab/Phi-3-medium-4k-instruct-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge