ISTA-DASLab/Mixtral-8x7B-Instruct-v0_1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-Instruct-v0.1.

7 commits

1 linked in READMEs

updated Feb 27, 2024

See the code

README

Official AQLM quantization of mistralai/Mixtral-8x7B-Instruct-v0.1.

For this quantization, we used 1 codebook of 16 bits.

Selected evaluation results for this model:

Model	AQLM scheme	WinoGrande	PiQA	HellaSwag	ArcE	ArcC	Model size, Gb	Hub link
Mixtral-8x7B-Instruct-v0.1 (THIS)	1x16	0.7593	0.8043	0.6179	0.7768	0.4793	12.6	Link

To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the official GitHub repo.

aqlm

conversational

endpoints_compatible

mixtral

safetensors

text-generation

text-generation-inference

transformers

Similar repos

ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-v0.1.

ISTA-DASLab/Mistral-7B-v0.1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of Mistral-7B-v0.1.

ISTA-DASLab/Mistral-7B-Instruct-v0.2-AQLM-2Bit-2x8

Model

Official AQLM quantization of [mistralai/Mistral-7B-Instruct-v0.2

ISTA-DASLab/Mistral-7B-v0.1-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-8x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Llama-2-70b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge

ISTA-DASLab/Mixtral-8x7B-Instruct-v0_1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-Instruct-v0.1.

7 commits

1 linked in READMEs

updated Feb 27, 2024

See the code

README

Official AQLM quantization of mistralai/Mixtral-8x7B-Instruct-v0.1.

For this quantization, we used 1 codebook of 16 bits.

Selected evaluation results for this model:

Model	AQLM scheme	WinoGrande	PiQA	HellaSwag	ArcE	ArcC	Model size, Gb	Hub link
Mixtral-8x7B-Instruct-v0.1 (THIS)	1x16	0.7593	0.8043	0.6179	0.7768	0.4793	12.6	Link

To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the official GitHub repo.

aqlm

conversational

endpoints_compatible

mixtral

safetensors

text-generation

text-generation-inference

transformers

Similar repos

ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-v0.1.

ISTA-DASLab/Mistral-7B-v0.1-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of Mistral-7B-v0.1.

ISTA-DASLab/Mistral-7B-Instruct-v0.2-AQLM-2Bit-2x8

Model

Official AQLM quantization of [mistralai/Mistral-7B-Instruct-v0.2

ISTA-DASLab/Mistral-7B-v0.1-AQLM-PV-2Bit-1x16-hf

Model

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-8x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

ISTA-DASLab/Llama-2-70b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge