ISTA-DASLab/Meta-Llama-3.1-70B-Instruct-AQLM-PV-2Bit-1x16

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-70B-Instruct

6 commits

1 linked in READMEs

updated Sep 17, 2024

See the code

README

Official AQLM quantization of meta-llama/Meta-Llama-3.1-70B-Instruct finetuned with PV-Tuning.

For this quantization, we used 1 codebook of 16 bits and groupsize of 8.

Results:

Model	Quantization	MMLU (5-shot)	ArcC	ArcE	Hellaswag	PiQA	Winogrande	Model size, Gb
	fp16	0.8213	0.6246	0.8683	0.6516	0.8313	0.7908	141
	1x16g8	0.7814	0.5478	0.8270	0.6284	0.8036	0.7814	21.9

aqlm

conversational

endpoints_compatible

facebook

llama

llama-3

Similar repos

ISTA-DASLab/Meta-Llama-3.1-8B-AQLM-PV-1Bit-1x16-hf

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-8B

ISTA-DASLab/Meta-Llama-3.1-8B-AQLM-PV-2Bit-1x16-hf

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-8B

ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3-70B-Instruct

ISTA-DASLab/Llama-3.2-1B-Instruct-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-1B

ISTA-DASLab/Meta-Llama-3-70B-AQLM-2Bit-1x16

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3-70B

ISTA-DASLab/Meta-Llama-3.1-8B-Instruct-AQLM-PV-2Bit-2x8-hf

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-8B-Instruct

ISTA-DASLab/Llama-3.2-1B-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-1B

ISTA-DASLab/Llama-3.2-3B-Instruct-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-3B

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge

ISTA-DASLab/Meta-Llama-3.1-70B-Instruct-AQLM-PV-2Bit-1x16

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-70B-Instruct

6 commits

1 linked in READMEs

updated Sep 17, 2024

See the code

README

Official AQLM quantization of meta-llama/Meta-Llama-3.1-70B-Instruct finetuned with PV-Tuning.

For this quantization, we used 1 codebook of 16 bits and groupsize of 8.

Results:

Model	Quantization	MMLU (5-shot)	ArcC	ArcE	Hellaswag	PiQA	Winogrande	Model size, Gb
	fp16	0.8213	0.6246	0.8683	0.6516	0.8313	0.7908	141
	1x16g8	0.7814	0.5478	0.8270	0.6284	0.8036	0.7814	21.9

aqlm

conversational

endpoints_compatible

facebook

llama

llama-3

Similar repos

ISTA-DASLab/Meta-Llama-3.1-8B-AQLM-PV-1Bit-1x16-hf

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-8B

ISTA-DASLab/Meta-Llama-3.1-8B-AQLM-PV-2Bit-1x16-hf

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-8B

ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3-70B-Instruct

ISTA-DASLab/Llama-3.2-1B-Instruct-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-1B

ISTA-DASLab/Meta-Llama-3-70B-AQLM-2Bit-1x16

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3-70B

ISTA-DASLab/Meta-Llama-3.1-8B-Instruct-AQLM-PV-2Bit-2x8-hf

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-8B-Instruct

ISTA-DASLab/Llama-3.2-1B-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-1B

ISTA-DASLab/Llama-3.2-3B-Instruct-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-3B

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge