Shadowgraph
TrendingBuzzResearch
Chrome extension
Sign in

ISTA-DASLab/Meta-Llama-3.1-8B-AQLM-PV-2Bit-1x16-hf

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-8B

3

5 commits

1 linked in READMEs

updated Aug 9, 2024

See the code

README

Official AQLM quantization of meta-llama/Meta-Llama-3.1-8B finetuned with PV-Tuning.

For this quantization, we used 1 codebook of 16 bits and groupsize of 8.

Results:

ModelQuantizationMMLU (5-shot)ArcCArcEHellaswagPiQAWinograndeModel size, Gb
meta-llama/Meta-Llama-3.1-8BNone0.65210.51450.81440.59980.80140.735616.1
1x16g80.57460.44540.77440.56290.79540.70244.1

Note

We used lm-eval=0.4.0 for evaluation.

UPD (09.08.2024)

Uploaded new version finetuned on more data for longer with better quality.

aqlm
conversational
endpoints_compatible
facebook
llama
llama-3
meta
safetensors
text-generation
text-generation-inference
transformers

Similar repos

ISTA-DASLab/Meta-Llama-3.1-8B-AQLM-PV-1Bit-1x16-hf

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-8B

1

ISTA-DASLab/Meta-Llama-3.1-8B-Instruct-AQLM-PV-2Bit-2x8-hf

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-8B-Instruct

2

ISTA-DASLab/Meta-Llama-3.1-70B-Instruct-AQLM-PV-2Bit-1x16

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-70B-Instruct

45

ISTA-DASLab/Meta-Llama-3-8B-AQLM-2Bit-1x16

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3-8B

8

ISTA-DASLab/Llama-3.2-1B-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-1B

0

ISTA-DASLab/Llama-3.2-3B-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-3B

0

ISTA-DASLab/Llama-3.2-1B-Instruct-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-1B

1

ISTA-DASLab/Llama-3.2-3B-Instruct-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-3B

0

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge

ISTA-DASLab/Meta-Llama-3.1-8B-AQLM-PV-2Bit-1x16-hf

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-8B

3

5 commits

1 linked in READMEs

updated Aug 9, 2024

See the code

README

Official AQLM quantization of meta-llama/Meta-Llama-3.1-8B finetuned with PV-Tuning.

For this quantization, we used 1 codebook of 16 bits and groupsize of 8.

Results:

ModelQuantizationMMLU (5-shot)ArcCArcEHellaswagPiQAWinograndeModel size, Gb
meta-llama/Meta-Llama-3.1-8BNone0.65210.51450.81440.59980.80140.735616.1
1x16g80.57460.44540.77440.56290.79540.70244.1

Note

We used lm-eval=0.4.0 for evaluation.

UPD (09.08.2024)

Uploaded new version finetuned on more data for longer with better quality.

aqlm
conversational
endpoints_compatible
facebook
llama
llama-3
meta
safetensors
text-generation
text-generation-inference
transformers

Similar repos

ISTA-DASLab/Meta-Llama-3.1-8B-AQLM-PV-1Bit-1x16-hf

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-8B

1

ISTA-DASLab/Meta-Llama-3.1-8B-Instruct-AQLM-PV-2Bit-2x8-hf

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-8B-Instruct

2

ISTA-DASLab/Meta-Llama-3.1-70B-Instruct-AQLM-PV-2Bit-1x16

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3.1-70B-Instruct

45

ISTA-DASLab/Meta-Llama-3-8B-AQLM-2Bit-1x16

Model

Official AQLM quantization of [meta-llama/Meta-Llama-3-8B

8

ISTA-DASLab/Llama-3.2-1B-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-1B

0

ISTA-DASLab/Llama-3.2-3B-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-3B

0

ISTA-DASLab/Llama-3.2-1B-Instruct-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-1B

1

ISTA-DASLab/Llama-3.2-3B-Instruct-AQLM-PV-2Bit-2x8

Model

Official AQLM quantization of [meta-llama/Llama-3.2-3B

0

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge