Shadowgraph
TrendingBuzzResearch
Chrome extension
Sign in

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

2

14 commits

1 linked in READMEs

updated Mar 11, 2024

See the code

README

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

For this quantization, we used 2 codebooks of 8 bits.

Selected evaluation results for this and other models:

ModelAQLM schemeWikiText 2 PPLModel size, GbHub link
Llama-2-7b1x165.922.4Link
Llama-2-7b (THIS)2x86.692.2Link
Llama-2-7b8x86.612.2Link
Llama-2-13b1x165.224.1Link
Llama-2-70b1x163.8318.8Link
Llama-2-70b2x84.2118.2Link
Mixtral-8x7b1x163.3512.6Link
Mixtral-8x7b-Instruct1x16-12.6Link

UPD (20.02.2024). We applied global finetuning on top of quantized model and improved results compared to first revision.

To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the official GitHub repo.

8-bit
aqlm
endpoints_compatible
llama
safetensors
text-generation
text-generation-inference
transformers

Similar repos

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-8x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

0

ISTA-DASLab/Llama-2-70b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

1

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

5

ISTA-DASLab/Llama-2-70b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-70b-hf.

6

ISTA-DASLab/Llama-2-13b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-13b-hf.

0

ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-v0.1.

23

ISTA-DASLab/Llama-2-13b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-13b-hf.

0

ISTA-DASLab/Llama-2-7b-AQLM-PV-2Bit-2x8-hf

Model

An official quantization of meta-llama/Llama-2-7b using PV-Tuning on top of AQLM.

0

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

2

14 commits

1 linked in READMEs

updated Mar 11, 2024

See the code

README

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

For this quantization, we used 2 codebooks of 8 bits.

Selected evaluation results for this and other models:

ModelAQLM schemeWikiText 2 PPLModel size, GbHub link
Llama-2-7b1x165.922.4Link
Llama-2-7b (THIS)2x86.692.2Link
Llama-2-7b8x86.612.2Link
Llama-2-13b1x165.224.1Link
Llama-2-70b1x163.8318.8Link
Llama-2-70b2x84.2118.2Link
Mixtral-8x7b1x163.3512.6Link
Mixtral-8x7b-Instruct1x16-12.6Link

UPD (20.02.2024). We applied global finetuning on top of quantized model and improved results compared to first revision.

To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the official GitHub repo.

8-bit
aqlm
endpoints_compatible
llama
safetensors
text-generation
text-generation-inference
transformers

Similar repos

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-8x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

0

ISTA-DASLab/Llama-2-70b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

1

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

5

ISTA-DASLab/Llama-2-70b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-70b-hf.

6

ISTA-DASLab/Llama-2-13b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-13b-hf.

0

ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-v0.1.

23

ISTA-DASLab/Llama-2-13b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-13b-hf.

0

ISTA-DASLab/Llama-2-7b-AQLM-PV-2Bit-2x8-hf

Model

An official quantization of meta-llama/Llama-2-7b using PV-Tuning on top of AQLM.

0

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge