Shadowgraph
TrendingBuzzResearch
Chrome extension
Sign in

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-8x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

0

12 commits

1 linked in READMEs

updated Mar 11, 2024

See the code

README

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

For this quantization, we used 2 codebooks of 8 bits.

Selected evaluation results for this and other models:

ModelAQLM schemeWikiText 2 PPLModel size, GbHub link
Llama-2-7b1x165.922.4Link
Llama-2-7b2x86.692.2Link
Llama-2-7b (THIS)8x86.612.2Link
Llama-2-13b1x165.224.1Link
Llama-2-70b1x163.8318.8Link
Llama-2-70b2x84.2118.2Link
Mixtral-8x7b1x163.3512.6Link
Mixtral-8x7b-Instruct1x16-12.6Link

To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the official GitHub repo.

8-bit
aqlm
endpoints_compatible
llama
safetensors
text-generation
text-generation-inference
transformers

Similar repos

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

2

ISTA-DASLab/Llama-2-70b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

1

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

5

ISTA-DASLab/Llama-2-70b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-70b-hf.

6

ISTA-DASLab/Llama-2-13b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-13b-hf.

0

ISTA-DASLab/Llama-2-13b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-13b-hf.

0

ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-v0.1.

23

ISTA-DASLab/Llama-2-7b-AQLM-PV-2Bit-2x8-hf

Model

An official quantization of meta-llama/Llama-2-7b using PV-Tuning on top of AQLM.

0

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-8x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

0

12 commits

1 linked in READMEs

updated Mar 11, 2024

See the code

README

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

For this quantization, we used 2 codebooks of 8 bits.

Selected evaluation results for this and other models:

ModelAQLM schemeWikiText 2 PPLModel size, GbHub link
Llama-2-7b1x165.922.4Link
Llama-2-7b2x86.692.2Link
Llama-2-7b (THIS)8x86.612.2Link
Llama-2-13b1x165.224.1Link
Llama-2-70b1x163.8318.8Link
Llama-2-70b2x84.2118.2Link
Mixtral-8x7b1x163.3512.6Link
Mixtral-8x7b-Instruct1x16-12.6Link

To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the official GitHub repo.

8-bit
aqlm
endpoints_compatible
llama
safetensors
text-generation
text-generation-inference
transformers

Similar repos

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

2

ISTA-DASLab/Llama-2-70b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

1

ISTA-DASLab/Llama-2-7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-7b-hf.

5

ISTA-DASLab/Llama-2-70b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-70b-hf.

6

ISTA-DASLab/Llama-2-13b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of meta-llama/Llama-2-13b-hf.

0

ISTA-DASLab/Llama-2-13b-AQLM-2Bit-2x8-hf

Model

Official AQLM quantization of meta-llama/Llama-2-13b-hf.

0

ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf

Model

Official AQLM quantization of mistralai/Mixtral-8x7B-v0.1.

23

ISTA-DASLab/Llama-2-7b-AQLM-PV-2Bit-2x8-hf

Model

An official quantization of meta-llama/Llama-2-7b using PV-Tuning on top of AQLM.

0

Linked in READMEs

Vahe1994/AQLM

Official Pytorch repository for Extreme Compression of Large Language Models via Additive…

1,330

README badge