Quantized LLM Model Distribution
25 repos
Efficient deployment of large language models through 4-bit quantization (GPTQ) and related compression techniques. This cluster focuses on making state-of-the-art models like Mistral, Llama, and specialized fine-tunes (WizardLM, Orca, SynthIA) runnable on consumer hardware by reducing model sizes while preserving performance. Repositories here deal with model serialization formats (safetensors), quantization pipelines, and the huggingface transformers infrastructure needed to serve these compressed models.
4-bit ·483
gptq ·483
transformers ·476
safetensors ·463
text-generation ·454
text-generation-inference ·377
llama ·358
en ·291
custom_code ·88
conversational ·81
