48 repos across 2 sub-areas
C++ libraries and tools for efficient large language model inference, centered on the GGML tensor library and the llama.cpp project. This cluster covers quantization, model loading, and optimized inference for LLMs across different hardware platforms, with supporting Python bindings and utilities for running models locally. Developers here focus on making LLMs practical and accessible through efficient, portable implementations.
Cluster 635319
28 repos
LLM Inference Optimization & Quantization
20 repos
C++ implementations and optimizations for running large language models locally, with a focus on inference acceleration, memory efficiency, and quantization techniques. The cluster centers on llama.cpp and its variants, exploring adaptive key-value caching, turboquantization, and hybrid inference strategies to reduce computational and memory overhead while maintaining model quality.