49 repos across 3 sub-areas
Libraries, models, and frameworks for efficient large language model inference and text generation at scale. The cluster centers on techniques like speculative decoding and optimized tensor operations (via safetensors), with a focus on making transformer-based models faster and more practical to deploy. Primary repos include Qwen-series models optimized for inference performance, alongside text-generation-inference frameworks and related transformer utilities for production applications.
Cluster 462276
31 repos
LLM Inference Optimization
15 repos
Techniques and implementations for accelerating large language model inference through methods like speculative decoding, quantization, and GPU optimization. The cluster centers on Qwen model variants (particularly 27B parameters) optimized for consumer GPUs via tools like DFlash2 and NVIDIA FP4 quantization, alongside broader frameworks for deploying and experimenting with optimized inference pipelines locally. Most repositories are Python-based with CUDA kernel implementations, reflecting the practical focus on making capable models run faster and cheaper on real hardware.
Cluster 462277
3 repos