48 repos across 2 sub-areas
Techniques and implementations for accelerating large language model inference through speculative decoding, quantization, and efficient text generation. The cluster centers on optimized model variants (Qwen and Gemma series) using quantization formats like 4-bit and specialized inference frameworks, with a focus on reducing latency and memory overhead while maintaining generation quality. Repositories demonstrate practical applications of these optimization strategies across different model architectures and sizes.
Cluster 638877
39 repos
Qwen LLM Inference Optimization
9 repos
Optimized inference implementations and deployments of Alibaba's Qwen language models, with focus on techniques like speculative decoding, quantization (FP8), and hardware-specific acceleration for consumer GPUs (RTX 3090) and AMD ROCm platforms. The cluster centers on vLLM-based serving frameworks and model-specific optimizations that reduce latency and memory overhead for running large language models in resource-constrained environments.