Speculative Decoding for LLM Inference

6 repos

Techniques and implementations for accelerating large language model text generation through speculative decoding, where a smaller draft model generates candidate tokens that a larger model verifies in parallel. This approach significantly reduces inference latency by batching verification steps. The cluster includes multiple model implementations (Kimi, GLM variants with Eagle3 draft models) and supporting infrastructure using formats like safetensors and frameworks like torchspec for efficient token generation and draft-model orchestration.

Related clusters