LLM Inference Optimization & Serving

23 repos

Frameworks and tools for efficiently serving and running large language models in production, with emphasis on inference optimization, speculative decoding, and specialized architectures like MoE and Llama variants. The cluster centers on SGLang and related systems that enable fast, flexible LLM inference through techniques like structured generation, dynamic batching, and hardware-aware optimization. Most repositories are Python-based implementations spanning model serving, inference engines, and compiler-level optimizations.

Python · 23

inference ·36,621

moe ·35,890

qwen ·35,789

llama ·35,766

blackwell ·35,766

gpt-oss ·35,766

diffusion ·35,766

deepseek ·35,766

cuda ·35,766

attention ·35,766

Related clusters

LLM Serving and Inference Optimization36 Cluster 43929710 GPU-Accelerated ML & CUDA Kernels37 Cluster 46227631 Cluster 4569083 Cluster 4475684

Kwai-Keye/sglang

SGLang is a fast serving framework for large language models and vision language models.

Python

11,771 commits

sgl-project/sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

Python

35,766

15,603 commits

tails-mpt/sglang

Fork of sgl-project/sglang with tails-mpt customizations

Python

10,187 commits

togethercomputer/xorl-sglang

No description

Python

9,759 commits

lkm2835/sglang

SGLang is a fast serving framework for large language models and vision language models.

Python

15,431 commits

HYGON-AI/sglang-das

基于 sgl-project/sglang 上游社区版本建设的 das 下游适配仓库，面向 hcu 平台进行功能适配与优化。

Python

15,272 commits

fort726/sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

Python

9,462 commits

guqiong96/Lsglang

Lsglang is a special extension of sglang that fully utilizes CPU and GPU computing resources with…

Python

124

15,447 commits

xltzsoft/deepseek-v4-sm89

面向 8×RTX 4090（SM89）的 DeepSeek-V4-Flash-0731 MXFP4 SGLang 实验分支

Python

14,156 commits

jpezzulli/sglang-rtxpro6000

Personal Optimized SGLang runtime for Qwen3.8-27B/DFlash2 and Qwen3.8 Flash-Next on one RTX PRO…

Python

15,291 commits

scottgl9/sglang-spark-gb10-optimizations

SGLang optimizations for NVIDIA Spark (GB10) — SM121 Grace Blackwell

Python

14,141 commits

huoji120/QWEN-EXO-booster

QWEN-BASE model-native memory and continuous learning system,base on SGLang

Python

13,811 commits

microsoft/llm-42

[Accepted to SOSP 2026] Fast Deterministic LLM Inference

Python

9,300 commits

radixark/miles

Miles is an enterprise-facing reinforcement learning framework for LLM and VLM post-training,…

Python

2,779

2,115 commits

radixark/miles_diffusion

Miles-diffusion is an post-training framework for large-scale diffusion model training and…

Python

1,140 commits

THUDM/slime

slime is an LLM post-training framework for RL Scaling.

Python

8,435

1,613 commits

verl-project/verl

verl/HybridFlow: A Flexible and Efficient RL Post-Training Framework

Python

23,389

2,679 commits

volcengine/verl

verl/HybridFlow: A Flexible and Efficient RL Post-Training Framework

Python

23,379

2,683 commits

ShChen233/cogflow_code

No description

Python

1,223 commits

RLsys-Foundation/TritonForge

🔥 LLM-powered GPU kernel synthesis: Train models to convert PyTorch ops into optimized Triton…

Python

152

495 commits

fzyzcjy/torch_memory_saver

Allow torch tensor memory to be released and resumed later

Python

274

576 commits

intel/models

Intel® AI Reference Models: contains Intel optimizations for running deep learning workloads on…

Python

731

3,049 commits

intel/intel-extension-for-pytorch

A Python package for extending the official PyTorch that can easily obtain performance on Intel…

Python

2,009

2,720 commits