Awesome-Large-Scale-LLM-Serving

🔥 Must-read papers for managing LLM serving clusters.

This repository contains papers on optimizing the efficiency of large language model (LLM) serving clusters, including request scheduling, auto-scaling, and serverless computing-oriented storage optimization and communication optimization techniques.

Papers on large language model cluster management typically involve optimizations from multiple perspectives simultaneously. For example, they may jointly optimize request scheduling and resource allocation. We categorize these papers based on their primary focus.

We sincerely welcome everyone to collect papers presented at top-tier conferences in the fields of systems and artificial intelligence and submit pull requests.

Contents

0. Survey
1. Request & Job Scheduling
2. Resource Management
3. Serverless LLM Serving
4. Prefill-Decoding Disaggregation
5. Communication Optimization

📜 Papers

You can directly click on the title to jump to the corresponding PDF link location

0. Survey

Survey of LLM Serving Systems

• [Arxiv'25] Taming the Titans: A Survey of Efficient LLM Inference Serving. Soochow University & Huawei

1. Request & Job Scheduling

Inter-instance and intra-instance request/job scheduling

• [Arxiv'25] SLOs-Serve: Optimized Serving of Multi-SLO LLMs. Carnegie Mellon University & Google
• [Arxiv'25] LLMSched: Uncertainty-Aware Workload Scheduling for Compound LLM Applications. Shanghai Jiao Tong University
• [Arxiv'25] Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Serving. Nanjing University & Alibaba Group
• [Arxiv'25] Niyama: Breaking the Silos of LLM Inference Serving. Microsoft Research
• [Arxiv'25] Hierarchical Prediction-based Management for LMaaS Systems. The Chinese University of Hong Kong
• [Arxiv'25] AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications. University of Virginia
• [Arxiv'25] Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMs. Massachusetts Institute of Technology & Databricks
• [ASPLOS'25] POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference. University of Washington & Microsoft Research
• [EuroSys'25] Fast State Restoration in LLM Serving with HCache. Tsinghua University
• [ICLR'25] Preble: Efficient Distributed Prompt Scheduling for LLM Serving. University of California at San Diego
• [Arxiv'24] Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments. University of Illinois at Urbana-Champaign & IBM Research
• [Arxiv'24] Aladdin: Joint Placement and Scaling for SLO-Aware LLM Serving. Stony Brook University & Azure Research
• [Arxiv'24] ConServe: Harvesting GPUs for Low-Latency and High-Throughput Large Language Model Serving. University of California at Los Angeles & Intel
• [Arxiv'24] Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services. University of Michigan & University of Illinois at Urbana-Champaign & Cisco
• [Arxiv'24] Mélange: Cost-Efficient Large Language Model Serving by Exploiting GPU Heterogeneity. University of California at Berkeley & National University of Singapore
• [ATC'24] Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. National University of Singapore & Shanghai Jiaotong University & Huawei Cloud
• [ASPLOS'24] ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference. Hanyang University & KT Corporation
• [ICML'24] HexGen: Generative Inference of Large Language Model over Heterogeneous Environment. The Hong Kong University of Science and Technology & ETH Zurich & Carnegie Mellon University
• [OSDI'24] Parrot: Efficient Serving of LLM-based Applications with Semantic Variable. Shanghai Jiao Tong University & Microsoft Research
• [OSDI'24] dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving. Peking University & Shanghai Artificial Intelligence Laboratory
• [OSDI'24] Llumnix: Dynamic Scheduling for Large Language Model Serving. Shanghai Jiao Tong University & Microsoft Research
• [SoCC'24] Queue Management for SLO-Oriented Large Language Model Serving. University of Illinois at Urbana-Champaign & IBM Research

2. Resource Management

Autoscaling and placement of LLM serving instances

• [Arxiv'25] DEEPFLOW: Serverless Large Language Model Serving at Scale. Huawei & Peking University
• [Arxiv'25] Hierarchical Autoscaling for Large Language Model Serving with Chiron. University of Illinois at Urbana-Champaign & IBM Research
• [Arxiv'25] Serving Models, Fast and Slow: Optimizing Heterogeneous LLM Inferencing Workloads at Scale. University of Illinois at Urbana-Champaign & Microsoft & Indian Institute of Science
• [Arxiv'24] DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency. University of Illinois at Urbana-Champaign & Microsoft Azure Research
• [Arxiv'24] ENOVA: Autoscaling towards Cost-effective and Stable Serverless LLM Serving. EmergingAI & Sun Yat-Sen University & Exascale Labs & University of Toronto
• [ICML'24] MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving. The Chinese University of Hong Kong & Shanghai Artificial Intelligence Laboratory & Huazhong University of Science and Technology & Shanghai Jiao Tong University & Peking University & University of California at Berkeley & University of California at San Diego
• [ASPLOS'24] SpotServe: Serving Generative Large Language Models on Preemptible Instances. Carnegie Mellon University & Peking University & The Chinese University of Hong Kong

3. Serverless LLM Serving

Accelerating LLM loading and service startup

• [Arxiv'25] ServerlessLoRA: Minimizing Latency and Cost in Serverless Inference for LoRA-Based LLMs. Shanghai Jiao Tong University & Stevens Institute of Technology
• [Arxiv'25] 𝜆Scale: Enabling Fast Scaling for Serverless Large Language Model Inference. The Chinese University of Hong Kong & University of Virginia & Hong Kong University of Science and Technology & Alibaba Group & Nokia Bell Labs
• [Arxiv'25] Towards Swift Serverless LLM Cold Starts with ParaServe. Peking University & Alibaba Group
• [ASPLOS'25] Medusa: Accelerating Serverless LLM Inference with Materialization. Tsinghua University
• [Arxiv'24] Fast and Live Model Auto Scaling with O(1) Host Caching. Shanghai Jiao Tong University & Huawei
• [OSDI'24] ServerlessLLM: Low-Latency Serverless Inference for Large Language Models. University of Edinburgh & Nanyang Technological University

4. Prefill-Decoding Disaggregation

Enhancing p/d disaggregation-based LLM serving

• [Arxiv'25] DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving. National University of Singapore & University of California at Berkeley & University of Science and Technology of China
• [Arxiv'25] Optimizing SLO-oriented LLM Serving with PD-Multiplexing. Shanghai Jiao Tong University & Huawei
• [Arxiv'25] KVDirect: Distributed Disaggregated LLM Inference. Rutgers University & ByteDance
• [ICLR'25] HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment. The Hong Kong University of Science and Technology
• [ICASSP'25] DynamicAttention: Dynamic KV Cache for Disaggregate LLM Inference. Ant Group
• [FAST'25] MOONCAKE: Trading More Storage for Less Computation – A KVCache-centric Architecture for Serving LLM Chatbot. Moonshot AI & Tsinghua University
• [Arxiv'24] Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads. University of Chinese Academy of Sciences & Huawei
• [Arxiv'24] MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool. Huawei & University of Chinese Academy of Sciences & Peking University
• [Arxiv'24] GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissions. Purdue University & University of Waterloo
• [Arxiv'24] P/D-Serve: Serving Disaggregated Large Language Model at Scale. Huawei
• [OSDI'24] DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. Peking University & StepFun & University of California at San Diego
• [SOSP'24] LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism. Peking University & Shanghai AI Lab

5. Communication Optimization

Optimizing transmission of KV Cache

• [HotInfra'24] Do Large Language Models Need a Content Delivery Network? University of Chicago
• [SIGCOMM'24] Cachegen: Kv cache compression and streaming for fast large language model serving. University of Chicago & Microsoft & Stanford University

Acknowledegments

Please contact me if I miss your name on the list, and I will add you back ASAP!

Contributors

Star History