LLM-inference-optimization-paper

Summary of some awesome works for optimizing LLM inference

This summary will including three parts:

1. some repositories that you can follow
2. some representative person or labs that you can follow
3. some important works in the different research interests

Repositories

For example, LLMSys-PaperList contains many excellent articles, and is keeping updating (which I believe is the most important for a paperlist). Awesome-LLM-Inference and Awesome_LLM_Accelerate-PaperList are also worth reading.

Besides, awesome-AI-system works also very well. And you can find other repositories in its content.

The log "Large Transformer Model Inference Optimization" helps me a lot at the beginning.

This log OpenAI Keynote on Building Scalable AI Infrastructure seems to be a laeding guidance.

Person/Lab

Follow others' research, and find yourself's idea.

It is not my intention to judge the work of these pioneers, and I understand that the shortness of my knowledge will lead me to leave out many important people. If you have a different opinion, please feel free to communicate with me through the issue.
In no particular order!!
Damn, I can't remember the names of foreigners.

Zhihao JIA: FlexFlow and other imporessive work, important role in MLSys, affiliated with CMU
Tianqi CHEN: TVM, XGBoost, and other imporessive work, important role in Machine Learning System and ML compilers, affiliated with CMU
Song HAN: many important work in efficient ML including sparsity and quantization. btw, the class TinyML and Efficient Deep Learning Computing is highly recommanded, affiliated with MIT Zhen DONG: many important work in quantization and high-performance ML, affiliated with UCB
Tri DAO: author of FlashAttention, affiliated with Princeton
Ce ZHANG: famous in efficient MLsys, affiliated with UChicago
Ion Stoica: Alpa, Ray, Spark, et.al.

SPCL: Scalable Parallel Computing Lab, affiliated with ETHz
Luo MAI: affiliated with University of Edinburgh

IPADS: focus more on PURE systems, buut also make great progress in MLSys, affiliated with SJTU
EPCC: Emerging Parallel Computing Center, parallel computing and MLSys are Naturally combined, affiliated with SJTU

Xin JIN: FastServe and LLMCad are impressive work, affiliated with PKU
Bin CUI: important role in MLSys including DL, GNN, and MoE, affiliated with PKU
Jidong ZHAI: leading many important work in MLSys, affiliated with THU
Lingxiao MA: with many important work in MLSys on Top-Conference, affiliated with MSRA
Cheng LI: high performce system and MLSys, affiliated with USTC
Xupeng Miao: SpotServe, SpecInfer, HET, et.al

Chuan WU: with some important work in distributed machine learning systems, affiliated with HKU James CHENG: affiliated with CUHK
Kai CHEN: database works well with MLSys, affiliated with HKUST
Lei CHEN: database works well with MLSys, many papers so I recommand u to focus on his Top-Conference paper, affiliated with HKUST
Yang YOU: leader of Colossal-AI, affiliated with NUS
Wei WANG: work in System and MLSys, affiliated with HKUST

Work

I hope to conlude these impressive works based on their research direction.
But my summary must not be informative enough, and I am looking forward to your addition.

Perhaps someone should write a detailed survey.

Periodically check the "cited by" of the papers with ⭐ will be helpful.
Paragraphs with 💡 are not perfect.

Survey/Evaluations/Benchmarks 💡

• ⭐ Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models: evaluations helps you find the bottleneck
• ⭐ Full Stack Optimization of Transformer Inference: a Survey: a survey by UCB
• ⭐ Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems: worth a read
• ⭐ Deep Learning Workload Scheduling in GPU Datacenters: A Survey: survey for GPU Datacenters DL Workload Scheduling
• ⭐ Towards Efficient and Reliable LLM Serving: A Real-World Workload Study: a benchmark for LLM serving
• ⭐ LLM Inference Unveiled: Survey and Roofline Model Insights: both survey and analysis
• A SURVEY OF RESOURCE-EFFICIENT LLM AND MULTIMODAL FOUNDATION MODELS: worth reading
• Training and Serving System of Foundation Models: A Comprehensive Survey
• Model Compression and Efficient Inference for Large Language Models: A Survey
• ⭐ Towards Coarse-to-Fine Evaluation of Inference Efficiency for Large Language Models
• ⭐ A Survey on Efficient Inference for Large Language Models: worth reading
• Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models
• ⭐ Navigating Challenges and Technical Debt in Large Language Models Deployment: important
• The CAP Principle for LLM Serving: anothor angle
• Demystifying Data Management for Large Language Models: talking about database in LLM, by Xupeng MIAO, accepted by SIDMOD'24
• Benchmarking LLM Inference Backends: vLLM, LMDeploy, MLC-LLM, TensorRT-LLM, and TGI: with code
• A Survey on Mixture of Experts
• Analyzing LLM performance: The impact of high-bandwidth memory on model inference: analyze of inference
• Inference Optimization of Foundation Models on AI Accelerators
• LLM Inference Serving: Survey of Recent Advances and Opportunities: newest
• A Survey on Mixture of Experts
• LLM Inference Serving: Survey of Recent Advances and Opportunities: better than nothing
• Contemporary Model Compression on Large Language Models Inference: survey in model compression
• ⭐ Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning: bring insights for MLSys
• Resource-efficient Algorithms and Systems of Foundation Models: A Survey
• ⭐ A Survey on Inference Optimization Techniques for Mixture of Experts Models: asurvey on MoE models
• Deploying Foundation Model Powered Agent Services: A Survey: survey for AI agent service
• Resource-efficient Algorithms and Systems of Foundation Models: A Survey
• A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey: survey on PEFT
• A Comprehensive Survey on Distributed Deep Learning Training: Parallelism Strategies, Frameworks, and Network Interconnects

Make useful benchmark or evaluation is helfpul.

• MLPerf Inference Benchmark: inference github, a well-known benchmark

• llmperf: evaluate both performance and correctness, but based on ray

• The Importance of Workload Choice in Evaluating LLM Inference Systems: important angles in LLM inference systems

• Vidur: A Large-Scale Simulation Framework For LLM Inference: test the performance of LLM inference

• Metron: Holistic Performance Evaluation Framework for LLM Inference Systems: an evaluation framework

• LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale: a Simulator

• LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators: inference + hardware

• Towards Efficient Large Multimodal Model Serving: a survey on mm serving, and a decoupled serving architecture that enables independent resource allocation and adaptive scaling for each stage

• LLMCompass: Enabling Efficient Hardware Design for Large Language Model Inference: a performance evaluation framework, can be used to estimate the time cost

• Predicting LLM Inference Latency: A Roofline-Driven ML Method: predict inference performance based on Roofline

• GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments: a work for predict LLMSys performance

• TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems: simulator provide some performance analysis

Technical reports of the enterprise

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning: deepseek: mla + moe
• Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs: moe training with lower-specification hardware
• SAGESERVE: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling: one of the first such public studies of Internet-scale LLM workloads: managing mixed workloads, combines short-term request routing to data centers with long-term scaling of GPU VMs and model placement with higher lead times, and co-optimizes the routing and resource allocation problem

Interesting NEW Frameworks in Parallel Decoding

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads, pdf

prior paper: Blockwise Parallel Decoding for Deep Autoregressive Models

Break the Sequential Dependency of LLM Inference Using Lookahead Decoding: by lookahead decoding

Both frameworks use parallel decoding, and deserve a more detailed research.

Benchmark LLM Inference framework

• vllm-project/aibrix

Papers for Parallel Decoding

There are some interesting papers about parallel decoding.

• Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster
• Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding
• ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel Decoding
• APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding: how to make it auto-parallel?

Complex Inference

In fact, I'm not so familiar with with topic. But perhaps OpenAI 4o1 used this...
Spend more time inferencing than pre-training

• ⭐ Large Language Monkeys: Scaling Inference Compute with Repeated Sampling: Starter material, apply repeated sampling
• ⭐ Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters: Starter material, scaling LLM Test-Time to improve accuracy
• Strategic Chain-of-Thought: Guiding Accurate Reasoning in LLMs through Strategy Elicitation: seems fewer people have explore the efficiency of CoT; a two-stage method gives me some throught
• Fast Best-of-N Decoding via Speculative Rejection: optimize alignment in inference, accepted by NIPS'24
• S*: Test Time Scaling for Code Generation: perhaps can do some acceleration on Test Time Scaling
• Thinking Short and Right Over Thinking Long: Serving LLM Reasoning Efficiently and Accurately: manage the CoT
• Kinetics: Rethinking Test-Time Scaling Laws: it is more efficient to scale model size up to a threshold before investing more compute in test-time scaling, and propose sparse attention, which lowers per-token cost and enables longer generations and more parallel samples within the same resource budget
• BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute: for small models, generating multiple responses and selecting the best can enhance quality while remaining cheaper than a single large-model response, does such idea really make sense for users?
• Are We Scaling the Right Thing? A System Perspective on Test-Time Scaling: a system-driven perspective on TTS, analyzing how reasoning models scale against practical metrics
• PASCAL: A Phase-Aware Scheduling Algorithm for Serving Reasoning-based Large Language Models: a phase-aware scheduling algorithm that prioritizes reasoning to reduce TTFT while using controlled preemption and token pacing during answering to preserve Quality-of-Experience, accepted by HPCA'26

GPT-o1

This topic is about GPT-o1, aka the strawberry.

• ⭐ Reverse engineering OpenAI’s o1: a leading blog for introduction in OpenAI’s o1
• ⭐ Chain-of-Thought Prompting Elicits Reasoning in Large Language Models: base work
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models: a improment based on CoT
• Large Language Model Guided Tree-of-Thought: also a ToT
• Let's Verify Step by Step: verify by step can be helpful
• Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models: what is Language Agent Tree Search (LATS)? accepted by ICML'24
• Critique-out-Loud Reward Models
• Generative Verifiers: Reward Modeling as Next-Token Prediction: a verifier, by DeepMind

Speculative Decoding

Also named as Speculative Sampling, model collaboration.

• ⭐ Accelerating Large Language Model Decoding with Speculative Sampling: opening of Speculative Decoding, by DeepMind
• ⭐ Fast inference from transformers via speculative decoding: work of similar period with the upper one, by Google, accepted by ICML'23
• SpecInfer: Accelerating Generative Large Language Model Serving with Speculative Inference and Token Tree Verification: paper under guidance of Zhihao JIA, use Tree decoding and a set of draft models
• LLMCad: Fast and Scalable On-device Large Language Model Inference: paper under guidance of Xin JIN, speculative decoding for on-device LLM inference based on tree decoding and other optimizations
• Speculative Decoding with Big Little Decoder: similar to speculative decoding, accepted in NIPS'23
• Online Speculative Decoding: update draft model online
• Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decoding: the trade-off analyse deserves a reading
• The Synergy of Speculative Decoding and Batching in Serving Large Language Models: analyse for combining the spec decoding with batching
• REST: Retrieval-Based Speculative Decoding: use retrieval for spec decoding, some familiar names in the authors list
• Cascade Speculative Drafting for Even Faster LLM Inference: by UIUC
• Multi-Candidate Speculative Decoding: multiple draft models
• ⭐ Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding: survey for Speculative Decoding
• BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding: a work with Yang YOU's name
• Decoding Speculative Decoding: provide some insight into the selection of draft models
• Ouroboros: Speculative Decoding with Large Model Enhanced Drafting: perhaps tree specualtive decoding?
• ⭐ Speculative Streaming: Fast LLM Inference without Auxiliary Models: a promising method for speculative decoding
• Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding: accelerating spec decoding
• Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokens: accelerate spec decoding with Fusing all tokens
• Minions: Accelerating Large Language Model Inference with Adaptive and Collective Speculative Decoding: using several SSMs, adaptive SSM prediction length, pipelining SSM decode and LLM verify
• Recurrent Drafter for Fast Speculative Decoding in Large Language Models
• Optimal Block-Level Draft Verification for Accelerating Speculative Decoding
• Accelerating LLM Inference with Staged Speculative Decoding: token tree and a second stage of speculative decoding
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding: combine KV cache with spec decoding
• EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models: algorithm optimization in spec decoding
• SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices: any difference with specinfer?
• Optimizing Speculative Decoding for Serving Large Language Models Using Goodput: model the speculative decoding length
• MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding: spec decoding for long-context
• QSpec: Speculative Decoding with Complementary Quantization Schemes: spec decoding with quantization, a novel A+B
• Cerberus: Efficient Inference with Adaptive Parallel Decoding and Sequential Knowledge Enhancement: optimization ob Medusa
• The N-Grammys: Accelerating autoregressive inference with learning-free batched speculation: use learning-free, negligible-cost draft strategies, namely N-grams obtained from the model weights and the context
• EdgeLLM: Fast On-device LLM Inference with Speculative Decoding: seem a extended work of LLMCad
• AdaServe: SLO-Customized LLM Serving with Fine-Grained Speculative Decodin: a speculation-and-selection scheme, that first constructs candidate token trees for each request and then dynamically selects tokens to meet individual SLO constraints
• SpecServe: Efficient and SLO-Aware Large Language Model Serving with Adaptive Speculative Decoding: dynamically adjusts speculative strategies according to real-time request loads and system configurations
• ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts: combining multi-level speculative decoding with MXFP4 quantized drafts, simple but work
• SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models: using multiple heterogeneous SSMs with a learning-based algorithm for SSM selection, request decomposition method to minimize batching overhead during LLM verification, pipelining speculation and verification phases on GPU
• ⭐ SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning: target at reason model and CoT, under guidence of Zhihao JIA; maybe refer to multi-agent?
• SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models: Multi-Level Speculative Decoding, under guidence of Jidong Zhai
• SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving: deploy draft model on edge divices and shared edge server to verify tokens
• FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference: speculation could be used by VLM
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios: SpecFormer, a novel architecture that integrates unidirectional and bidirectional attention mechanisms, to combine the autoregressive model's ability to extract information from the entire input sequence with the parallel generation benefits of non-autoregressive models
• AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving: dynamically adjusts speculative strategies according to real-time request loads and system configurations, accepted by SoCC'25
• WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching: offload draft model computation to edge device
• Towards Efficient LLM Inference via Collective and Adaptive Speculative Decoding: harnesses multiple SSMs to collaboratively predict LLM outputs, uses a pipelined execution to hide the latency of SSM speculation, proposes a mechanism to dynamically determine the optimal speculation length of SSM at runtime, accepted by SC'25

different model collaboration

• Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding: use both LLM and SLM

Skeleton-of-Thought

• Adaptive Skeleton Graph Decoding: successor of Skeleton-of-Thought

3D Parallelism 💡

Some knowledege about data parallel, model tensor parallel, and model pipeline parallel will help in this track.

• ⭐ Efficiently Scaling Transformer Inference: use model parallel to accelerating inference, by Google, in MLSys'23

• HexGen: Generative Inference of Foundation Model over Heterogeneous Decentralized Environment: a distributed inference engine that supports asymmetric partitioning of the inference computation

• InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding: Efficient Long-sequence training

• Liger: Interleaving Intra- and Inter-Operator Parallelism for Distributed Large Model Inference: accepted by PPoPP'24

• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs: full-stack approach of LLM training

• DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers: sequence parallel by Yang YOU

• LoongServe: Efficiently Serving Long-context Large Language Models with Elastic Sequence Parallelism: Elastic Sequence Parallelism?

• GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism: this could be potential in inference

• TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models: pipeline parallism

• QUART: Latency-Aware FaaS System for Pipelining Large Model Inference: pipeline in serving and fast expanding

• Mnemosyne: Parallelization Strategies for Efficiently Serving Multi-Million Context Length LLM Inference Requests Without Approximations: optimize sequence parallel

• CSPS: A Communication-Efficient Sequence-Parallelism based Serving System for Transformer based Models with Long Prompts: optimize sequence parallel

• ⭐ PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation: pipeline parallelism and speculation, accepted by SC'24

• HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment: algorithm analyse for resource allocation, parallel strategy and kv transfer in disaggreagting llm system

• ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughput: explores design spaces to suggest architectures that meet the requirements of both vendors and users

• Seesaw: High-throughput LLM Inference via Model Re-sharding: dynamic model re-sharding, facilitates the dynamic reconfiguration of parallelization strategies across prefill-decode stages, accepted by MLSYS'25

• PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training: fill the bubbles with other GPU workload

• ⭐ gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling: fine-grained scheduling policy that independently regulates the quantities of prefill and decode tokens, to balance the pipelien stage in PP

• Medha: Efficiently Serving Multi-Million Context Length LLM Inference Requests Without Approximations: Sequence Pipeline Parallelism (SPP) to reduce time-to-first-token by pipelining prefill chunks, and KV-Cache Parallelism (KVP) to lower time-peroutput-token by distributing decoding across servers

• WLB-LLM: workload-balanced 4D parallelism for large language model training: solve the workload imbalance issue at the pipeline parallelism and context parallelism levels, accepted by OSDI'25

• Mercury: Unlocking Multi-GPU Operator Optimization for LLMs via Remote Memory Scheduling: utilize an abstraction that treats remote GPU memory as an explicitly managed extension of the memory hierarchy, expanding the available storage and communication resources beyond local HBM, accepted by SOSP'25

• TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference: disaggregates the prefill and decode phases in the temporal dimension, so as to eliminate pipeline bubbles caused by the phase switching in pipeline parallelism in serving

• Serving LLM in Distributed GPU Cluster With Fine-Grain Pipeline Constraints: dynamically adjusts time constraints based on workload patterns to balance pipeline stages

• FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters: dynamically reconfigures pipeline architectures during runtime, accepted by EuroSys'26

• Dynamic Micro-Batch and Token-Budget Scheduling for IoT-Scale Pipeline-Parallel LLM Inference: runtime-adaptive scheduler that jointly tunes token budgets and micro-batch counts to balance prefill/decode workloads and minimize pipeline bubbles

• Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism: Chunkwise Dynamic Sequence Parallelism (CDSP), a fine-grained parallelism strategy that assigns SP sizes across intra-request token segments

Communication Overlap

• Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models: overlap comm with comp, similar to Liger
• Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communication Partitioning: accepted by ASPLOS'24
• T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives: many work about overlap in LLM, accepted by ASPLOS'24
• FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion: Fine-grained decomposition, perhaps provide some experiment result
• Kraken: Inherently Parallel Transformers For Efficient Multi-Device Inference: modify the model design for fast decoding, based on comm-comp overlapping
• NanoFlow: Towards Optimal Large Language Model Serving Throughput: overlaping based on nano-batch, with some interesting engineer implemntation, accepted by OSDI'25
• Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping: overlapping, provided by Deepspeed team
• PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Serving: overlap communication with model-weights/KV-cache prefetch
• Concerto: Automatic Communication Optimization and Scheduling for Large-Scale Deep Learning: use compilation to schedule overlap, accepted by ASPLOS'25
• TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives: TileLink to efficiently generate overlapped kernels for LLMs using tile-centric primitives and mappings, accepted by MLSYS'25
• FlashOverlap: A Lightweight Design for Efficiently Overlapping Communication and Computation: a novel signaling mechanism to identify tile-wise data dependency without interrupting the computation process, and reorders data to contiguous addresses, enabling communication by simply calling NCCL APIs
• Mitigating Contention in Stream Multiprocessors for Pipelined Mixture of Experts: An SM-Aware Scheduling Approach: build model of computation and communication, schedule tiles with consideration of SM useage, accepted by ICDCS'25

Prefill-Decode disaggregation

Ignore some of the earliest papers and focus on the latest work to optimize this.

• ⭐ Seesaw: High-throughput LLM Inference via Model Re-sharding: dynamic model re-sharding to facilitates the dynamic reconfiguration of parallelization strategies across stages, reduce the overhead caused by frequent stage transitions (seems like Elastic Scheduling)
• DynamicAttention: Dynamic KV Cache for Disaggregate LLM Inference: DynamicAttention, it allocates a continuous virtual GPU memory space at startup, but does not actually allocate physical GPU memory?
• ⭐ semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage: disaggregated computation and unified storage, a computation resource controller to achieve disaggregated computation at the streaming multi-processor (SM) level, and a unified memory manager to manage the asynchronous memory access from both phases
• ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads: requests grouping, disaggregation and resource scheduling
• Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture: dynamically adjusts the number of instances handling prefill and decode tasks based on real-time cluster performance metrics
• DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving: arbitrarily splits each request at any token boundary into at most two cooperating segments, then use a two-level scheduling framework then balances micro-request load across unified GPU instances
• Hybe: GPU-NPU Hybrid System for Efficient LLM Inference with Million-Token Context Window: utilizes the preexisting GPU for the prefill stage and employs lightweight NPUs during the decode stage, utilizes NVIDIA H100 GPU with inference-optimized vLLM library and implement Hybe NPU in 4nm process with equal HBM specification, accepted by ISCA'25
• WindServe: Efficient Phase-Disaggregated LLM Serving with Stream-based Dynamic Scheduling: stream-based, fine-grained dynamic resource scheduling to enhance utilization and performance for disaggregated LLM serving, accepted by ISCA'25
• Tropical: Enhancing SLO Attainment in Disaggregated LLM Serving via SLO-Aware Multiplexing: multiplexing prefill and decode on same hardware with an sevice-level objectives (SLO)-aware multiplexing strategy that balances the queuing time and the interference
• Nexus: Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving: dynamically partitions GPU resources across prefill and decode phases, while jointly considering compute capacity, memory footprint, and bandwidth contention, under guidence of Zhihao JIA
• Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving: dynamic choose GPUs' roles of disaggregation-aggregation, and consider the SLO
• Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference: a P/D autoscaling framework, a topology-aware scheduler that adapts to heterogeneous hardware and network constraints with a novel metric-driven policy derived from the first large-scale empirical study of autoscaling signals in production
• xLLM Technical Report: a novel decoupled service-engine architecture on P/D disaggregation, and co-optimizes system and algorithm designs to fully saturate computing resources, from JD
• Adaptive Rescheduling in Prefill-Decode Disaggregated LLM Inference: dynamic prefill-decode resource scheduling
• A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving: a dynamic LLM inference system that adjusts instance allocations to achieve an optimal prefill-to-decoding (P/D) ratio based on real-time load monitoring
• DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing: decouple prefill and decode execution only when needed through fine-grained, adaptive SM partitioning that provides phase isolation only when contention threatens latency service level objectives
• DisHelis: Optimizing Deployment of Disaggregated LLMs Inference Serving Over Heterogeneous Environments via Hierarchical Max-Flow: split resources for prefill/decode stages and dynamic role switch
• RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation: multiplex GPU for executing prefill and decode concurently based AMD CU masking
• DOPO: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving: achieve an optimal prefill-to-decoding (P/D) ratio based on real-time load monitoring

Prune & Sparsity 💡

An enduring topic in efficient machine learning.
We mainly focus on Semi-structured and Structured pruning becasue they can accelerate computing.

• ⭐ Accelerating Sparse Deep Neural Networks: use N:M sparsity to fully utilize the hardware for accelerating, by Nvidia

• ⭐ Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time: interesting paper in using sparsity, under guidence of Tri DAO and Ce ZHANG, accepted in ICML'23

• Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers

• Dynamic N:M Fine-Grained Structured Sparse Attention Mechanism: accepted by PPoPP'23

• ⭐ PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformation: A novel way to deal with dynamic sparsity may be used for GNN and MoE, accepted by SOSP'23

• DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving: seem a follow-up work of Deja Vu, also focus on KV-Cache

• FFSplit: Split Feed-Forward Network For Optimizing Accuracy-Efficiency Trade-off in Language Model Inferenc: sparsity in FFN

• ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models: a simple and effective sparsification method named "ProSparse"

• Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters: work for powerinfo

• Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations: pruning for LLM

• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention: inference framework based on sparse attention, by Microsoft

• ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models: use ReLU to imporve Sparsity, just like powerinfer

• CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activation: algorithm optimization that can utilize sparsity to accelerate inference

• Star Attention: Efficient LLM Inference over Long Sequences: a two-phase block-sparse approximation

• Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries: use Sparse Coding over Universal Dictionaries to compress KV cache, it's novelty

• SHARP: Accelerating Language Model Inference by SHaring Adjacent layers with Recovery Parameters: algorithm to replace a layer with the previous Adjacent layer and Recovery Parameters(based on finetune), to decrease memory overhead

• Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking: accepted by MLSYS'25

• SpInfer: Leveraging Low-Level Sparsity for Efficient Large Language Model Inference on GPUs: Tensor-Core-Aware Bitmap Encoding (TCA-BME) and sparse Gemm kernel, make unstructured pruning's theoretical advantages translate into practical performance gains, EuroSys'25

• Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Cores: EuroSys'25

• LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention: Efficient long-context LLM serving with unified block sparse attention, up to 3.3x faster decoding than TensorRT-LLM, accpeted by MLSYS'25

• SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving: for Dynamic sparse attention, solve the challenges: fragmented KV cache access, HBM cache contention, and high HBM demands of hybrid batching

• DualSpar: A Dual-Granularity Memory Framework with Adaptive Sparsity for Efficient LLM Inference: it binds token importance to KV cache block granularity and incorporates system load and request length into sparsity decision to optimize LLM inference with sparse KV cache, accepted by ICCD'25

Quantization 💡

Low-precision for memory and computing efficiency.

• Understanding and Overcoming the Challenges of Efficient Transformer Quantization
• ⭐ LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale: by UW
• ⭐ SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models: paper under guidance of Song HAN
• ⭐ AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration: paper under guidance of Song HAN
• Atom: Low-bit Quantization for Efficient and Accurate LLM Serving: paper under guidance of Tianqi CHEN, quantization is not important, designing how to quantify is important, in review of MLSys'24
• FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs
• QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models
• Understanding the Impact of Post-Training Quantization on Large Language Models: tech report will help
• ⭐ LLM-FP4: 4-Bit Floating-Point Quantized Transformers: by HKUST, accepted in EMNLP'23
• ⭐ Enabling Fast 2-bit LLM on GPUs: Memory Alignment, Sparse Outlier, and Asynchronous Dequantization: by SJTU, accepted in DAC'24
• INT4 Wight + FP8 KV-Cache: optimization for LLM inference: INT4 Wight + FP8 KV-Cache + Continues batching
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization: quant KV cache
• QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference: simple and crude optimization work
• LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization: for Heterogeneous Clusters and Adaptive Quantization, under guidence of Chuan WU, accepted by PPoPP'24(poster)
• IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact: use pivot token
• QAQ: Quality Adaptive Quantization for LLM KV Cache
• QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving: quantization in inference, under guidence of Song HAN
• Does compressing activations help model parallel training?: analyse in compressing(including pruning and quantization) in MP training, accepted by MLSys'24
• Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression: compress KV cache with quantization
• Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs: with targeted activate function
• FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design: FPx quantization, accepted by ATC'24
• Demystifying the Compression of Mixture-of-Experts Through a Unified Framework: combine quantization with MoE
• Does Compressing Activations Help Model Parallel Training?: quantization Activation?
• PQCache: Product Quantization-based KVCache for Long Context LLM Inference: apply quantization and Maximum Inner-Product Search for KV Cache compression
• Fast Matrix Multiplications for Lookup Table-Quantized LLMs: provide efficient kernels for lookup quantization
• Ladder: Enabling Efficient Low-Precision Deep Learning Computing through Hardware-aware Tensor Transformation: a computation optimization for Low-Precision
• Quant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric Algorithm-System Co-Design on Modern GPUs: a computation optimization for 6-bit LLM
• Mixture of Experts with Mixture of Precisions for Tuning Quality of Service: quantization on MoE models
• Zero-Delay QKV Compression for Mitigating KV Cache and Network Bottlenecks in LLM Inference: compress the KV Cache
• ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models: quantization matrix multiplication of arbitrary precision combinations based on BTC (Binary TensorCore) equivalents
• Progressive Mixed-Precision Decoding for Efficient LLM Inference: gradual lowering of precision deeper in the generated sequence, together with a spectrum of precision-switching schedulers
• COMET: Towards Partical W4A4KV4 LLMs Serving: provide quantization algorithm, quantization kernel and SM schedule method
• MixQ: Taming Dynamic Outliers in Mixed-Precision Quantization by Online Prediction: quantization with outliers, optimization on AWQ, accepted by SC'24
• Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference: low-bit compression to accelerate communication
• Unifying KV Cache Compression for Large Language Models with LeanKV: combine quantization and sparity to compress KV cache
• MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design: mix quantization, effectively assigning the larger bit-width to output features that need it most to achieve good accuracy with low memory consumption
• KVTuner: Sensitivity-Aware Layer-wise Mixed Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference: KVTuner to adaptively search for the optimal hardware-friendly layer-wise KV quantization precision pairs for coarse-grained KV cache with multi-objective optimization and directly utilize the offline searched configurations during online inference
• HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference: quantization to decrease kvc transfer overhead in disaggregation and eliminate kv dequantization
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models: Mixed-precision Auto-Regressive LINear kernels, accepted by PPoPP'25
• MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators: augments highly quantized MoEs with a mixture of low-rank compensators, provide 3-bit tensorcore kernels, accepted by MLSYS'25
• PacQ: A SIMT Microarchitecture for Efficient Dataflow in Hyper-asymmetric GEMMs: accelerator design, but may be helpful
• Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference: based on mixed-precision quantization to the key-value (KV) cache in LLMs based on token granularity, do quantization on KV cache chunk-level
• Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization: employs an online-offline hybrid approach, setting outlier thresholds offline, which are then used to determine the quantization scale online
• SQuat: Subspace-orthogonal KV Cache Quantization: a more efficient quantization algorithm(?)
• Tilus: A Virtual Machine for Arbitrary Low-Precision GPGPU Computation in LLM Serving: support Arbitrary Low-Precision computation with high performance
• Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization: quantization on KV cache, accepted by ISCA'25
• QFactory: Accelerating Quantized Large Language Model Serving with Qtile Graphs: an efficient compilation framework designed to generate high-performance quantized kernels, accepted by ATC'25
• DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization: stores the residual matrix—the difference between full-precision and quantized weights—in CPU, and dynamically fetches the residuals for only a small portion of the weights, aaccepted by OSDI'25
• Efficient and Workload-Aware LLM Serving via Runtime Layer Swapping and KV Cache Resizing: replace layers weights with different quantization type, and dynamic adjust KV cache capacity
• ⭐ Efficient Mixed-Precision Large Language Model Inference with TurboMind: a General Matrix Multiply (GEMM) pipeline that optimizes matrix operations through offline weight packing and online acceleration, and an attention pipeline that enables efficient attention computation with arbitrary Query, Key, and Value precision combinations
• LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving: accepted by SC'25
• ⭐ BitDecoding: Unlocking Tensor Cores for Long-Context LLMs with Low-Bit KV Cache: efficiently decode low-bit KV caches by cooperatively leveraging CUDA cores and Tensor Cores, accepted by HPCA'26
• High-Throughput Non-uniformly Quantized 3-bit LLM Inference: hardware-aligned bit shuffling and fused dequantization-multiplication pipeline that effectively maps workloads on both CUDA and Tensor Cores, accepted by PPoPP'26
• JanusQuant: Accurate and Efficient 2-bit KV Cache Quantization for Long-Context Inference: based on RtSmooth, a novel runtime smoothing quantization algorithm that mitigates outlier-induced accuracy loss via adaptive transformation, accepted by PPoPP'26

Batch Processing

Perhaps the most important way for improving the throughput in LLM inference.
This blog Dissecting Batching Effects in GPT Inference helps me a lot at the beginning.

Update2023/12/12: I'd like to use Continues Batching to take place of the Dynamic Batching I used before. The name Dynamic Batching is more likely to be used in Triton.

• ⭐ Orca: A Distributed Serving System for Transformer-Based Generative Models: Continues batch processing without redundant computing, accepted in OSDI'23
• Fast Distributed Inference Serving for Large Language Models: considering Job Completion Time(JCT) in LLM serving, paper under guidance of Xin JIN
• Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline: schedule based on response length prediction by LLM, paper under guidance of Yang YOU
• S3: Increasing GPU Utilization during Generative Inference for Higher Throughput: idea similar to above, by Harvard University
• SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills: blocking the prefill phase and reduce pipeline bubbles, by MSRIndia
• Flover: A Temporal Fusion Framework for Efficient Autoregressive Model Parallel Inference: accepted by HiPC'23
• Handling heavy-tailed input of transformer inference on GPUs: accepted by ICS'22
• CoFB: latency-constrained co-scheduling of flows and batches for deep learning inference service on the CPU–GPU system: Some form of inference service
• TCB: Accelerating Transformer Inference Services with Request Concatenation: perhaps similar to ByteTransformer, accepted by ICPP'22
• Fairness in Serving Large Language Models: under guidence of Ion Stoica, accepted by OSDI'24
• Characterizing and understanding deep neural network batching systems on GPUs: benchmarking is important
• Hydragen: High-Throughput LLM Inference with Shared Prefixes
• RelayAttention for Efficient Large Language Model Serving with Long System Prompts: think about the memory access of KV cache
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve: follow-up work of sarathi
• Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction: predict length
• LiveMind: Low-latency Large Language Models with Simultaneous Inference: perform inferences with incomplete prompts, to take advantage of streaming prompt
• A Queueing Theoretic Perspective on Low-Latency LLM Inference with Variable Token Length: theoretical analysis of latency
• ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG
• Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models: seems similar to ORCA or bytetransformer?
• BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching: optimization on ORCA, dynamic re-batching
• EcoServe: Maximizing Multi-Resource Utilization with SLO Guarantees in LLM Serving: A fusion monster with a variety of optimization techniques
• ⭐ AcceLLM: Accelerating LLM Inference using Redundancy for Load Balancing and Data Locality: what's Redundancy
• Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batching: formalize as an optimization problem and adjust the batch size based on this

Chunked prefill

• Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration: fine-grained chunked prefill with decode, but what is SM-masked stream?
• Efficient LLM Inference via Chunked Prefills: analyse on chunked prefill

Computing Optimization

This part include some impressive work optimizing LLM computing by observing the underlying computing properties. Such as FlashAttention, et.al.

FlashAttention Family

• ⭐ FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness: one of the most important work these years, both simple and easy to use, by Tri DAO
• ⭐ FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning: you'd better not ignore it
• ⭐ Flash-Decoding for long-context inference: you'd better not ignore it, too
• ⭐ Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity: successor to FlashAttention in inference, accepted by VLDB'24
• ⭐ FlashDecoding++: Faster Large Language Model Inference on GPUs: worth reading, FLashDecoding follow-up
• SubGen: Token Generation in Sublinear Time and Memory
• DeFT: Flash Tree-attention with IO-Awareness for Efficient Tree-search-based LLM Inference
• Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers: modification in self-attention
• FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
• Flex Attention: A Programming Model for Generating Optimized Attention Kernels: auto-generated attention kernel
• XY-Serve: End-to-End Versatile Production Serving for Dynamic LLM Workloads: decomposing computations into unified, hardware-friendly, fine-grained meta primitives to optimize both attention and linear computation
• MetaAttention: A Unified and Performant Attention Framework across Hardware Backends: attention can be abstracted into two operations: relevance scoring and aggregation, then design a cross-backend attention runtime around these operations that generalizes to variants of attention with customizable operators, accepted by PPoPP'26
• FlashAttention-T: Towards Fully Tensorized Attention by Exploiting Tensor-Vector Parallelism: offload critical softmax primitives to idle tensor units, accepted by PPoPP'26

Optimization focus on Auto-regressive Decoding

• Splitwise: Efficient generative LLM inference using phase splitting: splitting prefill and decode in a map-reduce style, by UW and Microsoft
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving: also split the prefill and decode, accepted by OSDI'24
• Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads: seems a combination of SARATHI and Splitwise
• ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference: similar to splitwise, accepted by ASPLOS'24
• Splitwiser: Efficient LLM Inference with Constrained Resources
• ToEx: Accelerating Generation Stage of Transformer-based Language Models via Token-adaptive Early Exit: Token-adaptive Early Exit

Kernels Optimization

• Automatic Task Parallelization of Dataflow Graphs in ML/DL models
• MonoNN: Enabling a New Monolithic Optimization Space for Neural Network Inference Tasks on Modern GPU-Centric Architectures: compilation optimization on compuataion graph
• POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference: optimize attention kernel in mix-batching
• Focus: High-Performant and Customizable Attention Engine for LLM Serving: flexible attention engine, advised by Chen Tianqi and accepted by MLSYS'25
• ML-Triton, A Multi-Level Compilation and Language Extension to Triton GPU Programming: Multi-level Triton
• TyphoonMLA: A Mixed Naive-Absorb MLA Kernel For Shared Prefix: MLA kernel for shared prefix in particular
• HyTiS: Hybrid Tile Scheduling for GPU GEMM with Enhanced Wave Utilization and Cache Locality: GEMM optimization, accepted by SC'25

Memory Manage

This part is inspired by PagedAttention of vLLM. And there are many Top-Conference paper discussing the memory management in DL computing on GPUs.

• ⭐ Efficient Memory Management for Large Language Model Serving with PagedAttention: memory page management for the KV-Cache in Attention-type model, accepted by SOSP'23 (many papers will cite the vLLM project instead of their paper, which makes it harder for us to find its citated by)

• ⭐ AutoScratch: ML-Optimized Cache Management for Inference-Oriented GPUs: cache management for inference, accepted by MLSys'23

• Improving Computation and Memory Efficiency for Real-world Transformer Inference on GPUs: block-based data layout, accepted by TACO'October-2023

• AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems: a unique observation that there is rich similarity in attention computation across inference sequences

• BPIPE: memory-balanced pipeline parallelism for training large language models: memory balance perhaps can work well in inferencce, by SNU, accepted by ICML'23

• Improving Large Language Model Throughput with Efficient LongTerm Memory Management: perhaps a new view

• CacheGen: Fast Context Loading for Language Model Applications

• Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache

• Quantized Side Tuning: Fast and Memory-Efficient Tuning of Quantized Large Language Models: consider the memory consumption in fine-tuning

• Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference

• Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference: compress KV Cache

• LLM as a System Service on Mobile Devices: LLM as a service on Mobile devices

• DistMind: Efficient Resource Disaggregation for Deep Learning Workloads: by Xin JIN, accepted by ToN'Jan24

• ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching: sparsity in KV Cache, accepted by ISCA'24

• AttentionStore: Cost-effective Attention Reuse across Multi-turn Conversations in Large Language Model Serving: a hierarchical KV caching system that leverages cost-effective memory/storage mediums to save KV caches for all requests

• vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention: improve PagedAttention

• Layer-Condensed KV Cache for Efficient Inference of Large Language Models: only computes and caches the KVs of a small number of layers

• MiniCache: KV Cache Compression in Depth Dimension for Large Language Models: compress KV cache

• CacheBlend: Fast Large Language Model Serving with Cached Knowledge Fusion: very popular idea recently

• Block Transformer: Global-to-Local Language Modeling for Fast Inference: build KV Cache block from many tokens' KV Cache

• MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool: KV Cache management in P/D disaggregation arch

• Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention: multi-round chat and memory management, accepted by ATC'24

• Stateful Large Language Model Serving with Pensieve: similar to cachedattention

• Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving: P/D disaggregation archtecture and KV Cache management

• P/D-Serve: Serving Disaggregated Large Language Model at Scale: a P/D based system, with D2D access optimization

• InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management: offload KV Cache

• Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption: a survey for optimizing KV Cache

• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving: tensor management especially for llm inference

• Token-Picker: Accelerating Attention in Text Generation with Minimized Memory Transfer via Probability Estimation: remove unimportant tokens in KV Cache

• CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving: compression and streaming transfering of KV Cache, accepted by SIGCOMM'24

• Compute Or Load KV Cache? Why Not Both?: recompute and load together for long context

• LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management: manage KV Cache by layers

• Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching: compress KV cache and multi-level memory

• EPIC: Efficient Position-Independent Context Caching for Serving Large Language Models: better prefix-cache

• ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference: Low-rank KV cache and dynamic rebuild KV cache

• ⭐ VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration: the first work I see that optimize KV cache in vision models

• ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction: KV cache page evict and recall, accepted by NIPS'24

• SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation: Optimization on Zero? redesign the data flow of heterogeneous hardware and sharded model training to minimize the excessive communication overhead, accepted by NIPS'24

• ⭐ KunServe: Elastic and Efficient Large Language Model Serving with Parameter-centric Memory Management: memory management for KV cache and parameter, seems a novel work considering the weights migration

• SYMPHONY: Improving Memory Management for LLM Inference Workloads: dynamically migrates K,V caches to enable finegrained scheduling of inference requests

• Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management: efficiently migrate requests and their KV cache among GPUs

• Efficient LLM Inference with Activation Checkpointing and Hybrid Caching: recompute+cache for KV cache management, only recompute attention(no projection)

• Memory Offloading for Large Language Model Inference with Latency SLO Guarantees: offload kv cache to CPU memory

• Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving: sparse attention is hot recently, dynamic kvcache budget and efficient kvc loading from CPU

• Efficient and scalable huge embedding model training via distributed cache management: staleness and skewed popularity distributions based cache

• BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference: different kv heads have different importance, then offload and compress

• Fast State Restoration in LLM Serving with HCache: cache for offloading kvc to CPU, accepted by EuroSys'25

• Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference: use model replication to improve serving throughput and GPU utilization?

• Characterizing the Behavior and Impact of KV Caching on Transformer Inferences under Concurrency: instrument vLLM to measure and analyze fine-grain metrics (token throughput, KV cache memory access patterns, load balancing of the forward passes), during different inference stages (prefill, decode, batching and KV cache eviction policies) in several scenarios

• Mitigating KV Cache Competition to Enhance User Experience in LLM Inference: mitigating KV Cache competition with several technology

• Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache: KV cache reusing is able to save cloud cost across a range of workloads with long context

• KVSort: Drastically Improving LLM Inference Performance via KV Cache Compression: error-bounded lossy compression on sorted KV vectors

• FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework: dynamic batching and kv cache pool in MM kv cache compression, guided by Jidong ZHAI

• Accelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management: multi-level KV cache management(an idea lack innovation) and request reorder, accepted by ASPLOS'25

• Aqua: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains: memory management framework for a sudden increase in the number of inference requests to a cloud-hosted LLM, accepted by ASPLOS'25

• ⭐ Jenga: Effective Memory Management for Serving LLM with Heterogeneity: optimization on PagedAttention, targeted at heterogeneous embeddings in LLMs, accepted by SOSP'25

• Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching: kv cache load/offload?

• Hardware-based Heterogeneous Memory Management for Large Language Model Inference: an asymmetric memory architecture consisting of capacity-centric and bandwidth-centric memory with computation units attached to each memory device, more like a hardware paper

• Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving: Survey and Analyze Key-Value Cache Compression Techniques for Large Language Model Serving, accepted by MLSYS'25

• FastTree: Optimizing Attention Kernel and Runtime for Tree-Structured LLM Inference: FastTree, which introduces GPU kernels tailored for efficiently processing queries that share contexts through the radix tree

• Characterizing the Behavior and Impact of KV Caching on Transformer Inferences Under Concurrency: a paper analyze KV cache and its impact based on vLLM, accepted by IPDPS'25

• MCaM: Efficient LLM Inference with Multi-tier KV Cache Management: use DRAM to store history KV cache, accepted by ICDCS'25

• KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider: systematic characterization of the KV$ workload patterns from Aliyun, accepted by ATC'25

• DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction: compress kv cache in three levels of differentiation in the KV cache: (1) the differing impact of keys and values on attention computation, (2) the varying importance of tokens, and (3) the diverse dynamic sparsity patterns across attention heads, accepted by SOSP'25

• eLLM: Elastic Memory Management Framework for Efficient LLM Serving: dynamic elastic KV cache pool, under guidence of Mingyi GUO

• Disk-Based Shared KV Cache Management for Fast Inference in Multi-Instance LLM RAG Systems: store some prefix kv cache on Disk to reduce prefill overhead

• LLM Serving With Efficient KV-Cache Management Using Triggered Operations: lower overhead KV cache storage and retrieval with SmartNICs capable of triggered operations, bypassing the CPU and network stack and improving data-transfer latency

• Oneiros: KV Cache Optimization through Parameter Remapping for Multi-tenant LLM Serving: avoids KV cache swapping by remapping, and thereby repurposing, the memory allocated to model parameters for KV cache, exploiting the high CPU-GPU bandwidth offered by the modern hardware (such as GH hardware), accepted by SoCC'25

• FineServe: Precision-Aware KV Slab and Two-Level Scheduling for Heterogeneous Precision LLM Serving: with KV Slab, a precision-aware adaptive memory management technique dynamically allocating KV cache based on model quantization characteristics, significantly reducing GPU memory fragmentation

• Throughput-Oriented LLM Inference via KV-Activation Hybrid Caching with a Single GPU: stores activation checkpoints instead of keys and values during intermediate inference stages

• Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading: understanding KV cache offloading

• RoMe: Row Granularity Access Memory System for Large Language Models: optimize memory access from hardware spec by using larger Row Granularity, accepted by HPCA'26

• 10Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Training: solve memory constraints in training by intelligently coordinating memory usage across GPU, CPU, and NVMe tiers, accepted by SoCC'25

Prefix Sharing

note: some papers about prefix sharing is not in this section

• LLM Query Scheduling with Prefix Reuse and Latency Constraints: balancing prefix reuse and fairness in query scheduling
• Marconi: Prefix Caching for the Era of Hybrid LLMs: prefix caching target at State Space Models
• ⭐ TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving: a unified segment-level prefix cache pool, TokenLake, which uses a declarative cache interface to expose requests' query tensors, prefix caches, and cache-aware operations to TokenLake for efficient pooling, under guidence of Xin JIN
• Strata: Hierarchical Context Caching for Long Context Language Model Serving: a hierarchical context caching framework designed for efficient long context LLM serving, introduces GPU-assisted I/O to combat KV cache fragmentation, employs cache-aware request scheduling to balance compute with I/O latency and overlapping unavoidable stalls
• PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernel: fully exploit prefix sharing in attention computation by reduce redundant KV cache loading from GMEM to SMEM with an optimized kernel, accepted by ASPLOS'26

Inference on hardware: GPUs, CPUs or based on SSD

• Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective: a helpful survey
• ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughput: balances throughput and latency under different hardware
• Understanding and Optimizing Multi-Stage AI Inference Pipelines: a Heterogeneous Multi-stage LLM inference Execution Simulator

Underlying optimization for GPU

• Reducing shared memory footprint to leverage high throughput on Tensor Cores and its flexible API extension library: implement some APIs to reduce the shared memory footprint, accepted in HPC Asia'23
• Benchmarking and Dissecting the Nvidia Hopper GPU Architecture: help us understand GPUs
• SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving: optimizing energy consuming based on lower GPU frequency
• Foreseer: Knowledge-Driven Acceleration of Memory-Bound Matrix Multiplications for Large Language Model Inference: similar to cutlass, optimization on intel GPU
• Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels: for Ascend GPU (perhaps also work for NVIDIA?)
• MEPipe: Democratizing LLM Training with Memory-Efficient Slice-Level Pipeline Scheduling on Cost-Effective Accelerators: maybe inference on RTX4090?
• PASK: Cold Start Mitigation for Inference with Proactive and Selective Kernel Loading on GPUs: DAC's paper for hardware
• Performance Implications of Pipelining the Data Transfer in CPU-GPU Heterogeneous Systems: talk about CPU-GPU data transfer with pin memory, UVM and asyncMemcpy
• LLMShare: Optimizing LLM Inference Serving with Hardware Architecture Exploration: a simulator that models the performance and cost across different hardware solutions and introduce a customized design space exploration framework to identify optimal setups efficiently, accepted by DAC'25
• μShare: Non-Intrusive Kernel Co-Locating on NVIDIA GPUs: enables intra-SM scattered co-locating of kernels to solve the problem that a kernel saturating only a subset of intra-SM hardware resources while leaving others underutilized
• FlashFuser: Expanding the Scale of Kernel Fusion for Compute-Intensive Operators via Inter-Core Connection: leverage the distributed shared memory to help kernel fusion to reduce memory access, accepted by HPCA'26
• Exploiting Efficient Mapping and Pipelined Execution for Accelerating SpMV on Tensor Cores: efficient Tensor Core mapping and optimized pipeline execution for SpMV, accepted by PPoPP'26
• HELM: Characterizing Unified Memory Accesses to Improve GPU Performance under Memory Oversubscription: characterize unified memory access patterns across diverse applications to select optimal UM migration and placement policies under oversubscription, accepted by SC'25

GPU sharing

• ⭐ Hardware Compute Partitioning on NVIDIA GPUs: spatially partition the computing units of NVIDIA GPUs transparently, worth reading
• Improving GPU Sharing Performance through Adaptive Bubbleless Spatial-Temporal Sharing: Bless leverages precise computing resource management and fine-grained kernel scheduling to ensure stringent quota guarantees and reduce latency fairly for applications with varying GPU quotas, accepted by EuroSys'25
• Efficient Performance-Aware GPU Sharing with Compatibility and Isolation through Kernel Space Interception: a kernel-space GPU-sharing scheme that ensures compatibility and isolation, accepted by ATC'25
• ⭐ LithOS: An Operating System for Efficient Machine Learning on GPUs: amazing work, also helps understand GPU system, accepted by SOSP'25
• Prism: Unleashing GPU Sharing for Cost-Efficient Multi-LLM Serving: tackles a key limitation of existing systems: the lack of cross-model memory coordination to unleashe the full potential of GPU sharing, under guidence of Ion Stoica
• Hummingbird: SLO-Oriented GPU Preemption at Microsecond-scale: microsecond-scale GPU preemption to harveste idle GPU time slices, seems low-priority tasks give up time to the high-priority tasks
• Multiplexed Heterogeneous LLM Serving via Stage-Aligned Parallelism: GPU sharing for for serving heterogeneous LLMs, accepted by SoCC'25
• Understanding GPU Resource Interference One Level Deeper: contension for shared GPU resources in block schedulers, compute units, L1/L2 caches, and memory bandwidth, accepted by SoCC'25
• BOER: Enhancing Resource Utilization for Deep Learning Inference with Hybrid Spatial GPU Sharing: combines MPS atop MIG partitions to reduce interference and enhance GPU utilization, accepted by SC'25

CPUs or based on SSD

Heterogeneous scenarios or single PC are becoming increasingly important.

Making optimization for the calculating on CPU or SSD will have different methods.

• Efficient LLM Inference on CPUs: LLMs with quantization on CPUs, by Intel, accepted by NIPS'23

• Inference Performance Optimization for Large Language Models on CPUs: xFasterTransformer, LLMs inference optimization on CPUs, by Intel

• Distributed Inference Performance Optimization for LLMs on CPUs: similar work to above, by Intel

• Exploiting Intel Advanced Matrix Extensions (AMX) for Large Language Model Inference: inference on CPU based on advanced hardware

• TURNIP: A "Nondeterministic" GPU Runtime with CPU RAM Offload: free to run operations such as GPU kernel calls in many different orders

• Improving Throughput-oriented Generative Inference with CPUs: cooperate of CPUs and GPU, accepted by APSys'23

• Chrion: Optimizing Recurrent Neural Network Inference by Collaboratively Utilizing CPUs and GPUs: execute the operators on the CPU and GPU in parallel, by SJTU

• EdgeNN: Efficient Neural Network Inference for CPU-GPU Integrated Edge Devices: inference on edge devices, accepted by ICDE'23

• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU: by SJTU IPADS

• LLM in a flash: Efficient Large Language Model Inference with Limited Memory: by Apple

• Efficient LLM inference solution on Intel GPU: intel GPU is interesting

• FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines: efficient serving with CPU-GPU system

• Efficient and Economic Large Language Model Inference with Attention Offloading: similar to FastDecode

• Glinthawk: A Two-Tiered Architecture for High-Throughput LLM Inference: similar to fastdecode: cpu for attention and gpu for others

• Petals: Collaborative Inference and Fine-tuning of Large Models: looks like heterogeneous resources are being utilized

• Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures: analysis performance on loosely-coupled (PCIe A100/H100) and closely-coupled (GH200) systems

• Efficient Inference on CPU

• CPU inference

• NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention

• ⭐ A Quantitative Analysis and Guidelines of Data Streaming Accelerator in Modern Intel Xeon Scalable Processors: use CPU for DL, accepted by ASPLOS'24

• LM-Offload: Performance Model-Guided Generative Inference of Large Language Models with Parallelism Control: based on offload

• T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge: computation on CPU with quantization

• TBA: Faster Large Language Model Training Using SSD-Based Activation Offloading: how to use SSD?

• InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference: offload KV Cache to CSD(Computational Storage Drive)

• TwinPilots: A New Computing Paradigm for GPU-CPU Parallel LLM Inference: some idea in using CPU

• Improving Throughput-oriented LLM Inference with CPU Computations: pipeline in CPU-GPU inference

• Understanding Performance Implications of LLM Inference on CPUs: analyse of using CPU for inference

• GPUs, CPUs, and... NICs: Rethinking the Network's Role in Serving Complex AI Pipelines: NIC can be important, especially in communication

• LIA: A Single-GPU LLM Inference Acceleration with Cooperative AMX-Enabled CPU-GPU Computation and CXL Offloading: leveraging cooperative AMX-enabled CPU-GPU computation and CXL offloading, accepted by ISCA'25

• Pie: Pooling CPU Memory for LLM Inference: use CPU memory to enlarge batchsize to improve throughput, by Ion Stoica

• NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference: offload KV cache and attention to CPU for larger batchsize, similar to fastdecode, by Ion Stoica, accepted by MLSYS'25

• Task Scheduling for Efficient Inference of Large Language Models on Single Moderate GPU Systems: more likely inference on personal device

• Efficient LLM Inference with I/O-Aware Partial KV Cache Recomputation: use recomputation and transfer to re-produce KV cache; can use their run-time and split parallelism

• Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Serving: Monolithic 3D-Stackable DRAM (Mono3D DRAM), near-memory processing (NMP), and GPU acceleration, accepted by MICRO'25

• Managing Scalable Direct Storage Accesses for GPUs with GoFS: a GPU-orchestrated file system (GoFS) for scaling the direct storage accesses for GPU programs, by fully offloading the storage management to the GPU without host-side control, accepted by SOSP'25

• KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models: compute some experts in MoE on CPU, accepted by SOSP'25

• Sandwich: Separating Prefill-Decode Compilation for Efficient CPU LLM Serving: a hardware-centric CPU-based LLM serving engine that uses different execution plans for the prefill and decode phases and optimizes them separately, under guidence of Chuan WU

• SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference: leveraging underutilized CPU resources to offload auxiliary computation and communication, with CPU sampling, a token-safe execution model, and structure-aware transmission

• Q-Infer: Towards Efficient GPU-CPU Collaborative LLM Inference via Sparsity-Aware Dynamic Scheduling: utilize sparsity to conduct GPU-CPU collaborative inference

• LILo: Harnessing the On-chip Accelerators in Intel CPUs for Compressed LLM Inference Acceleration: leveraging an on-chip lossless compression accelerator in the latest Intel CPUs, to accelerate inference under memory capacity constraints, accepted by HPCA'26

• AGILE: Lightweight and Efficient Asynchronous GPU-SSD Integration: a lightweight asynchronous GPU-centric I/O library that eliminates deadlock risks, integrates a flexible HBM-based software cache, and overlaps computation and I/O, accepted by SC'25

Inference on personal device

Inspired by AI PC, open up a new area.
Including edge systems now.

• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU: inference a 30B model with a 16GB GPU, accepted by ICML'23
• LLM as a System Service on Mobile Devices: an intro for LLM on private devices
• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU: based on sparsity in NN Layers
• ⭐ LLM for Mobile: An Initial Roadmap: a road map
• PowerInfer-2: Fast Large Language Model Inference on a Smartphone: work on smartphone
• Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM: on edge devices, accepted by MICRO'24
• ⭐ HeteroLLM: Accelerating Large Language Model Inference on Mobile SoCs with Heterogeneous AI Accelerators: features on mobile SoCs, tensor partition strategy, to do Heterogeneous AI inference
• PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networks: cloud(LLM)-edge(SmallLM) collaboration
• FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference: offloading based framework, asynchronous prefetching, balanced memory locking, and flexible tensor preservation
• Fast On-device LLM Inference with NPUs: chunked prefill, offload outlier to CPU/GPU, schedule computation to NPU/CPU/GPU, accepted by ASPLOS'25
• FlexInfer: Flexible LLM Inference with CPU Computations: offload kvc and weights to CPU, accepted by MLSYS'25
• An Adaptive and Scalable Framework for Resource-Efficient Deployment of Mixture of Experts in LLM-Based Intelligent IoT Networks: deploy MoE on IoT, but the strategies are commonly used
• A Novel Hat-Shaped Device-Cloud Collaborative Inference Framework for Large Language Models: can learn the edge-cloud serving from this paper, based on speculation decode
• HERA: Hybrid Edge-cloud Resource Allocation for Cost-Efficient AI Agents: assign sub-tasks of LLM agent to local SLM and cloud-side LLM
• Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC: for agentic LLM workloads on memory-unified heterogeneous SoCs, process loads "Reactive tasks, initiated by users, demand immediate, low-latency responses, while proactive tasks operate invisibly and prioritize throughput"
• Elastic On-Device LLM Service: elasticizes both the model and the prompt dimension of a full LLM (what's this? I'm not so familiar with on-device LLM inference)
• Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference: heterogeneous parallel mechanisms to fully exploit both GPU and NPU computational power and memory bandwidth, fast synchronization mechanism between heterogeneous processors that leverages the unified memory architecture, accepted by SOSP'25

Heterogeneous or decentralized environments

• FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs: decentrailized system on consumer-level GPUs, through there will be some problems

• Distributed Inference and Fine-tuning of Large Language Models Over The Internet: some techs in this paper will be instructive

• ⭐ HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices: heterogeneous parallel computing using CPUs and GPUs

• Metis: Fast Automatic Distributed Training on Heterogeneous GPUs: accepted by ATC'24

• Helix: Distributed Serving of Large Language Models via Max-Flow on Heterogeneous GPUs: we can get performance model for Heterogeneous GPUs cluster and learn the algorithm analyse

• Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity: making heterogeneity-aware GPU provisioning decisions for LLM serving

• MoLink: Distributed and Efficient Serving Framework for Large Models: deploy LLMs on consumer-level GPUs with network optimization, for pipeline parallel serving, accepted by EMNLP'25

• AXLearn: Modular Large Model Training on Heterogeneous Infrastructure: train on Heterogenous hardware

• Deploy Efficient Large Language Model Distributed Inference Pipeline for Heterogeneous GPUs: a heuristic algorithm and implement a system that automatically deploys an efficient inference pipeline on heterogeneous GPUs

• LLM-Mesh: Enabling Elastic Sharing for Serverless LLM Inference: a serverless inference scheme for small-to-mid-sized LLMs that enables elastic sharing across heterogeneous hardware

• Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism: it selectively parallelizes compute-intensive operations to reduce latency and dynamically distributes Attention computations to low-end GPUs at a head granularity, and an online load dispatching policy that continuously optimizes serving performance by carefully balancing network latency, computational load, and memory intensity

• SplitQuant: Resource-Efficient LLM Offline Serving on Heterogeneous GPUs via Phase-Aware Model Partition and Adaptive Quantization: co-designing quantization and model partitioning strategies for heterogeneous environments

• Cronus: Efficient LLM inference on Heterogeneous GPU Clusters via Partially Disaggregated Prefill: partitions each prefill stage and executes its initial portion on the low-end GPU, while overlapping the remaining prefill and decode stages of earlier requests on the high-end GPU

• Cauchy: A Cost-Efficient LLM Serving System through Adaptive Heterogeneous Deployment: adaptively deploys prefill and decode computation to the most suitable heterogeneous GPUs and dynamically schedules user requests, accepted by SoCC'25

• ⭐ FlowMesh: A Service Fabric for Composable LLM Workflows: AI deployment increasingly resembles a pipeline of data transformation, fine-tuning, and agent interactions, so propose FlowMesh, a multi-tenant service fabric that executes and optimizes these workloads as one shared service instead of isolated pipelines; decomposes workflows into fine-grained operators

• Hybrid Learning and Optimization-Based Dynamic Scheduling for DL Workloads on Heterogeneous GPU Clusters: an application-agnostic reinforcement learning (RL)-based scheduling framework that dynamically prioritizes and allocates DL jobs on heterogeneous GPU clusters, accepted by SoCC'25

Communication Optimization

• HiCCL: A Hierarchical Collective Communication Library: we can study some lower-level technologies, such as communication, accepted by IPDPS'25
• Unified Designs of Multi-Rail-Aware MPI Allreduce and Alltoall Operations Across Diverse GPU and Interconnect Systems: optimization on allreduce/alltoall kernel but seems lack of innovation, accepted by IPDPS'25
• ⭐ Chimera: Communication Fusion for Hybrid Parallelism in Large Language Models: nalyze the communication processes of each LLM parallelism pattern, identify the communication redundancy in hybrid parallelism and eliminate redundancy by fusing adjacent communication operators during parallelism transformation, accepted by ISCA'25
• Enabling efficient GPU communication over multiple NICs with FuseLink: extends inter-server network by integrating high-speed intra-server connections, and leverages GPUs to efficiently relay traffic to idle NICs, accepted by OSDI'25
• SyCCL: Exploiting Symmetry for Efficient Collective Communication Scheduling: leverages collective and topology symmetries to decompose the original collective communication demand into smaller sub-demands within smaller topology subsets, accepted by SIGCOMM'25
• ODRP: On-Demand Remote Paging with Programmable RDMA: smartly chains native RDMA data path primitives to offload all memory access and management operations onto the RDMA-capable NIC (RNIC), accepted by NSDI'25
• WIC: Hiding Producer-Consumer Synchronization Delays with Warp-Level Interrupt-based GPU Communications: early-started polling leads to the unnecessary occupation of computational resources, so propose Warp-level Interrupt-based Communication, accepted by ATC'25
• Characterizing Communication Patterns in Distributed Large Language Model Inference: characterize communication behavior across different parallelization configurations
• Scalable and Fast Inference Serving via Hybrid Communication Scheduling on Heterogeneous Networks: a fast and scalable inference serving system by employing an offline planner for joint computation allocation and communication scheduling, along with an online scheduler for dynamic traffic management and load balancing

Algorithm Optimization 💡

In this part, researchers provide some algorithm-based method to optimizing LLM inference.

• H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models: accepted by NIPS'23
• Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time: consider the different importance of tokens in KV Cache, similar to H2O
• ⭐ SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference: skipping maybe an useful method like spec decoding
• Inference with Reference: Lossless Acceleration of Large Language Models: also a potential optimization
• Efficient Streaming Language Models with Attention Sinks: streaming LLM for infinite sequence lengths, by MIT and under guidence of Song HAN
• Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference: also important tokens, just like H2O, accepted by MLSys'24
• Q-Hitter: A Better Token Oracle for Efficient LLM Inference via Sparse-Quantized KV Cache: an optimization to H2O, accepted by MLSys'24
• RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval: use approximate nearest neighbor search to search the most relevant KV cache
• CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMs: based on observation: adjacent query tokens tend to focus on similar subsets of the past Key-Value (KV) cache
• TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention: sparse attention
• SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation: algorithm optimization for less KV Cache
• Activation Sequence Caching: High-Throughput and Memory-Efficient Generative Inference with a Single GPU: use characterization results to optimize KV Cache management

Industrial Inference Frameworks 💡

• ⭐ DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale: you must know DeepSpeed
• DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference
• DeepSpeed Model Implementations for Inference (MII)
• ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs: developed by ByteDance, accepted by IPDPS'23
• TurboTransformers: an efficient GPU serving system for transformer models: by Tencent Inc, accepted by PPoPP'21
• Accelerating Generative AI with PyTorch II: GPT, Fast: a blog in PyTorch, use only PyTorch code, gpt-fast
• FlexFlow Serve: Low-Latency, High-Performance LLM Serving: based on FlexFlow
• FlashInfer: Kernel Library for LLM Serving
• FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
• Efficiently Programming Large Language Models using SGLang: we can get some optimization from here
• Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models: different parallel, by Tencent
• Characterizing Cloud-Native LLM Inference at ByteDance and Exposing Optimization Challenges and Opportunities for Future AI Accelerators: analyse on acceleraters from bytedance, accepted by HPCA'26

LLM Serving 💡

LLM server providers will focus on this part. Engineering practices are just as important as algorithm optimization.

• ⭐ AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving: accepted by OSDI'23

• ⭐ STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining: Elastic will be important in the future, accepted by ASPLOS'23

• INFaaS: Automated Model-less Inference Serving: accepted by ATC'21

• Tabi: An Efficient Multi-Level Inference System for Large Language Models: under guidence of Kai CHEN, accepted by EuroSys'23

• FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance: cost is the service provider cares most

• FaaSwap: SLO-Aware, GPU-Efficient Serverless Inference via Model Swapping

• Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning: accepted by NSDI'23

• Cocktail: A Multidimensional Optimization for Model Serving in Cloud: model ensembling, accepted by NSDI'22

• SLA-Driven ML INFERENCE FRAMEWORK FOR CLOUDS WITH HETEROGENEOUS ACCELERATORS: accepted by MLSys'22

• FaST-GShare: Enabling Efficient Spatio-Temporal GPU Sharing in Serverless Computing for Deep Learning Inference: accepted by ICPP'23

• Flashpoint: A Low-latency Serverless Platform for Deep Learning Inference Serving

• Serving deep learning models in a serverless platform

• BATCH: Machine Learning Inference Serving on Serverless Platforms with Adaptive Batching: accepted by SC'20

• MArk: exploiting cloud services for cost-effective, SLO-aware machine learning inference serving: accepted by ATC'19

• ⭐ MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters: challenges and solutions in real-world scenarios, accepted by NSDI'22

• SuperServe: Fine-Grained Inference Serving for Unpredictable Workloads: serve multiple models, under the guidence of Ion Stoica, accepted by NSDI'25

• Learned Best-Effort LLM Serving: a best-effort serving system of UCB

• Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences: accepted by OSDI'22, enables microsecond-scale kernel preemption and controlled concurrent execution in GPU scheduling

• PipeSwitch: fast pipelined context switching for deep learning applications: PipeSwitch, a system that enables unused cycles of an inference application to be filled by training or other inference applications, accepted by OSDI'20

• ⭐ Paella: Low-latency Model Serving with Software-defined GPU Scheduling: how the tasks are scheduled to GPUs, accepted by SOSP'23

• OTAS: An Elastic Transformer Serving System via Token Adaptation: elastic in serving while considering SLO

• DeltaZip: Multi-Tenant Language Model Serving via Delta Compression: Multi-Tenant is interesting

• ServerlessLLM: Locality-Enhanced Serverless Inference for Large Language Models: find different problems in serving LLMs

• Fast and Efficient Model Serving Using Multi-GPUs with Direct-Host-Access: accepted by EuroSys'23

• Towards Pareto Optimal Throughput in Small Language Model Serving: Small Language Model Serving

• MOPAR: A Model Partitioning Framework for Deep Learning Inference Services on Serverless Platforms

• Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services: idea of QoE

• FlexLLM: A System for Co-Serving Large Language Model Inference and Parameter-Efficient Finetuning: how to find novel questions?

• Deferred Continuous Batching in Resource-Efficient Large Language Model Serving: similar to FlexLLM

• LLMServingSim: A Simulation Infrastructure for LLM Inference Serving Systems: provide some features about LLM serving

• Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving: Improvements to ORCA(SLS) and FastServe(ILS)

• Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems: consider serving efficiency from energy view

• Power-aware Deep Learning Model Serving with μ-Serve: consider energy

• Eloquent: A More Robust Transmission Scheme for LLM Token Streaming: a new token transmission scheme, useful in chatbot

• Responsive ML inference in multi-tenanted environments using AQUA: serving several LLMs based on time-sharing GPUs cycles, in multi-tenanted environments

• Towards SLO-Optimized LLM Serving via Automatic Inference Engine Tuning: effect of hyper-parameters in inference engine

• Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Scheduling: request schedule

• Efficient LLM Scheduling by Learning to Rank: rank request based on output length predict and schedule

• Responsive ML inference in multi-tenanted environments using AQUA: offload context to other GPUs in multi-tenant environment

• UELLM: A Unified and Efficient Approach for LLM Inference Serving: serving optimization in MaaS clouds

• One Queue Is All You Need: Resolving Head-of-Line Blocking in Large Language Model Serving: shcduling the requests

• ConServe: Harvesting GPUs for Low-Latency and High-Throughput Large Language Model Serving: harvest stranded GPU resources for offline LLM inference tasks

• LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services: accepted by SC'24

• Revisiting SLO and Goodput Metrics in LLM Serving: check metrics SLO and Goodput in LLM serving

• Hops: Fine-grained heterogeneous sensing, efficient and fair Deep Learning cluster scheduling system: schedule tasks in multi-tenant deep learning (DL) cluster, accepted by SoCC'24

• ⭐ Ensuring Fair LLM Serving Amid Diverse Applications: ensures fair LLM access across diverse applications, with a copilot trace analysis

• BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching: exploits the relaxed latency requirements in offline batch inference to reorder and overlap requests with varied resource demands while ensuring high prefix sharing

• BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching: similar to blendserve

• iServe: An Intent-based Serving System for LLMs: use cost model to dynamically set deployment configuration

• TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms: seems a Practical work in engineering? Take into account temperature and power consumption

• ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments: a novel scheduling algorithm, which optimizes the deployment plan of LLM serving to accommodate the heterogeneous resource and network bandwidth conditions in cloud environments, and fluctuating online conditions

• ⭐ MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism: we can learn for the expert-attention disaggregation, accepted by SIGCOMM'25

• SkyServe: Serving AI Models across Regions and Clouds with Spot Instances: seems subsequent work on spotserve, serve AI models over a mixture of spot and on-demand replicas, EuroSys'25

• Past-Future Scheduler for LLM Serving under SLA Guarantees: efficient requests scheduler via considering the historical distribution of request output lengths and calculating memory occupancy at each future time point, and the framework LightLLM

• Deferred prefill for throughput maximization in LLM inference: looks a bit counter-intuitive

• Performance Aware LLM Load Balancer for Mixed Workloads: a heuristic-guided, reinforcement learning-based router with a trainable response-length predictor and a novel formulation for estimating the impact of mixing different workloads

• Niyama : Breaking the Silos of LLM Inference Serving: request schedule paper

• ⭐ Optimizing SLO-oriented LLM Serving with PD-Multiplexing: PD multiplexing, enabling in-place and phase-decoupled compute partition, seems different from simple multiplexing

• Ascendra: Dynamic Request Prioritization for Efficient LLM Serving: set high or low priority for requests

• PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications: target at prefill-only workload, which only output one token, accepted by SOSP'25

• ⭐ ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production: show the real production LLM workload

• Efficient LLM Serving on Hybrid Real-time and Best-effort Requests: collocate the Real-time and Best-effort Requests, propose request scheduling and KV cache sharing

• SOLA: Optimizing SLO Attainment for Large Language Model Serving with State-Aware Scheduling: a state-aware scheduling that optimizes the SLO attainment in LLM serving

• Weaver: Efficient Multi-LLM Serving with Attention Offloading: workload weaving, which offloads attention operators of hot models to running cold models, achieving high GPU memory utilization with low communication cost, accepted by ATC'25

• BestServe: Serving Strategies with Optimal Goodput in Collocation and Disaggregation Architectures: ranking serving strategies by estimating goodput under various operating scenarios

• Adaptive Request Scheduling for CodeLLM Serving with SLA Guarantees: denamic requests batching considering SLA

• BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving: grouping requests into size-homogeneous buckets based on sequence length

• Adaptively Robust LLM Inference Optimization under Prediction Uncertainty: based on output length predictor

• A System-level Abstraction and Service for Flourishing AI-powered Applications: introduce a virtual capability layer with a unified API that efficiently powers various applications with AI technology, promoting a capability-centric approach to solve the inefficient resource management and unnecessary implementation details for developers

• Prompt-Aware Scheduling for Low-Latency LLM Serving: based on a trained output length predictor

• TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling: preemptive request scheduling and proactive key-value (KV) cache management, accepted by EuroSys'26

• Synera: Synergistic LLM Serving across Device and Cloud at Scale: device-cloud synergistic LLM serving system with optimization on offloading decisions, pipeline stalls, and batching bottlenecks

• Vortex: Hosting ML Inference and Knowledge Retrieval Services With Tight Latency and Throughput Requirements: deploying ML inference and knowledge retrieval as services(not very clear)

• Laser: Unlocking Layer-Level Scheduling for Efficient Multi-SLO LLM Serving: scheduling layer-grained execution for efficient multi-SLO LLM serving, instead of iteration-level, accepted by PPoPP'26

LLM as microservice

• ⭐ A System for Microserving of LLMs: seems a idea and industrial practice that makes sense

Serverless LLM serving

• DeepFlow: Serverless Large Language Model Serving at Scale: provide fine-grained LLM service

• ⭐ Towards Swift Serverless LLM Cold Starts with ParaServe: pipeline parallelism and dynamic adjust parallelism strategy, and accelerate cold-start

• λScale: Enabling Fast Scaling for Serverless Large Language Model Inference: serverless inference system to achieve fast model scaling, by fast model multicast, inference execution during model transmission and dynamically constructs execution pipelines

• Medusa: Accelerating Serverless LLM Inference with Materialization: target at cold-start of LLM serverlesss, to solve the available KV cache blocks profiling and cuda graph capture problems, accepted by ASPLOS'25

• SMore: Enhancing GPU Utilization in Deep Learning Clusters by Serverless-based Co-location Scheduling: serverless computing reveals an opportunity to optimize gpu utilization with fine-grained resource allocation

• PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling: rapidly launch inference services in response to bursty requests without preemptively over-provisioning GPUs

• FluidFaaS: A Dynamic Pipelined Solution for Serverless Computing with Strong Isolation-based GPU Sharing: serverless computing + MIG to fully use GPU, accepted by HPDC'25

• DEEPSERVE: Serverless Large Language Model Serving at Scale: Industrial-grade framework for serverless LLM serving, accepted by ATC'25

• Torpor: GPU-Enabled Serverless Computing for Low-Latency, Resource-Efficient Inference: uses various techniques, including asynchronous API redirection, GPU runtime sharing, pipelined model execution, and efficient GPU memory management, to minimize latency overhead caused by model swapping, accepted by ATC'25

• FlashServe: Cost-Efficient Serverless Inference Scheduling for Large Language Models via Tiered Memory Management and Predictive Autoscaling: fast model loading to reduce cold-start, even support adapter multiplexing (not clear the difference with ServerlessLoRA)

• Towards Resource-Efficient Serverless LLM Inference with SLINFER: support serverless LLM deployment tailored for small- to mid-sized LLMs that enables elastic and on-demand sharing, and further leveraging CPUs, accepted by HPCA'26

Aligning Systems

• PUZZLE: Efficiently Aligning Large Language Models through Light-Weight Context Switch

Comm kernels

• Enabling Elastic Model Serving with MultiWorld: optimizing collective communication lib for LLM inference
• Flexible Scheduling of Network and Computing Resources for Distributed AI Tasks
• AdapCC: Making Collective Communication in Distributed Machine Learning Adaptive: communicating strategy based on runtime, ICDCS'24
• Crux: GPU-Efficient Communication Scheduling for Deep Learning Training: a communication scheduler that aims to maximize GPU computation utilization by mitigating the communication contention among DLT jobs, SIGCOMM'24

Dynamic resource

• TENPLEX: Changing Resources of Deep Learning Jobs using Parallelizable Tensor Collections: by Luo MAI, similar to SpotServe?
• SpotServe: Serving Generative Large Language Models on Preemptible Instances: by Xupeng MIAO and under guidence of Zhihao JIA
• Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances: by team of SpotServe
• FaPES: Enabling Efficient Elastic Scaling for Serverless Machine Learning Platforms: a FaaS-oriented Performance-aware Elastic Scaling system to enable efficient resource allocation in serverless platforms for ML jobs, accepted by SoCC'24
• Serving Models, Fast and Slow:Optimizing Heterogeneous LLM Inferencing Workloads at Scale: resource allocation at cluster and data center scale
• BLITZSCALE: fast and live large model autoscaling with O(1) host caching: loading parameters through the compute network between GPUs, autoscaling in a fine-grained layer-level, accepted by OSDI'25
• Aegaeon: Effective GPU Pooling for Concurrent LLM Serving on the Market: reduces auto-scaling overhead by 97% through component reuse, explicit memory management, and fine-grained KV cache synchronization
• Cloud Native System for LLM Inference Serving: use cloud native system to elastic scaling
• Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling: an elastic system that facilitates dynamic and fine-grained scaling through dynamically regulates module-level resource allocation and performance optimization
• Temporal-Aware GPU Resource Allocation for Distributed LLM Inference via Reinforcement Learning: a two-layer design: a macro-level scheduler leverages reinforcement learning and optimal transport to coordinate inter-region task distribution, while a micro-level allocator refines task-to-server assignments within each region to reduce latency and switching costs
• ⭐ From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models: an operator-level autoscaling framework, which allocates resources at finer (operator)-granularity, optimizing the scaling, batching, and placement based on individual operator profiles (furthermore, scaling to multi-agent system)
• MaverIQ: Fingerprint-Guided Extrapolation and Fragmentation-Aware Layering for Intent-Based LLM Serving: use Layer-based fingerprint to analyse deployment configuration performance, propose insight: unbanlanced layer deployment for pipeline parallel in inference give limited effect on latency (not very make sense), accepted by SC'25
• ZipBatch: Multi-Tenant GPU Batching with Dual-Resource Regulation: elastic fine-grained GPU resource provisioning for different tenant to reduce mismatch between heterogeneous burstiness and immutable resource configurations, accepted by SoCC'25

Request Scheduling

• Compass: A Decentralized Scheduler for Latency-Sensitive ML Workflows: scheduler for latency-sensitive request
• Llumnix: Dynamic Scheduling for Large Language Model Serving: scheduling in multi instances may by helpful for me now
• Arlo: Serving Transformer-based Language Models with Dynamic Input Lengths: solve Dynamic Input Lengths by multi-instance and request scheduling
• Intelligent Router for LLMSandwich Workloads: Improving Performance Through Workload-Aware Scheduling: scheduling based on a output length predictor
• Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMs: request scheduling in cluster and on instance
• Fast Inference for Augmented Large Language Models: schedule for Augmented LLM
• ALISE: Accelerating Large Language Model Serving with Speculative Scheduling: prediction-based scheduling + memory management + quantization's hodgepodge
• The Effect of Scheduling and Preemption on the Efficiency of LLM Inference Serving: cost model in request scheduling
• Queue Management for SLO-Oriented Large Language Model Serving: schedule for request with differnt models and differnet SLO requirements
• FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving: fairness and request switch
• HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location: request co-location to maximize serving throughput and prevent starvation, without compromising online serving latency
• Locality-aware Fair Scheduling in LLM Serving
• Queueing, Predictions, and LLMs: Challenges and Open Problems: prediction-based queueing and serving
• Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents: throughput-optimal scheduling analyse
• Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving: a memory-efficient hidden cache? and schedule to use a biger batch
• LLMSched: Uncertainty-Aware Workload Scheduling for Compound LLM Applications: an uncertainty-aware scheduling framework for emerging compound LLM applications, accepted by ICDCS'25
• SkyLB: A Locality-Aware Cross-Region Load Balancer for LLM Inference: schedule requests on instances on different regions to suite the regional diurnal patterns, under guidence of Ion Stoica
• Libra: Novel LLM Token Streaming via Region-Based Task Scheduling and Token Bundling: introduce user regions to relieve the network issues, since users from the same region (e.g., universities and business zones) are likely to share similar behaviors to access LLM serving systems
• A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Serving: use a structurally-informed online performance model that provides accurate, forward-looking per-step latency and capacity estimations to schedule requests in and among instances
• SLICE: SLO-Driven Scheduling for LLM Inference on Edge Computing Devices: an innovative scheduling solution designed for edge computing scenarios with differentiated SLO requirements, combined with a utility-maximizing request scheduling algorithm with a dynamic iterative control mechanism for generation rates
• FairBatching: Fairness-Aware Batch Formation for LLM Inference: an adaptive Prefill/Decode batch capacity determination mechanism, which dynamically adjusts the computational budget to improve the GPU utilization without triggering SLO violations, optimization on sarathi, under guidence of Haibo CHEN
• Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live: a serving system to optimize job completion time for multi-turn agent workloads by combining tool-aware KV cache timeout with program-level scheduling, under guidence of Ion Stoica
• LLM Inference Scheduling: A Survey of Techniques, Frameworks, and Trade-offs: a survey for request scheduling, from Huawei
• Large-Scale LLM Inference with Heterogeneous Workloads: Prefill-Decode Contention and Asymptotically Optimal Control: scheduling workload by formulating LLM inference as a multiclass many-server queueing network with state-dependent service rates

Shared Prefix Serving

• ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition: share prefix and optimize KV Cache

Serving for LoRA

• S-LoRA: Serving Thousands of Concurrent LoRA Adapters: beginninf of Serving for LoRA, under the guidence of Ion Stoica: accepted by MLSys'24
• Dynamic LoRA Serving System for Offline Context Learning: successor of S-LoRA
• CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference: serving LoRA is becoming more and more important, accepted by ATC'25
• TOPPINGS: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference: CARAServer's ATC version
• PUNICA: MULTI-TENANT LORA SERVING: accepted by MLSys'24
• Petals: Collaborative Inference and Fine-tuning of Large Models
• LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design: maybe useful, kernel optimization
• dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving: accepted by OSDI'24
• Enhancing LoRA Model Serving Capacity via Adaptive Operator Scheduling for Multi-Tenancy on GPU: optimize SGMV kernels
• V-LoRA: An Efficient and Flexible System Boosts Vision Applications with LoRA LMM: LoRA for vision models, and optimize LoRA kernels, accepted by EuroSys'25
• Efficient Multi-task LLM Quantization and Serving for Multiple LoRA Adapters: facilitates the sharing of a single quantized model for multiple LoRA adapters, accepted by NIPS'24
• Comparative Analysis and Optimization of LoRA Adapter Co-serving for Large Language Models: more like a survey for LoRA serving
• Dynamic Operator Optimization for Efficient Multi-Tenant LoRA Model Serving: introducing Dynamic Operator Optimization (Dop), an advanced automated optimization technique designed to dynamically optimize the Segmented Gather Matrix-Vector Multiplication (SGMV) operator based on specific scenarios, accepted by AAAI'25
• DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMs: compress model deltas to serves multiple full-parameter fine-tuned models(maybe not LoRA fine-tune?)
• ServerlessLoRA: Minimizing Latency and Cost in Serverless Inference for LoRA-Based LLMs: in fact similar to S-LoRA, on the background of serverless LLM+LoRA model
• Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management: GPU memory management on both KV cache and LoRA weights
• Katz: Efficient Workflow Serving for Diffusion Models with Many Adapters: take ControlNet off the critical path with a ControlNet-as-a-Service design in T2I service, accepted by ATC'25
• ExpertWeave: Efficiently Serving Expert-Specialized Fine-Tuned Adapters at Scale: serve multiple LoRAs for MoE models
• Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments: cache LoRA adapters on GPU and LoRA scheduling, accepted by MICRO'26
• Maximizing GPU Efficiency via Optimal Adapter Caching: An Analytical Approach for Multi-Tenant LLM Serving: an analytical, AI-driven pipeline that accurately determines the optimal allocation of adapters in single-node setups
• EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices: Deploying fine-tuned LLMs on multi-tenant edge devices, accepted by MobiSys'25
• Cannikin: No Lagger of SLO in Concurrent Multiple LoRA LLM Serving: optimizes the minimum of the SLO attainments of all LoRA services in the serving system, denoted as lagger-SLO attainment, under guidence of Xin JIN, accepted by TPDS'25
• Rock: Serving Multimodal Models in Cloud with Heterogeneous-Aware Resource Orchestration for Thousands of LoRA Adapters: serving thousands of LoRA adapters for multimodal models, accepted by CLUSTER'25
• Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving: unified LLM Fine-tuning and Serving, accepted by NIPS'25
• Predictive-LoRA: A Proactive and Fragmentation-Aware Serverless Inference System for LLMs: proactive, prediction-based LoRA loading, and LoRA weights management to reduce memory fragmentation
• MixLoRA: An Efficient Multi-Tenant Framework for Concurrently Serving Diverse LoRA Models in Large Language Models: dispatch cuda streams to different LoRA rank to optimize sgmv kernel, accepted by ICPP'25
• Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving: optimize LoRA to Block-Diagonal to eliminate communication overhead in S-LoRA tensor parallel
• AuLoRA: Fine-Grained Loading and Computation Orchestration for Efficient LoRA LLM Serving: layer-wise LoRA loading and more fine-grained overlapping, and dynamic LoRA batching, accepted by ICCD'25
• Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems: a workload-aware dynamic adapter placement and routing framework designed to tame rank diversity, by dynamically rebalancing adapters across GPUs and leveraging GPU Direct RDMA for remote access

---

For LoRA but not serving

• ASPEN: High-Throughput LoRA Fine-Tuning of Large Language Models with a Single GPU
• LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin: potential new style of LoRA
• Higher Layers Need More LoRA Experts
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
• FlexLLM: A System for Co-Serving Large Language Model Inference and Parameter-Efficient Finetuning: how to find novel questions?
• LoRA Meets Dropout under a Unified Framework: Analyze LoRA algorithmically
• HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning: algorithm optimization for LoRA
• SBoRA: Low-Rank Adaptation with Regional Weight Updates: an algorithm optimization for LoRA
• A Survey on LoRA of Large Language Models: survey of LoRAs, incluing parallel LoRA computing and Multi-LoRA, github
• mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs: can study the LoRA-aware pipeline parallelism scheme, github, accepted by VLDB'25
• MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts: LoRA based MoE, github
• GongBu: Easily Fine-tuning LLMs for Domain-specific Adaptation: LLM fine-tuning tools
• Adapters Selector: Cross-domains and Multi-tasks LoRA Modules Integration Usage Method: select several LoRAs for a content
• SplitLLM: Hierarchical Split Learning for Large Language Model over Wireless Network: split learning(?) train lora weights in wireless network environment, store lora in edge servers?
• Revolutionizing Large Model Fine-Tuning: The Role of LoRA in Parameter-Efficient Adaptation: a survey, can provide some reference
• HyC-LoRA: Memory Efficient LoRA Fine-tuning with \textbf{Hy}brid Activation \textbf{C}ompression: optimize fine-tune memory overhead by quantization, accepted by MLSYS'25
• ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory: fine-tune
• HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models: split-learnng + LoRA, fine-tune on client device, set different rank for different weights
• Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management: explore the dependencies between requests and LoRAs to reduce TTFT
• Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics: LoRA algorithm analyze
• Ratel: Optimizing Holistic Data Movement to Fine-tune 100B Model on a Consumer GPU: with 1) active gradient offloading, and 2) holistic traffic-aware activation swapping mechanism, accepted by ICDE'25
• Communication-Efficient MoE Fine-Tuning with Locality-Aware Expert Placement: fine-tune on MoE models with locality awareness (not sure difference with training), accepted by ICDCS'25
• JENGA: Enhancing LLM Long-Context Fine-tuning with Contextual Token Sparsity: utilize sparsity for fine-tuning, accepted by ATC'25
• mTuner: Accelerating Parameter-Efficient Fine-Tuning on Multi-GPU Servers with Elastic Tensor: use elastic tensors to do optimizations such as improving temporal memory utilization, relaxing data dependence, and accumulating runtime tensors in a memory-adaptive way, accepted by ATC'25
• PLoRA: Efficient LoRA Hyperparameter Tuning for Large Models: automatically orchestrates concurrent LoRA fine-tuning jobs under given hardware and model constraints
• LoRAFusion: Efficient LoRA Fine-Tuning for LLMs: concurrently fine-tune multiple independent LoRA adapters that share the same base model on the same set of GPUs, optimization on both kernels and scheduling, accepted by EuroSys'26
• Symbiosis: Multi-Adapter Inference and Fine-Tuning: enabling the as-a-service deployment of the base model to do concurrently fine-tune and inference
• LoRAQuant: Mixed-Precision Quantization of LoRA to Ultra-Low Bits: post-train quantization on LoRA adapters

Combining fine-tuning/training with inference

• Deferred Continuous Batching in Resource-Efficient Large Language Model Serving
• Latency-Guaranteed Co-Location of Inference and Training for Reducing Data Center Expenses: place training and inference together, control the inference latency to the desired SLO, while maximizing the throughput of the training jobs co-located on the same GPUs, accepted by ICDCS'24
• Colocating ML Inference and Training with Fast GPU Memory Handover: handle the GPU memory useage in both inference and training, accepted by ATC'25
• Resource Multiplexing in Tuning and Serving Large Language Models: a new iteration-level multitasking scheduling mechanism, an Autograd engine that transforms a tuning task into a suspendable pipeline, and an inference engine capable of batching inference and tuning requests, accepted by ATC'25
• LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems: co-locating and managing concurrent LLM serving and training workloads with offline profiling, execution prediction mechanisms, and runtime scheduling to dynamically adapt resource allocation

Serving Long-Context

Long-Context is a hot point recently.

• Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis
• Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference: like a update for H2O or Dejevu, et.al, each attention head have different memory budget
• Context Parallelism for Scalable Million-Token Inference
• TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection: select some important KV cache to take part in attention computation
• DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism: consider the dynamic nature of training data, specifically, the variability in sequence lengths and token relationships (i.e., attention patterns) across samples, accepted by SOSP'25
• IC-Cache: Efficient Large Language Model Serving via In-context Caching: leveraging historical request-response pairs from larger models as in-context examples, introduces intricate trade-offs between response quality, latency, and system throughput, accepted by SOSP'25
• CascadeInfer: Low-Latency and Load-Balanced LLM Serving via Length-Aware Scheduling: dynamically reschedules requests across multiple instances serving the same LLM to mitigate per-instance length heterogeneity by partitioning these instances into length-specialized groups, each handling requests within a designated length range, naturally forming a pipeline

Complex ML loads

Process differnet ML loads in a cluster.

• PAL: A Variability-Aware Policy for Scheduling ML Workloads in GPU Clusters: serve multiple different loads in GPU cluster, accepted by SC'24
• PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption: why Encryption in LLM inference? by IPADS, accepted by ASPLOS'25
• Topology-aware Preemptive Scheduling for Co-located LLM Workloads: schedule different workloads

RAG with LLM

• ⭐ Chameleon: a heterogeneous and disaggregated accelerator system for retrieval-augmented language models: retrieval will be helpful, but how to use it?
• Generative Dense Retrieval: Memory Can Be a Burden: accepted by EACL'24
• ⭐ Accelerating Retrieval-Augmented Language Model Serving with Speculation: also a paper for RaLM
• RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation: improve RAG inference with cache, under guidence of Xin JIN
• FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research
• Accelerating Retrieval-Augmented Language Model Serving with Speculation: help understand RaLM
• NinjaLLM: Fast, Scalable and Cost-effective RAG using Amazon SageMaker and AWS Trainium and Inferentia2
• Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting: RAG with spec decoding, different draft models with different RAG
• CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion: optimize KV cache reuse(prefix cache)
• RAGServe: Fast Quality-Aware RAG Systems with Configuration Adaptation: trade-off between latency and quantity
• Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation: combine RAG with prefix cache
• RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving: analyse RAG algorithm then optimize system
• CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion: reuses the precomputed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache, accepted by EuroSys'25
• Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs: KV cache for RAG knowledge which is stored on disk
• HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Serving: HedraRAG, a runtime system built on a graph-based abstraction that exposes optimization opportunities across stage-level parallelism, intra-request similarity, and inter-request skewness, implement these optimizations by dynamic graph transformations and finally mapped onto hybrid CPU-GPU pipelines, accepted by SOSP'25

Combine MoE with LLM inference

Here are two repositories have some papers for MoE: Papers: MoE/Ensemble, and MOE papers to read

• ⭐ DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale: accepted by ICML'22

• Accelerating Distributed MoE Training and Inference with Lina: both training and inference, accepted by ATC'23

• MegaBlocks: Efficient Sparse Training with Mixture-of-Experts: accepted by MLSys'23

• Tutel: Adaptive Mixture-of-Experts at Scale: accepted by MLSys'23

• Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference: accepted by ISCA'24

• Optimizing Mixture of Experts using Dynamic Recompilations: under guidence of Zhihao JIA

• Serving MoE Models on Resource-constrained Edge Devices via Dynamic Expert Swapping: expert swapping is interesting

• Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference: some hot optimizations for inference, accepted by NIPS'24

• Exploiting Transformer Activation Sparsity with Dynamic Inference

• SE-MoE: A Scalable and Efficient Mixture-of-Experts Distributed Training and Inference System

• Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production: accepted by ACL'22

• Fast Inference of Mixture-of-Experts Language Models with Offloading: combine moe with offloading

• ⭐ MoE-Infinity: Activation-Aware Expert Offloading for Efficient MoE Serving: under guidence of Luo MAI, provided some features and design in moe inference

• Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

• FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement: train MoE with new schedule plan, maybe work for inference

• Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference

• EdgeMoE: Fast On-Device Inference of MoE-based Large Language Models: quantized experts and expers management

• Toward Inference-optimal Mixture-of-Expert Large Language Models: some analysis for training moe based on inference cost

• [Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules]: comm optimization in MoE, accepted by InfoCom'24

• SiDA: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models: based on offload, accepted by MLSys'24

• Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy: introduce some features of MoE, accepted by ICLR'24

• Demystifying the Compression of Mixture-of-Experts Through a Unified Framework: introduce some features of MoE too

• Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models: introduction paper

• Efficient All-to-All Collective Communication Schedules for Direct-Connect Topologies: all_to_all comm, HPDC'24

• Scattered Mixture-of-Experts Implementation: ScatterMoE, an implementation of Sparse MoE

• Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts: the Shortcut-connection looks more like a algorithm optimization, and provide oppotunity for overlapping

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model: a opsen-source work and it inferences based expert-parallel

• SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget: MoE experts offloading, at the cost of reduced accuracy

• ProMoE: Fast MoE-based LLM Serving using Proactive Caching: optimization on Pre-gated MoE, by IPADS

• Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design: pre-gating router decoupled from the MoE backbone that facilitates system-friendly pre-computing and lookahead scheduling, NIPS'24

• MoEsaic: Shared Mixture of Experts: share Expert among different MoE instance, "MoE's modular architecture lets users compose their model from popular off-the-shelf experts" is a new scenario

• HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference: use quantization to decrease uncached MoE load overhead, on edge devices

• ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference: prediction and offload based optimization

• MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs: use offload-pipeline to accelerate inference moe on single GPU

• ⭐ MoE-CAP: Cost-Accuracy-Performance Benchmarking for Mixture-of-Experts Systems: benchmarking for MoE systems

• ⭐ Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection: damn! I had considered this before:( . key insight is that expert importance varies significantly across tokens and inference phases, utilize this to solve the all-activate problem

• ⭐ EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference: Gemm implemention optimization and alltoall communication overlap

• ⭐ Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling: optimize all2all order, co-locate experts from different models

• MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing: utilize the expert dependency to opmizate GPU load balance and alltoall latency

• fMoE: Fine-Grained Expert Offloading for Large Mixture-of-Experts Serving: fine-grained expert offload, prefetch and cache

• ⭐ Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts: fine-grained task schduling and computation-alltoall overlap

• MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs: offload MoE weights to CPU by layers, accepted by ASPLOS'25

• eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference: predict to preload experts from cpu, use same expert for subsequent prompts and skip routing for some tasks

• MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints: CPU-GPU based MoE inference

• Faster MoE LLM Inference for Extremely Large Models: less activated experts for faster inference

• ⭐ Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchrony: By dynamically queuing tokens at each layer (referred to as μ-queuing), GPUs avoid waiting for straggling experts and instead continuously process whichever layer is ready

• ⭐ Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving: fine-grained resource scaling and provides inherent fault tolerance by decoupling to EaaS, powered by a high-performance, CPU-free peer-to-peer communication library

• [] BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMs: merge experts, adjust the processing of certain tokens, which means adjusting the algorithm to improve efficiency

• A3D-MoE: Acceleration of Large Language Models with Mixture of Experts via 3D Heterogeneous Integration: hardware-software co-design but the technical aspects seem a bit strange

• MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs: deconstructs monolithic experts into fine-grained "sub-experts" and then schedule

• MixServe: An Automatic Distributed Serving System for MoE Models with Hybrid Parallelism Based on Fused Communication Algorithm: TP-EP hybrid parallelism based on fused AR-A2A communication algorithm that overlaps intra-node AR communication and inter-node A2A communication, accepted by ICDCS'26

MoE training

• ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling: scheduling comp and comm in MoE training, perhaps useful for MoE inference. accepted by EuroSys'24

• ST-MoE: Designing Stable and Transferable Sparse Expert Models: a start work in MoE

• MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services

• Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models: algorithm change in MoE

• Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping: Computation-Communication Overlapping, accepted by MLSys'24

• Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training: training with offload, ICML'24

• MPMoE: Memory Efficient MoE for Pre-Trained Models With Adaptive Pipeline Parallelism

• Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules: Dedicated Schedules for MP+EP+ESP MoE training, maybe work for infernece

• Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing: load is stabilized in the middle and late stages of training, but may not wrok greatly for insference

• SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Offline and Online Parallelization: parallel strategy of MoE, accepted by ATC'23

• APTMoE: Affinity-Aware Pipeline Tuning for MoE Models on Bandwidth-Constrained GPU Nodes: fine-tune MoE models with CPU and some algorithm insights, accepted by SC'24

• Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing: prediction the expert workload to optimize training

• FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models: There isn't much of a novel technology(?), accepted by ASPLOS'25

• Mast: Efficient Training of Mixture-of-Experts Transformers with Task Pipelining and Ordering: optimize overlapping in MoE training, accepted by ICDCS'25

• PopFetcher: Towards Accelerated Mixture-of-Experts Training Via Popularity Based Expert-Wise Prefetch: expert prefetch, accepted by ATC'25

Inference with multimodal

• MOSEL: Inference Serving Using Dynamic Modality Selection: improving system throughput by 3.6x with an accuracy guarantee and shortening job completion times by 11x
• Generative AI Beyond LLMs: System Implications of Multi-Modal Generation: by META
• Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizations: by Google
• Accelerating Text-to-image Editing via Cache-enabled Sparse Diffusion Inference: optimization for diffusion models by cache
• DISTMM: Accelerating distributed multimodal model training: helpful although it is made for training, accepted by NSDI'24
• Addressing Model and Data Heterogeneity in Multimodal Large Language Model Training: distributed MM trainging
• Efficiently serving large multimedia models using EPD Disaggregation
• MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Serving: position-independent caching, with both reuse and recompute, may lead to performance loss
• Characterizing and Efficiently Accelerating Multimodal Generation Model Inference: some insights
• ModServe: Scalable and Resource-Efficient Large Multimodal Model Serving: provide comprehensive systems analysis of two prominent LMM architectures, decoder-only and cross-attention
• HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Serving: a Hybrid Encode-Prefill-Decode (EPD) Disaggregation architecture
• ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism: introduce Elastic Multimodal Parallelism (EMP), a new serving paradigm that elastically adapts to resource heterogeneity across request types and inference stages, under guidence of Yang YOU
• RServe: Overlapping Encoding and Prefill for Efficient LMM Inference: orchestrates intra- and inter-request pipelines for LLM inference
• Nova: Real-Time Agentic Vision-Language Model Serving with Adaptive Cross-Stage Parallelization: for VLM serving
• Argus: Quality-Aware High-Throughput Text-to-Image Inference Serving System: selects the right level of approximation for each prompt to maintain quality while meeting throughput targets
• ⭐ vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models: disaggregate multimodal serving into different stages for any-to-any models, support batching, GPU allocation and communication connector
• OnePiece: A Large-Scale Distributed Inference System with RDMA for Complex AI-Generated Content (AIGC) Workflows: split MultiModal pipeline into fine-grained microservice, and use GPU RDMA for communication
• Cornserve: Efficiently Serving Any-to-Any Multimodal Models: allows a computation graph consists of heterogeneous components, then use a planer to to find an optimized deployment plan, finally run it with a distributed runtime
• Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing: optimize the three-stage pipeline: multimodal preprocessing, vision encoding, and LLM inference

Training in Multimodal

• DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models: disaggregation in MM training, under guidence of Xin JIN
• Efficient Multi-Task Large Model Training via Data Heterogeneity-aware Model Management: efficient MM model training
• Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling: ASPLOS'25

Diffusion Models

• Approximate Caching for Efficiently Serving Text-to-Image Diffusion Models: serving Diffusion models, accepted by NSDI'24
• DiffusionPipe: Training Large Diffusion Models with Efficient Pipelines: accepted by MLSys'24
• SwiftDiffusion: Efficient Diffusion Model Serving with Add-on Modules: more papers in diffusion models
• PATCHEDSERVE: A Patch Management Framework for SLO-Optimized Hybrid Resolution Diffusion Serving: algorithm-based framework
• DiffServe: Efficiently Serving Text-to-Image Diffusion Models with Query-Aware Model Scaling: address the problem of serving text-to-image generation diffusion models in a query-aware resource-efficient manner by serving "easy" queries using a lightweight diffusion model without compromising image generation quality
• FlashPS: Efficient Generative Image Editing with Mask-aware Caching and Scheduling: towards a very specfic application: Generative image editing using diffusion models, accepted by EuroSys'26
• TridentServe: A Stage-level Serving System for Diffusion Pipelines: automatically, dynamically derives the placement plan for the encode-diffuse-decode three-stage, with dispatch plan, under guidence of Bin CUI
• Understanding Diffusion Model Serving in Production: A Top-Down Analysis of Workload, Scheduling, and Resource Efficiency: the actual industrial trace of multimodal, propose a targeted optimization system, accepted by SoCC'25
• ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving: analyse the variable, heavy-tailed request distributions, and decouples model stages for different LMM architectures, accepted by SoCC'25
• MixFusion: A Patch-Level Parallel Serving System for Mixed-Resolution Diffusion Models: fine-grained patch-level parallelism to enable efficient batching of mixed-resolution requests and a patch-tailored cache management policy to exploit the patch-level locality benefits, accepted by PPoPP'26
• ChituDiffusion: A Data-Characteristic-Aware Serving System for Diffusion Models: leverages the locality of data properties to recompose a diffusion pipeline into dGraphs with shared optimization opportunities, enabling thorough compile-time and runtime co-optimizations, accepted by PPoPP'26

Compound Inference Systems

What is this? maybe multiple LLM?

• Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems: a new scenario, by Stanford
• ALTO: An Efficient Network Orchestrator for Compound AI Systems: also new to me, by Stanford
• Proteus: A High-Throughput Inference-Serving System with Accuracy Scaling: accuracy scaling is interesting, accepted by ASPLOS'24
• ⭐ MuxServe: Flexible Multiplexing for Efficient Multiple LLM Serving: multiple LLMs
• ROUTERBENCH: A Benchmark for Multi-LLM Routing System: but what is multi-LLM?
• Expert Router: Orchestrating Efficient Language Model Inference through Prompt Classification
• BlockLLM: Multi-tenant Finer-grained Serving for Large Language Models
• Prompt Cache: Modular Attention Reuse for Low-Latency Inference: prompt KV cache reuse, accepted by MLSys'24
• Preble: Efficient Distributed Prompt Scheduling for LLM Serving: similar to BlockLLM?
• Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Execution: for LLM-based Applications
• Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems
• RouteLLM: Learning to Route LLMs with Preference Data: use multiple LLMs for efficient serving
• USHER: Holistic Interference Avoidance for Resource Optimized ML Inference: inference several models simultaneously
• CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory: a new scenario Collaboration-of-Experts instead of mixture-of-experts, provide some new oppotunities, acceped by ASPLOS'25
• ⭐ SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference: enables service-aware and latency-optimized LLM sharing on same device
• HybridServe: Efficient Serving of Large AI Models with Confidence-Based Cascade Routing: use multiple small model to achieve the accuracy of serving solely with giant DNNs(?)
• Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts: algorithm to route user requests from mobile and IoT devices to an appropriate LLM service (i.e., edge LLM expert) to ensure acceptable quality-of-service (QoS)
• Efficient Routing of Inference Requests across LLM Instances in Cloud-Edge Computing: a novel routing algorithm based on the Non-dominated Sorting Genetic Algorithm II (NSGA-II) to distribute inference requests across heterogeneous LLM instances in a cloud-edge computing environment

Embodied AI agent

• Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Execution: disaggregates the perception and generation and provides controlled pipeline parallelism for them, and establishes a public context for perception and generation to share

Reasoning LLM

• Reasoning Language Model Inference Serving Unveiled: An Empirical Study: model quantization methods and speculative decoding can improve service system efficiency with small compromise to RLLM accuracy, while prefix caching, KV cache quantization may even degrade accuracy or serving performance for small RLLM

LLM Application

• Teola: Towards End-to-End Optimization of LLM-based Applications: endd-to-end optimization
• Parrot: Efficient Serving of LLM-based Applications with Semantic Variable: accepted by OSDI'24
• Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications: many LLM apps share GPU, accepted by EuroSys'24
• Fast Inference for Augmented Large Language Models: seems a subclass of multi-agent
• Towards End-to-End Optimization of LLM-based Applications with Ayo: utilizes task primitives as the basic units and represents each query's workflow as a primitive-level dataflow graph, enables optimizations in parallelization, pipelining across primitives of different modules, and enhances scheduling to improve application-level performance
• Improving the End-to-End Efficiency of Offline Inference for Multi-LLM Applications Based on Sampling and Simulation: multi-LLM's end-to-end running
• Tempo: Application-aware LLM Serving with Mixed SLO Requirements: Meet SLO requirements for all services in the system and accelerate the overall process, can be used in multi-agent application
• Pie: A Programmable Serving System for Emerging LLM Applications: decomposes the traditional generation loop into fine-grained service handlers exposed via an API and delegates control of the generation process to user-provided programs, accepted by SOSP'25
• Efficient Serving of LLM Applications with Probabilistic Demand Modeling: the resource demands of LLM applications can be modeled in a general and accurate manner with Probabilistic Demand Graph (PDGraph), and applies the Gittins policy to determine the scheduling order that can minimize the average application completion time
• Justitia: Fair and Efficient Scheduling for LLM Applications: schedule a series of LLM inferences (LLM applications) on shared GPUs

Multi-agent system

• The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective: a survey for agents serving
• Why Do Multi-Agent LLM Systems Fail?: learn algorithm from it
• ⭐ Autellix: An Efficient Serving Engine for LLM Agents as General Programs: multi-agent has something similar to LLM application, scheduling and preemption
• Kairos: Low-latency Multi-Agent Serving with Shared LLMs and Excessive Loads in the Public Cloud: existing works have low serving performance for these multi-agent applications, mainly due to the ignorance of inter-agent latency and resource differences for request scheduling, this paper solve it
• KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows: a workflow-aware KV cache management framework tailored for agentic workloads instead of simple LRU policy, and a CPU-GPU prefetch machanism
• ⭐ Batch Query Processing and Optimization for Agentic Workflows: represents each workflow as a structured query plan DAG and constructs a consolidated graph for batched queries that exposes shared computation
• What Limits Agentic Systems Efficiency?: based on that web environment latency can contribute as much as 53.7% to the overall latency in a web-based agentic system
• A CPU-Centric Perspective on Agentic AI: tool processing on CPUs can take up to 90.6% of the total latency, really amazing idea
• DyOrc: Efficient Serving of Dynamic Machine Learning Workflows: models as services and an efficient serving system for dynamic machine learning workflows with conditionally activated model components, to optimize the workflow system, accepted by SoCC'25
• Aragog: Just-in-Time Model Routing for Scalable Serving of Agentic Workflows: optimize the strategy which is cost-aware assignment of workflow agents to specific LLMs, through both static and dynamic assignment
• Astraea: A State-Aware Scheduling Engine for LLM-Powered Agents: a scheduling engine targets at the request's JCT
• The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective: comprehensive system-level analysis of AI agents, quantifying their resource usage, latency behavior, energy consumption, and datacenter-wide power consumption demands across diverse agent designs and test-time scaling strategies, accepted by HPCA'26

Fault Tolerance

• Characterization of Large Language Model Development in the Datacenter: fault-tolerant serving in the future?
• Lazarus: Resilient and Elastic Training of Mixture-of-Experts Models with Adaptive Expert Placement: Fault Tolerance in MoE training
• Partial Experts Checkpoint: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training: checkpointing in MoE
• Robust LLM Training Infrastructure at ByteDance: from bytedance, in training, accepted by SOSP'25
• Enhancing reliability in AI inference services: An empirical study on real production incidents: identifies dominant failure modes (in our dataset ~60% inference engine failures, within that category ~40% timeouts)
• THORN-ML: Transparent Hardware Offloaded Resilient Networks for RDMA based Distributed ML Workloads: fault tolerance for network

Energy Optimization

It is usually related to CPU-GPU heterogeneity and GPU power consumption.

• DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
• Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems
• AGFT: An Adaptive GPU Frequency Tuner for Real-Time LLM Inference Optimization: employs online reinforcement learning to autonomously learn an optimal frequency tuning policy
• GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving: minimizes GPU energy by explicitly separating prefill and decode control
• VoltanaLLM: Feedback-Driven Frequency Control and State-Space Routing for Energy-Efficient LLM Serving: tune GPU frequency
• Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use: a survey to understand GPU energy cost
• A Universal Load Balancing Principle and Its Application to Large Language Model Serving: optimize load balance to save energy

Early Exits

• Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving: early exits, accepted by SOSP'24
• Improving DNN Inference Throughput Using Practical, Per-Input Compute Adaptation: early exits and some system optimization, accepted by SOSP'24
• SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting: using speculative models to reduce the search space of early exiting predictor, accepted by ISCA'25
• Accelerating Large Language Model Inference via Early-Exiting Algorithms: can help learns early-exits

RLHF

• OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework: framework for RLHF
• HybridFlow: A Flexible and Efficient RLHF Framework: framework for RLHF, accepted by EuroSys'25
• RLHFuse: Efficient RLHF Training for Large Language Models with Inter- and Intra-Stage Fusion
• Systems Opportunities for LLM Fine-Tuning using Reinforcement Learning: optimization for LLM Fine-Tuning using Reinforcement Learning
• ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation: accepted by MLSYS'25
• Optimizing RLHF Training for Large Language Models with Stage Fusion: breaks the traditional view of RLHF workflow as a composition of individual tasks, splitting each task into finer-grained subtasks, and performing stage fusion to improve GPU utilization, accepted by NSDI'25
• Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Models: use RL to fine-tune LLM, could be a potential baseline framework

Some Interesting Idea

Wise men learn by others.

• Orca 2: Teaching Small Language Models How to Reason
• FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference: optimization for retrieval-augmented language model
• Optimizing Dynamic Neural Networks with Brainstorm: this idea has the potential to go further, accepted by OSDI'23
• Ring Attention with Blockwise Transformers for Near-Infinite Context: Ring Attention?
• Reducing Activation Recomputation in Large Transformer Models: by NVIDIA
• Cheaply Estimating Inference Efficiency Metrics for Autoregressive Transformer Models: an interesting performance metric, accepted by NIPS'23
• FEC: Efficient Deep Recommendation Model Training with Flexible Embedding Communication: accepted by SIGMOD'23
• Efficient Multi-GPU Graph Processing with Remote Work Stealing: accepted by ICDE'23
• ARK: GPU-driven Code Execution for Distributed Deep Learning: accepted by NSDI'23
• Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphs: accepted by MLSys'22
• Golgi: Performance-Aware, Resource-Efficient Function Scheduling for Serverless Computing: Scheduling for Serverless Computing
• FastFold: Optimizing AlphaFold Training and Inference on GPU Clusters: expand to other ML models instead of LLM
• Arrow Matrix Decomposition: A Novel Approach for Communication-Efficient Sparse Matrix Multiplication
• FinGPT-HPC: Efficient Pretraining and Finetuning Large Language Models for Financial Applications with High-Performance Computing
• Two-Face: Combining Collective and One-Sided Communication for Efficient Distributed SpMM: efficient SpMM, accepted by ASPLOS'24
• GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching: GPU memory pool, accepted by ASPLOS'24
• QuickLLaMA: Query-aware Inference Acceleration for Large Language Models: an inference-friendly LLaMA architecture
• Marconi: Prefix Caching for the Era of Hybrid LLMs: prefix cache for new model arch like combine attention with SSM
• Comprehensive Deadlock Prevention for GPU Collective Communication: communication library
• Quilt: Resource-aware Merging of Serverless Workflows: maybe useful on GPU
• Uno: A One-Stop Solution for Inter- and Intra-Data Center Congestion Control and Reliable Connectivity: network in data centers

Dataflow

I'd like to create a separate area for data flows. It's just my preference.

• ⭐ FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks: dataflow in inference
• Pathways: Asynchronous Distributed Dataflow for ML: accepted by MLSys'22
• VirtualFlow: Decoupling Deep Learning Models from the Underlying Hardware: accepted by MLSys'22
• NeuStream: Bridging Deep Learning Serving and Stream Processing: dataflow in DNN serving, accepted by EuroSys'25

How about data pre-processing overhead in training?

• Pecan: Cost-Efficient ML Data Preprocessing with Automatic Transformation Ordering and Hybrid Placement

GNN

Just my preference.

• Boosting Distributed Full-graph GNN Training with Asynchronous One-bit Communication
• GNNPipe: Scaling Deep GNN Training with Pipelined Model Parallelism
• PckGNN: Optimizing Aggregation Operators with Packing Strategies in Graph Neural Networks: accepted by IPDPS'24
• NPA: Improving Large-scale Graph Neural Networks with Non-parametric Attention: SIGMOD'24
• Eliminating Data Processing Bottlenecks in GNN Training over Large Graphs via Two-level Feature Compression: compress node features in graph, accepted by VLDB'24
• Mega: More Efficient Graph Attention for GNNs: optimize graph attention efficiency, ICDCS'24
• TORCHGT: A Holistic System for Large-Scale Graph Transformer Training: graph transformer model

Blockchain

Just my preference, too.

• Weaving the Cosmos: WASM-Powered Interchain Communication for AI Enabled Smart Contracts