🚀 From Static to Systematic:
A Survey on Efficient Inference Across LLM, RL, and Agentic Eras

---

📚 Table of Contents

• Latest News
• Overview
  • Three Eras of Inference
  • The Three-Tax View
  • Cross-Era Roadmap
• Paper List
  • Model-Level
    • Model Compression
    • Structural Optimization
  • Algorithm-Level
    • KV Cache Management
    • Speculative Decoding
    • Thinking & Reasoning Efficiency
    • Early Exit & Layer Skipping
  • System-Level
    • Batching & Scheduling
    • Memory Management
    • Hardware & Distributed
• Citation
• Contact

---

🔥 Latest News

• [2026-05-12] 🚀 Repository launched to track efficient inference techniques across the LLM, RL, and Agentic eras.
• [2026-05-12] 🎉 Our survey paper "From Static to Systematic: A Survey on Efficient Inference Across LLM, RL, and Agentic Eras" is now available.

---

📖 Overview

This repository accompanies our COLM 2026 survey, which takes an evolutionary perspective on efficient LLM inference: rather than cataloguing techniques in isolation, we trace how each technique class transfers, adapts, or breaks as the computational paradigm shifts from single-pass generation, to reasoning-time rollouts, to persistent agentic loops.

Three Eras of Inference

① LLM Era — Single-pass, open-loop generation.

Given a prompt x, the model autoregressively produces y token by token. Prefill caches KV in one forward pass; decode attends to all cached tokens, making it memory-bandwidth-bound.

② RL Era — Inference-time reasoning + training-time rollout.

Large Reasoning Models (e.g., DeepSeek-R1, Qwen3) emit thousands of CoT tokens before answering. RL algorithms like GRPO/DAPO require G on-policy trajectories per gradient step, making rollout the dominant pipeline cost — subject to policy non-stationarity, trajectory length heterogeneity, and synchronization barriers.

③ Agentic Era — Persistent multi-step decision loops.

Systems like Claude Code and OpenClaw place the LLM in a loop of generate → tool call → observe → append. Context grows cumulatively; tool latency is unpredictable; KV state may be shared across model instances.

The Three-Tax View

We introduce the Three-Tax view as a unified lens for the cumulative efficiency burden each era introduces:

The eras are cumulative: each inherits prior taxes and introduces new ones.

Cross-Era Roadmap

---

📑 Paper List

Tables follow the paper's three-level taxonomy: Model-Level → Algorithm-Level → System-Level. Within each technique node we further split papers into the LLM Era (foundational methods) and the RL/Agentic Era (era-specific adaptations or newly emerged methods). The Tax label per node indicates which inference tax(es) the technique primarily addresses (🟦 K = KV Tax, 🟩 T = Thinking Tax, 🟥 A = Action Tax).

---

🧱 Model-Level

Model-level methods target the static structure of the deployed model — its parameters, precision, and architectural connectivity. They produce a smaller or sparser artifact that downstream serving stacks consume directly.

Model Compression

Tax addressed: 🟦 K (LLM Era), 🟩 T (RL/Agentic Era)

LLM Era methods reduce parameter count or numerical precision without retraining (Pruning, Quantization, Knowledge Distillation). RL/Agentic Era shifts to training-aware co-evolution: quantization noise is repurposed as policy-gradient regularizer, and full CoT traces become the distillation supervision signal.

LLM Era

RL / Agentic Era

---

Structural Optimization

Tax addressed: 🟦 K (LLM Era); 🟦 K + 🟩 T (Natively Trainable Sparse Attention); 🟦 K + 🟩 T + 🟥 A (Hybrid Architectures)

LLM Era methods redesign architectures for sparse or sub-linear execution (MoE, low-rank KV, sparse attention). RL/Agentic Era shifts focus from external context to self-generated reasoning steps: natively trainable sparse attention reduces the KV footprint of long CoT, and hybrid linear/full-attention architectures balance global modeling with per-step throughput for long-horizon rollouts.

LLM Era

RL / Agentic Era

---

⚙️ Algorithm-Level

Algorithm-level methods modify what and how much the model computes per token / per turn / per agent step — without changing model weights.

KV Cache Management

Tax addressed: 🟦 K (LLM Era); 🟦 K + 🟩 T (RL-guided Compression); 🟦 K + 🟥 A (Cross-turn Persistent Reuse, Cross-agent KV Sharing)

LLM Era focuses on cache reduction (eviction, sparse selection, quantization) and prefix-sharing for local reuse. RL/Agentic Era redefines KV as a long-lived state spanning rollouts and multi-agent interactions: reward-guided compression replaces heuristic eviction, and KV becomes a transferable, persistent asset across turns and across model instances.

LLM Era

RL / Agentic Era

---

Speculative Decoding

Tax addressed: 🟦 K + 🟩 T (token-level SD, accelerates CoT generation); 🟩 T (Trajectory-level Epoch Reuse, exploits policy laziness); 🟥 A (Action-space Speculation, exploits tool latency)

The SD paradigm preserves its draft-and-verify skeleton across all three eras, but the exploited form of redundancy changes fundamentally each time: token locality (LLM) → policy laziness (RL) → tool-latency gaps (Agentic).

LLM Era

RL / Agentic Era

---

Thinking & Reasoning Efficiency

Tax addressed: 🟦 K (Prompt Compression, SoT); 🟩 T (Test-Time Scaling, CoT Compression, Dynamic Compute, Latent CoT); 🟥 A (Agentic Plan Caching)

LLM Era methods target surface-level redundancy with task-agnostic heuristics on tokens (prompt compression, parallel decoding, search). RL/Agentic Era shifts to internalized adaptive control — the model learns to compress its own reasoning, allocate compute by difficulty, internalize CoT in latent space, or cache reusable sub-plans across tasks.

LLM Era

RL / Agentic Era

---

Early Exit & Layer Skipping

Tax addressed: 🟦 K + 🟩 T (Early Exit); 🟦 K (Layer Skipping); 🟩 T (Training-free Intervention, Learnable Exit); 🟩 T + 🟥 A (Agentic Dynamic Compute)

LLM Era methods skip redundant computation at the token level — adaptive exit predictors and self-speculative drafters that reuse shallow layers. RL/Agentic Era shifts the target from physical layer traversal to cognitive trajectory truncation: detecting when CoT is sufficient, RL-trained brevity, and proactive termination of ineffective multi-step trajectories.

LLM Era

RL / Agentic Era

---

🖥️ System-Level

System-level methods optimize how model executions are orchestrated across hardware — batching, memory placement, kernels, and distribution.

Batching & Scheduling

Tax addressed: 🟦 K (LLM Era iteration-level orchestration); 🟩 T (Long-tail Rollout, Async Decoupling); 🟥 A (Workflow-level Scheduling)

LLM Era targets iteration-level orchestration and KV-memory-aware batching (continuous batching, paged memory, SLO-aware scheduling). RL/Agentic Era introduces new scheduling units: long-tail rollout segmentation, async rollout-training decoupling, and DAG-based end-to-end agentic workflow scheduling.

LLM Era

RL / Agentic Era

---

Memory Management

Tax addressed: 🟦 K (Hierarchical Offloading, PD Disaggregation); 🟩 T (Parameter Reallocation, Rollout-aware Offloading); 🟦 K + 🟥 A (Memory-aware Orchestration)

LLM Era targets hierarchical offloading and PD disaggregation to handle contexts beyond accelerator capacity. RL/Agentic Era introduces pipeline-level pressure: dynamic parameter reallocation across Actor/Critic/Reference models, decoupled rollout infrastructure, and KV retention coupled with multi-agent workflow state persistence.

LLM Era

RL / Agentic Era

---

Hardware & Distributed

Tax addressed: 🟦 K (LLM Era IO-aware kernels, tensor/pipeline parallelism); 🟩 T (Asynchronous Disaggregated Architectures); 🟩 T + 🟥 A (Heterogeneous Cluster Scheduling)

LLM Era breaks memory walls and communication bottlenecks (FlashAttention family, unified KV engines, tensor/pipeline parallelism). RL/Agentic Era faces pipeline bubbles between memory-bound rollouts and compute-bound training: asynchronous disaggregated stacks with interruptible rollouts and heterogeneous cluster scheduling.

LLM Era

RL / Agentic Era

---

🌟 Citation

If you find this survey or repository useful for your research, please cite:

@article{wangstatic,
  title={From Static to Systematic: A Survey on Efficient Inference Across LLM, RL, and Agentic Eras},
  author={Wang, Zhenglin and Li, Zhenyang and Wang, Zongwei and Liu, Tong and Wu, Jialong and Liang, Sichu and Zhang, Congzhi and Sun, Haowen and Ma, Ziyang and Lai, Yilong and others}
}

📧 Contact

We welcome contributions! If you would like to add a paper, fix a categorization, or report any issue:

• Open a GitHub Issue or Pull Request, or
• Email the corresponding author Zhenglin Wang at zhenglin@seu.edu.cn.

Tracing how efficient inference evolves across eras, and why. — The spiral continues.