Large Language Model Based Long Context Modeling Papers and Blogs

This repository curates papers and blogs on long-context language modeling, covering surveys; efficient attention; KV-cache optimization; recurrent transformers and state-space models; position encoding & length extrapolation; long-context training; long-term memory; retrieval-augmented generation; in-context learning; context and model compression; long reasoning (long CoT); long video & image; long-horizon agents; long-text generation; inference acceleration; benchmarks & evaluation; and technical reports.

🔥 Must-read papers for LLM-based Long Context Modeling.

🔥⚡🔥 Thanks for all the great contributors on GitHub!

🚀🤝🚀 I have the privilege of joining [LCLM-Horizon] and collaborating with them on providing a very complete and comprehensive scholarly survey (A Comprehensive Survey on Long Context Language Modeling) and repository (A-Comprehensive-Survey-For-Long-Context-Language-Modeling) dedicated to Long Context Language Modeling. I look forward to collaborating with them to advance research and deepen understanding in this area!

flowchart LR
  LCLM["Long-Context Modeling"]
  LCLM --> A["Attention & KV Cache"]
  LCLM --> T["Training & Alignment"]
  LCLM --> M["Memory & RAG"]
  LCLM --> C["Compression"]
  LCLM --> R["Reasoning & Generation"]
  LCLM --> V["Multimodal / Video"]
  LCLM --> E["Evaluation & Acceleration"]

  A --> A1["Sparse / Linear / IO-aware Attention"]
  A --> A2["Eviction / Quantization / Offloading"]
  T --> T1["Continual Pretraining / Long-SFT"]
  T --> T2["Adaptation & RL for Long Context"]
  M --> M1["Long-Term Memory"]
  M --> M2["RAG / Hybrid Long-Context"]
  C --> C1["Context Compression"]
  C --> C2["Model Compression"]
  R --> R1["Long CoT"]
  R --> R2["Long-Form Text Generation"]

If you find our repository and survey useful for your research, please consider citing the following paper:

@article{liu2025comprehensive,
  title={A Comprehensive Survey on Long Context Language Modeling},
  author={Liu, Jiaheng and Zhu, Dawei and Bai, Zhiqi and He, Yancheng and Liao, Huanxuan and Que, Haoran and Wang, Zekun and Zhang, Chenchen and Zhang, Ge and Zhang, Jiebin and others},
  journal={arXiv preprint arXiv:2503.17407},
  year={2025}
}

Contents

• 📢 News
  • Week Papers
  • Month Papers
• 📜 Papers
  • 1. Survey Papers
  • 2. Efficient Attention
  • 3. KV-Cache Optimization
  • 4. Recurrent Transformers
  • 5. State Space Models & Hybrids
  • 6. Position Encoding & Length Extrapolation
  • 7. Long-Context Training
  • 8. Long-Term Memory
  • 9. Retrieval-Augmented Generation
  • 10. In-Context Learning (Many-shot / Long-ICL)
  • 11. Context Compression
  • 12. Model Compression for Long Context
  • 13. Long Reasoning (Long CoT)
  • 14. Long Video & Image
  • 15. Long-Horizon Agents
  • 16. Long-form Text Generation
  • 17. Inference Acceleration & Serving
  • 18. Benchmarks & Evaluation
  • 19. Technical Reports (Long-Context Models)
  • 20. Blogs & Tutorials
• Acknowledgements
  • Contributors
  • Star History

📢 News

Week Papers

• [2026.08.14]

  • Paper: SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning
  • Paper: KV Cache Compression Through the Lens of Transform Coding
  • Paper: MemoryLake on MemoryArena: A Matched Study of Agent Memory Backends
  • Paper: AgentRewind: Recoverable Execution for Long-Horizon LLM Agents
  • Paper: ScienceFlow: A long-horizon agent for ML research, scientific discovery and beyond
  • Paper: MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning
  • Paper: Handover of In-Context Learning State Across Session Boundaries

• [2026.08.13]

  • Paper: The Query Knows What to Forget: A Second Erase Direction for Linear Attention
  • Paper: When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation
  • Paper: RippleMem: From Isolated Retrieval to Associative Recollection for Long-Term Agent Memory
  • Paper: vToken: Token-Level Virtualization for Reclaimable KV Caches
  • Paper: SCOPE: Subspace Clustering with Online Per-Head Top-K Estimation for Sparse Video Attention
  • Paper: LycheeMemory V2: Efficient Long-Term Memory for LLM Agents via Semantic Segment-Level Consolidation
  • Paper: AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design
  • Paper: PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives
  • Paper: Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development
  • Paper: Beyond Retrieval: Query-Conditioned Reuse of Long-Horizon Agent Trajectories
  • Paper: CoverPrune: Coverage-Driven Token Pruning for 3D VLMs via Optimal Transport
  • Paper: NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video
  • Paper: EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memory

• [2026.08.12]

  • Paper: EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory
  • Paper: Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
  • Paper: MARCH: Scaling Recurrent Memory with Content-Routed State Anchors
  • Paper: Disentangling the Expressivity of RoPE
  • Paper: LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration
  • Paper: Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus
  • Paper: The Sleeping Agent: What Gist-Based Context Compression Loses and Why
  • Paper: Governed Persistent Memory: Source-Bound State Semantics and Fail-Closed Release for Long-Horizon Agents
  • Paper: Towards a Formal Definition of Agent Memory: Basis, Span, Optimality, and the Sequential Memory Problem
  • Paper: Beyond Memory: A Transactional Continuity Kernel for Long-Lived AI Agents
  • Paper: LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation
  • Paper: Hybrid Gated Attention
  • Paper: Claim-Level Reliability Assessment for Efficient Test-Time Reasoning
  • Paper: Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling

• [2026.08.11]

  • Paper: Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks
  • Paper: Neural Introspection Gating for Adaptive KV-Cache Reuse in Vision-Language-Action Models
  • Paper: ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover
  • Paper: When Vision Becomes Text: Visual Token Pruning via Cross-Modal Residual Guidance in VLMs
  • Paper: SparSTAR: Sparse Attention for SpaceTime AutoRegressive Video Synthesis
  • Paper: Self-Correcting Long-Horizon Search Agents via Tree-Structured Memory
  • Paper: StreamFlow: Dynamic Memory Flows for Streaming Video Understanding
  • Paper: InSight-doc: Agentic Visual Perception for Long-Document Understanding
  • Paper: R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video
  • Paper: ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling

• [2026.08.10]

  • Paper: Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
  • Paper: MixFormer: Linear Transformer with Mixture of Memory Experts
  • Paper: KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models
  • Paper: MESA:Task-Adaptive Multi-Structure Evidence Selection for Long-Horizon Agent Memory
  • Paper: Not All Visual Tokens Are Equally Safe to Remove:Consequence-Sensitive Visual Token Compression
  • Paper: Evo-Bench: Can Language Models Improve Agent Harness?
  • Paper: BDH-CQ: In-Context Learning with Recurrent Latent Reasoning
  • Paper: Motif 3: Technical Report

• [2026.08.09]

  • Paper: DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference
  • Paper: RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation
  • Paper: VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling
  • Paper: Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling
  • Paper: VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference
  • Paper: Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses
  • Paper: Not Worth Another Token: Marginal Value Estimation for Efficient Deep Research Agents

• [2026.08.08]

  • Paper: OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
  • Paper: SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Coding
  • Paper: CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents
  • Paper: SuperLocalMemory 4.0: The Governed Memory Operating System for AI Agents

• [2026.08.07]

  • Paper: CoinRAG: Contextualized Information Nugget KV Cache Reuse for Long-Context RAG
  • Paper: HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management
  • Paper: Every Cache Entry Earns Its Place: Global Allocation of Resolution and Coverage for KV Cache Compression
  • Paper: Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
  • Paper: StateFlow: Sequence Pipeline Parallelism for Long-Context Modeling with Linear Recurrence
  • Paper: The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents
  • Paper: RoRA: Role-Oriented Regional Allocation for Visual Token Pruning in MLLMs
  • Paper: Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
  • Paper: Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding
  • Paper: An AI4AI Framework for Visual Token Pruning
  • Paper: DocMemo: Dynamic Evidence Discovery via Probabilistic Memory-Guided Retrieval for Multi-Modal Document Understanding
  • Paper: MemOPD: On-Policy Distillation through Memory State Alignment for Long-Horizon Agents
  • Paper: Long-Horizon Agent Trajectory Attribution: A Unified Benchmark and Fine-Grained Annotation Framework
  • Paper: MemPrism: Task-Conditioned Relational Memory Views for Long-Horizon Agents
  • Paper: HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses
  • Paper: Explicit, Not Longer: What Makes Epistemic Stance Survive Memory Compression
  • Paper: CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing
  • Blog: Efficient Decode Context Parallelism with vLLM for Long Context Workloads

• [2026.08.06]

  • Paper: Retrofitting Linear Attention into Diffusion Language Models
  • Paper: Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding
  • Paper: Toward Reliable Context Compression for Long-Horizon Agents: An Empirical Study of Execution Instability
  • Paper: StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding
  • Paper: One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding
  • Paper: TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
  • Paper: Runtime Observability for Heterogeneous Attention Memory
  • Paper: Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning

• [2026.08.05]

  • Paper: Recursive Synthesis for Long-Horizon Terminal Tasks
  • Paper: QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
  • Paper: OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
  • Paper: Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression
  • Paper: Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning
  • Paper: MemoryCPT: An End-to-End Agent Memory Framework for Cost-Performance Trade-off
  • Paper: Caching for the Future: Scrub Jay Episodic Memory Principles for Agent Memory Systems
  • Paper: Not All Redundant Tokens Are Alike: Analyzing Visual Token Pruning through Token Roles
  • Paper: ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment
  • Paper: EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents
  • Paper: Thinking with Anchors: Grounded and Efficient Document Reasoning
  • Paper: Chained Recursive Language Models for Multi-Iteration Reasoning
  • Paper: Training-Free Hashing-Based Attention via Binary Principal Components

• [2026.08.04]

  • Paper: Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
  • Paper: TimeRLM: Recursive Language Models Enable Precise Anomaly Localization in Long-Context Time-Series
  • Paper: Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks
  • Paper: TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
  • Paper: PI-Mem: Pushing Long-Context Reasoning to 3.6M Tokens with Parallel-Iterative Memory
  • Paper: Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse
  • Paper: Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention
  • Paper: SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference
  • Paper: RUTA: Principled Visual Token Allocation via Rate-Utility Optimization
  • Paper: Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Models
  • Paper: GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Models
  • Paper: StreamDAM: Presence-Aware Memory for Real-Time Streaming Video Object Segmentation
  • Paper: When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding
  • Paper: Muon Meets Mamba: Spectral Optimization for State Space Models
  • Paper: Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility
  • Paper: EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners
  • Paper: When Do Fewer Visual Tokens Accelerate Multimodal Inference? A Break-Even Study Across Decision Locations and Hardware
  • Paper: Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin
  • Paper: Interpretable Adaptive Sampling for LLM Test-Time Scaling
  • Paper: SAKI: Score-Aware Low-Rank Key Indexing with Random-Matrix Noise Correction for KV Retrieval
  • Paper: OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agents

Month Papers

📜 Papers

Paper entries live under papers/ so this README stays under GitHub's homepage size limit. For an interactive chapter reader (search + in-page paper cards), open the project homepage.

Acknowledgements

Please contact me if I miss your names in the list, I will add you back ASAP!

Contributors

Star History