Awesome-Context-Compression-LLMs 🗜️

A curated collection of research papers focused on enhancing the efficiency of Large Language Models (LLMs) through context compression techniques. These methods aim to reduce token usage, compress latent states, and optimize memory footprints (KV Cache).

📑 Table of Contents

• Introduction
• Taxonomy
• Explicit Context Compression (Prompt/Input Level)
• Implicit Context Compression (Latent/Reasoning Level)
• Inference-Time KV Compression (Memory/Cache Level)
• Surveys
• Contributing
• Star History

🎯 Introduction

As LLMs scale to handle longer contexts and more complex tasks, efficient context management becomes crucial. This repository organizes papers into three distinct categories based on where and how compression occurs:

1. Explicit Compression: Operates on input tokens before/during encoding
2. Implicit Compression: Compresses into latent representations
3. KV Compression: Optimizes the Key-Value cache during inference

🗂️ Taxonomy

Context Compression Methods
├── Explicit Context Compression (Input Level)
│   ├── Token Pruning (LLMLingua, Selective-Context)
│   ├── Summarization-based
│   └── Information-theoretic Selection
│
├── Implicit Context Compression (Latent Level)
│   ├── Soft Prompt (AutoCompressor)
│   ├── Autoencoder-based (ICAE, CoCom)
│   └── Latent Reasoning (Coconut)
│
└── Inference-Time KV Compression (Cache Level)
    ├── Eviction Policies (H2O, TOVA, SnapKV)
    ├── Quantization (KIVI)
    └── Sparse Attention (StreamingLLM)

---

📝 Explicit Context Compression (Prompt/Input Level)

Definition: Methods that operate primarily on the input text or input tokens. They select, prune, or summarize the context before or during the initial encoding to shorten the input sequence length. The goal is often to fit more context into the window or reduce API costs.

Keywords: Prompt Compression, Token Pruning, Summarization, Information Entropy, Token Selection, Coarse-grained Pruning

---

🧠 Implicit Context Compression (Latent/Reasoning Level)

Definition: Methods that compress context into soft vectors, embeddings, or latent states. This includes encoding long text into compact vector representations and Latent Reasoning where the "Chain of Thought" or intermediate reasoning steps are performed in the latent space (not outputting tokens) to reduce generation overhead.

Keywords: Soft Prompt, Autoencoder, Memory Vectors, Latent Space Reasoning, Continuous Chain of Thought, Internal State Compression

---

⚡ Inference-Time KV Compression (Memory/Cache Level)

Definition: Methods that specifically target the Key-Value (KV) Cache during the generation phase. They aim to reduce GPU memory usage and latency by evicting "unimportant" KV pairs, quantizing the cache, or using sparse attention patterns. These methods usually happen on-the-fly during inference.

Keywords: KV Cache Eviction, Heavy Hitters, Sparse Attention, Cache Quantization, Budget-constrained Generation, Infinite Context, Streaming Inference

---

📚 Surveys

---

🤝 Contributing

We welcome contributions! Please follow these steps:

1. Fork the repository
2. Add your paper following the table format
3. Ensure the paper is correctly categorized
4. Submit a Pull Request

Categorization Guidelines

• Cat 1 (Explicit): If the paper discusses compressing prompts before sending to the model/API
• Cat 2 (Implicit): If the paper compresses into latent vectors or performs latent reasoning
• Cat 3 (KV Cache): If the paper manages GPU memory by manipulating KV pairs during inference

---

⭐ Star History

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🙏 Acknowledgements

Special thanks to all researchers whose work is featured in this repository. Your contributions to making LLMs more efficient benefit the entire community.

---

If you find this repository helpful, please consider giving it a ⭐!