Shadowgraph
TrendingAbout
Sign in

HarryWu99/llm_kvcache_sparsity

Implement some method of LLM KV Cache Sparsity

Python

41

2 commits

updated Jun 6, 2024

See the code

README

LLM KV Cache Sparsity

Implement some method of LLM KV Cache Sparsity, including:

  1. Efficient Streaming Language Models with Attention Sinks, also called "SinkCache"
  2. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  3. SnapKV: LLM Knows What You are Looking for Before Generation

To Run

pip install -r requirements.txt
# edit longbench loading method `load_from_disk` in example/test.py
python example/test.py --sparsity_method snapkv

The result file will write to results folder.

Then you can use longbench_eval/eval.py to get the scores.

The core code for KV Cache eviction is in models/kv_clusters.py

Results

Todo

Similar repos

October2001/Awesome-KV-Cache-Compression

📰 Must-read papers on KV Cache Compression (constantly updating 🤗).

744

Zefan-Cai/Awesome-LLM-KV-Cache

Awesome-LLM-KV-Cache: A curated list of 📙Awesome LLM KV Cache Papers with Codes.

468

snu-mllab/KVzip

[NeurIPS'25 Oral] Query-agnostic KV cache eviction: 3–4× reduction in memory and 2× decrease in…

225

NVlabs/RocketKV

[ICML 2025] RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression

53

SalesforceAIResearch/ThinK

ThinK: Thinner Key Cache by Query-Driven Pruning

30

FasterDecoding/SnapKV

335

CURRENTF/SparseEngine

A sparse-first inference engine (SparseEngine).

77

FMInference/H2O

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

534

Languages

Python

100.0%

HarryWu99/llm_kvcache_sparsity

Implement some method of LLM KV Cache Sparsity

Python

41

2 commits

updated Jun 6, 2024

See the code

README

LLM KV Cache Sparsity

Implement some method of LLM KV Cache Sparsity, including:

  1. Efficient Streaming Language Models with Attention Sinks, also called "SinkCache"
  2. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  3. SnapKV: LLM Knows What You are Looking for Before Generation

To Run

pip install -r requirements.txt
# edit longbench loading method `load_from_disk` in example/test.py
python example/test.py --sparsity_method snapkv

The result file will write to results folder.

Then you can use longbench_eval/eval.py to get the scores.

The core code for KV Cache eviction is in models/kv_clusters.py

Results

Todo

Similar repos

October2001/Awesome-KV-Cache-Compression

📰 Must-read papers on KV Cache Compression (constantly updating 🤗).

744

Zefan-Cai/Awesome-LLM-KV-Cache

Awesome-LLM-KV-Cache: A curated list of 📙Awesome LLM KV Cache Papers with Codes.

468

snu-mllab/KVzip

[NeurIPS'25 Oral] Query-agnostic KV cache eviction: 3–4× reduction in memory and 2× decrease in…

225

NVlabs/RocketKV

[ICML 2025] RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression

53

SalesforceAIResearch/ThinK

ThinK: Thinner Key Cache by Query-Driven Pruning

30

FasterDecoding/SnapKV

335

CURRENTF/SparseEngine

A sparse-first inference engine (SparseEngine).

77

FMInference/H2O

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

534

Languages

Python

100.0%