SCBench

[Paper] [Code] [Project Page]

SCBench (SharedContextBench) is a comprehensive benchmark to evaluate efficient long-context methods in a KV cache-centric perspective, analyzing their performance across the full KV cache lifecycle (generation, compression, retrieval, and loading) in real-world scenarios where context memory (KV cache) is shared and reused across multiple requests.

🎯 Quick Start

Load Data

You can download and load the SCBench data through the Hugging Face datasets (🤗 HF Repo), and run the experiments in the Github (💻 SCBench):

from datasets import load_dataset

datasets = ["scbench_kv", "scbench_prefix_suffix", "scbench_vt", "scbench_repoqa", "scbench_qa_eng", "scbench_qa_chn", "scbench_choice_eng", "scbench_many_shot", "scbench_summary", "scbench_mf", "scbench_summary_with_needles", "scbench_repoqa_and_kv"]

for dataset in datasets:
    data = load_dataset('microsoft/SCBench', dataset, split='test')

Data Format

All data in SCBench are standardized to the following format:

{
    "id": "Random id for each piece of data.",
    "context": "The long context required for the task, such as repo-code, long-document, and many-shot.",
    "multi_turns": [{"input": "multi-turn question.", "answer": "multi-turn reference answer."}],
}

Experiments

We implement Multi-Turn and Multi-Request modes with HF and vLLM in GreedySearch and GreedySearch_vllm two class. Please refer the follow scripts to run the experiments.

Run the benchmark

First, build the environment, see basic environment.

Run the test:

bash scripts/test_llama.sh

Run multiple tasks in one command:

bash scripts/run_all_tasks.sh

Specify the max sequence length, max number of turns, and number of eval examples:

• --max_seq_length: The maximum sequence length for the test.
• --max_turns: The maximum number of turns for the test.
• --num_eval_examples: The number of test examples to use, use all examples in default.

Run with efficient long-context methods:

• --attn_type: The attention type to use.
• --kv_type: The KV cache type to use.

For example, run with MInference and SnapKV:

bash scripts/test_minference_with_snapkv.sh

The supported efficient long-context methods are as follows:

attn_type:

• dense: Dense attention
• minference: MInference
• a_shape: A-Shape
• tri_shape: Tri-Shape

kv_type:

• dense: Dense KV cache
• kivi: KIVI
• snapkv: SnapKV
• quest: Quest
• pyramidkv: PyramidKV
• streamingllm: StreamingLLM

You will need to build specific environment for different attention types and KV cache types, see section Environment for more details.

Dataset

SCBench covers 12 diverse tasks that test four key long-context capabilities: string retrieval, semantic retrieval, global information processing, and multi-tasking.

String Retrieval

• Retr.KV: Tests key-value lookup in large JSON objects with random, incompressible content
• Retr.Prefix-Suffix: Evaluates finding strings with specific prefix and suffix patterns
• Retr.MultiHop: Assesses multi-hop variable tracing capabilities in long inputs

Semantic Retrieval

• Code.RepoQA: Function retrieval from large codebases based on natural language descriptions
• Language QA: Includes English QA, Chinese QA, and multi-choice questions on long texts
• Requires semantic understanding on length inputs

Global Information Processing

• Many-shot ICL: Tests in-context learning with hundreds of examples
• Math.Find: Statistical tasks on large arrays
• En.Sum: Summarization of documents
• Requires global information processing or aggregation

Multi-Tasking

• Mix.Sum+NIAH: Combines summarization with needle-in-haystack search
• Mix.RepoQA+KV: Integrates code function retrieval with key-value lookup
• Requires multi-tasking or multi-step reasoning

Two Shared Context Modes

The benchmark evaluates these tasks across two shared context modes:

• Multi-turn Mode: Caches context within single sessions
• Multi-request Mode: Shares context across multiple sessions

Compared to previous long-context benchmarks

Our SCBench is the first long-context benchmark that covers single-turn, multi-turn, and multi-request scenarios. In addition, our impelmentation also involves KV cache reuse techniques, thereby providing a more comprehensive analysis on the full KV cache lifecycle of efficient long-context methods.

Results and Findings

Our SCBench reveals that the following key insights:

Finding 1: Sub-O(n) Memory is Problematic in Multi-Request/Multi-Turn Decoding

• Sparse decoding methods with sub-O(n) memory perform well on first queries but lose accuracy in subsequent requests
• Methods maintaining O(n) memory with sub-O(n²) computation during pre-filling can better approximate full attention accuracy across multiple queries

Finding 2: Task Performance Shows Varying Decline Patterns

• Sparse KV cache methods excel in tasks requiring global information processing
• O(n) memory is essential for tasks involving exact match retrieval

Finding 3: Performance vs Compression Rate

• All methods show performance degradation as compression rates increase
• Sub-O(n) memory methods exhibit significant drop at 1/4 compression rate
• Methods like RetrievalAttention and KIVI that maintain O(n) memory with sparse decoding show better resilience at higher compression rates

Finding 4: Issues with Long-Generation Scenarios

• Attention distribution shifts significantly as generation length and number of rounds increase
• This out-of-distribution (OOD) issue impacts performance even for O(n) memory methods

Finding 5: Dynamic vs Static Patterns

• Dynamic sparse patterns generally outperform static patterns

Citation

@article{li2024scbench,
    title={SCBench: A KV cache-centric analysis of long-context methods},
    author={Li, Yucheng and Jiang, Huiqiang and Wu, Qianhui and Luo, Xufang and Ahn, Surin and Zhang, Chengruidong and Abdi, Amir H and Li, Dongsheng and Gao, Jianfeng and Yang, Yuqing and Qiu, Lili},
    journal={arXiv preprint arXiv:2412.10319},
    year={2024}
}