RULER Evaluation Data

This dataset contains pre-generated JSONL files for the RULER long-context evaluation benchmark, used in Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference (ICML 2026 Spotlight). RULER is designed to evaluate effective context length and long-context behavior beyond simple retrieval, covering retrieval, multi-hop tracing, aggregation, and question answering style tasks.

The files are organized by target context length:

.
├── 4096/
├── 8192/
├── 16384/
├── 32768/
└── 49152/

Each directory contains one JSONL file per RULER task. The 4096, 8192, 16384, and 32768 directories contain 16 task files each. The 49152 directory contains 13 task files and does not include scat_arith_1.jsonl, scat_arith_2.jsonl, or scat_arith_3.jsonl.

Dataset Details

• Total examples: 34,900
• Context lengths: 4,096, 8,192, 16,384, 32,768, and 49,152 tokens

Available task files:

• cwe.jsonl
• fwe.jsonl
• niah_multikey_1.jsonl
• niah_multikey_2.jsonl
• niah_multikey_3.jsonl
• niah_multiquery.jsonl
• niah_multivalue.jsonl
• niah_single_1.jsonl
• niah_single_2.jsonl
• niah_single_3.jsonl
• qa_1.jsonl
• qa_2.jsonl
• scat_arith_1.jsonl
• scat_arith_2.jsonl
• scat_arith_3.jsonl
• vt.jsonl

Schema

Each row is a JSON object. Common fields are:

• index: integer example index within the file
• input: prompt text to pass to the model
• outputs: list of accepted answer strings
• length: measured sequence length

Some files also include:

• length_w_model_temp: measured sequence length including model-specific template text
• answer_prefix: prefix used for constrained or formatted answer generation

Example:

{
  "index": 0,
  "input": "...",
  "outputs": ["answer"],
  "length": 32760,
  "length_w_model_temp": 32760,
  "answer_prefix": " The answer is"
}

Loading

Load one task file directly:

from datasets import load_dataset

dataset = load_dataset(
    "json",
    data_files="32768/niah_single_1.jsonl",
    split="train",
)

Load multiple context lengths or tasks:

from datasets import load_dataset

data_files = {
    "4096_niah_single_1": "4096/niah_single_1.jsonl",
    "8192_niah_single_1": "8192/niah_single_1.jsonl",
    "16384_niah_single_1": "16384/niah_single_1.jsonl",
    "32768_niah_single_1": "32768/niah_single_1.jsonl",
    "49152_niah_single_1": "49152/niah_single_1.jsonl",
}

dataset = load_dataset("json", data_files=data_files)

Intended Use

This dataset is intended for evaluating long-context language models with RULER-compatible evaluation scripts. It is not intended for model training, user profiling, or decisions affecting people.

RULER benchmark:

• Repository: https://github.com/NVIDIA/RULER
• Paper: https://arxiv.org/abs/2404.06654

Citation

@inproceedings{xionglong,
  title={Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference},
  author={Xiong, Siheng and Zou, Joe and Fekri, Faramarz and Cho, Yae Jee},
  booktitle={Forty-third International Conference on Machine Learning}
}