Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs

This huggingface page contains data for the paper: Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs

Abstract

Evaluating the abilities of large language models (LLMs) for tasks that require long-term memory and thus long-context reasoning, for example in conversational settings, is hampered by the existing benchmarks, which often lack narrative coherence, cover narrow domains, and only test simple recall-oriented tasks. This paper introduces a comprehensive solution to these challenges. First, we present a novel framework for automatically generating long (up to 10M tokens), coherent, and topically diverse conversations, accompanied by probing questions targeting a wide range of memory abilities. From this, we construct BEAM, a new benchmark comprising 100 conversations and 2,000 validated questions. Second, to enhance model performance, we propose LIGHT–a framework inspired by human cognition that equips LLMs with three complementary memory systems: a long-term episodic memory, a short-term working memory, and a scratchpad for accumulating salient facts. Our experiments on BEAM reveal that even LLMs with 1M token context windows (with and without retrieval-augmentation) struggle as dialogues lengthen. In contrast, LIGHT consistently improves performance across various models, achieving an average improvement of 3.5%–12.69% over the strongest baselines, depending on the backbone LLM. An ablation study further confirms the contribution of each memory component.

Dataset Description

BEAM is a comprehensive dataset for evaluating long-term memory in language models. It contains multi-scale conversations (128K, 500K, 1M, and 10M tokens) across diverse domains, including general, coding, and math topics, and is used to evaluate ten distinct memory abilities.

Dataset Structure

Each conversation includes:

• Seed Information: Category, title, theme, and subtopics
• Narratives: Conversation labels
• Conversation Plan: Storyline of the chat that unflods chronologically
• User Questions: User questions for assistant
• Chat Data: Full conversation history
• Probing Questions: 10 different question types for evaluation
• User Profile: User information and relationships
• Plans: For 10M conversations, includes 10 different conversation plans

10M Conversations Special Structure

10M conversations have a unique structure with multiple plans:

• Main Conversation: Overall conversation data
• Plans Array: 10 different conversation plans (plan-0 through plan-9)
• Each Plan Contains:
  • Plan-specific topic information
  • Plan-specific narratives and user profile
  • Plan-specific conversation plan
  • Plan-specific user questions
  • Plan-specific chat data

Question Types

1. Abstention: Evaluates whether a model withholds answers when evidence is missing
2. Contradiction Resolution: Tests the capacity to detect and reconcile inconsistent statements across widely separated turns, maintaining global coherence
3. Event Ordering: Assesses whether a model can recognize and reconstruct the sequence of evolving information in the dialogue
4. Information Extraction: Measures recall of entities and factual details in long histories
5. Instruction Following: Examines sustained adherence to user-specified constraints over long contexts
6. Knowledge Update: Evaluates revising stored facts as new ones appear
7. Multi-Session Reasoning: Probes inference that integrates evidence across multiple, non-adjacent dialogue segments
8. Preference Following: Captures personalized responses that adapt to evolving preferences
9. Summarization: Assesses the ability to abstract and compress dialogue content
10. Temporal Reasoning: Tests reasoning about explicit and implicit time relations

Usage

import ast
from datasets import load_dataset

# Load the dataset
dataset = load_dataset("Mohammadta/BEAM")

# Access different chat sizes
dataset_100k = dataset["100K"]
dataset_500k = dataset["500K"]
dataset_1m = dataset["1M"]

# Example: Get a regular conversation (100K, 500K, 1M)
conversation = dataset_100k[0]
print(conversation["conversation_seed"])  # Dict
print(conversation["chat"])               # List
print(conversation["probing_questions"])  # String

# Direct access to dict objects (no parsing needed)
profile_data = conversation["user_profile"]
print(f"User info: {profile_data.get('user_info', '')}")
print(f"Relationships: {profile_data.get('user_relationships', '')}")

seed_data = conversation["conversation_seed"]
print(f"Category: {seed_data.get('category', '')}")
print(f"Title: {seed_data.get('title', '')}")

# probing_questions is a string
questions_string = conversation["probing_questions"]
probing_questions_data = ast.literal_eval(questions_string)
print(f"Probing questions: {probing_questions_data}")

# Direct access to lists
chat_data = conversation["chat"]
print(f"Chats: {chat_data}")

user_questions = conversation["user_questions"]
print(f"User questions: {user_questions}")

📄 License

The BEAM dataset is released under the
Creative Commons Attribution–ShareAlike 4.0 International License (CC BY-SA 4.0).

Full license text: https://creativecommons.org/licenses/by-sa/4.0/

Citation

If you use this dataset in your research, please cite:

Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs

@misc{tavakoli2025milliontokensbenchmarkingenhancing,
      title={Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs}, 
      author={Mohammad Tavakoli and Alireza Salemi and Carrie Ye and Mohamed Abdalla and Hamed Zamani and J Ross Mitchell},
      year={2025},
      eprint={2510.27246},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.27246}, 
}