SubtleMemory: A Benchmark for Fine-Grained Relational Memory Discrimination in Long-Horizon AI Agents

Overview | Data | Data Construction | Evaluation | Integration Guide | Results | Citation

Overview

Long-running assistants accumulate many similar memories about a user, a task, or an external fact. These memories may complement one another, apply only under different contexts, or directly conflict. SubtleMemory tests whether memory systems preserve and use these fine-grained relations instead of only recalling isolated facts.

SubtleMemory contains 10 persona-level histories, 1,090 relation-controlled semantic variant sets, and 1,522 evaluation instances. Each instance asks a downstream question whose answer depends on the relation among target-relevant memories.

Framework Features

• Relation-controlled data construction: embed latent relation-controlled semantic artifacts implicitly into user histories, so the target relation is only revealed through later memory-dependent questions.
• SubtleMemory data: include the generated benchmark data under data/subtlememory, with one folder per persona.
• Unified evaluation: run standalone memory systems, framework-native memory agents, and OpenClaw-style plugin agents under the same add -> finalize -> search -> answer -> evaluate protocol.
• Failure diagnosis: inspect stage artifacts to separate memory writing, finalization, retrieval, answer generation, and judging failures.
• Readback analysis: for adapters that support it, compare normal search with optional readback mode, which uses question provenance to expose memory objects written from target sessions.
• Relation-sensitive reporting: evaluate relation-level behavior rather than only final answer accuracy.

News

• 2026-08-21: SubtleMemory has been accepted to EMNLP 2026 Findings!
• 2026-06-05: arXiv preprint is online: https://arxiv.org/abs/2606.05761
• 2026-06-04: Project page is online: https://yummytanmo.github.io/SubtleMemory/

Repository Layout

Install

Use Python 3.10 or newer. From the repository root:

uv sync
cp env.template .env

Edit .env with the credentials needed by the systems you want to evaluate or use for data construction. Common variables include:

• ANSWER_LLM_API_KEY, ANSWER_LLM_BASE_URL, ANSWER_LLM_MODEL
• JUDGE_LLM_API_KEY, JUDGE_LLM_BASE_URL, JUDGE_LLM_MODEL
• GENERATION_BASE_URL, GENERATION_API_KEY, FILTER_BASE_URL, FILTER_API_KEY
• Memory-backend credentials such as MEM0_API_KEY, MEMOS_KEY, EVERMEMOS_API_KEY, ZEP_API_KEY, and local runtime paths for OpenClaw, MIRIX, MetaClaw, A-Mem, and MemoBase

Data

The generated SubtleMemory bench data is available on Hugging Face:

• Dataset page: Yummytanmo/SubtleMemory
• Bench instance viewer: bench_instances/persona_0
• History session viewer: history_sessions/persona_0

The Hugging Face release preserves the repository data layout and exposes persona_0 through persona_9 as dataset splits. It provides two configs: bench_instances for benchmark cases and QA instances, and history_sessions for chronological conversation sessions.

from datasets import load_dataset

bench_p0 = load_dataset(
    "Yummytanmo/SubtleMemory",
    "bench_instances",
    split="persona_0",
)

history_p0 = load_dataset(
    "Yummytanmo/SubtleMemory",
    "history_sessions",
    split="persona_0",
)

The same data is also stored in this repository at:

data/subtlememory/
  persona_0/
    bench_instances.json
    history_sessions.json
  ...
  persona_9/
    bench_instances.json
    history_sessions.json

Each persona bundle contains chronological history sessions and benchmark instances with relation labels, source facts, target session IDs, correct answers, and incorrect answers. The evaluation loader converts these bundles to the LoCoMo-style runtime format used by the staged pipeline.

For evaluation, copy the generated bench data into the evaluation data location:

mkdir -p evaluation/data/subtlememory
rsync -a data/subtlememory/ evaluation/data/subtlememory/

Data Construction

SubtleMemory is built through a staged construction pipeline that starts from open-source seed data and ends with persona-scoped benchmark bundles.

High-resolution PDF: assets/data_construction.pdf

The construction workflow follows five stages:

1. Select and normalize semantic seed data.
2. Generate relation-controlled semantic variants.
3. Embed variants into implicit, natural multi-turn sessions.
4. Construct evaluation instances and QA targets.
5. Assemble chronological user histories and final bench bundles.

The main construction entry points are:

uv run python construction/user-related/main.py build
uv run python construction/user-unrelated/main.py generate-to-filter
uv run python construction/user-unrelated/main.py filter
uv run python construction/user-unrelated/main.py build-passed-dataset
uv run python construction/merge_user_related_unrelated.py --help

See construction/README.md for the construction workflow, expected intermediate paths, and links to detailed prompt and data-flow documents.

Evaluation

After copying the data into evaluation/data/subtlememory, run a small smoke slice:

uv run python -m evaluation.cli \
  --dataset subtlememory \
  --system mem0 \
  --run-name mem0-smoke \
  --from-conv 4 \
  --to-conv 5 \
  --smoke \
  --smoke-messages 10 \
  --smoke-questions 1

Validate the completed run:

uv run python -m evaluation.validate_run \
  --output-dir evaluation/results/subtlememory-mem0-mem0-smoke/subtlememory-mem0-mem0-smoke-api

Run the full staged pipeline for a configured system:

uv run python -m evaluation.cli \
  --dataset subtlememory \
  --system memos \
  --run-name memos-main

Run only later stages after ingestion has completed:

uv run python -m evaluation.cli \
  --dataset subtlememory \
  --system memos \
  --run-name memos-main \
  --stages search answer evaluate

Results

The main leaderboard and aggregate benchmark results are available on the project page:

https://yummytanmo.github.io/SubtleMemory/#results

Readback Mode

Standard api mode asks the memory system to retrieve evidence through its normal search API. readback mode instead uses question provenance, especially session_ids, to read back memory objects written from the target sessions. Readback is optional diagnostic support; a memory system can be benchmarked in normal api mode without implementing it.

api mode:      query -> provider search API -> answer -> evaluate
readback mode: session_ids -> stored memory objects -> answer -> evaluate

Run api first so the system performs add and finalize:

uv run python -m evaluation.cli \
  --dataset subtlememory \
  --system memos \
  --run-name memos-readback-demo \
  --from-conv 4 \
  --to-conv 5 \
  --smoke \
  --smoke-messages 10 \
  --smoke-questions 1

Then run readback while reusing the same add/finalize artifacts:

uv run python -m evaluation.cli \
  --dataset subtlememory \
  --system memos \
  --run-name memos-readback-demo \
  --search-mode readback \
  --reuse-add-artifacts-from evaluation/results/subtlememory-memos-memos-readback-demo/subtlememory-memos-memos-readback-demo-api \
  --from-conv 4 \
  --to-conv 5 \
  --smoke \
  --smoke-messages 10 \
  --smoke-questions 1 \
  --stages search answer evaluate

Readback-specific details are saved to readback_search_readback.jsonl, one row per question with session IDs, status, read objects, and errors.

Supported Systems

Each system YAML lives under evaluation/config/systems/. Most settings are configured through .env, so the same config file can be reused across machines.

To add and benchmark a new memory system, see the detailed evaluation integration guide. It explains the system YAML, adapter contract, api versus optional readback search modes, registry wiring, artifact expectations, validation flow, and a Mem0 worked example.

Outputs

Each run writes a mode-specific artifact directory under evaluation/results/. Important files include:

The pipeline writes checkpoints so interrupted runs can usually resume without rerunning completed stages.

Acknowledgements

This repository is adapted from the EverOS evaluation framework. We thank EverOS for the base evaluation pipeline, staged artifact design, adapter abstractions, and config-driven experiment workflow that SubtleMemory builds on.

SubtleMemory evaluates and integrates with several open-source memory and agent systems. Please cite and follow the licenses of the corresponding upstream projects when using those adapters.

Citation

If you find SubtleMemory useful, please cite the paper:

@misc{wang2026subtlememory,
  title = {SubtleMemory: A Benchmark for Fine-Grained Relational Memory Discrimination in Long-Horizon AI Agents},
  author = {Wenxuan Wang and Haoyu Sun and Fukuan Hou and Mingyang Song and Weinan Zhang and Yu Cheng and Yang Yang},
  year = {2026},
  eprint = {2606.05761},
  archivePrefix = {arXiv},
  primaryClass = {cs.AI},
  url = {https://arxiv.org/abs/2606.05761}
}