SubtleMemory

SubtleMemory is a synthetic benchmark for fine-grained relational memory discrimination in long-horizon AI agents. It tests whether memory systems can preserve and use the relation among target-relevant memories rather than only retrieve isolated facts.

This dataset release preserves the original repository file structure:

data/subtlememory/
  persona_0/
    bench_instances.json
    history_sessions.json
  ...
  persona_9/
    bench_instances.json
    history_sessions.json

The Hugging Face Dataset Viewer is configured with persona-level splits:

• bench_instances: one split per persona, containing benchmark cases and QA instances.
• history_sessions: one split per persona, containing the chronological conversation sessions.

Counts

The benchmark contains 1,522 QA instances across the 1,090 benchmark cases.

Fields

bench_instances.json contains:

• case_id, instance_id, persona_id, source, and topic
• relation_type and relation_subtype
• case, facts, and session_ids
• qas, where each QA item contains query, correct_answers, and incorrect_answers
• persona_str, the synthetic persona profile serialized as JSON text

history_sessions.json contains:

• session_id, case_id, persona_id, source, order, and timestamp
• conversation_type, conversation_flow, and persona_signal_level
• history, a list of {role, content} conversation turns

Synthetic Data Notice

The data is synthetic benchmark data generated for research use. Persona identities, histories, contact details, account numbers, file paths, and other sensitive-looking values are fabricated and should not be interpreted as real personal data.

Usage

from datasets import load_dataset

bench_p0 = load_dataset(
    "Yummytanmo/SubtleMemory",
    "bench_instances",
    split="persona_0",
)

history_p0 = load_dataset(
    "Yummytanmo/SubtleMemory",
    "history_sessions",
    split="persona_0",
)

Links

• Project page: https://yummytanmo.github.io/SubtleMemory/
• Code repository: https://github.com/Yummytanmo/SubtleMemory
• arXiv: https://arxiv.org/abs/2606.05761

License

Apache-2.0.