🗨️ User (session 5): "I'm thinking I'll try to relax by listening to the music Luke recommended the other day."

🔎 Prior sessions: the system had recommended Bruno Mars, Ryuichi Sakamoto, Hikaru Utada, Yumi Matsutoya, Haruka Nakamura.

❌ Deployed reply (full history in context): "That sounds like a perfect way to unwind. I hope the music brings you some calm…" — 0/4 facts referenced

✅ Same model, asked directly "What Japanese artists did you recommend?" → "Ryuichi Sakamoto, Hikaru Utada, Yumi Matsutoya, Haruka Nakamura." — correct

{"natural": {"pos_001": "…your reply…", "pos_002": "…"},
 "direct_qa": {"pos_001": ["…reply to sub_question 0…", "…"]}}   // optional

data/        instances.test.jsonl · negatives.test.jsonl (corpus/users configs live on HF)
eval/        run_judge.py · run_judge_negatives.py · prompts/ · metadata.json
baselines/   run_fullcontext.py · run_mem0.py · run_letta.py · results/

Judge is GPT-5.4-nano at temperature 0, validated against human annotators (swap_judge() to change it). pip install openai, plus mem0ai / letta for those baselines.

Values in %; judge fixed at GPT-5.4-nano. Paper numbers use 73 instances; the released 72 drop one contaminated probe (≤ 1.4 pp effect). Mem0/Letta Natural Integration corrected on 2026-10-07 (eval schema 1.2): the arXiv v1 values (58.3 / 56.9) were scored with judge inputs that did not match the paper's; with matched inputs they fall in the same range as the other rows. Per-instance outputs in baselines/results/.

None of these baselines volunteers or fabricates prior-session memory on uncued turns; the split is there to catch systems tuned to over-trigger recall.

benchmark row:

{
  "instance_id": "pos_001",
  "user_id": "annotator_11",
  "event_type": "user_re_provision",        // or "user_memory_probe"
  "trigger_session_idx": 4,
  "input": {
    "sessions": [                            // ALL sessions before the trigger session, verbatim
      {"session_idx": 0, "date": "2025/November/05",
       "turns": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
    ],
    "current_session": {                     // the trigger session, truncated at the trigger turn
      "session_idx": 4, "date": "2025/November/09",
      "turns": [/* ... */, {"role": "user", "content": "As I wrote yesterday, ..."}]
    },
    "trigger_quote": "As I wrote yesterday, ..."
  },
  "target": {
    "ground_truth": "In session 3 (the previous day) the user ...",
    "sub_questions": [{"question": "...", "answer": "..."}]   // 3–5 fact questions (316 total)
  }
}

The assistant's original reply is deliberately not included — it is not a gold answer, and it would invite string-matching. Dates matter: many triggers are temporal ("yesterday", "the other day").

negatives row: same input schema; target is {"memory_needed": false, "pcd_score": 0, "prior_session_summaries": [...]} plus matched_positive. Selection: no detected memory event, a substantive user turn, and a GPT-5.4 prior-context-dependence check = 0; 70/72 come from the same user as their positive.

corpus row (one session): user_id, chronological session_idx (joins to trigger_session_idx), date, summary, daily_summary, user_feedback (1–7 rating + comment), full turns with telemetry. 30 sessions have empty turns; ignore talking_index.

users row: num_sessions, date_range, total_turns, mean_rating, monthly_summaries.

1. Deployment. 40 proficient English speakers talked daily with "Luke" (GPT-4.1-mini) for ~4 months under 7 randomized memory conditions and rated every session 1–7.
2. Detection. GPT-5.4 scanned all 1,872 sessions for explicit memory moments; every candidate was verified by human annotators (95.5% precision).
3. Benchmark. The 72 user-cued moments were paired with their referenced prior context and decomposed into 316 fact questions. Judges validated against humans (Natural Integration human–human κ = 0.57; fact questions Fleiss κ = 0.65; judge stable across models, κ = 0.89 vs GPT-5.5).
4. Negatives. Matched turns with no memory cue, added for the public release to measure false positives.

Memory moments are rare — ~1.4% of user turns — which is exactly why they are worth testing on: existing benchmarks pair externally authored questions with 15–24% of turns.

• Real diary conversations from an IRB-approved longitudinal study; participants consented to public release for research use, self-curated what they shared, and had a withdrawal window (one person withdrew; their data is excluded).
• Pseudonymized: participants are annotator_NN; all personal names, pet names, and residence-level places / small local organizations replaced with consistent substitutes (same name → same substitute everywhere, including gold facts and questions), after a GPT-5.4 screen of every turn and human review. Public figures, brands, and large cities unchanged. Paper numbers were computed before pseudonymization.
• License: data CC BY-NC 4.0 (non-commercial research only); evaluation code MIT.