StreamMemBench

Streaming Evaluation of Agent Memory for Future-Oriented Assistance

Paper • Quickstart • Data • Evaluation • Memory Systems • Citation

---

📢 News

• 2026-06-12 Paper released on arXiv. StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance
• Code and benchmark data released.

---

🔍 Overview

StreamMemBench is an open benchmark for evaluating streaming persona memory systems. It turns time-ordered multimodal life streams into evidence-anchored tasks, then evaluates whether a memory system can use initial evidence, incorporate feedback, and reuse corrected memory in follow-up interactions.

Highlights

• Two-step task sequence centered around evidence anchors drawn from EgoLife egocentric streams. An initial task tests evidence use, and a follow-up task tests whether feedback and interaction experience are reused.
• Four diagnostic metrics covering fidelity, initial evidence use, feedback incorporation, and followup reuse. All are normalized to [0, 1].
• 8 memory systems evaluated across two backbone models, including RAG baselines (rag_raw, rag_extracted) and upstream memory systems (mem0, memos, evermemos, memskill, memoryos, a_mem).
• 6 participants × 7 days of egocentric stream data, yielding 2,854 evidence-task items and 16,214 tasks.
• Chinese and English benchmark versions with shared English public schema fields.

---

🚀 Quickstart

Install the package and run the deterministic smoke baseline.

python -m pip install -e .
streammembench eval --system rag_raw --config configs/evaluation/smoke.yaml --max-items 1 --output results/quickstart/smoke.jsonl
streammembench aggregate --results results/quickstart

For API-backed construction and evaluation, copy .env.example to .env and set LLM_API_KEY. The default endpoint is DeepSeek. Set LLM_BASE_URL and LLM_MODEL to use another OpenAI-compatible provider.

---

📦 Install

python -m pip install -e .

---

🖥 CLI

streammembench build --config configs/datasets/streammembench_v1.yaml
streammembench prepare --config configs/construction/egolife_v1.yaml --participant A1_JAKE --day DAY1
streammembench construct --config configs/construction/egolife_v1.yaml --participant A1_JAKE --day DAY1 --max-segments 1
streammembench eval --system rag_raw --config configs/evaluation/smoke.yaml --output results/local/rag_raw.jsonl
streammembench eval --system rag_raw --config configs/evaluation/paper_sample.yaml --output results/local/paper_sample.jsonl
streammembench systems
streammembench aggregate --results results/local
streammembench reproduce --system rag_raw --config configs/evaluation/release_sample.yaml --output-dir results/reproduction

The registered systems include two RAG baselines (rag_raw, rag_extracted) and memory systems (mem0, memos, evermemos, memskill, memoryos, a_mem). rag_raw runs without external API keys; rag_extracted uses an extraction model when configured. Third-party memory systems require their upstream packages, local checkouts, credentials, or service endpoints as documented in docs/memory_systems.md.

For eval and reproduce, the CLI automatically merges the main benchmark config with the registered system config under configs/memory_systems/. Use --system-config to point at a different system config for a local run.

See configs/README.md for how public configs map to the original research pipeline.

---

📊 Data

data/streammembench_v1 contains the converted full benchmark release.

• 6 participants
• 7 days
• 42 segment files
• 3,347 stream segments with source observations in segments/
• 2,854 evidence-task items, covering 2,854 unique stream segments with accepted evidence anchors
• 493 stream segments without a public evidence-task item after extraction/review
• 8,107 evidence anchors
• 16,214 tasks

stream_segments denotes the complete set of released stream windows provided as input to memory systems. evidence_task_items denotes the evaluated subset for which at least one evidence anchor and its associated task sequence were accepted during benchmark construction.

StreamMemBench is released in Chinese and English under a shared public schema with English field names. The two language versions are provided in data/streammembench_v1 and data/streammembench_v1_en, respectively, and share the same participant coverage, temporal segmentation, and task-set size.

For dataset scope, release boundaries, and known limitations, see docs/dataset_card.md.

Source and Splits

StreamMemBench v1 is derived from the EgoLife project. EgoLife should be cited alongside StreamMemBench when using this data.

• Paper
• Project page
• Data collection
• Video dataset
• Code

The converted benchmark uses six EgoLife participants across seven days. Public stream files are split by participant/day and then by the processed five-minute stream windows. See docs/data_sources.md.

---

📋 Dataset Item

Each benchmark item contains one stream segment and one or more evidence anchors.

{
  "participant": "A1_JAKE",
  "day": "DAY1",
  "segment_id": 1,
  "time_range": "11:10:00 - 11:15:00",
  "stream_segment": {
    "text": "...",
    "observations": []
  },
  "evidence_anchors": [
    {
      "evidence_id": "jake_workstation_setup",
      "evidence_statement": "...",
      "evidence_type": "stated_or_inferred",
      "subject": "Jake",
      "supporting_observations": [],
      "source_span": {
        "time_window": "11:10:00 - 11:15:00",
        "clip": "A1_JAKE/DAY1",
        "raw_indices": []
      },
      "tasks": {
        "initial_task": {
          "user_request": "...",
          "expected_behavior": "..."
        },
        "followup_task": {
          "user_request": "...",
          "expected_behavior": "..."
        }
      }
    }
  ]
}

The public schema includes the following fields.

• evidence, evidence_anchor, evidence_statement
• supporting_observations
• initial_task, followup_task
• feedback
• fidelity, initial_evidence_use, feedback_incorporation, followup_reuse

---

🏗 Construction

Construction supports the Chinese and English benchmark workflows with shared English public schema fields. Prompt templates are under prompts/, and their stage mapping is documented in prompts/README.md.

The released data preserves outputs from the two-round evidence/task construction and review pipeline. See docs/construction_pipeline.md for the source-to-release flow and multi-stage artifact layout.

---

🔄 Build Flow

The stable release flow is shown below.

EgoLife raw/processed events
  -> five-minute stream segments
  -> evidence anchor construction
  -> two-round construction trace
  -> public evidence-task files
  -> memory-system evaluation
  -> aggregate metrics

Stable public artifacts are saved in the following locations.

• data/streammembench_v1/segments/{participant}/{day}.json
• data/streammembench_v1/evidence_tasks/{participant}/{day}.clean.json
• results/construction/construction_logs/ for newly generated construction traces
• data/source/egolife_text/{dense_caption,transcript}.json for local stream reconstruction
• results/{run_id}/... for generated evaluation outputs

---

🧠 Memory Systems

Memory systems implement MemorySystem and are evaluated as agent-like systems. A system can wrap an in-process object, a local package, or a remote API as long as it can add stream records to memory, generate responses, revise with feedback, store the evaluated interaction, and optionally retrieve memory records for analysis. See docs/memory_systems.md.

Registered system names are listed below.

• rag_raw, rag_extracted
• mem0, memos, evermemos, memskill, memoryos, a_mem

rag_raw and rag_extracted are complete local baselines. mem0, memos, memoryos, a_mem, evermemos, and memskill have direct entry points under src/streammembench/memory/systems/; they require the corresponding upstream package, local checkout, or service configuration before their results should be reported.

---

📈 Evaluation

Evaluation results use four canonical metric keys.

• fidelity
• initial_evidence_use
• feedback_incorporation
• followup_reuse

All metric values are normalized to [0, 1]. See docs/evaluation.md.

Each evaluation result contains run_id, system_name, participant, day, evidence_id, the four metric fields, initial_answer, feedback, revised_answer, followup_answer, and metadata. revised_answer is empty when the feedback simulator judges that the first answer needs no correction.

For the prediction/result JSONL contract and label-leakage rules, see docs/submission_format.md.

---

🏗 Repository Layout

configs/                 Dataset, evaluation, and memory-system configs
data/                    Released Chinese/English benchmark data and smoke sample
prompts/                 Construction and evaluation prompt templates
src/streammembench/      Python package
docs/                    Dataset and reproduction documentation
results/                 Local result output directory

---

Citation

If you use StreamMemBench in your research, please cite the paper.

@misc{streammembench2026,
  title     = {StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance},
  author    = {Guanming Liu and Yuqi Ren and Hansu Gu and Peng Zhang and Weihang Wang and Jiahao Liu and Ning Gu and Tun Lu},
  year      = {2026},
  eprint    = {2606.14571},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI}
}

StreamMemBench v1 is derived from EgoLife. Please also cite EgoLife when using the benchmark data.

---

License

The StreamMemBench code is released under the MIT License. Benchmark data is derived from EgoLife and remains subject to the upstream EgoLife license and Hugging Face access terms.

---

Acknowledgements

StreamMemBench builds on the EgoLife project. We thank the EgoLife authors for making their egocentric stream data publicly available.

The benchmark integrates or evaluates upstream memory-system projects including A-Mem, Mem0, MemOS, MemoryOS, MemSkill, and EverMemOS. We thank their authors for releasing code that supports reproducible comparison.

For questions and feedback, open an issue on GitHub or contact the maintainers.