MemoryBench Experiment Results

Paper • Code • Dataset

---

Overview

This repository contains experiment results for MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems. MemoryBench evaluates whether LLM systems can learn from accumulated user feedback during service time. The official benchmark data is hosted at THUIR/MemoryBench.

This repository is an artifact archive for published runs. It stores model predictions, per-sample evaluation details, aggregate summaries, and sanitized run configurations so that result tables can be inspected and reused without rerunning the full experiments.

What's Included

The archive currently contains result groups across four backbone models: Qwen3-8B, Qwen3-32B, Mistral-Small-3.2-24B-Instruct-2506, and DeepSeek-V4-Flash. Each result group corresponds to one combination of:

• experiment: currently off-policy
• backbone model
• dataset split: domain or task
• set name: for example Open-Domain, Academic&Knowledge, Long-Long
• memory system baseline

Layout

Each result group is stored as:

{exp}-{model}/{domain|task}/{set_name}/{baseline}/
├── evaluate_details.json
├── predict.json
├── run_config.json
└── summary.json

evaluations/llm-judge/{exp}-{model}/{domain|task}/{set_name}/{baseline}/
├── evaluate_details.json
└── summary.json

The default files contain the original evaluation results. LLM-as-judge results are stored under evaluations/llm-judge/. The LLM judge results reuse the original predict.json and run_config.json.

File meanings:

• summary.json: aggregate metrics, including the summary field used for result tables.
• evaluate_details.json: per-sample evaluation details.
• predict.json: model predictions.
• run_config.json: sanitized run configuration for provenance.

Usage

The easiest way to consume this repository is through the MemoryBench Python API.

from memorybench import MemoryBenchResults

results = MemoryBenchResults.from_hf(
    repo_id="THUIR/MemoryBench-Results",
)

summary = results.load_summary(
    exp="off-policy",
    model="Qwen3-8B",
    dataset_type="domain",
    set_name="Open-Domain",
    baseline="a_mem",
)
print(summary["summary"])

You can also load predictions, evaluation details, and run configurations:

predicts = results.load_predict(
    exp="off-policy",
    model="Qwen3-8B",
    dataset_type="domain",
    set_name="Open-Domain",
    baseline="a_mem",
)

details = results.load_evaluate_details(
    exp="off-policy",
    model="Qwen3-8B",
    dataset_type="domain",
    set_name="Open-Domain",
    baseline="a_mem",
)

config = results.load_run_config(
    exp="off-policy",
    model="Qwen3-8B",
    dataset_type="domain",
    set_name="Open-Domain",
    baseline="a_mem",
)

To build a table from one metric in the summary field:

from memorybench import load_result_summary_table

table = load_result_summary_table(
    metric="weighted_average",
    exp="off-policy",
    models=["Qwen3-8B", "Qwen3-32B"],
    dataset_type="domain",
    set_name=None,
    baselines=["wo_memory", "bm25_message", "a_mem"],
    repo_id="THUIR/MemoryBench-Results",
)
print(table)

By default, summary-table APIs return a pandas.DataFrame. Pass as_pandas=False to return a list of dictionaries.

Citation

If you use MemoryBench or these result artifacts in your research, please cite:

@inproceedings{ai2026memorybench,
  title     = {MemoryBench: A Benchmark for Memory and Continual Learning in {LLM} Systems},
  author    = {Qingyao Ai and Yichen Tang and Changyue Wang and Jianming Long and Weihang Su and Yiqun LIU},
  booktitle = {Forty-third International Conference on Machine Learning},
  year      = {2026},
  url       = {https://openreview.net/forum?id=If4X4W2HWx}
}