recalld-benchmarks

Public, reproducible memory benchmarks for Recalld, run on the Agent Memory Benchmark (AMB), Vectorize's open-source harness. We did not build our own harness. The harness is pinned to one commit and every change we made to it is a patch in this repository, each marked with whether it can affect scoring. Answer and judge prompts are defined by the dataset (not per-provider adapters), so scores reflect memory quality rather than custom answerer tuning.

Raw reports are committed unedited in results/, gzipped.

Recalld is a product of Bit Robotics Ltd, which publishes this repository, and Recalld is one of the systems measured in it. Read What we changed in AMB before quoting any number: three of the seven patches touch code that affects scoring, and one of them changes ingestion order for every provider.

Published results

Both runs cover all ten locomo10 conversations and their 1540 non-adversarial questions. LoCoMo has 1986 questions in total; AMB excludes the 446 adversarial ones, and so do we.

Each result is for the engine as it was on the run date. The engine is updated regularly, so current results may differ slightly; we re-run after changes that affect recall.

Same harness, dataset, answer model, judge model, retrieval limit, context mode and memory stores. /memory/recall runs an LLM selection pass server-side and returns only the excerpts it picked; /memory/search returns the top-k excerpts by vector similarity with no LLM. The runs are four weeks apart and we did not record the Recalld server revision for either, so the endpoint is the intended difference, not a proven sole one. Recall's retrieved context is 6.7x smaller. We make no claim about which endpoint is more accurate: see the next section.

Read this before quoting a number

The judge accepts "I don't know". AMB's LoCoMo judge prompt has a clause for questions whose gold answer says the information is missing. None of these 1540 questions is of that kind, but the default judge model applies the clause anyway: an answer such as "The context lacks the information" is often marked correct although the gold answer is a concrete fact. A rough text search of the committed reports finds about 90 such rows in the recall run and about 30 in the search run. We kept AMB's judge prompt and default judge model unchanged so that our numbers stay comparable with other AMB results, which are judged the same way. The consequence is that the headline accuracies are upper bounds, and that the leniency helps recall more than search. Every row's answer, gold answer and judge reasoning is in the reports, so this can be checked.

Three recall rows retrieved nothing. conv-41_q69, conv-42_q92 and conv-44_q43 have the context {"sources": []}. The answer model said it could not answer and the judge marked all three correct, for the reason above. When these runs were made our validation only caught blank strings, so they passed. The rows are left in and the denominator is unchanged. verify_retrievals.py now rejects an empty source list, and summarize_results.py lists these rows.

Category labels in the reports are AMB's, and three of them are swapped. Checked against the LoCoMo evaluation code and the question counts in the paper, AMB's single-hop is LoCoMo's multi-hop, its multi-hop is open-domain, and its open-domain is single-hop. The raw reports keep AMB's labels. Read them as follows:

07-locomo-category-labels corrects the labels for future runs. It does not change any score.

Recompute both from the committed reports, without an API key:

uv run python scripts/summarize_results.py

avg_context_tokens is what the answer model is given per question. It is not the memory system's total compute. See the run notes for Recalld's own server-side token spend on the same run.

Comparing against other systems

AMB ships its authors' own LoCoMo runs on these same 1540 questions, in results-manifest.json. Those are the fairest external reference points, since we did not run them:

One caveat before using that table. Our runs apply 03-locomo-session-order, which loads a conversation's sessions in chronological rather than alphabetical order. The AMB runs above predate that patch and do not have it. We have not measured what difference it makes to a score, so treat the accuracy column as indicative rather than a like-for-like ranking.

The LoCoMo entries in AMB's external_results.json, including its Mem0 figures, are third-party blog claims rather than runs on this harness. They are not comparable with anything above.

What is measured

The LoCoMo benchmark: long conversations between two people, spread over many sessions. After conversations are stored in a memory system, the harness asks questions about them. A memory system scores well if retrieved context leads to correct answers.

For each conversation unit, AMB:

1. Ingests all session documents into the memory provider
2. Retrieves relevant memories for each question (top-k)
3. Generates an answer using the dataset's published RAG prompt (same for all providers)
4. Judges the answer against ground truth using the dataset's published judge prompt

Providers in this repo

Only the two Recalld adapters have published numbers so far. The Mem0 OSS adapter is included so the comparison can be run, by us or by anyone else, not because we are quoting results from it. We have not published a Mem0 OSS run; do not read this repository as a Recalld-vs-Mem0 claim.

For third-party baselines on the same harness, use AMB's own published results rather than anything in this repository.

Fixed configuration

What we changed in AMB

The harness is not vendored. scripts/setup.py clones it at the pinned commit, copies in files that do not exist upstream, then applies every patch in patches/ in filename order. Nothing else is modified.

Added files (overlay/), no upstream file is touched:

• src/memory_bench/memory/: recalld.py, mem0_oss.py, locomo_parse.py
• tests/: unit tests for the Recalld adapter

Patches. The stack is sequential; 01 and 02 both edit src/memory_bench/memory/__init__.py:

Patches 03, 04 and 07 look like upstream bugs to us rather than benchmark-specific tweaks; we would rather they land in AMB than live here.

Reproduce it

You need: Python >= 3.11, uv, git, Docker (only for Mem0 OSS), a Recalld API key, and a Gemini API key (for answer + judge).

git clone https://github.com/bit-robotics/recalld-benchmarks
cd recalld-benchmarks
uv run python scripts/setup.py

Fill in your keys in amb/.env (created from .env.example). To reproduce the published configuration, also uncomment the block marked "Published-run configuration" in that file: it sets the answer model, the judge model and RECALLD_ANSWER_CONTEXT=sources. Without it AMB uses its own default answerer and the adapter uses facts mode, which is a different experiment. The published runs routed Gemini through Vertex AI because AI Studio's daily request cap stops a full run partway; the GEMINI_BACKEND=vertex block in .env.example shows how. An AI Studio key is enough for the validation gate and single-conversation runs.

Check the adapters before spending any API budget:

cd amb
uv run --with pytest pytest tests -q
cd ..

Every command block in this README starts from the repo root.

Running Mem0 OSS

Start the open-source Mem0 server following the mem0 repository so it listens on http://localhost:8888 (or set MEM0_OSS_BASE_URL). The adapter targets server API v2.0.11.

• Set AUTH_DISABLED=true on the Mem0 server. The adapter sends no authentication header, so a server with auth enabled rejects its requests.
• Record server version/commit and extraction model in the run's notes.md.

Validation gate: always run this first

Before any full run, test retrieval on a small slice:

cd amb
uv run omb run --dataset locomo --split locomo10 --memory recalld --mode rag --unit conv-30 --query-limit 5
cd ..
uv run python scripts/verify_retrievals.py recalld

For recalld-recall:

cd amb
uv run omb run --dataset locomo --split locomo10 --memory recalld-recall --mode rag --unit conv-30 --query-limit 5
cd ..
uv run python scripts/verify_retrievals.py recalld-recall --mode rag

Run the gate for each provider you intend to benchmark (mem0-oss works the same way as recalld), and only continue when it passes.

A run whose retrieval silently returns nothing still produces a plausible-looking score, because the answer model guesses or the judge accepts "I don't know". verify_retrievals.py exits non-zero if any question got empty context, including an empty source list; summarize_results.py lists such rows under the totals.

Evaluate custom context for one question

To score arbitrary plaintext (e.g. output from any memory tool) against a gold answer without running ingestion or retrieval:

uv run python scripts/eval_context_file.py \
  --dataset locomo --split locomo10 --mode rag \
  --unit conv-30 --query-id conv-30_q0 \
  path/to/context.txt

Recalld retrieval sweeps (ingest once, score all k/endpoint combos)

To compare /memory/search and /memory/recall at multiple top-k values without re-ingesting the same conversation, use the sweep runner. It ingests once on the first matrix entry, preserves Recalld agents, then runs fully scored AMB experiments (answer + judge) for every combination.

# One conversation: ingest once, then run 8 fully scored AMB experiments
uv run python scripts/run_recalld_sweep.py --unit conv-30 --k 10 25 50 100

# Search endpoint only (smoke test)
uv run python scripts/run_recalld_sweep.py --unit conv-30 --k 10 --endpoints search --query-limit 5

# Resume unfinished combinations without re-ingesting
uv run python scripts/run_recalld_sweep.py --unit conv-30 --k 10 25 50 100 --resume --sweep-id <id>

# Explicitly remove only this sweep's preserved Recalld agents
uv run python scripts/run_recalld_sweep.py --sweep-id <id> --cleanup

Defaults: endpoints search,recall; k values 10,25,50,100; dataset locomo; split locomo10; mode rag. Each combination gets a unique AMB --name such as recalld-search-k25-<sweep-id> and writes to amb/outputs/locomo/<name>/rag/locomo10.json.

Sweep state (commands, accuracy, answer/judge models) is stored under experiments/sweeps/<sweep-id>/state.json. Agent IDs are shared via experiments/sweeps/<sweep-id>/recalld-agents.json.

Verify each sweep output independently:

uv run python scripts/verify_retrievals.py --name recalld-search-k25-<sweep-id> --mode rag
uv run python scripts/verify_retrievals.py --name recalld-recall-k50-<sweep-id> --mode rag

Note: AMB's published leaderboard numbers use each provider's default context size (Recalld defaults to k=10). Sweep k values are an explicit ablation: record endpoint, k, and avg_context_tokens from the output JSON when publishing.

Full comparison runs

The published runs are one AMB run per conversation, ten per provider. Give each its own --name so the reports do not overwrite each other:

cd amb
for unit in conv-26 conv-30 conv-41 conv-42 conv-43 conv-44 conv-47 conv-48 conv-49 conv-50; do
  uv run omb run --dataset locomo --split locomo10 --memory recalld-recall --mode rag \
    --unit "$unit" --name "recalld-recall-$unit"
done
cd ..

Swap --memory recalld-recall for recalld (search) or mem0-oss, and change the --name prefix to match. Each report lands in amb/outputs/locomo/<name>/rag/locomo10.json. Verify every one:

uv run python scripts/verify_retrievals.py --name recalld-recall-conv-26 --mode rag

Then copy the ten reports into one directory under results/ (see results/README.md) and aggregate them:

uv run python scripts/summarize_results.py results/<run-dir>

This re-ingests every conversation. The published reports did not: they scored stores built by earlier runs with --skip-ingestion, and we did not record the Recalld server revision or prompt versions in use when those stores were built. A fresh reproduction measures today's Recalld, not the August build, and will not match question for question.

Repository layout

overlay/    Files added to the AMB clone: provider adapters and their tests
patches/    Sequential patches applied to AMB, in filename order
scripts/    Setup, verification, sweeps and result aggregation
results/    Committed raw reports (gzipped) with per-run notes
amb/        The AMB clone itself: created by setup.py, never committed

Corrections welcome

If you work on Mem0 or any compared system and believe a setting here misrepresents your product, please open an issue or pull request. We will re-run with corrected settings and publish new numbers alongside the old ones. The same applies to our own numbers: if you can show a patch in patches/ biases a result, we want to know. See CONTRIBUTING.md.

Licences and attribution

This repository (adapters, patches, scripts, notes) is released under the MIT License, copyright Bit Robotics Ltd.

LoCoMo is by Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri and Yuwei Fang (ACL 2024), and is licensed CC BY-NC 4.0. This repository does not ship the dataset; AMB downloads it. The committed reports do, however, reproduce parts of it: each question, its gold answer, and the conversation excerpts the memory system retrieved for it. They are included so that the published numbers can be verified, and are unmodified apart from gzip compression. If you reuse the reports, the CC BY-NC 4.0 terms apply to that content.

AMB is not vendored here and had no licence file at the pinned commit. The files in patches/ contain the minimal upstream context needed for git apply; everything else in the harness is fetched from Vectorize's repository at setup time. See CITATION.cff for how to cite all three.