plur-bench

Reproducible benchmarks for the PLUR.ai memory engine, across three axes of memory quality (see below). Apache-2.0, fully local, keyless CI.

The retrieval track runs per-question canonical-doc-scored benchmarks on real LongMemEval-S — the same protocol gbrain, mem0, Letta, and the LongMemEval paper use — sleep- and crash-resistant via per-question JSON checkpoints. Designed for an A100 research host but runs on any CPU with patience.

---

What this measures — three axes

Memory quality is not one number. plur-bench keeps three distinct axes in separate tables and never conflates them:

Retrieval recall and end-to-end answer accuracy are themselves different axes (separate tables below). The editability and agent-task axes answer what recall cannot: whether the memory's design — correctability, decay, injection — changes outcomes. Each axis ships an offline path (*:test / *:smoke / *:demo, no API key) and a real path that runs on a configured box.

---

Headline result (LongMemEval-S, N=500, chunk granularity)

Measured with this harness using PLUR's vendored production retrieval code, at chunk granularity — the only setting comparable to gbrain (see Granularity).

PLUR matches the best published open retrieval result on LongMemEval. The harness was validated against gbrain's number (#4): run under gbrain's embedder and unit, PLUR reproduces 97.0% R@5 vs their 97.4–97.6% — within 1 pp.

These are Recall@K (retrieval) figures: the fraction of questions whose ground-truth evidence lands in the top-K. That is not the same axis as the end-to-end answer-accuracy numbers other systems headline (mem0/Mastra ~94–95%); the two are not directly comparable. An LLM-judge accuracy tier is available via --scoring llm_judge — see Scoring modes.

Why chunk granularity. Earlier turn-granularity numbers (BM25 93.2%, hybrid 96.2%) are not comparable to gbrain — finer units make "a unit from the right session is in the top-5" easier to hit. gbrain indexes one ~300-word chunk per session; matching that (#3) was the prerequisite for any honest comparison, and the harness equivalence was then validated (#4). Always publish chunk-granularity numbers against gbrain.

Per-category, from the validated openai-3-large control run (hybrid, chunk):

---

Comparison vs published numbers

Auto-generated by pnpm report from run manifests + config/published-numbers.yaml. Do not edit by hand.

Retrieval recall and end-to-end accuracy are different axes and are listed in separate tables — never compared directly. A third axis — agent-task outcomes (does memory change what the agent does, not just what it retrieves) — lives in agent-bench/: a with/without-PLUR A/B harness (ported from datacore-bench). Try it offline: pnpm agent-bench:demo.

Retrieval — Recall@K

End-to-end accuracy (LLM-judge)

No local run manifests found in results/ — run a benchmark to add this-harness rows. Published rows above always render.

Regenerate with pnpm report (reads run manifests + config/published-numbers.yaml). See Methodology metadata for what makes a row comparable.

---

TL;DR — how to reproduce

git clone https://github.com/plur-ai/plur-bench
cd plur-bench
pnpm install
cp .env.example .env                            # OPENAI_API_KEY for openai-3-large + llm_judge
./scripts/fetch-corpus.sh                       # ~260 MB LongMemEval-S (verifies sha256)

# Retrieval (R@K) at CHUNK granularity — the only setting comparable to gbrain:
caffeinate -i -s pnpm bm25:chunk                # ~25s for all 500 questions
caffeinate -i -s pnpm hybrid:chunk              # ~5 hours CPU (bge-small embedding)
caffeinate -i -s pnpm rerank:chunk              # ~3 days CPU — recommend A100

pnpm report                                     # regenerate the comparison table

End-to-end accuracy (LLM-judge) and controls — see Scoring modes:

PLUR_JUDGE=mock pnpm bm25:chunk --scoring llm_judge   # offline dry-run, no key
OPENAI_API_KEY=… pnpm rerank:chunk --scoring llm_judge --runs 10
pnpm full-context   # control ceiling   ·   pnpm no-memory   # control floor

Batched / resumable (checkpoints persist; stop and re-run the same command):

caffeinate -i -s pnpm rerank:chunk --limit 50   # first 50; resume by re-running

Or Docker:

docker compose up benchmark-runner                          # CPU
docker compose --profile gpu up benchmark-runner-gpu        # A100

---

Prompt to paste into Claude Code / Codex (research-machine workflow)

I'm on the PLUR research machine running benchmarks. The repo is plur-bench; the harness is benchmark/per-question.ts. The harness is feature-complete — the remaining work is producing the definitive numbers on this box (A100 + OPENAI_API_KEY). Always run at --granularity chunk (the only setting comparable to gbrain; harness equivalence is validated, ~97% R@5).

Please:

1. Corpus: run ./scripts/fetch-corpus.sh; it verifies the sha256. If config/datasets.yaml shows the dataset UNPINNED, record the printed digest there and re-run (issue #23).
2. GPU: confirm nvidia-smi shows the A100 and PLUR_ONNX_PROVIDER=cuda loads.
3. Retrieval (R@K), chunk granularity, with checkpoints:

   pnpm bm25:chunk
   pnpm hybrid:chunk
   PLUR_ONNX_PROVIDER=cuda PLUR_DISABLE_REACTIVATION=1 pnpm rerank:chunk
   PLUR_EMBEDDER=openai-3-large PLUR_OPENAI_3_LARGE_DIM=1536 pnpm rerank:chunk   # headline stack
   pnpm bm25:chunk:dedup / hybrid:chunk:dedup / rerank:chunk:dedup               # gbrain best-per-page parity (#21)
   

4. End-to-end accuracy (vs mem0/Zep/Letta/Mastra), OPENAI_API_KEY set:

   pnpm rerank:chunk --scoring llm_judge --runs 10
   pnpm full-context   # control ceiling (mandatory)   ·   pnpm no-memory   # control floor
   pnpm bm25 --corpus locomo --preset mem0-paper --scoring llm_judge          # vs mem0 paper (#27)
   

5. Each question writes a checkpoint; runs resume by re-running the same command. Each run also writes results/<run-id>/manifest.json (full methodology + accuracy/cost).
6. Run pnpm report to regenerate the comparison table from the manifests + config/published-numbers.yaml, and post the table. Keep R@K and accuracy in separate columns — they are different axes.
7. Significance + equivalence (the H1 gate): the per-question arms are PAIRED on question_id, so a bare ladder of point estimates is not a verdict. Run pnpm stats compare between adjacent arms to get the McNemar exact p, per-arm Wilson 95% CIs, and the TOST equivalence verdict:

   pnpm stats compare results/hybrid-chunk-cp results/rerank-chunk-cp   # is the reranker's lift significant?
   pnpm stats compare results/bm25-chunk-cp   results/hybrid-chunk-cp   # is dense > sparse significant?
   

Reference (chunk granularity, validated): gbrain hybrid R@5 = 97.6%; PLUR openai-3-large control R@5 = 97.0% (within 1pp). gbrain's "BM25 19.8%" is mislabelled Postgres FTS — not a real BM25 baseline.

---

Methodology — what makes this comparable to gbrain

---

Significance & equivalence (the H1 gate)

A ladder of point estimates (BM25 → +dense → +reranker) is not a verdict. The arms are measured on the same questions, so they are paired — the right tests are paired ones. scripts/paired_stats.py (pnpm stats) reads two arms' per-question checkpoint dirs, joins on question_id, and reports:

• McNemar exact test (two-sided binomial on the discordant pairs) — is the difference between two arms significant? At our operating point (~97% R@5 on N=500) the discordant counts are tiny, so a 0.6pp reranker "lift" is almost certainly not significant.
• Wilson 95% CIs per arm — the honest uncertainty band on each recall.
• TOST equivalence (default margin ±2pp) — the test that actually backs the public claim. "Local parity with the cloud frontier" is an equivalence claim, and equivalence has to be tested directly, not inferred from a non-significant difference. TOST asks: is the paired difference provably inside ±2pp?

pnpm stats compare results/hybrid-chunk-cp results/rerank-chunk-cp
pnpm stats:test     # known-answer unit tests for the estimators

Retrieval R@K is deterministic (fixed embeddings, deterministic BM25/RRF/ reranker), so there is no seed variance on the retrieval axis — the binomial Wilson CI is the uncertainty, and re-running gives the identical number. Cross- seed SD (pnpm stats seeds ...) is meaningful only on the accuracy axis (--scoring llm_judge, sampled answer generation), never on R@K. Don't report a "±SD over seeds" on a recall number — it is structurally 0.

This is the defensible framing the numbers support: dense embedding is the workhorse; hybrid and the reranker are at statistical parity with it on this benchmark, fully local. Never "we beat the frontier."

---

Corpus integrity & reproducibility

A benchmark number means nothing unless two people can prove they ran on the same bytes. config/datasets.yaml is the single source of truth for each corpus — its source, the file the harness reads, its license, question count, and a pinned sha256:

datasets:
  longmemeval-s:
    name: LongMemEval-S
    source: hf:xiaowu0162/longmemeval
    file: data/longmemeval_s.json        # path within the HuggingFace repo
    local: corpus/longmemeval_s.json     # where the harness reads it
    sha256: ""                           # pinned digest (empty = not yet pinned)
    n_questions: 500

• Fetch verifies. ./scripts/fetch-corpus.sh downloads the corpus, places it at corpus/longmemeval_s.json, and checks its sha256 against the pin. A corrupt, partial, or wrong file fails the run; it never silently accepts unverified bytes.
• First-run pinning. Until sha256 is filled, fetch/verify prints the computed digest and stops (UNPINNED). Confirm it is the official release, paste the digest into config/datasets.yaml, commit — from then on every run is gated against it.
• Every run is traceable. The harness logs corpus_sha256=… at load and, if the dataset is pinned, confirms the match (or warns loudly on mismatch). This digest is what #7 embeds in each run manifest as dataset.sha256.

pnpm verify-corpus                       # check corpus/longmemeval_s.json against the pin
# or an arbitrary file:
npx tsx scripts/verify-corpus.ts --file <path> --expect <sha256>

Exit codes: 0 match · 1 mismatch · 2 unpinned (digest printed) · 3 missing/args.

LongMemEval ships several variants (longmemeval-cleaned, longmemeval-v2); we pin the exact file we measured on rather than tracking "latest" — see the comments in config/datasets.yaml.

---

Methodology metadata & run manifests

A number is only comparable to another if you can see exactly how each was produced. The Zep↔mem0 dispute is the cautionary tale: their scores for each other diverged by up to 25 pp purely from adapter/judge/protocol differences. So every run records its full methodology.

Each run writes results/<run-id>/manifest.json — the citable artifact — containing a versioned methodology block plus the aggregate and per-category stats:

{
  "schema_version": 1,
  "dataset":     { "name": "longmemeval_s", "sha256": "…", "source": "hf:xiaowu0162/longmemeval", "n_questions": 500 },
  "protocol":    { "style": "per-question", "granularity": "chunk", "top_k": 10, "session_dedup": false },
  "system":      { "name": "plur", "mode": "hybrid", "embedder": "bge-small-en-v1.5", "reranker": null, "vendored_core_ref": null, "adapter_config": {} },
  "scoring":     { "metric": "canonical_doc_r@k", "judge_model": null, "judge_prompt_sha256": null, "runs": 1 },
  "environment": { "harness_git_sha": "…", "node": "v22.…", "os": "darwin 25.…", "hardware": "…", "lockfile_sha256": "…" },
  "comparability": { "target": null, "caveats": [] }
}

• Methodology hash. A short fingerprint over the protocol-affecting fields (dataset, protocol, system, scoring metric/judge — not environment or run counts, so two machines on the same protocol agree). It is stamped into every per-question checkpoint.
• Mixed-protocol guard. On resume, a checkpoint whose stamped hash differs from the current run is rejected — you cannot accidentally pool results measured under different settings into one directory. (Checkpoints written before this stamping warn rather than reject.)
• The dataset.sha256 is the corpus pin from Corpus integrity; comparability.target references an entry in config/published-numbers.yaml, and pnpm report flags whether the run's protocol actually matches that target (see Comparison vs published numbers).

---

Scoring modes — retrieval vs end-to-end accuracy

--scoring selects what a "correct" answer means:

The two are different axes and not interchangeable — canonical_doc is retrieval recall; llm_judge is answer accuracy.

llm_judge pipeline: the top-K retrieved units are formatted into a context block → an answer model generates a prediction → a judge model rules it correct/incorrect against the gold answer, repeated --runs N times for accuracy ± SD (binary correct/incorrect, mean over questions; matches mem0's paper protocol).

# offline, no API key — deterministic mock judge (CI / smoke):
PLUR_JUDGE=mock pnpm bm25 --scoring llm_judge --runs 3

# real judge (default gpt-4o) + answer model (default gpt-4o-mini):
OPENAI_API_KEY=sk-… pnpm hybrid --scoring llm_judge --runs 10

• Pluggable judges (src/plur-core/judges/, mirrors embedders/rerankers): PLUR_JUDGE=openai|anthropic|mock, PLUR_JUDGE_MODEL, PLUR_ANSWER_MODEL. Anthropic is a stub; swapping providers is config-only.
• Versioned judge prompt — config/prompts/judge-binary-v1.txt; its sha256 is recorded in methodology.scoring.judge_prompt_sha256.
• API-key gated — --scoring llm_judge with the OpenAI judge exits cleanly if OPENAI_API_KEY is unset; tests and CI never require a key (mock judge).
• Resumable — per-question judge verdicts are checkpointed; raising --runs re-judges, a smaller dir is transparently upgraded. The manifest carries accuracy (mean ± SD, per-run) and cost (tokens in/out).

Control baselines

Two honest control rows the field tends to omit (both --scoring llm_judge only):

PLUR_JUDGE=mock pnpm full-context   # or pnpm no-memory

• They do no retrieval, so Recall@K is null in the manifest (not 0% — it is not applicable). Accuracy is the figure; token cost is the point: full-context sends ~115k tokens/question on LongMemEval-S, PLUR retrieves a few hundred. That efficiency gap is the headline these controls exist to expose.
• Manifests carry system.name = "control:full-context" | "control:no-memory" so the comparison report (#8) renders them as reference rows in every LLM-judge table.

---

LOCOMO & the mem0-paper preset

LOCOMO lets PLUR be compared to mem0's paper numbers (LOCOMO J = 66.88 base / 68.44 graph, arXiv:2504.19413). It is the most-cited and most-disputed dataset in the space — we run it with its flaws documented, not avoided.

# fetch locomo10.json yourself (CC BY-NC — see below), then:
PLUR_JUDGE=mock pnpm bm25 --corpus locomo --preset mem0-paper --scoring llm_judge

• --format locomo converts LOCOMO into the per-question shape: each conversation's sessions become the haystack; a question's evidence dia-ids (D<session>:<turn>) become its answer_session_ids. That one mapping makes both scoring tiers work unchanged — canonical_doc R@K scores evidence-session membership, llm_judge answers from the retrieved turns.
• --preset mem0-paper pins the comparability-critical choices: llm_judge scoring, --runs 10, --top-k 30, judge + answer model gpt-4o-mini, category-5 (adversarial) excluded, and comparability.target = mem0-paper-locomo. Explicit flags still override; env still wins for models.
• --corpus locomo resolves to the locomo entry in config/datasets.yaml (path + format). Use --corpus <path> --format locomo for an arbitrary file.
• The adapter records its speaker/role + dia-id mapping in the manifest's system.adapter_config — this is the exact point where mem0's run of Zep was disputed, so it must be auditable.

Caveats (documented, not hidden):

• License: the mc10 LOCOMO release is CC BY-NC 4.0 (non-commercial). It is not vendored here — fetch it yourself; the sha is unpinned until you do.
• Full-context is mandatory in any LOCOMO table: conversations are 16k–26k tokens, so they fit a modern context window and the full-context control tends to beat memory systems. Always include --mode full-context.
• The dispute: Zep's LOCOMO J has been reported as 58.4 / 66.0 / 75.1 / 84 depending on the harness — root-caused to adapter errors (speaker/role mapping, timestamps in text vs created_at, sequential vs parallel search). Never cite a single Zep LOCOMO number without naming the harness. Recorded as zep-locomo-disputed in config/published-numbers.yaml.

Schema note: the adapter follows the snap-research/locomo locomo10.json field layout; confirm the mapping against the release on the first real run.

---

Granularity — the comparability lever

How a conversation is sliced into search units decides what the numbers mean. Controlled with --granularity:

chunk mode reproduces gbrain v0.28.8's chunker — 300-word slices, 50-word overlap, 6000-character cap (their src/core/chunkers/recursive.ts). (gbrain's contextual-retrieval feature shipped later, in v0.40.3.0, so the published 97.6% is plain chunk-hybrid — no special wrapper to replicate.)

pnpm bm25:chunk        # or: pnpm hybrid:chunk / pnpm rerank:chunk
# equivalently:
npx tsx benchmark/per-question.ts --mode bm25 --granularity chunk --checkpoint-dir results/bm25-chunk-cp

Checkpoints are namespaced per granularity, so chunk and turn runs can share a results directory without overwriting each other.

Only chunk-granularity numbers should be published against gbrain. Track: #3 (this granularity fix) and #4 (proving the harness reproduces gbrain's 97.6% by running openai-3-large at chunk granularity before trusting the airgapped bge-small number).

---

Session dedup — exact best-per-page parity

Granularity matches the ingestion unit; this matches the scoring unit. At chunk granularity a single long session can occupy several of the top-K slots, so a raw top-5 might cover only 2–3 distinct sessions. gbrain dedups to the best chunk per page (its best_per_page CTE) before applying the limit, so its top-K always represents K distinct sessions. Without dedup, plur-bench's R@K is therefore slightly conservative relative to gbrain.

--dedup-sessions collapses the ranked results to the highest-ranked unit per session before scoring, restoring exact protocol parity:

pnpm bm25:chunk:dedup     # or: pnpm hybrid:chunk:dedup / pnpm rerank:chunk:dedup
# equivalently:
npx tsx benchmark/per-question.ts --mode hybrid --granularity chunk --dedup-sessions \
  --checkpoint-dir results/hybrid-chunk-dedup-cp

Properties (see benchmark/dedup.test.ts):

• Default off — existing checkpointed runs stay reproducible.
• hit_1 is unchanged — the top-ranked session is never moved.
• hit_5 / hit_10 can only rise — dedup never hides a session, so the number is >= the non-dedup number; it removes a downward bias, it does not flatter PLUR beyond gbrain's own rule.
• No-op at session granularity (units are already one per session); the effect is largest at chunk.

Checkpoints are namespaced by both granularity and dedup, so dedup and non-dedup runs can share a results directory without overwriting each other.

---

Repository structure

plur-bench/
├── README.md                # this file
├── Dockerfile               # node + pnpm + py + system deps
├── docker-compose.yml       # 3 profiles: cpu, gpu (A100), postgres
├── entrypoint.sh            # modes: matrix (default), single, shell
├── package.json
├── benchmark/
│   └── per-question.ts      # the working harness (per-question, canonical-doc, resumable)
├── src/plur-core/           # vendored PLUR retrieval code
│   ├── fts.ts               # searchEngrams (PLUR BM25)
│   ├── hybrid-search.ts     # rrfMerge, applyReranker, hybridSearchWithMeta
│   ├── embeddings.ts        # legacy cache-backed embedding search
│   ├── embedders/           # minilm, bge-small, bge-base, embedding-gemma, openai-3-large
│   ├── rerankers/           # bge-reranker-v2-m3, jina-reranker-v2-base
│   ├── schemas/             # Engram zod schema
│   ├── sync.ts logger.ts ...
├── config/
│   └── matrix.yaml          # for the matrix runner (optional, second-tier)
├── corpus/                  # fetched at setup (gitignored)
│   └── .gitkeep
├── results/                 # checkpoints + summaries (gitignored)
│   └── .gitkeep
└── .env.example             # OPENAI_API_KEY (openai-3-large + llm_judge), etc.

---

Checkpoint resume — how it works

--checkpoint-dir <path> is the key flag. Each completed question writes <dir>/<question_id>[.<granularity>][.dedup].json (the turn, non-dedup run keeps the legacy <question_id>.json name):

{"question_id":"e47becba","category":"single-session-user","mode":"rerank","granularity":"chunk","dedup_sessions":true,"hit_1":true,"hit_5":true,"hit_10":true,"latency_ms":195234}

On startup the harness loads existing checkpoints into the running stats, then skips any question with a matching checkpoint. This means:

• Sleep / lid close: stops, resume seamlessly
• Kill / crash: stops, resume seamlessly
• Reboot: stops, resume seamlessly
• Want to run in batches: --limit 50 then --limit 100 then --limit 150 — each pass processes only what's not yet checkpointed

Atomic write (<file>.tmp + rename) means a kill mid-write doesn't corrupt the checkpoint file.

---

On the GPU rerank story

The reranker is a 568M-param cross-encoder. On CPU each query scores ~50 candidate (query, doc) pairs through the model — that's the 196 s/query observed locally. On A100:

• onnxruntime-node-cuda puts inference on GPU
• bge-reranker-v2-m3 in q8 quantization, batched 50 pairs per call → expected ~2-5 s/query
• Full 500 questions → ~30-60 min wall time

The Dockerfile builds with onnxruntime-node by default; for GPU set PLUR_ONNX_PROVIDER=cuda and the compose gpu profile mounts NVIDIA devices.

---

Continuous integration

CI is a keyless gate — it never downloads the real corpus and never calls an external API (judge logic is covered by the mock; real-judge runs are manual). .github/workflows/ci.yml runs on every push and PR:

pnpm typecheck     # tsc --noEmit
pnpm test          # unit tests (schema, scoring, judge-mock, report, …)
pnpm ci-smoke      # run the harness on corpus/fixture-longmemeval.json, then assert:
                   #   • a schema-valid manifest is produced
                   #   • fixture R@5 lands in the pinned band [0.8, 1.0]
                   #   • the rendered report matches test/golden/fixture-comparison.md

A PR that breaks the manifest schema, scoring, or the report generator fails the gate. If you intentionally change the registry or report format, regenerate the golden with pnpm ci-smoke:bless and commit it.

---

Next bench-repo improvements (for the colleague)

1. GPU verification — confirm onnxruntime-node-cuda actually accelerates bge-reranker-v2-m3 on the A100 (the 10-50× claim needs a measurement)
2. LLM-as-judge scoring mode — ✅ shipped: --scoring llm_judge (see Scoring modes). Remaining: run it with a real judge on the full corpus and record the accuracy figure.
3. OpenAI-3-large run — PLUR_EMBEDDER=openai-3-large with the Matryoshka 1536d truncation we discussed. Expected R@5 ≈ 97-98% (matching or beating gbrain's headline).
4. plur sync --reembed integration — currently we re-embed per question; on the A100 this is fine but could be cached across the corpus if memory allows.

---

License

Apache 2.0.