gnosis-membench

Benchmark harness for the gnosis memory service. Runs LongMemEval_S and LOCOMO through a consistent ingest → retrieval → answer → grade pipeline. Results are logged in RESULTS.md, the append-only run ledger.

Current standing — LongMemEval_S L-35 (full 500-Q, 2026-08-10) — new best overall: gpt-4o backbone + gpt-4o judge. Reuses L-31 Neo4j data; answer.py only changes.

L-31 (2026-08-09) — KU structural fix:

• KU 70.8% → 80.6% (+9.8pp) via write-time SUPERSEDES edges + valid_to IS NULL filter
• Overall 71.0%; regressions (SSA −3.6pp, temporal −7.9pp, MS −4.2pp) confirmed as ingest variation

L-32 (2026-08-10) — enumeration clause fix + multi-query expansion for MS:

• MS 54.5% → 59.5% (+5.0pp) via 2-sub-query LLM expansion for aggregative multi-session questions
• KU 80.6% → 81.9% (+1.4pp); overall 72.6% (+1.6pp vs L-31)

L-33 (2026-08-10) — extended pattern + 4 sub-queries:

• Extended _AGGREGATIVE_PATTERN to include average|percentage|how long (6 pattern-miss failures now covered)
• Sub-queries increased 2→4; dedup via seen: set[str] (was substring scan)
• Overall 74.2% (+1.6pp vs L-32, +0.6pp vs L-25b)

L-34 (2026-08-10) — math instruction for aggregative MS questions:

• Appended [instruction] to retrieved context for aggregative multi-session questions: list every value, compute step by step
• MS 60.3% → 66.1% (+5.8pp, +7 questions) — targets wrong-sum failures from L-32 analysis
• Overall 74.2% (ties L-33)

L-35 (2026-08-10) — conservative math instruction + pattern extensions (new best overall):

• Conservative _MATH_NOTE with 3-check filter (direct match, time period, dedup); pattern extended with increase, page count
• MS flat (66.1%, 6 fixed / 6 broken — check (1) too vague → abstention failures); temporal +4.7pp (71.7%)
• Overall 75.2% (+1.0pp, new best vs L-33/L-34 at 74.2%)

LOCOMO standing (Run 23, full-10, 2026-07-04): excl-adv J 66.9–68.9 at parity with mem0 (66.88), leading on single-hop, temporal, adversarial, and multi-hop F1. Open-domain remains the LOCOMO gap (J 29.2 vs frontier ~74–77).

See RESULTS.md for the full run ledger.

What the harness measures

Each benchmark conversation gets an isolated gnosis scope:

benchmark JSON --ingest--> gnosis (/v1/memories, infer=true)
question ------retrieve-> /v1/memory/context OR /v1/memories/search
retrieved text --answer-> answer model (never sees raw history)
hypothesis ----grade---> benchmark scorer + LLM judge

• One gnosis user_id per benchmark conversation; one session_id per session.
• Ingest posts one extraction-mode add per consecutive two-turn chunk.
• context and search are separate retrieval conditions.
• LOCOMO J score uses a local mem0-style CORRECT/WRONG judge for categories 1–4; category 5 (adversarial) uses the official substring rule. LongMemEval uses the official per-type judge prompts.

Measurement scopes

Do not mix scopes when comparing scores:

Setup

Prerequisites: Python 3.12+, uv, Docker Compose, an OpenAI-compatible endpoint.

cd stack
cp .env.example .env
# Edit stack/.env with your endpoint, models, and gnosis feature flags.
docker compose -p membench-local -f compose.yaml up --build -d
curl -fsS http://localhost:8080/ready

Use -p membench-local for state isolation. A new project name gives a fresh Neo4j store. Tear down with docker compose -p membench-local -f compose.yaml down -v.

Local-only warning: stack/compose.yaml publishes Neo4j (7474/7687) and gnosis (8080) on all host interfaces and ships placeholder credentials (membench-token, membench-neo4j, the operator token). Never expose this stack on a shared, public, or production network. Bind the host side of those ports to 127.0.0.1 with a Compose override and replace every default credential before any non-disposable run.

Running benchmarks

All commands run from membench/. Dataset downloads require network access.

cd membench
uv sync
uv run membench download --benchmark locomo
uv run membench download --benchmark longmemeval_s

membench reads its configuration from the process environment; it does not load stack/.env (that file configures the Compose services only). Export the answer/judge endpoint and models in the shell that runs uv run membench, otherwise the harness falls back to the public OpenAI default with an empty key:

export OPENAI_BASE_URL=https://inference-api.nvidia.com/v1   # or your endpoint
export OPENAI_API_KEY="$INFERENCE_API_KEY"
export MEMBENCH_ANSWER_MODEL=azure/openai/gpt-4o
export MEMBENCH_JUDGE_MODEL=azure/openai/gpt-4o

LOCOMO subset-3 gate (internal regression, ~71 J)

uv run membench run \
  --benchmark locomo \
  --subset 3 \
  --data-file ../data/locomo10.json \
  --conditions context \
  --out ../results/locomo/frozen-subset3

Required stack/.env for the frozen gate:

GNOSIS_FACT_EXTRACTION_ENABLED=true
GNOSIS_ENTITY_GRAPH_ENABLED=true
GNOSIS_ADAPTIVE_ROUTING_ENABLED=true
GNOSIS_CHAIN_OF_NOTE_ENABLED=true
GNOSIS_LLM=openai/gpt-5.5
GNOSIS_EMBEDDING=local-qwen3-embedding-0.6b
GNOSIS_EMBEDDING_DIMENSIONS=1024

LongMemEval_S frozen-100 (primary optimization target)

The frozen ID list is tracked, but the subset JSON is gitignored. Regenerate it from longmemeval_s_cleaned.json (written by the download above) before the first run:

uv run python scripts/make_lme_subset100.py

uv run membench run \
  --benchmark longmemeval_s \
  --data-file ../data/longmemeval_s_subset100.json \
  --conditions context \
  --inline-dates \
  --out ../results/longmemeval_s/frozen-100

Additional stack/.env for LME_S (append to LOCOMO flags above, replacing embedder):

GNOSIS_EMBEDDING=openai/azure/openai/text-embedding-3-large
GNOSIS_EMBEDDING_DIMENSIONS=3072
GNOSIS_SCOPED_DENSE_RETRIEVAL_ENABLED=true
GNOSIS_DENSE_SCOPE_POOL=10000

Resume an interrupted run

Resume requires the same Compose project/Neo4j volume and the same --out directory. Do not change either independently — that produces a hybrid result, not a resume.

uv run membench run \
  --benchmark locomo \
  --subset 3 \
  --data-file ../data/locomo10.json \
  --conditions context \
  --stages ingest,answer,grade \
  --out ../results/locomo/frozen-subset3

Regrade without re-retrieving

uv run membench run \
  --benchmark locomo \
  --subset 3 \
  --data-file ../data/locomo10.json \
  --conditions context \
  --stages grade \
  --out ../results/locomo/frozen-subset3

Experiment ledger

See RESULTS.md for the full run ledger with raw scores.

Costs and concurrency

Ingest adds are separate: ceil(turns / 2) POSTs per session. With fact extraction enabled, each add triggers one extraction LLM call (~30 s for LME_S-sized turns).

Default concurrency: MEMBENCH_CONCURRENCY=8. Use --grade-concurrency and --ingest-concurrency to override independently. Keep within your endpoint's rate limit.

Responses API shim

Some endpoints accept only streaming Responses API requests. Run the shim locally:

OPENAI_BASE_URL=https://provider.example/v1 \
OPENAI_API_KEY="$YOUR_KEY" \
uv run python -m membench.responses_shim --port 14100

export OPENAI_BASE_URL=http://localhost:14100/v1

Configuration reference

Aging protocol

membench aging is a synthetic maintenance protocol that generates update chains, contradictions, stable controls, and distractors, ingests them in timestamp order, and scores ranked retrieval deterministically — no LLM judge.

uv run membench aging \
  --seed 7 --users 8 --update-versions 3 \
  --config-label supersession_off \
  --out ../results/aging/off

See RESULTS.md for the full aging protocol spec and the membench aging --help output for all flags.

Outputs

Each run directory contains:

• ingest_state.json — completed conversation IDs (for resume)
• ingest_state_adds.jsonl — adds already written, skipped when a conversation is retried
• answers_<condition>.jsonl — retrieved context and answer records
• graded_<condition>.jsonl — scores and judge responses
• results.json — run settings and aggregate metrics
• report.md — per-category report

Run output and downloaded data are .gitignored. RESULTS.md is the durable, human-readable benchmark ledger.

Repository layout

stack/compose.yaml          Neo4j + gnosis Compose stack
data/                       downloaded data (ignored) + frozen LME ID list
membench/
  src/membench/datasets.py  download/load/normalize datasets
  src/membench/ingest.py    ordered turn-pair ingestion + resume state
  src/membench/answer.py    context/search retrieval + answer prompts
  src/membench/grade.py     LongMemEval and LOCOMO scoring
  src/membench/aging.py     synthetic supersession protocol
  src/membench/run.py       CLI entry point
  src/membench/responses_shim.py  Responses-to-chat compatibility shim
  tests/                    fixture-based unit tests
docs/
  frontier-2026.md          competitive landscape analysis (updated 2026-08-06)
  extraction-design.md      edu-v1/v2.0 fact extraction design (implemented)
  gaps-abstention-maintenance.md  abstention + knowledge-update gap analysis
  knowledge-update.md       KU roadmap — L-23 baseline → L-25b (70.8%) → L-31 (81.9%, complete)
  multihop-techniques.md    multi-hop retrieval techniques
RESULTS.md                  append-only benchmark run ledger

Development

cd membench
uv sync
uv run ruff check src tests
uv run ruff format --check src tests
uv run pytest -q

CI installs Python 3.13, runs these four checks, and does not start Compose, download data, or make paid API calls.

Research sources

See docs/frontier-2026.md for the full competitive analysis (updated July–August 2026), including verified scores, technique dissections, and the ranked next-technique roadmap for gnosis.

Key sources: LOCOMO · LongMemEval · Mnemis · EverMemOS · Chronos · Zep/Graphiti · EMem · Memory-R2 · "Is Grep All You Need?" · MemCon · Memanto · Graph-Native Bitemporal (L-31 blueprint) · Ground Truth First · Scrub Jay Episodic Memory · AgentMemBench · Beyond Memory Leaderboards