ostk-recall

A local, single-binary search-and-recall service for your own files and chat history. It indexes notes, source code, and assistant session logs into a local corpus, runs hybrid semantic + keyword retrieval over them, and exposes the results to any MCP client (Claude Desktop, Cursor, Claude Code, the ostk kernel, and others). Data and indexes stay on your machine; nothing is sent off-box.

Built on model2vec-rs for static embeddings, LanceDB (Arrow + Tantivy BM25) for the vector and full-text store, SQLite (via rusqlite) for the ingest manifest, audit log, and concept/thread ledger, and fastembed-rs for the optional cross-encoder reranker.

What it does

• Ingests many source kinds into one corpus — markdown trees, source code, Claude Code, Gemini, and Codex CLI session logs, arbitrary globs, Claude.ai .zip exports, and ostk .ostk/ directories.
• Hybrid retrieval — model2vec dense vectors and Tantivy BM25, fused with Reciprocal Rank Fusion, with an optional cross-encoder rerank pass.
• A concept ledger — typed nodes and attributed, directed edges, durable in SQLite. Grow them from markdown frontmatter at scan time, or write them directly over MCP. Each edge records its origin (authored / observed / promoted) and derives a conductance from confidence and recency rather than storing a weight. Diffusion also walks the latent (vector-similarity) half of the graph; an off-diagonal bridge walked during consolidation is promoted into a weak reified edge that must then earn its conductance through use, or decay.
• A live attention runtime — a turn observer, an auto-weaver that links new chunks to thread anchors, and an idle curator that fades inactive threads. It also maintains an ambient "memory lens" (an MCP resource) aligned to whatever the current attention vector is focused on.
• Two agent-facing MCP tools (recall and remember) plus a resources surface, served over stdio or a shared local-socket daemon. Historical tool names remain hidden compatibility aliases for one transition cycle.

Status

Pre-alpha but functional; the maintainer runs it as a daily driver.

Not yet built:

• Per-file offset cursors for claude_code / gemini incremental scan (append-only JSONL currently falls back to a full-source rescan when poked by the watcher; content-addressed chunk ids keep that idempotent, just wasteful).
• MCP transports beyond stdio (HTTP / SSE).

Install

Release binaries

Pre-built binaries for Linux x86_64, macOS arm64, and Windows x86_64 ship with every tagged release: https://github.com/os-tack/ostk-recall/releases. Untar (unzip on Windows), put the binary on your PATH, done.

From source

git clone https://github.com/os-tack/ostk-recall
cd ostk-recall
make install     # → ~/.cargo/bin/ostk-recall

make install wraps cargo install --path crates/cli --locked --force. make help lists every target; make version shows the installed version and the workspace git sha.

Build-time native dependency (pulled in transitively, but the system tool must be present):

• macOS: brew install protobuf
• Debian / Ubuntu: apt-get install protobuf-compiler

Configure

Default path: ${XDG_CONFIG_HOME:-~/.config}/ostk-recall/config.toml.

Minimal config:

[corpus]
root = "~/.local/share/ostk-recall"

[embedder]
model = "minishlab/potion-retrieval-32M"   # repo id "org/name" — the prefix is required

[[sources]]
kind = "markdown"
project = "notes"
paths = ["~/notes"]

config.example.toml is a complete, commented reference for every option — globals ([corpus], [embedder], [reranker], [runtime], [lens], [weaver], [[record_rule]]), all source kinds, the relational entity_type / edges scanner, and the [watch] block.

Quickstart

ostk-recall init                # create the corpus root, download the model
ostk-recall scan                # ingest the configured sources
ostk-recall serve               # run the daemon: MCP over a local socket + ambient lens
ostk-recall serve --watch       # daemon + in-process file-watcher (keeps the corpus fresh)
ostk-recall serve --stdio       # direct stdio MCP transport (this process is the server)
ostk-recall connect             # bridge a stdio MCP client to a running daemon
ostk-recall watch               # standalone file-watcher → pokes a running daemon
ostk-recall weave               # weave bulk-scanned content into the thread graph
ostk-recall verify              # reconcile counts across store + manifest
ostk-recall optimize            # compact and fold indexes
ostk-recall lens show           # print the current ambient memory lens
ostk-recall lens status         # inspect lens freshness and resource behavior

ostk-recall --help documents the full CLI; ostk-recall <verb> --help covers each verb.

Examples

The examples/ directory has four self-contained, runnable setups — each with sample content, a config.toml, a run.sh, and a README walking the scan → query loop:

Each keeps its corpus inside the example directory (gitignored), so running them never touches your real corpus.

Source kinds

A markdown or code source can also set entity_type and edges to seed typed concept nodes and authored edges from each file's frontmatter at scan time — see examples/02-personal-kb and config.example.toml.

MCP tools

Two agent-facing tools, callable from any MCP client — either by spawning ostk-recall serve --stdio directly, or, when a shared daemon is running, through the ostk-recall connect bridge. serve also exposes an MCP resources surface serving the ambient ostk://memory-lens.

tools/list advertises exactly recall and remember. A canonical request selects one operation with action, for example:

{"name":"recall","arguments":{"action":"search","scope":{"project":"my-project"},"query":"shared storage backend","limit":5}}
{"name":"remember","arguments":{"action":"record","scope":{"project":"my-project"},"kind":"fact","text":"Shared deployments use PostgreSQL.","subject":"storage.shared","predicate":"backend","value":"postgres","idempotency_key":"storage-shared-v1"}}

The boundary is intentional: recall reads memory; remember deliberately changes it. Legacy recall_*, memory_*, attention_*, and thread_* names remain dispatchable during the compatibility window but are hidden from tools/list. Deprecated flat scope fields and nested params are likewise accepted but not advertised; canonical calls use action, scope, and action-specific fields.

recall — read and inspect

Recall may log access telemetry, but it never creates or revises semantic memory. Every search response includes a conflicts array and explicit conflict_coverage; unstructured code, prose, and conversation results are not silently presented as contradiction-checked. Healthy per-call coverage stays compact and points to the standing contract in recall(action="status"); retrieval mechanics live under diagnostics.claim_retrieval instead of being repeated as conflict boilerplate.

Canonical calls return the complete envelope once in MCP structuredContent. The required text content is only a short fallback summary, avoiding duplicate context in clients that expose both fields. surface(view="now") includes focus plus compact active assertions; it does not inject concept-activation or thread-resonance rankings. Those remain available through their explicit views.

remember — intentional memory changes

Facts and decisions may carry a structured subject, predicate, value, validity interval, and source-evidence snapshot. Conflicts are deterministic when structured claims share a scoped key but assert incompatible overlapping values. Semantic claim retrieval is enabled, but semantic/NLI conflict candidates over arbitrary prose remain intentionally deferred: similarity can find an assertion, but it does not prove that two assertions contradict. See the architecture decision. recall(action="status") therefore reports semantic potential-conflict nomination as unavailable rather than returning a misleading empty set.

Claims should be atomic enough to retrieve, correct, and supersede independently. A longer synthesis is still an ordinary claim with its honest epistemic kind—for example, a narrative grouping may be a note, while a runbook synthesis remains a procedure. split creates atomic children and durable child --part_of--> parent links; it does not invent a special synthesis kind. Ordered children also carry previous/next continuity, so sentence-safe projections do not lose the rest of a passage when one child is retrieved. With the default keep_parent:true, grouping does not supersede either side; suppressing the broad parent is an explicit lifecycle choice (keep_parent:false). Search returns matching children with compact relationship and continuity IDs, never the parent body hidden in result metadata. Use recall(action="get", kind="claim") to expand the hierarchy and traverse its complete ordered sequence.

The ostk-project decision adapter applies the same rule during ingest. A compound .ostk/decisions.jsonl record becomes a structural parent plus bounded outcome/rationale children; ordinary search returns the children and keeps the parent available through hierarchy traversal. The next ordinary incremental scan migrates the v0.9.3 monolithic projection only after those children are durable—no destructive --reingest is required. Exact same-key source revisions invalidate their predecessor before recording the replacement. Older source-derived claims with trustworthy source timestamps receive bounded ranking pressure, reported on the result, but age never changes their truth state or silently chooses a winner.

Long assistant transcript messages use the same retrieval principle without rewriting history. The original authored block remains byte-for-byte as an addressable archival parent. Deterministic section children follow markdown headings, paragraph topic shifts, and sentence boundaries; their dense input also receives a bounded preceding user question, while stored and returned text remains authored-only. Ordinary recall ranks the children and suppresses the duplicate archival parent, but direct relationship lookup can still recover the complete original block. At startup, the daemon additively backfills these projections from retained corpus rows, including transcripts whose source JSONL has rotated away. It never deletes or stales a parent and is idempotent on replay.

{"action":"record","scope":{"project":"my-project"},"kind":"note","text":"Storage synthesis: SQLite is the local default; PostgreSQL is required for shared deployments.","idempotency_key":"storage-synthesis-v1"}
{"action":"split","scope":{"project":"my-project"},"id":42,"children":[{"kind":"fact","text":"SQLite is the local storage default.","subject":"storage.local","predicate":"backend","value":"sqlite"},{"kind":"constraint","text":"Shared deployments require PostgreSQL.","subject":"storage.shared","predicate":"backend","value":"postgres"}],"keep_parent":true,"reason":"separate independently retrievable assertions","idempotency_key":"storage-synthesis-v1-split"}

Ordinary record calls do not accept a free-form string evidence: use structured support for an exact source coordinate, or remember.relate when recording durable derivation/grouping provenance. This prevents ungrounded text from being silently accepted and discarded.

Memory lens

ostk://memory-lens is a small, changing markdown portfolio aligned with recently attended conversation. It combines bounded corpus evidence with a bounded lane of active operator assertions and eligible source-derived claims. It is the collective ambient view: read-only, shared by daemon clients, and deliberately unaffected by any one caller's focus pin. Explicit focus still shapes that caller's recall and memory surface.

The server publishes the resource; the MCP client controls context placement. A conforming client discovers and reads it, subscribes, then re-reads after an update notification:

resources/list
resources/read        {"uri":"ostk://memory-lens"}
resources/subscribe   {"uri":"ostk://memory-lens"}
# server later sends notifications/resources/updated
resources/read        {"uri":"ostk://memory-lens"}
resources/unsubscribe {"uri":"ostk://memory-lens"}

The lens excludes harness/audit apparatus from ambient surfacing, collapses duplicate derived excerpts, rotates recently surfaced chunks, and stays within its configured token budget. Claim-ledger mutations independently invalidate the resource generation and notify subscribers even when attention has not drifted. Claim IDs and revisions are stable follow-up handles; rendering a claim is observed, but use of any individual entry is not inferred. Excluded content remains available to explicit recall. ostk-recall lens show reads the persisted snapshot without becoming an MCP client; lens status reports its freshness and lifecycle contract.

Salience health treats entropy collapse and measured active-vs-decided drift as failures. “Surfaced without observed use” remains visible as an advisory: the daemon audits successful Lens reads against the stable chunk and claim handles in that rendered snapshot, but does not infer that the client used or benefited from any individual entry. It therefore does not declare legitimate transcript-derived concepts unhealthy from missing use feedback.

Connecting clients

connect never starts the server. The normal shared setup is one long-lived ostk-recall serve --watch process plus any number of MCP clients configured to spawn ostk-recall connect. The bridge injects the caller project, heals across daemon restarts, replays the MCP handshake, and materializes paged results.

Alternatively, serve --stdio is a self-contained server owned by one client. A .serve.lock admits only one server per corpus, so use either the shared daemon (connect clients) or per-client serve --stdio, not both.

Claude Desktop

~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "ostk-recall": { "command": "ostk-recall", "args": ["serve", "--stdio"] }
  }
}

Cursor

~/.cursor/mcp.json (or project-local .cursor/mcp.json): same mcpServers entry as above.

Claude Code

.mcp.json at user or project level:

{
  "mcpServers": {
    "ostk-recall": { "type": "stdio", "command": "ostk-recall", "args": ["serve", "--stdio"] }
  }
}

To join a shared daemon instead, replace the args with ["connect"].

ostk

ostk v6.0.0+ ships fcp-recall in its driver defaults. Its kernel-side legacy mem_fault_recall adapter remains compatibility plumbing; agent-facing MCP clients see the canonical recall and remember tools. Keep ostk-recall on PATH. For earlier ostk, register via HUMANFILE:

DRIVER recall ostk-recall serve --stdio

Live updates

The daemon binds a scan-trigger socket at corpus.root/recall.sock (named pipe \\.\pipe\ostk-recall-recall on Windows). ostk-recall serve --watch runs the watcher in-process; alternatively run a separate ostk-recall watch. Either way, edits under a configured source path are debounced and re-ingested without a manual rescan:

[watch]
enabled = true
# debounce_ms defaults: 800 (Linux), 1200 (Windows), 1500 (macOS).
# projects = ["notes", "docs"]   # optional allowlist; defaults to all sources.
mode = "incremental"             # path-aware ingest; default "legacy" rescans per kick.

At startup the watcher registers filesystem watches first, then catches up the watched ostk-project audit journals before settling into live updates. Catch-up waits for the shared scan lock and retries transient failures with bounded backoff, closing the gap created by journal appends while the daemon was down. The watcher status file reports startup_catchup_attempts, startup_catchup_errors, startup_catchup_paths, and last_startup_catchup.

As of the v0.6 TurnEnd gate, only watched conversation transcripts (claude_code / gemini) drive the live attention runtime; other sources are ingested and searchable but do not wake the observer or weaver.

Consolidation

serve handles live cognition (a watched transcript turn wakes the observer and weaver). Offline consolidation is a separate, operator-scheduled step — run it from cron / launchd / systemd, not inside serve:

ostk-recall weave --since 24h                 # bind recent arrivals to thread anchors
ostk-recall weave --consolidate --since 1w    # deep re-weave, bridge, merge, promote, fade
ostk-recall optimize                          # compact fragments

weave --consolidate runs the full cycle over its --since window; the window is the consolidation horizon, so pick it per schedule tier. The CLI is the only contract — the scheduler is yours. Set HF_HUB_OFFLINE=1 for offline runs once the model is cached.

Architecture

                                          ┌─► MCP (stdio / socket) ──► clients
  sources ──► scanners ──► pipeline ──► store    (2 tools)             ▲
   (fs)      (8 kinds)    chunk+embed   │                              │
                          + merge_insert │                              │
                                         │  ┌──────────────┐           │
                                         ├──┤ corpus.lance ├───────────┤ recall
                                         │  │ + Tantivy    │           │
                                         │  └──────────────┘           │
                                         │  ┌──────────────┐           │
                                         ├──┤ manifest +   │           │
                                         │  │ audit_events │           │
                                         │  └──────────────┘           │
                                         │  ┌──────────────┐           │
                                         └──┤ threads.sqlite├──────────┤ remember
                                            │ claims + chain│           │
                                            └──────┬───────┘           │
              ┌────────────────────────────────────┴─────┐             │
              ▼                  ▼                  ▼                    │
       ┌────────────┐   ┌────────────┐   ┌──────────────┐              │
       │TurnObserver│   │ AutoWeaver │   │ IdleCurator  │──────────────┘
       │on TurnEnd  │   │on TurnEnd  │   │ timer-driven │
       │→ membrane  │   │→ evidence  │   │→ fade +      │
       │  chunks    │   │  links     │   │  tension     │
       └────────────┘   └────────────┘   └──────────────┘
                                                ▲
                          fs events ──► watch ──┘  (debounced → trigger.sock)

Read path. The daemon serves MCP to many clients over a local socket; stdio clients reach it through connect. serve --stdio is the alternative direct transport for a client that prefers to spawn its own server.

Write path. The daemon binds recall.sock; a watcher debounces filesystem events and delivers changed paths; a scan mutex serialises every trigger so per-path ingest never overlaps the single writer.

Attention loop. On a watched transcript TurnEnd, TurnObserver emits membrane chunks via Pipeline::ingest_synthetic and AutoWeaver writes derived evidence links — both gate on IngestEvent::is_turn_end(), so bulk ingest is skipped. ostk-recall weave runs the same weaver over bulk content on demand. IdleCurator fades inactive threads on a timer. The InMemoryAttention score tier is per-process and rebuilds from the threads.sqlite chain ledger on boot.

Stack: model2vec-rs (embedder) ▶ LanceDB + Tantivy (store) + SQLite (manifest, audit, ledger) ▶ pipeline (scan ▶ chunk ▶ embed ▶ merge_insert) ▶ query (dense

• BM25 with RRF, then optional cross-encoder rerank) ▶ attention runtime ▶ MCP server.

See docs/architecture.md and docs/spec/driver-protocol.md for full writeups.

Development

make check                                   # fmt-check + clippy + tests (CI parity)
make test                                    # tests only
make lint-strict                             # clippy with -D warnings
OSTK_RECALL_E2E=1 cargo test --workspace     # network-enabled tests (model download)

License

Dual-licensed under either:

• MIT License
• Apache License, Version 2.0 (with NOTICE)

at your option. Contributions are accepted under the same terms (Apache-2.0 §5).