agentic-rag

Retrieval-augmented generation where retrieval is a tool the model decides to call, not a pipeline stage that always fires. Hybrid BM25 + dense vectors, fused with Reciprocal Rank Fusion, exposed over MCP, with an eval harness that measures both whether retrieval works and whether the model refuses when the corpus cannot answer.

No API key. Auth comes from the already-authenticated Claude Code CLI, and embeddings run locally.

Why it is shaped this way

Retrieval is a decision, not a stage. The usual shape is: embed the question, staple the top 5 chunks to the prompt, send it. That retrieves exactly once whether or not the question needs it, and it cannot go back for more. Here the model gets a tool and a JSON schema and chooses whether to search, what to search for, and whether the first answer was enough. Measured: a two-part question produced three searches with escalating specificity; "what is the capital of Australia" produced one search, then a refusal.

Chunking is cut on headings, and the heading travels into the vector. A passage that says "roughness is per-shape, do not copy 0.50" is meaningless without the heading above it. So chunks break on markdown headings, the full heading path is prepended before embedding, and that same path is what a citation points at. Retrieval quality and citation quality come from one decision.

Hybrid, because each half is blind where the other sees. Dense retrieval smears rare literal tokens: ask this corpus for bpy or 0.001 and a 256-dim vector has nothing, while BM25 hits exactly. Ask "how do I make chrome look scratched" and BM25 has no lexical overlap while the vectors do.

RRF rather than a weighted blend. BM25 scores are unbounded and corpus-dependent, cosine sits in [-1,1]. Blending them means inventing a scale factor and re-tuning it whenever the corpus changes. RRF discards magnitudes and fuses ranks, so there is no weight to tune.

Measured results

Corpus: a single 190KB / 2,774-line technical reference, 202 chunks. 13 questions: 10 answerable from the corpus, 3 deliberately not.

RETRIEVAL over 10 answerable questions, k=5
  recall@5 : 8/10 = 0.80
  MRR      : 0.725

GROUNDING over 3 unanswerable questions
  refusal rate: 3/3 = 1.00

The two misses are reported because they are the useful part, and they share one cause. "What is the compositor for" returns nothing from the Compositing section, and "which file format should I export a 3D model in" returns nothing from the Pipeline and Formats section, even though both sections exist. In each case the terms appear more often in passing elsewhere than in their own definitional section, so BM25 rewards the discussion and a short generic query gives the dense side little to grip. Short generic queries against a corpus that mentions the term elsewhere are this retriever's weakest case. That is a property of the method, not a bug to paper over.

The first version of this eval scored 0.90, on a question set that included two questions written from the indexed document's own internal sections. Replacing them with two ordinary questions dropped the score to 0.80. That is the more honest number, and the drop is itself the finding: an eval written from the corpus flatters the retriever.

The refusals are worth reading twice: all three unanswerable questions were searched before being refused (2, 2 and 1 tool calls). The model checked rather than assuming, then said the corpus does not cover it.

What it does not do

• No reranking. A cross-encoder over the fused top-30 is the obvious next step and is not built.
• No query rewriting, which is the most likely fix for both misses above.
• Embeddings are 256-dim static vectors (potion-base-8M). They are fast, local and free; they are not a large embedding model.
• One corpus, one file type (markdown). No PDF or HTML loaders.
• Evaluated on 13 questions, which is enough to find a failure mode and not enough to publish a benchmark.

Run it

pip install model2vec numpy
python ingest.py path/to/corpus.md --db index.db
python retrieve.py "your query"            # retrieval only, no model
python ask.py "your question"              # model decides when to search
python evals/run_evals.py                  # recall@k and MRR, no model calls
python evals/run_evals.py --grounding      # adds the refusal check

Layout

ingest.py       heading-aware chunking, local embedding, SQLite store
retrieve.py     BM25 + dense + RRF
mcp_server.py   exposes search_docs as a tool over MCP (JSON-RPC on stdio)
ask.py          runs the model with the tool attached; logs what it called
evals/          gold questions, recall@k, MRR, refusal rate
tool_calls.jsonl  append-only log of every search the model chose to make

tool_calls.jsonl exists because the first version counted tool calls by grepping the subprocess's stderr and always got zero, including on answers that were plainly grounded and cited real chunk ids. The MCP server runs as a child of the CLI and its stderr never reaches the parent. A silence was being read as "the tool never fired". The server now writes its own durable log and the caller reads that instead.