RAGCheck

Where does your RAG pipeline actually break?

A RAG (Retrieval-Augmented Generation) pipeline with a hand-built evaluation harness — not just a chatbot demo, but a system for measuring where and why retrieval and generation fail, including adversarial testing and a validated scoring methodology.

Most RAG tutorials stop at "it works on my example query." This project starts there and keeps going: gold-standard evaluation sets, retrieval metrics computed from scratch, hallucination/faithfulness scoring, poisoned- context adversarial tests, and a documented case where the evaluation harness itself was found to have a bug — diagnosed, fixed, and validated.

---

Stack

• Frontend: React (Vite) — dashboard for eval results and experiment comparisons
• Backend: Node.js + Express
• Database: PostgreSQL + pgvector — vector similarity search and relational data (gold Q&A sets, eval runs, experiment logs) in a single database
• ORM: Prisma
• Embeddings: bge-small via @xenova/transformers, run locally
• Chunking: hand-rolled recursive splitter, with separator-aware packing
• Vector search: native pgvector cosine distance (<=>), HNSW index
• Lexical search: hand-rolled BM25, fused with dense retrieval via Reciprocal Rank Fusion (RRF)
• Generation: Groq API (openai/gpt-oss-120b), OpenAI-compatible client, hand-written rate limiting and backoff
• Evaluation: hand-written retrieval metrics (hit@k, MRR, precision/recall) and hand-written faithfulness/hallucination scoring — no black-box eval library used as the primary metric source

---

What this pipeline does

1. Ingests source documents and chunks them (with tunable chunk sizes)
2. Embeds chunks and stores them in Postgres via pgvector
3. Retrieves relevant chunks for a query (dense, BM25, or hybrid fusion)
4. Generates an answer from retrieved context via an LLM
5. Scores that answer for faithfulness (is it grounded in context?) and correctness (does it match the known answer?)
6. Stress-tests the whole thing with unanswerable questions and deliberately poisoned context

---

Experiments

Chunking

The naive splitter fragmented text into sub-sentence pieces, producing embeddings that captured incomplete ideas and hurt retrieval quality. Separator-aware packing respects natural boundaries (sentences, paragraphs), consistently landing under the target size — a property of boundary-aware splitting, not a bug.

Retrieval: dense vs. BM25 vs. hybrid

hit@1 / gold-in-window@5, by difficulty:

Key finding: dense embeddings under-retrieve rare, structurally-templated entries (e.g. chemical elements) at rank 1 — medium-difficulty questions topped out at 41.7% hit@1 on dense alone. Adding BM25 via RRF fusion closed this gap entirely on the lexical side; a dense-biased fusion weighting (2:1) additionally recovered a compound-question regression introduced by naive 1:1 fusion, without sacrificing the medium-tier gain.

The more important finding: gold-in-window@5 = 1.000 across every retrieval arm, including plain dense, for both medium and compound questions. The correct chunk was always inside the top-5 context window actually passed to the generator — meaning the 41.7% hit@1 number, while a real retrieval-ranking weakness, never once produced an empty context window for the LLM. hit@1 mattered for measurement, not for what the model actually saw.

Generation: does better retrieval improve answers?

Controlled probe, same 22 questions, constant scorer:

Re-running the identical configuration twice produced a +3 swing purely from generator nondeterminism at temperature 0. The hybrid treatment's +1 gain over the control is inside that noise floor.

Decision: hybrid retrieval was not adopted as the generation source. It remains the best-performing retrieval configuration on record, but the measured generation-quality gain did not clear the noise bar established by the control run — a real result is not the same as a number that happens to be higher.

Faithfulness and adversarial testing

• 258-question baseline eval: mean faithfulness 0.934 (247 grounded-correct, 10 grounded-wrong, 1 off-context)
• 20-question unanswerable set: tests whether the model correctly refuses when no retrieved passage supports an answer, using fictional/provisional entities to rule out the model falling back on pretrained knowledge
• 14-question poisoned-context set: deliberately injects false claims into retrieved context at a fixed rank position to test whether the model repeats them uncritically — 9/9 questions resisted the injected claim after a scoring correction (see below)

The scorer bug

Mid-project, a verdict on one question ("radium") looked like a model hallucination. Investigation traced it to the evaluation harness itself: the answer-matching function required exact phrase order, but the gold answer was a legitimate "X and Y" construction where English allows either order. The fix was scoped narrowly — a relaxed pass that only triggers when the gold answer contains a conjunction, allows phrase-level (not word-level) reordering, and can only ever upgrade a wrong verdict to correct, never the reverse.

The fix was validated, not assumed: a reconstruction script rebuilt the exact context each answer was originally scored against from stored data, recomputed faithfulness fresh, and confirmed byte-identical agreement with the original stored scores across all 292 rows — proving the fix didn't silently change unrelated results.

Result: exactly 2 verdicts changed in the entire dataset, both the same question. The corrected finding reversed the original story — the model had been correctly resisting the poisoned claim all along; the scorer, not the model, had been wrong.

A separate, earlier claim ("a retrieval miss on a Potassium question") was later found to have no corresponding object in the dataset and was retracted from the project's findings once discovered.

---

What this project demonstrates

• Building retrieval metrics (hit@k, MRR, precision/recall) from first principles instead of importing a library as a black box
• Designing adversarial and unanswerable test sets specifically to rule out models falling back on pretrained knowledge instead of grounded retrieval
• Diagnosing a bug in the evaluation harness itself, fixing it narrowly, and mathematically validating the fix before trusting any downstream number
• Correctly distinguishing a real effect from noise (generator nondeterminism) before making an adoption decision, rather than declaring victory on the first favorable-looking number
• Retracting an earlier claim once the underlying data didn't support it

---

What's next

• Generation-side prompting improvements for comparative questions (the remaining known failure mode — e.g. "which is denser, X or Y" — traced to the generator's extraction/comparison step, not retrieval)
• A small regression-test suite locking in the adversarial-resistance behavior found during poison testing
• Per-difficulty re-chunking as an alternative lever on the medium-tail weakness, not yet tested against the fusion-based fix

---

Repo structure

ragcheck/
├── server/
│   ├── ingestion/   (chunking, embedding)
│   ├── retrieval/   (pgvector + BM25 + RRF fusion)
│   ├── eval/         (metrics: hit-rate.ts, mrr.ts, faithfulness.ts, rescore.ts)
│   └── db/           (Prisma schema, migrations)
├── client/            (React dashboard for results)
├── data/              (gold Q&A set, raw documents)
└── DECISIONS.md       (running log of every architectural decision and why)