🛡️ CertiRAG

Certified Retrieval-Augmented Generation with Fail-Closed Verification

CertiRAG decomposes LLM-generated answers into atomic claims, verifies each claim against retrieved evidence using NLI models, and produces a tamper-evident audit certificate. Unverified or contradicted claims are blocked before reaching the user — a fail-closed design that provably prevents unfaithful content from being displayed.

Table of Contents

• Why CertiRAG?
• Key Features
• Architecture Overview
• Verification Guarantee (Theorem 1)
• Project Structure
• Installation
  • Prerequisites
  • LITE Mode (CPU / Codespaces)
  • FULL Mode (GPU / Colab Pro)
  • Development Install
• Configuration
  • Environment Variables
  • YAML Configuration Files
  • Configuration Reference
• Quick Start Guide
  • Step 1: Ingest Documents
  • Step 2: Run Verification
  • Step 3: Inspect the Certificate
• Using CertiRAG as a Library
  • Basic Usage
  • Custom Configuration
  • Working with Schemas
  • Using Individual Components
• CLI Reference
• Data Schemas
  • EvidenceChunk & EvidenceSpan
  • ClaimIR & Claim
  • VerificationResult & EvidenceScore
  • AuditCertificate & RendererDecision
• Pipeline Deep Dive
  • Stage 1: Hybrid Retrieval
  • Stage 2: Claim Compilation
  • Stage 3: Claim Normalization
  • Stage 4: Verification
  • Stage 5: Minimal Sufficient Evidence (MSE)
  • Stage 6: Rendering & Policy
  • Stage 7: Certificate Generation
• Evaluation Framework
• Web UI
• Docker Deployment
• Testing
• Development Workflow
• Notebooks
• Contributing
• Citation
• License

Why CertiRAG?

Standard RAG systems retrieve documents and generate answers, but they cannot guarantee that the generated text is faithful to the source evidence. This leads to:

CertiRAG solves all of these by introducing a formal verification layer between generation and display:

Traditional RAG:  Query → Retrieve → Generate → Display (hope for the best)
CertiRAG:         Query → Retrieve → Generate → Decompose → Verify → Certify → Display (guaranteed)

Every claim is individually verified. Every decision is recorded. Every certificate is tamper-evident. If a claim can't be proven, it doesn't get shown.

Key Features

Architecture Overview

┌──────────────────────────────────────────────────────────────────────────┐
│                          CertiRAG Pipeline                               │
│                                                                          │
│  ┌──────────┐    ┌───────────┐    ┌───────────┐    ┌──────────────────┐  │
│  │ Hybrid   │───▶│  Claim    │───▶│  Claim    │───▶│   Verifier       │  │
│  │ Retriever│    │  Compiler │    │ Normalizer│    │ (MiniCheck/NLI)  │  │
│  └──────────┘    └───────────┘    └───────────┘    └────────┬─────────┘  │
│       │                                                      │           │
│       │  BM25 + Dense                                        │           │
│       │  + RRF Fusion                                        ▼           │
│       │  + Reranking            ┌─────────┐     ┌────────────────────┐   │
│       │                         │   MSE   │◀────│  VerificationResult│   │
│       │                         │Selector │     │  per claim         │   │
│       │                         └────┬────┘     └────────────────────┘   │
│       │                              │                                   │
│       │                              ▼                                   │
│       │                    ┌──────────────────┐                          │
│       │                    │ Renderer Policy  │                          │
│       │                    │  (Theorem 1)     │                          │
│       │                    └────────┬─────────┘                          │
│       │                             │                                    │
│       │                             ▼                                    │
│       │                    ┌──────────────────┐                          │
│       │                    │   Certificate    │                          │
│       │                    │    Builder       │                          │
│       │                    └────────┬─────────┘                          │
│       │                             │                                    │
└───────┼─────────────────────────────┼────────────────────────────────────┘
        │                             ▼
   ┌────┴────┐              ┌──────────────────┐
   │Documents│              │ AuditCertificate │
   │ Corpus  │              │ + Rendered Answer│
   └─────────┘              └──────────────────┘

For detailed Mermaid diagrams of every subsystem, see docs/architecture.md.

Verification Guarantee (Theorem 1)

CertiRAG enforces a formal verification predicate on every claim:

$$ \text{VERIFIED}(c_i) \iff \underbrace{\hat{p}{\text{entail}}(c_i) \ge \tau_e}{\text{sufficient support}} ;\wedge; \underbrace{\hat{p}{\text{contradict}}(c_i) < \tau_c}{\text{no contradiction}} ;\wedge; \underbrace{|E_{\text{MSE}}(c_i)| \ge 1}_{\text{evidence exists}} $$

The guarantee: No claim reaches the user as "verified" unless it passes all three gates. This is enforced deterministically in RendererPolicy.decide() — no stochastic component.

Project Structure

CertiRAG/
├── certirag/                        # Core library (pip-installable)
│   ├── __init__.py                  # Package root, version = "0.1.0"
│   ├── config.py                    # Central config (Pydantic Settings + YAML + env)
│   ├── utils.py                     # Seeds, hashing, logging, text processing
│   ├── pipeline.py                  # End-to-end orchestrator (CertiRAGPipeline)
│   ├── cli.py                       # CLI: ingest / run / eval / validate / export-schemas
│   │
│   ├── schemas/                     # Pydantic v2 data contracts
│   │   ├── evidence.py              # EvidenceChunk, EvidenceSpan, SourceInfo
│   │   ├── claim_ir.py              # ClaimIR, Claim, CandidateEvidence, ClaimType
│   │   ├── verification.py          # VerificationResult, EvidenceScore, VerificationLabel
│   │   └── certificate.py           # AuditCertificate, RendererDecision, PolicySnapshot
│   │
│   ├── ingest/                      # Document ingestion pipeline
│   │   ├── chunker.py               # Sentence-aware chunking (regex / spaCy)
│   │   ├── embedder.py              # Dense embeddings (e5-base-v2 / OpenAI)
│   │   └── indexer.py               # BM25Index + DenseIndex + ChunkStore
│   │
│   ├── retrieve/                    # Evidence retrieval
│   │   ├── hybrid.py                # HybridRetriever (BM25+Dense+RRF+rerank)
│   │   └── span_extractor.py        # SpanExtractor + get_span_text()
│   │
│   ├── claim_ir/                    # Claim compilation & normalization
│   │   ├── compiler.py              # LLM-based claim decomposition
│   │   ├── gemini_compiler.py       # Gemini-based claim compiler
│   │   ├── normalizer.py            # Atomicity (split / dehedge / dedup)
│   │   └── validator.py             # JSON Schema validation for all contracts
│   │
│   ├── verify/                      # Verification system (7 backends)
│   │   ├── verifier.py              # BaseVerifier ABC + BenchmarkPrediction
│   │   ├── cascade_verifier.py      # 2-stage gate → resolver pipeline (novel)
│   │   ├── ensemble_verifier.py     # Weighted NLI ensemble (accurate/sota/large)
│   │   ├── hf_nli_verifier.py       # DeBERTa NLI (2-class + 3-class)
│   │   ├── groq_verifier.py         # Groq LLM-as-judge (Llama 3.3 70B)
│   │   ├── gemini_verifier.py       # Gemini LLM-as-judge
│   │   ├── minicheck.py             # MiniCheck (RoBERTa-large / word-overlap)
│   │   ├── nli_verifier.py          # Legacy NLI-DeBERTa wrapper
│   │   ├── llm_judge.py             # GPT-4o judge (ablation baseline)
│   │   ├── calibrator.py            # Platt scaling / isotonic calibration
│   │   └── mse.py                   # Minimal Sufficient Evidence selector
│   │
│   └── render/                      # Output rendering
│       ├── policy.py                # RendererPolicy (Theorem 1 enforcement)
│       └── certificate.py           # CertificateBuilder (SHA-256 sealing)
│
├── eval/                            # Evaluation harness
│   ├── benchmark.py                 # Pluggable benchmark runner + CLI
│   ├── scoring.py                   # Metrics: Acc, F1, AUROC, ECE, bootstrap CIs
│   ├── optimizer.py                 # Threshold grid search + optimization
│   ├── run_full_eval.py             # Full eval suite (all models × all datasets)
│   ├── ablations.py                 # 10 ablation configurations (A1–A10)
│   ├── baselines.py                 # StandardRAG, PosthocChecker, MultiQueryRAG
│   ├── plots.py                     # Publication-quality matplotlib figures
│   ├── metrics.py                   # claim_f1, faithfulness, AUROC, ECE, yield
│   ├── runner.py                    # EvalRunner orchestrator
│   └── datasets/                    # Benchmark data loaders
│       ├── loaders.py               # VitaminC, ANLI R1-R3, FEVER-NLI loaders
│       ├── alce.py                  # ALCE (Gao et al., 2023)
│       ├── ragtruth.py              # RAGTruth (Wu et al., 2024)
│       └── aggrefact.py             # AggreFact (Tang et al., 2022)
│
├── tests/                           # 134 tests (pytest)
│   ├── conftest.py                  # Shared fixtures & factories
│   ├── unit/                        # Schema, atomicity, metrics, renderer, spans
│   ├── integration/                 # E2E verification, golden paths, corruption
│   ├── adversarial/                 # Negation, spoofing, entity substitution
│   └── benchmark/                   # NLI dataset evaluation tests
│
├── eval_results/                    # Saved benchmark JSON results (21 files)
│
├── ui/                              # Web interface
│   └── app.py                       # Streamlit demo application
│
├── notebooks/                       # Jupyter analysis notebooks
│   ├── 01_data_exploration.ipynb    # Chunking, spans, indexing analysis
│   ├── 02_retrieval_analysis.ipynb  # BM25 vs Dense vs Hybrid comparison
│   ├── 03_verifier_calibration.ipynb # Calibration, reliability, thresholds
│   └── 04_publication_plots.ipynb   # Paper-ready tables, LaTeX export
│
├── docs/                            # Documentation
│   └── architecture.md              # 8 Mermaid architecture diagrams
│
├── configs/                         # Configuration presets
│   ├── default.yaml                 # Balanced defaults
│   ├── strict.yaml                  # High-precision (τ_e=0.7, τ_c=0.5)
│   └── mixed.yaml                   # Mixed display mode
│
├── ARCHITECTURE.md                  # System architecture documentation
├── EVALUATION.md                    # Comprehensive evaluation report (7 models × 5 datasets)
├── CONTRIBUTING.md                  # Contribution guidelines
├── CHANGELOG.md                     # Release history
├── pyproject.toml                   # Build config, dependencies, tool settings
├── Makefile                         # Development command shortcuts
├── Dockerfile                       # Multi-stage: CPU (lite) + GPU (full)
├── LICENSE                          # MIT License
└── README.md                        # This file

Installation

Prerequisites

• Python 3.10+ (tested on 3.10, 3.11, 3.12)
• pip 24.x+ (if using pip 26+, ensure setuptools compatibility)
• Git for cloning

LITE Mode (CPU / Codespaces)

Best for development, testing, and environments without GPUs. Uses OpenAI API for embeddings and claim compilation.

# Clone
git clone https://github.com/aayushakumar/CertiRAG.git
cd CertiRAG

# Install with LITE dependencies
pip install -e ".[lite,dev]"

# Set up environment
echo 'CERTIRAG_MODE=lite' > .env
echo 'CERTIRAG_OPENAI_API_KEY=sk-your-key-here' >> .env

LITE mode uses:

• OpenAI API for embeddings and claim decomposition
• NumPy-based cosine similarity (no FAISS)
• Word-overlap heuristic for verification (no GPU model)
• No cross-encoder reranking

FULL Mode (GPU / Colab Pro)

For production evaluation and paper experiments. Runs all models locally on GPU.

# Install with FULL dependencies (requires CUDA)
pip install -e ".[full,dev]"

FULL mode uses:

• sentence-transformers for local embeddings (e5-base-v2)
• FAISS GPU index for dense retrieval
• MiniCheck-RoBERTa-large for verification (local inference)
• Cross-encoder reranking (BAAI/bge-reranker-base)

Development Install

# Install everything (LITE + FULL + dev tools)
pip install -e ".[lite,full,dev]"

# Or use Make
make install-dev

Google Colab Pro Setup

# In a Colab notebook cell:
!git clone https://github.com/aayushakumar/CertiRAG.git
%cd CertiRAG
!pip install -e ".[full,dev]"

import os
os.environ["CERTIRAG_MODE"] = "full"
os.environ["CERTIRAG_OPENAI_API_KEY"] = "sk-..."  # For claim compilation

from certirag.pipeline import CertiRAGPipeline
pipeline = CertiRAGPipeline.from_config()

Configuration

CertiRAG uses a layered configuration system: defaults → YAML file → environment variables → constructor args.

Environment Variables

All environment variables use the CERTIRAG_ prefix:

Create a .env file in the project root:

# .env
CERTIRAG_MODE=lite
CERTIRAG_OPENAI_API_KEY=sk-your-key-here
CERTIRAG_OPENAI_MODEL=gpt-4o-mini
CERTIRAG_LOG_LEVEL=INFO

YAML Configuration Files

Override any setting via YAML. Three presets are included:

configs/default.yaml — Balanced for development:

mode: lite
chunk_size: 8
top_k: 5
tau_entail: 0.5
tau_contradict: 0.7
max_claims: 20
split_conjunctions: true
remove_hedges: true
dedup_threshold: 0.8
calibration_method: temperature

configs/strict.yaml — High precision:

mode: lite
tau_entail: 0.7
tau_contradict: 0.5
default_mode: strict

configs/mixed.yaml — Show unverified claims with warnings:

mode: lite
tau_entail: 0.5
tau_contradict: 0.7
default_mode: mixed

Use a config file:

certirag --config configs/strict.yaml run "What is the capital of France?"

Configuration Reference

Quick Start Guide

Step 1: Ingest Documents

Prepare your documents as JSONL (one JSON object per line with id, text, and optional source):

{"id": "doc001", "text": "The Eiffel Tower is a 330-metre tall wrought-iron lattice tower in Paris, France. Named after engineer Gustave Eiffel, whose company built the tower from 1887 to 1889.", "source": "wikipedia"}
{"id": "doc002", "text": "The Great Wall of China spans approximately 21,196 kilometres across the historical northern borders of China.", "source": "encyclopedia"}

Ingest via CLI:

certirag ingest --docs documents.jsonl --index-dir data/index

Or via Python:

from certirag.pipeline import CertiRAGPipeline

pipeline = CertiRAGPipeline.from_config()
pipeline.ingest([
    {"id": "doc001", "text": "The Eiffel Tower is 330 metres tall...", "source": "wikipedia"},
    {"id": "doc002", "text": "The Great Wall spans 21,196 km...", "source": "encyclopedia"},
])

Step 2: Run Verification

certirag run "How tall is the Eiffel Tower?" --display strict

Example output:

CertiRAG Verification Results
━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Question: How tall is the Eiffel Tower?

Claims:
  ✅ VERIFIED   "The Eiffel Tower is 330 metres tall."
                Evidence: doc001, span s0 (score: 0.94)
  🟡 UNVERIFIED "It is the tallest structure in Europe."
                Reason: Insufficient entailment (0.42 < 0.85)

Stats: 1 verified, 1 unverified, 0 blocked
Certificate: SHA-256 seal = a3f8c2d1...

Step 3: Inspect the Certificate

# Validate a saved certificate
certirag validate --input outputs/certificate_20260209.json --schema certificate

# Programmatic validation
from certirag.render.certificate import CertificateBuilder
from certirag.config import get_config

config = get_config()
builder = CertificateBuilder(config)
result = builder.verify_certificate("outputs/certificate.json")
print(f"Valid: {result['valid']}")       # True if untampered
print(f"Errors: {result['errors']}")     # [] if everything is OK

Using CertiRAG as a Library

Basic Usage

from certirag.pipeline import CertiRAGPipeline

# Initialize (reads config from env / .env / YAML)
pipeline = CertiRAGPipeline.from_config()

# Ingest your documents
pipeline.ingest([
    {"id": "paris", "text": "Paris is the capital of France. It has a population of 2.1 million."},
    {"id": "london", "text": "London is the capital of the United Kingdom and England."},
])

# Run verification on a question
result = pipeline.run("What is the capital of France?")

# Access results
print(f"Total claims: {result.stats['total']}")
print(f"Verified:     {result.stats['verified']}")
print(f"Blocked:      {result.stats['blocked']}")

# Iterate over individual decisions
for decision in result.renderer_decisions:
    print(f"  [{decision.render_state.value}] Claim {decision.claim_id}: "
          f"entail={decision.entail_score:.2f}, "
          f"contradict={decision.contradict_score:.2f}")

# Access the sealed certificate
cert = result.certificate
print(f"Certificate seal: {cert.integrity_hash[:16]}...")
print(f"Tamper check: {'✅ valid' if cert.verify_integrity() else '❌ TAMPERED'}")

Custom Configuration

from certirag.config import CertiRAGConfig, ExecutionMode

# Construct programmatically
config = CertiRAGConfig(
    mode=ExecutionMode.FULL,
    openai_api_key="sk-...",
)
pipeline = CertiRAGPipeline(config=config)

# Or load from YAML
from certirag.config import get_config
config = get_config("configs/strict.yaml")
pipeline = CertiRAGPipeline(config=config)

Working with Schemas

All data flows through strictly-typed Pydantic v2 models:

from certirag.schemas.evidence import EvidenceChunk, EvidenceSpan, SourceInfo
from certirag.schemas.claim_ir import ClaimIR, Claim, CandidateEvidence, ClaimType
from certirag.schemas.verification import VerificationResult, EvidenceScore, VerificationLabel
from certirag.schemas.certificate import (
    AuditCertificate, RendererDecision, PolicySnapshot, RenderState
)

# Create an evidence chunk
chunk = EvidenceChunk(
    chunk_id="c1",
    doc_id="doc001",
    text="Paris is the capital of France.",
    source=SourceInfo(title="Wikipedia"),
    spans=[
        EvidenceSpan(span_id="s0", start=0, end=31,
                     sentence="Paris is the capital of France.")
    ]
)

# JSON round-trip
json_str = chunk.model_dump_json(indent=2)
restored = EvidenceChunk.model_validate_json(json_str)
assert restored.chunk_id == chunk.chunk_id  # ✅ Always works

# Export JSON schemas for external validation tools
from certirag.claim_ir.validator import export_all_schemas
export_all_schemas("schemas/")

Using Individual Components

Each pipeline stage can be used independently:

# --- Chunking ---
from certirag.ingest.chunker import DocumentChunker
chunker = DocumentChunker(chunk_size=256, overlap=2)
chunks = chunker.chunk_document("Your document text...", doc_id="d1")

# --- Indexing ---
from certirag.ingest.indexer import BM25Index, ChunkStore
store = ChunkStore()
store.add_many(chunks)
bm25 = BM25Index()
bm25.build(chunks)
results = bm25.query("search query", top_k=5)  # [(chunk_id, score), ...]

# --- Retrieval ---
from certirag.retrieve.hybrid import SimpleRetriever
retriever = SimpleRetriever(bm25_index=bm25, chunk_store=store)
evidence = retriever.retrieve("What is X?", top_k=5)

# --- Claim Normalization ---
from certirag.claim_ir.normalizer import ClaimNormalizer
normalizer = ClaimNormalizer(split_conjunctions=True, dedup_threshold=0.92)
normalized_ir = normalizer.normalize(claim_ir)

# --- Verification ---
from certirag.verify.minicheck import MiniCheckVerifier
verifier = MiniCheckVerifier(mode="full", device="cuda")
results = verifier.verify_claims(claim_ir, evidence_chunks)

# --- Rendering ---
from certirag.render.policy import RendererPolicy
policy = RendererPolicy(tau_entail=0.85, tau_contradict=0.70)
decisions = policy.render(verification_results)
display = policy.filter_for_display(claim_ir, decisions, mode="strict")

# --- Calibration ---
from certirag.verify.calibrator import VerifierCalibrator
calibrator = VerifierCalibrator(method="isotonic")
calibrator.fit(raw_scores_array, true_labels_array)
calibrated = calibrator.calibrate(raw_scores_array)
print(f"ECE: {calibrator.compute_ece(calibrated, true_labels_array):.4f}")

# --- MSE Selection ---
from certirag.verify.mse import MSESelector
mse = MSESelector(tau_entail=0.85, max_spans=5)
trimmed = mse.select_batch(verification_results)
stats = mse.compute_stats(trimmed)
print(f"Compression: {stats['compression_ratio']:.2f}")

CLI Reference

certirag [--config <path>] [--mode {lite,full}] [-v/--verbose] <command> [args]

Commands

certirag ingest --docs <path> [--index-dir data/index]

certirag run "Your question" [--display {strict,mixed,debug}] [--output path.json]

certirag eval --benchmark {alce,ragtruth,aggrefact} --data-dir <path> [--subset <name>] [--max-examples <n>]

certirag validate --input <path> --schema {evidence,claim_ir,verification,certificate}

certirag export-schemas [--output-dir schemas/]

Data Schemas

EvidenceChunk & EvidenceSpan

class EvidenceSpan(BaseModel):
    span_id: str                    # Unique span identifier ("s0", "s1", ...)
    start: int                      # Start character offset (>= 0)
    end: int                        # End character offset (> 0, > start)
    sentence: str                   # The actual sentence text

class EvidenceChunk(BaseModel):
    chunk_id: str                   # Unique chunk identifier
    doc_id: str                     # Parent document ID
    text: str                       # Full chunk text
    source: SourceInfo              # Source metadata (title, url_or_path, section)
    spans: list[EvidenceSpan] = []  # Sentence-level spans
    retrieval: RetrievalScores = RetrievalScores()  # bm25, dense, rerank, rrf

Invariant: chunk.text[span.start:span.end] must equal span.sentence (±2 chars). Enforced by Pydantic validator.

ClaimIR & Claim

class Claim(BaseModel):
    id: str                                            # Claim ID ("c0", "c1", ...)
    text: str                                          # Claim text (non-empty)
    type: ClaimType = ClaimType.FACTUAL                # Classification
    fields: dict[str, Any] = {}                        # Extra metadata
    evidence_candidates: list[CandidateEvidence] = []  # Linked evidence
    hedge: bool = False                                # Contains hedging language
    unsupported_by_design: bool = False                # Intentionally unsupported

class ClaimIR(BaseModel):
    question: str                     # Original user question
    claims: list[Claim] = []          # Decomposed atomic claims
    model_id: Optional[str] = None    # LLM used for decomposition  
    raw_answer: Optional[str] = None  # Original generated answer

Claim types: FACTUAL · TEMPORAL_FACT · NUMERIC_FACT · CAUSAL · COMPARATIVE · DEFINITION · PROCEDURAL · OTHER

VerificationResult & EvidenceScore

class EvidenceScore(BaseModel):
    chunk_id: str                    # Evidence chunk ID
    span_id: str                     # Evidence span ID
    label: VerificationLabel         # ENTAILED / CONTRADICTED / NOT_ENOUGH_INFO
    score: float                     # Confidence score [0.0, 1.0]
    raw_score: float | None = None   # Pre-calibration score

class VerificationResult(BaseModel):
    claim_id: str                          # Claim being verified
    label: VerificationLabel               # Aggregate label
    score: float                           # Aggregate score [0.0, 1.0]
    best_evidence: list[EvidenceScore] = []  # MSE-selected evidence
    all_scores: list[EvidenceScore] = []     # All (claim, span) scores

AuditCertificate & RendererDecision

class RendererDecision(BaseModel):
    claim_id: str                    # Claim ID
    render_state: RenderState        # VERIFIED / UNVERIFIED / BLOCKED
    policy: PolicySnapshot           # Thresholds used for this decision
    reason: str                      # Human-readable explanation  
    entail_score: float = 0.0        # Max entailment score
    contradict_score: float = 0.0    # Max contradiction score
    evidence_count: int = 0          # Number of supporting evidence spans

class AuditCertificate(BaseModel):
    query_id: str                    # Unique query identifier
    question: str                    # Original question
    timestamp: str                   # ISO timestamp (auto-generated)
    config_hash: str                 # SHA-256 of configuration
    policy: PolicySnapshot | None    # Policy thresholds used
    stats: dict                      # {total, verified, blocked, unverified}
    integrity_hash: str              # SHA-256 seal for tamper detection

Pipeline Deep Dive

Stage 1: Hybrid Retrieval

Query → BM25 Index (sparse, lexical matching) → Top-50 candidates
      → Dense Index (embedding cosine similarity) → Top-50 candidates  
      → Reciprocal Rank Fusion (k=60) → Merged ranking
      → [FULL mode] Cross-Encoder Reranking → Final Top-k

RRF formula: $\text{RRF}(d) = \sum_{r \in \text{rankers}} \frac{1}{k + \text{rank}_r(d)}$ where $k = 60$.

Stage 2: Claim Compilation

Uses LLM (GPT-4o-mini default) with few-shot prompting to decompose generated answers into atomic, independently verifiable claims. Each claim is typed (factual, temporal, numeric, etc.).

Stage 3: Claim Normalization

1. Conjunction splitting: "Paris is the capital and largest city" → 2 claims
2. Hedge detection: Flags "might", "probably", "approximately"
3. Deduplication: TF-IDF cosine similarity > 0.92 → merge

Stage 4: Verification

Scores every (claim, evidence_span) pair using MiniCheck or NLI model → 3-way label (ENTAILED / CONTRADICTED / NOT_ENOUGH_INFO) with confidence score. Cost-capped at 240 pairs per query.

Stage 5: Minimal Sufficient Evidence (MSE)

Greedy algorithm: sort evidence by entailment score descending, accumulate until threshold is met. Reduces evidence by ~66% while maintaining verification guarantees.

Stage 6: Rendering & Policy

Deterministic Theorem 1 evaluation:

Display modes control visibility: strict (verified only), mixed (all labeled), debug (full scores).

Stage 7: Certificate Generation

Assembles all pipeline outputs into a sealed AuditCertificate:

• SHA-256 config hash for reproducibility
• SHA-256 integrity hash over the full certificate body
• Verification via certificate.verify_integrity() returns True/False

Evaluation Framework

Benchmark Results (February 2026)

CertiRAG's verification pipeline was evaluated across 5 model configurations on three public NLI benchmarks. Full analysis in BENCHMARK_RESULTS.md.

Key improvements (v1 → v2):

• Macro F1: 42.3% → 75.5% (+33 pp) — 3-class model detects contradictions
• ECE: 0.43 → 0.11 — ensemble calibration 4× better
• Contradiction F1: 0% → 75% — unlocked by MNLI+FEVER+ANLI training data
• 5 pluggable verifier presets — from fast heuristic to multi-model ensemble

Running Benchmarks

# Quick smoke test (~3 min CPU)
python -m eval.benchmark --dataset vitaminc --max-examples 50

# Full suite with single model (~35 min CPU)
python -m eval.benchmark --model ensemble_accurate --dataset all --max-examples 200

# Ensemble with calibration
python -m eval.benchmark --model ensemble_sota --calibrate --cal-size 100

# Full evaluation (all models x all datasets with optimizations)
python eval/run_full_eval.py --max-examples 200

# LLM-as-Judge via Groq (free, requires API key)
export GROQ_API_KEY="gsk_..."
python -m eval.benchmark --model groq --dataset vitaminc --max-examples 50

# Via pytest
pytest -m benchmark tests/benchmark/

Plug In Your Own Model

from eval.benchmark import BenchmarkRunner

runner = BenchmarkRunner(verifier=my_verifier, model_name="MyModel")
scores = runner.run("vitaminc", max_examples=100)
print(scores.summary_table())

Supported Benchmarks

Metrics

Running Evaluations

certirag eval --benchmark alce --data-dir data/alce --subset asqa
certirag eval --benchmark ragtruth --data-dir data/ragtruth
certirag eval --benchmark aggrefact --data-dir data/aggrefact
# Or all at once:
make eval

Ablation Studies

10 built-in ablation configurations available via eval.ablations.get_ablation_configs():

Web UI

CertiRAG includes a modern web interface for interactive verification demos.

# Start the UI
make demo
# Or directly:
streamlit run ui/app.py --server.port 8501

Features:

• Sidebar: mode selection, threshold sliders, display mode toggles
• Real-time claim verification with color-coded results
• Evidence inspector with span scores and progress bars
• Certificate viewer with integrity validation
• Works in LITE mode without GPU or API keys (uses simplified heuristics)

Docker Deployment

# Build CPU image
docker build -t certirag:latest .

# Build GPU image
docker build --target gpu -t certirag:gpu .

# Run (CPU)
docker run -p 8501:8501 \
  -e CERTIRAG_OPENAI_API_KEY=sk-... \
  -v $(pwd)/data:/app/data \
  certirag:latest

# Run (GPU)
docker run --gpus all -p 8501:8501 \
  -v $(pwd)/data:/app/data \
  -v $(pwd)/models:/app/models \
  certirag:gpu

Health check: http://localhost:8501/_stcore/health

Testing

104 tests across three categories, all passing:

make test              # Unit tests only (fast, < 2s)
make test-all          # All tests
make test-coverage     # With HTML coverage report

Development Workflow

make install-dev       # Install all dependencies
make format            # Auto-format with ruff
make lint              # Lint check
make typecheck         # mypy static analysis
make test-all          # Full test suite
make test-coverage     # Coverage report

# Full CI check
make lint && make typecheck && make test-all

Notebooks

pip install jupyter
jupyter notebook notebooks/

Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/your-feature
3. Install dev dependencies: pip install -e ".[lite,full,dev]"
4. Make changes with tests: ensure make lint && make test-all passes
5. Submit a pull request

Code style: Pydantic v2 models, type hints everywhere, Google-style docstrings, tests for all new functionality.

Limitations

• Evaluation scale: Benchmarks use N≈200 examples per dataset; bootstrap 95% CIs are ±4–6pp.
• English only: All verification models and datasets are English-language.
• Static evidence: Evaluations use pre-paired claim-evidence; real-world retrieval noise is not captured.
• Verifier accuracy ceiling: No model achieves 100% — a "verified" badge is not a guarantee of truth.
• CPU-only benchmarks: All published results are from 2-core CPU; GPU may yield different latency profiles.

See EVALUATION.md for full limitations and ethical considerations.

Citation

@article{certirag2026,
  title={CertiRAG: Certified Retrieval-Augmented Generation 
         with Fail-Closed Claim Verification},
  author={Kumar, Aayush},
  year={2026},
  url={https://github.com/aayushakumar/CertiRAG}
}

License

MIT License — see LICENSE for details.

Built with 🛡️ for trustworthy AI