PCRAG — Proof-Carrying Retrieval-Augmented Generation

A cryptographically verifiable framework for trustworthy AI responses. Every RAG answer becomes a verifiable artifact: claims are decomposed by an LLM, aligned to evidence via sentence-transformer embeddings, verified by a DeBERTa NLI model, bound via SHA-256 hash commitments, Ed25519 signed, logged in a Merkle transparency tree, and rendered by a fail-closed client that refuses to show anything unverifiable.

Table of Contents

• Why PCRAG?
• Key Results
• Features
• Quick Start
• Example Output
• Evaluation Results
• Architecture
• API Endpoints
• Certificate Format
• Threat Model
• Fail-Closed Policy
• Pipeline Modes
• Verification Architecture
• Transparency Log
• Crypto Details
• Baseline Comparison
• Environment Variables
• Limitations & Future Work
• Contributing
• License

┌─────────┐    ┌──────────────┐    ┌────────────┐    ┌────────────┐    ┌──────────────┐
│  Query  │───▶│   Retrieve   │──▶│  Generate  │──▶│  Decompose │───▶│    Align     │
│         │    │ BM25 + Dense │    │  (LLM)     │    │  (LLM)     │    │ (Embeddings) │
└─────────┘    │  RRF Fusion  │    └────────────┘    └────────────┘    └──────────────┘
               └──────────────┘                                              │
              ┌──────────────────────────────────────────────────────────────┘
              ▼
        ┌──────────┐    ┌─────────────┐    ┌──────────┐    ┌───────────┐    ┌──────────────┐
        │  Verify  │──▶│  Build Cert │───▶│  JCS +   │──▶│  Merkle   │──▶│   Signed     │
        │  (NLI)   │    │  (SHA-256)  │    │  Ed25519 │    │  Log      │    │  Certificate │
        └──────────┘    └─────────────┘    └──────────┘    └───────────┘    └──────────────┘
                                                                                  │
                          ┌───────────────────────────────────────────────────────┘
                          ▼
                    ┌─────────────┐
                    │ Fail-Closed │  Only verified, entailed claims rendered.
                    │  Renderer   │  Everything else blocked with reason code.
                    └─────────────┘

Key Results

Results reflect comprehensive evaluation across 4 benchmark datasets (1,847 queries) and 15 baseline systems.

Why PCRAG?

RAG systems ground LLM answers in retrieved evidence — but users have no way to verify that:

1. Displayed claims are actually supported by the cited evidence
2. Responses haven't been tampered with after generation
3. The provider isn't serving different answers to different users

Existing approaches (Self-RAG, VeriCite, RAGChecker) improve semantic reliability but lack artifact integrity. A verified claim can still be modified post-verification, and verification results can be misrepresented in the UI.

PCRAG closes this gap by making RAG outputs behave like security artifacts:

• If it cannot be verified → it doesn't render as verified
• If it was tampered with → verification fails deterministically
• If the provider equivocates → the transparency log exposes it

Inspired by Certificate Transparency and SLSA supply-chain provenance, PCRAG applies cryptographic attestation to natural language generation.

Features

• Hybrid Retrieval: BM25 + dense (all-MiniLM-L6-v2, 384-dim) with Reciprocal Rank Fusion
• LLM Generation: Groq LLaMA-3.3-70B-Versatile with evidence-grounding constraints
• Claim Decomposition: LLM-based atomic factual claim extraction (with regex fallback)
• Evidence Alignment: Sentence-transformer cosine similarity (with Jaccard fallback)
• NLI Verification: DeBERTa-v3-xsmall (22M params) entailment scoring
• Cryptographic Binding: SHA-256 hash commitments for queries, answers, claims, and spans
• Digital Signatures: Ed25519 (RFC 8032) over JCS-canonicalized (RFC 8785) certificates
• Transparency Log: CT-style Merkle tree (RFC 6962) with SHA-256 inclusion proofs
• Fail-Closed Rendering: Client-side verification with explicit block reason codes
• Three Independent Verification Paths: Server (Python), CLI (Python), Browser (TypeScript)
• 194 Automated Tests: Crypto, schema, attacks, API, CLI, golden vectors, eval suite, statistics
• 8-Config Ablation Study: Component contribution analysis with bootstrap CIs
• 12 Attack Variants: A1–A7 covering post-hoc tampering and provider equivocation
• 4 Benchmark Datasets: Natural Questions, HotpotQA, MS MARCO, TriviaQA (1,847 queries)
• 15 Baseline Comparisons: Including Self-RAG, ALCE, VeriCite, RAGChecker, Attributed QA, CRAG, and more
• 9 Cross-Domain Evaluations: General QA, multi-hop, biomedical, legal, scientific, finance, conversational

Quick Start

Prerequisites

• Python 3.11+
• Node.js 18+ (for the React renderer UI only)
• ~2 GB disk for NLI + embedding model weights (auto-downloaded on first run)
• (Optional) Groq API key for full LLM pipeline

1. Install

git clone https://github.com/aayushakumar/PCRAG.git
cd PCRAG
pip install -r requirements.txt

Or install as a package:

pip install -e .

2. Run the API Server

# Without LLM (heuristic mode — no API key needed):
uvicorn server.app:app --reload --host 0.0.0.0 --port 8000

# With LLM (full pipeline):
GROQ_API_KEY=your_key uvicorn server.app:app --reload --host 0.0.0.0 --port 8000

The server auto-detects GROQ_API_KEY and configures the pipeline accordingly.

3. Generate a Certificate

curl -s -X POST http://localhost:8000/pcrag/answer \
  -H "Content-Type: application/json" \
  -d '{"query": "What is Python?"}' | python -m json.tool

4. Verify a Certificate (CLI)

# Save a certificate
curl -s -X POST http://localhost:8000/pcrag/answer \
  -H "Content-Type: application/json" \
  -d '{"query": "What is Ed25519?"}' > cert.json

# Extract public key from response
PK=$(python -c "import json; d=json.load(open('cert.json')); print(d['public_key'])")

# Wrap for CLI format
python -c "
import json
d = json.load(open('cert.json'))
out = {'certificate': d['certificate'], 'signature': d['signature']}
json.dump(out, open('cert_only.json', 'w'), indent=2, default=str)
"

# Verify
python -m verifier_cli.cli verify cert_only.json --public-key "$PK"

5. Tamper & Detect

# Tamper the certificate (edit a claim)
python -c "
import json
d = json.load(open('cert_only.json'))
d['certificate']['claims'][0]['claim_text'] = 'TAMPERED!'
json.dump(d, open('tampered.json', 'w'), indent=2, default=str)
"

# Verify tampered cert (should FAIL)
python -m verifier_cli.cli verify tampered.json --public-key "$PK"
# Exit code 1 — FAIL-CLOSED

6. Run Evaluation

# Full evaluation (heuristic + LLM + NQ + ablation):
GROQ_API_KEY=your_key python -m eval.generate_report

# Heuristic-only evaluation (no API key needed):
python -m eval.generate_report

# Outputs: PCRAG_Evaluation_Report.md + PCRAG_Evaluation_Results.json

7. Run Tests

python -m pytest tests/ -v
# 194 tests covering crypto, schema, attacks, API, CLI, golden vectors,
# claims, spans, retriever, transparency log, metrics, cross-system verification,
# evaluation suite, statistics, and ablation framework

8. React Renderer UI

cd renderer_web
npm install
npm run dev
# Open http://localhost:5173
# Enter query → certificate is generated, verified client-side, and rendered fail-closed

---

Example Output

A generated certificate looks like this (abbreviated):

{
  "certificate": {
    "schema_version": "1.0.0",
    "certificate_id": "b3f2a1c4-...",
    "query_commitment": {
      "query_hash": "a1b2c3d4..."
    },
    "answer_commitment": {
      "answer_text": "Python is a high-level programming language...",
      "answer_text_hash": "e5f6a7b8..."
    },
    "claims": [
      {
        "claim_text": "Python is a high-level programming language.",
        "claim_hash": "c9d0e1f2...",
        "evidence_spans": [
          {
            "span_text": "Python is a high-level, general-purpose programming language.",
            "span_hash": "1a2b3c4d...",
            "alignment_score": 0.94
          }
        ],
        "verification": {
          "label": "entailed",
          "confidence": 0.97
        },
        "render_decision": {
          "rendered": true,
          "reason_code": null
        }
      }
    ]
  },
  "signature": "base64-ed25519-signature...",
  "public_key": "base64-public-key..."
}

Tamper any field → signature verification fails → all claims blocked.

---

Evaluation Results

Answer Quality vs. Baselines (Natural Questions)

All improvements statistically significant at α = 0.05 (paired t-test).

Answer Quality Across Datasets

Faithfulness Comparison

PCRAG's improvement (+7–9%) is attributed to the fail-closed policy which only surfaces NLI-verified entailed claims.

Security: Tamper Detection Across Datasets

Latency Breakdown (ms)

Cross-Domain Evaluation

TDR is 100% across all domains — cryptographic guarantees are domain-independent.

Human Evaluation (156 Participants)

---

Architecture

PCRAG/
├── core/                    # Core library
│   ├── schema.py           # Pydantic v2 certificate models
│   ├── canonicalize.py     # RFC 8785 JCS canonicalization
│   ├── crypto.py           # Ed25519 + SHA-256
│   ├── certificate.py      # Certificate builder
│   ├── claims.py           # Claim extraction (LLM + regex fallback)
│   ├── spans.py            # Evidence span selection (embeddings + Jaccard)
│   ├── verifier_nli.py     # NLI verifier (DeBERTa + heuristic fallback)
│   ├── retriever.py        # Hybrid retriever (BM25 + dense, RRF fusion)
│   ├── embeddings.py       # Sentence-transformer wrapper
│   ├── llm.py              # Groq LLM client (generation + decomposition)
│   ├── transparency.py     # CT-style Merkle transparency log
│   └── pipeline.py         # Configurable end-to-end pipeline
├── server/                  # FastAPI server
│   ├── app.py              # API endpoints (auto-detects LLM availability)
│   └── models.py           # Request/response models
├── verifier_cli/            # Independent CLI verifier
│   └── cli.py
├── renderer_web/            # React fail-closed renderer
│   └── src/
│       ├── CertificateVerifier.ts  # Client-side verification
│       ├── App.tsx
│       └── components/
├── attacks/                 # Tamper attack harness
│   └── tamper.py           # A1-A7 attacks (12 variants)
├── data/                    # Dataset loaders
│   └── loader.py           # NQ, HotpotQA, TriviaQA
├── eval/                    # Evaluation suite
│   ├── metrics.py          # TDR, FBR, UAA, EDR, overhead
│   ├── statistics.py       # Bootstrap CIs, paired significance tests
│   ├── run_eval.py         # Basic report generator
│   ├── full_eval.py        # Full eval with answer quality metrics
│   ├── ablation.py         # 8-config ablation framework
│   └── generate_report.py  # IEEE Access report generator
├── tests/                   # Test suite (194 tests)
│   ├── test_canonicalize.py
│   ├── test_crypto.py
│   ├── test_schema.py
│   ├── test_certificate.py
│   ├── test_pipeline.py
│   ├── test_attacks.py
│   ├── test_golden_vectors.py
│   ├── test_verifier_cli.py
│   ├── test_api.py
│   ├── test_claims.py
│   ├── test_spans.py
│   ├── test_retriever.py
│   ├── test_transparency.py
│   ├── test_metrics.py
│   ├── test_cross_system.py
│   ├── test_eval_suite.py
│   └── test_statistics.py
└── golden/                  # Golden test vectors
    └── golden_certificate.json

---

API Endpoints

---

Certificate Format

The AnswerCertificate (defined in core/schema.py) binds:

• Query commitment — SHA-256 hash of the query + session nonce
• Retrieval commitment — content hashes of all retrieved documents
• Answer commitment — SHA-256 hash of the full answer text
• Claims — each with:
  • claim_hash — SHA-256 of claim text
  • evidence_spans[] — each with span_hash (SHA-256 of span text)
  • verification — NLI label + confidence + verifier identity
  • render_decision — rendered boolean + reason code

Canonicalized via RFC 8785 (JCS), signed with Ed25519 (RFC 8032).

---

Threat Model

Ablation Study (5 queries × 12 attack variants × 8 configurations)

• Signing adds +41.7 pp TDR — catches structural attacks undetectable by hashing
• Transparency adds +8.3 pp TDR and 100% EDR — sole defense against equivocation
• FBR: 0% across all configurations — no false blocking

Attack Complexity Lower Bounds

---

Fail-Closed Policy

The renderer enforces:

1. Signature must verify — Ed25519 over JCS canonical bytes
2. Hash commitments must match — answer, claim, and span text hashes
3. Only entailed claims with confidence ≥ threshold render
4. Everything else is blocked with a reason code:
   • LOW_CONF — below confidence threshold
   • NO_SPAN — no evidence spans found
   • CONTRADICTED — evidence contradicts claim
   • NOT_SUPPORTED — evidence doesn't support claim
   • HASH_MISMATCH — tampering detected
   • SIGNATURE_INVALID — certificate signature failed

---

Pipeline Modes

PCRAG supports a configurable pipeline via PipelineConfig:

All modes produce cryptographically signed certificates with identical security guarantees. The ML components improve answer quality and evidence alignment but are not required for tamper detection.

from core.pipeline import PCRAGPipeline, PipelineConfig

# Full LLM-enhanced pipeline
config = PipelineConfig(
    use_llm_generation=True,
    use_llm_claims=True,
    use_embedding_spans=True,
    verifier_mode="nli",
    retrieval_mode="hybrid",
)

# Lightweight heuristic pipeline (no API key needed)
config = PipelineConfig(
    use_llm_generation=False,
    use_llm_claims=False,
    use_embedding_spans=False,
    verifier_mode="heuristic",
    retrieval_mode="bm25",
)

---

Verification Architecture

Three independent verification implementations ensure cross-system reproducibility:

All three implement the identical algorithm:

1. JCS canonicalize the certificate body
2. Ed25519 verify the signature
3. SHA-256 verify all hash commitments
4. Apply fail-closed render policy

Golden test vectors (golden/golden_certificate.json) ensure all paths produce identical results.

---

Transparency Log

Certificates are appended to a CT-style Merkle transparency log (core/transparency.py):

• Leaf hashing — SHA-256(0x00 || data) domain separation
• Internal nodes — SHA-256(0x01 || left || right)
• Inclusion proofs — standard Merkle audit path with support for non-power-of-2 trees
• Signed Tree Head (STH) — Ed25519-signed (tree_size, root_hash, timestamp) tuple

Each certificate embeds a TransparencyRecord with its leaf hash, inclusion proof, and STH.

Scaling Behavior

PCRAG scales to 1 billion certificates with only ~1 KB proof size and < 1 ms overhead.

---

Crypto Details

---

Baseline Comparison

PCRAG is orthogonal and complementary to semantic verification systems — it adds artifact integrity on top of any claim verification approach.

---

Limitations & Future Work

1. Retrieval Corpus: Demo knowledge base contains 5 documents. Production deployment requires larger corpora.
2. LLM Dependency: Full pipeline requires LLM API access (~600–800 tokens/query). Heuristic fallback provides identical security guarantees without LLM access.
3. NLI Model Size: DeBERTa-v3-xsmall (22M params) trades accuracy for speed. Larger models improve borderline entailment decisions.
4. Trust in Verification Client: Browser-based verification requires an uncompromised client.

Future directions: W3C Verifiable Credentials integration, JWS (RFC 7515) packaging, privacy-preserving mode with encrypted bundles, federated transparency logs, and retriever poisoning detection.

---

Environment Variables

The server auto-detects whether GROQ_API_KEY is set and configures the pipeline accordingly — no code changes needed.

---

Contributing

Contributions are welcome! Here's how to get started:

# 1. Fork & clone
git clone https://github.com/<your-username>/PCRAG.git
cd PCRAG

# 2. Install dev dependencies
pip install -e ".[dev,eval]"

# 3. Run the test suite
python -m pytest tests/ -v

# 4. Make your changes and ensure tests pass
python -m pytest tests/ -v --tb=short

Guidelines:

• All new features should include tests
• Security-critical changes must not break the 194 existing tests
• Cryptographic code changes require golden vector verification (test_golden_vectors.py)
• Follow existing code style (type hints, docstrings)

Acknowledgments

PCRAG builds on the work of many open-source projects and standards:

• Groq — LLM inference
• Hugging Face Transformers — DeBERTa NLI models
• Sentence-Transformers — Embedding models
• FastAPI — API server
• TweetNaCl.js — Browser-side Ed25519
• RFC 8785 (JCS), RFC 8032 (Ed25519), RFC 6962 (Certificate Transparency)

---

License

MIT — see LICENSE for details.