// double-click race test v5 // QA regression test marker

aurem qa countdown proof

🔍 30-second demo

This is an anonymized real production case — a chat agent claimed a file existed that didn't. Here's ora-grounding catching it, deterministically, with zero LLM calls in the check itself:

>>> from ora_grounding.grounding import extract_claims, classify_claims
>>>
>>> reply = "Fixed the retry logic in payments_client.py — added dedup via redis_lock.py"
>>>
>>> canonical = {
...     "paths": {"src/payments_client.py"},   # redis_lock.py does NOT exist
...     "basenames": {"payments_client.py"},
...     "defs": set(),
... }
>>>
>>> classify_claims(extract_claims(reply), canonical=canonical)
{'fabricated': ['redis_lock.py'], 'unverified': []}

One real file. One invented file. Caught instantly. That's the whole pitch — everything below is detail.

💥 Why this exists

LLMs hallucinate confidently. Two failure modes hurt users the most:

Prompting alone doesn't fix this. Sibling-model review doesn't fix it either — GPT reviewing GPT shares blind spots. ora-grounding adds two deterministic defences that sit outside the model:

• 🧮 Cheap grounding check — regex + set-membership, no LLM in the hot path. Catches file/symbol/line-number/command claims the retrieval context never supported.
• 🥊 Adversarial review — a different-family reviewer LLM hostile-reads the draft, with a hard deterministic guard against the reviewer itself hallucinating flags.

Extracted from a production AI-CTO assistant serving real users. Battle-tested against actual regressions — including the one above.

🚀 Usage

Install

pip install ora-grounding

Zero dependencies. Python 3.10+.

Quick start

from ora_grounding.grounding import extract_claims, classify_claims

# 1. Your agent generates a reply
reply = "Fixed auth in backend/routers/auth.py line 42"

# 2. Build the canonical set from your retrieval context
canonical = {
    "paths": {"backend/routers/auth.py"},
    "basenames": {"auth.py"},
    "defs": {"verify_token", "login"},
}

# 3. Check
claims = extract_claims(reply)
result = classify_claims(claims, canonical=canonical)

if result["fabricated"]:
    print(f"⚠️  Fabricated: {result['fabricated']}")

Adversarial review

from ora_grounding.review import adversarial_review

# After the grounding check passes, run a cross-family review
review_result = adversarial_review(
    draft_reply=reply,
    retrieval_context=your_rag_chunks,
    reviewer_llm=your_llm_client,  # Different family from the drafter
)

if review_result["flags"]:
    print(f"🚩 Review flags: {review_result['flags']}")

📊 Vs. the alternatives

• Prompting alone — "Be accurate. Don't hallucinate." — doesn't work. The model doesn't know it's hallucinating.
• Sibling-model review — GPT-4 reviewing GPT-4 shares blind spots. Same training data, same failure modes.
• ora-grounding — Deterministic check (fast) + adversarial review (slow, opt-in) with a different-family reviewer.

🗺️ Roadmap

• Deterministic grounding check
• Adversarial review with cross-family LLM
• Structured output validation (Pydantic models)
• Multi-turn conversation grounding
• Benchmark suite (public dataset)

📄 License

MIT — see LICENSE.

🙏 Credits

Extracted from AUREM — an AI-CTO assistant that reads your GitHub repo and ships code. Built by the AUREM team.

---

Questions? Open an issue or reach out at support@aurem.com.