factlabel

A nutrition label for AI-written content. factlabel checks what an AI agent writes about data against the data itself, blocks what doesn't hold up, and shows readers exactly what was checked and why.

The problem

AI agents now write the updates people read: fundraising reports, financial summaries, dashboards, briefings. The numbers usually come from real records. The story around them often doesn't. Three failure modes show up again and again:

• Invented figures. "More than 2,000 new donors joined." The ledger shows 9 donors in all.
• False attribution. "Driven by an explosion of grassroots momentum." 90% of the money came from one corporate gift.
• Spin. "Individual giving grew 40%." It fell 30.6%.

Each sentence sounds reasonable, and a reader has no way to check it. Asking another large language model to review the text doesn't solve this: it is slow, costly per word, and can be talked into agreeing.

factlabel vs. "ask an LLM to check it"

What factlabel does

factlabel sits between the agent and the reader.

1. Audits the draft against your source records and documents: every figure, dashboard tile, citation, cause-and-effect claim, and trend.
2. Enforces publish gates: each draft comes out as pass, review (a person must look), or block.
3. Annotates the agent's own page with numbered findings, a "why this was flagged" explanation for each, and a Trust Facts badge that summarizes the whole page.
4. Fixes it (optional): code corrects the dashboard numbers from the source data, Claude rewrites the text from the findings, and factlabel audits the rewrite from scratch until it passes or a person needs to step in.

Trust Facts

Behind the badge in the corner is a summary of the whole page, not just a grade: a headline, scores for numbers, sourcing, attribution, framing, and completeness, what checked out, and what was checked. Every finding is numbered to match a badge on the page.

Why this was flagged

Each finding shows the steps that produced it, marked as decided by code or by Jev, with the exact question Jev was asked and its answer probabilities, the evidence used, and the threshold that applied.

After the fix

The same page after the revise loop: every number traces back to the ledger, the real source of the growth is named, and the decline is stated plainly.

How it works

The audit runs in three decoupled tiers:

[agent draft + source data]
            │
            ▼
┌──────────────────────────────────────────────┐
│ Tier 1: deterministic (code only)            │
│  - recompute every metric, break every table │
│    down by its categories                    │
│  - extract and mask figures, match quotes    │
│  - spot causal connectors ("driven by")      │
│  - anomaly profile: the material facts an    │
│    honest draft must disclose                │
└──────────────────────┬───────────────────────┘
                       ▼
┌──────────────────────────────────────────────┐
│ Tier 2: parallel Jev audit matrix            │
│  blinded call (the text only, no data):      │
│   which metric each figure and tile reports, │
│   which cause a sentence credits, which way  │
│   it says each metric moved, how loaded its  │
│   wording is                                 │
│  grounded call (text + verified facts):      │
│   is each claim supported, does each         │
│   citation hold, is each material fact       │
│   disclosed                                  │
└──────────────────────┬───────────────────────┘
                       ▼
┌──────────────────────────────────────────────┐
│ Tier 3: supervisory mediation (code)         │
│  - composite rules over atomic answers       │
│  - confident violation      -> block (red)   │
│  - borderline or conflicting -> a person     │
│    reviews (amber)                           │
│  - verified in code, matched with confidence │
│    -> silent pass (green check)              │
└──────────────────────────────────────────────┘

The rule that matters most: a model never decides whether a number is right. Jev only tells code which source metric a sentence is talking about. Code does the comparison, and reads qualifiers literally ("over 90%" fails when the truth is 89%).

Blinded questions. Anything about what the text says is asked without showing Jev the data, so a wrong number in the draft, or the true number in the ledger, can't bias the reading. Whether the text holds up is asked separately, with the data.

Attribution in two passes. Pass A reads the sentence with its numbers masked and asks which cause it credits and for which outcome. Pass B is computed from the records: the credited cause's real share of that outcome, judged against the change when the sentence describes a change. Every reading Jev considers is weighted by its probability, so "individual donors (62%) or something not in the data (36%)" counts as 98% sure the credit isn't backed.

Material omissions. Tier 1 profiles the data for declines, concentration in one source, and growth that came mostly from one place. Each fact is checked with two atomic questions (how the draft treats it, and whether it acknowledges it) that must agree. Growth concentration only counts when the draft reports that growth.

Why Jev

Jev from TypeSafe AI is a decision model, not a chat model. It doesn't write text. It answers typed questions about a shared state in one parallel pass:

• Choice: pick one option from a list, with a probability for each.
• Score: place something on a rubric you define.
• Noul: the probability that a yes/no statement is true.

That fits an audit well. Every answer is one of the options you defined, so results are typed and machine-checkable. Each answer comes with a confidence, so uncertain calls can go to a person instead of blocking on their own. And it is fast and cheap enough to run on every draft: a full audit of the example page asks about 50 questions across two parallel calls, takes under a second, and costs about $0.0006.

Jev can still be wrong, and TypeSafe's published accuracy figures are its own. That's why numbers are checked by code, uncertain answers go to a person, and every judgment is shown to the reader rather than hidden behind a score.

Install

factlabel needs Python 3.10 or later. It isn't on PyPI yet, so install it from GitHub:

pip install "factlabel[claude] @ git+https://github.com/generallymatthew/factlabel.git"

The [claude] extra is only needed for the revise loop. To work on factlabel itself:

git clone https://github.com/generallymatthew/factlabel.git
cd factlabel
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest

The tests don't call any API, so no keys are needed to run them.

API keys

You bring your own keys. They are read from environment variables and never stored in the project.

The easiest way is a .env file (already git-ignored). Copy the template and fill it in:

cp .env.example .env
set -a; source .env; set +a

Quick start

Try it on the included example: a fundraising agent's Q3 update that gets the story wrong.

# 1. Audit the draft
factlabel audit examples/fundraising/spun_draft.json --html out/label.html

# 2. Annotate the agent's own dashboard page
factlabel annotate examples/fundraising/spun_draft.json examples/fundraising/dashboard.html \
  -o out/annotated.html

# 3. Have Claude fix it, re-auditing each version
factlabel revise examples/fundraising/spun_draft.json --out out/revised.json

# 4. Put the fixed draft back into the page, with Trust Facts
factlabel rebuild out/revised.json examples/fundraising/dashboard.html -o out/rebuilt.html --annotate

Open the HTML files in a browser. examples/fundraising/honest_draft.json is the same update written honestly; it passes.

No API key yet? --no-jev runs only the checks code can do on its own (numbers, dashboard tiles, quotes), and says clearly which checks were skipped.

Usage

Commands

The exit codes make factlabel easy to drop into CI or an agent's publish step.

Python

from factlabel import JevJudge, PublishBlocked, enforce, load_case

draft, sources, metric_specs = load_case("examples/fundraising/spun_draft.json")
try:
    report = enforce(draft, sources, metric_specs=metric_specs, judge=JevJudge())
except PublishBlocked as e:
    print(e)          # draft blocked (F, 0/100): FALSE_ATTRIBUTION, FIGURE_MISMATCH, ...
    report = e.report

To fix drafts with your own agent instead of Claude, pass any function (draft, report, sources) -> draft to revise(). feedback_for(report, sources) gives you findings written for an agent to act on:

from factlabel import JevJudge, feedback_for, load_case, revise

def my_agent(draft, report, sources):
    return my_agent_rewrite(draft, feedback_for(report, sources))

result = revise(draft, sources, my_agent, metric_specs=metric_specs, judge=JevJudge())
result.status     # "passed", "review", or "escalated"

Describing your data: the case file

A case file holds the draft, the source data, and the metrics to check against.

{
  "draft": {
    "narrative": "We raised $1.25M in Q3 ...",
    "dashboard": [{"label": "Total raised (Q3)", "value": 1250000, "unit": "USD"}],
    "citations": [{"claim": "...", "quote": "exact words", "source": "board_memo"}]
  },
  "sources": {
    "documents": {"board_memo": "Q3 results were dominated by ..."},
    "tables": {
      "donations": [{"donor": "Acme Corp", "type": "corporate", "quarter": "Q3", "amount": 1125000}]
    }
  },
  "metrics": {
    "q3_total_raised": {
      "op": "sum", "table": "donations", "field": "amount", "where": {"quarter": "Q3"},
      "unit": "USD", "description": "Total dollars raised in Q3"
    }
  }
}

Metric ops: sum, count, count_distinct, mean, min, max, share (a percent, with an optional within scope), pct_change (period_field, from, to), and value for numbers you compute yourself. Write clear descriptions: Jev uses them to match sentences to metrics.

factlabel also breaks every table down by its categories on its own (by type, by quarter, and so on), so a figure that comes from those breakdowns counts as sourced, and false attribution can be caught without you defining a metric for every slice.

Marking up your page

annotate and rebuild work on any HTML page. These optional attributes make matching exact:

<div data-factlabel-tile="Donors">
  <span>Donors</span>
  <b data-factlabel-value>2,140</b>
</div>

<div data-factlabel-narrative>
  <p>The agent's text goes here.</p>
</div>

Without them, tiles are found by their label text and sentences by text search, including sentences that run across tags like <strong>. Readers can deep-link to the panel with #trust-facts or to a finding with #finding-5.

What it checks

Supervisory tiers: a Jev-based violation at 85% confidence or more acts on its own; between 50% and 85%, or when two atomic answers disagree, a person reviews it; below 50% it isn't flagged but stays in the report's audit trace. A check passes silently only when code made the comparison and any matching Jev did was at least 90% sure. All of Jev's questions and every threshold live in one file, src/factlabel/questions.py.

Benchmark

factlabel eval scores the audit on paired drafts: a misleading draft and an honest draft of the same update, over the same data. Each misleading draft lists the failure types it should trigger, so precision and recall are measured per failure type.

factlabel eval evals/ --runs 3            # 30 drafts across fundraising, SaaS usage, marketing ROI
factlabel eval evals-holdout/ --runs 3    # a retail domain written after the engine was built

Results against live Jev (jev-1.13), 3 runs each:

Read these numbers with care. The engine was tuned on the main suite, so it measures fit, not generalization. The retail domain was written after tuning and run once blind: every misleading draft was blocked, but it exposed two real flaws (attribution judged against the level instead of the change, and a part of a metric read as the whole), and 2 of 3 honest drafts passed. After fixing both, it scores 100% too, so it is no longer truly held out. New example cases are the most useful contribution to this project.

Costs and dependencies

• Jev is a paid, closed, hosted service. factlabel's code is open source, but its judgment step calls TypeSafe's API with your key. The example audit costs about $0.0003. If you depend on factlabel, know that you depend on TypeSafe too. Everything that decides whether a number is right is plain code in this repository.
• Claude is optional. Only the revise loop uses it, and a rewrite costs a few cents with Claude Opus 5. Any other model or agent can be plugged in instead.
• Nothing else is sent anywhere. The annotated pages and labels are self-contained HTML files with no external requests.

Limitations

• Jev's answers are probabilities. Treat review as "a person must look", not "probably fine". Tune the thresholds on your own drafts before relying on them.
• Figure extraction is regex-based and English-only.
• Trim long source documents before auditing; accuracy drops as irrelevant context grows.
• The questions were tuned on four synthetic domains. Real agent drafts will find new failure modes; add them to the benchmark before changing the questions.

Contributing

Contributions are welcome, especially new example cases and better questions. See CONTRIBUTING.md. You don't need any API keys to run the tests.

License

Apache-2.0. Jev and Claude are used through their own APIs under their providers' terms; no part of either is included here.