---

trustbut is a local command-line tool that produces a receipt for a code change, whether an AI agent or a person wrote it. It reruns your tests in a clean checkout, runs every new or changed test against the code from before the change to see whether it fails there (if it passes either way, it doesn't test the change), and flags skipped tests, removed assertions and silenced type checks. Then it gives you a PASS / WARN / FAIL verdict. Wired into your agent, it can also stop the agent from finishing while the receipt is FAIL, so it keeps fixing until the evidence holds up (details).

It works with any agent (Claude Code, Codex, Cursor, Gemini CLI, OpenCode, Copilot) and 11 languages (JavaScript/TypeScript, Python, Go, Rust, Java, Kotlin, C#, Ruby, PHP, Dart/Flutter and Swift) because it only needs git and your test runner. No LLM, no server, no account. Your code never leaves your machine.

$ npx trustbut verify

## 🧾 Receipt: WARN

`3f2a91c` → `8be04d1` · runner: vitest

**Tests (clean checkout):** 41 passed · 0 failed · 1 skipped
**Proof:** 1 new/changed test proves the change ✅
**New/changed tests:** 1 prove the change · 1 pass without it

### Findings
- ⚠ **Test skipped** · `test/auth.test.ts:88`: it.skip('handles clock skew', () => {
- ⚠ **cast to any** · `src/auth/session.ts:41`: const claims = decoded as any;

### Review these first
1. `src/auth/session.ts`: auth/permissions, has findings
2. `package.json`: dependencies

When the suite fails at head, the receipt also lists the failing tests. If package.json declares dependencies that aren't installed in your node_modules (a stale install), a deps-not-installed warning names them, because both runs link and use your installed dependencies. See the finding ids in the spec's finding catalog.

The output is Markdown, so the same receipt reads cleanly in your terminal and renders properly in a pull request or a CI job summary.

Supported languages

trustbut detects the right runner automatically, including in monorepo subfolders. Setup steps (such as starting a database emulator), extra arguments and environment variables go in a .trustbut.json. Detection rules and the exact commands are in Supported languages and test runners.

Why I'm building this

AI agents now write a large share of the code I ship. Writing it is no longer the slow part. Checking it is.

When an agent says "done, all tests pass", that's the agent reporting on its own work. Reading the diff doesn't show what actually ran, or whether the new tests would fail if the change were wrong. I wanted a check from outside the agent that answers those two questions with evidence, not with another model's opinion.

The data says I'm not alone:

• In the 2026 Stack Overflow survey, 48% of developers trust AI output when they can easily validate it, but only 16% trust it for most tasks. The gap between those numbers is the problem trustbut works on.
• In 2025, 66% of developers named "almost right, but not quite" as their top frustration with AI tools (InfoWorld on the 2025 survey).
• Armin Ronacher put it well in The Final Bottleneck: generating code is now cheaper than reviewing it, and the human stays accountable.

The labs have also reported their models gaming tests in agentic coding:

• Anthropic's Claude 3.7 Sonnet system card reports the model special-casing tests, and sometimes modifying them, to make them pass in agentic coding environments like Claude Code. The Claude 4 system card reports this much reduced, but still present.
• OpenAI's research on monitoring reasoning models caught a model exiting early or skipping unit tests to look green.
• METR found frontier models patching the grading and scoring code of their evaluations, and ImpossibleBench measures how often models exploit tests on tasks that can't be solved honestly.

That's why trustbut also flags skipped tests, removed assertions and silenced checks. But in my own measurement of 201 real pull requests, merged agent work rarely showed these behaviours (see What we've measured). The question that holds up is the plainer one: does anything prove this change?

AI code reviewers read the diff and comment, but they don't rerun anything. A vendor checking its own agent is a weak audit, and most teams use more than one agent. So trustbut is independent, deterministic and local: it doesn't ask an AI whether the code looks right. It runs things and reports what it found.

What we've measured

I'm measuring whether trustbut is useful, in the open, in trustbut-index. It runs trustbut on real merged pull requests from established open-source projects: PRs written by agents (Copilot, Codex, Claude Code, Cursor, Devin, Jules) and, as a control, PRs written by people in the same projects. These are pilot results, not the final study.

trustbut 0.7.0 on 201 pull requests:

¹ 18 of the 109 runs had a failing test with no result from before the change, so this count is a lower bound.

What this says so far:

• Agents didn't break suites or fake their tests in these PRs. Their new tests were at least as likely as people's to test the change. Merged PRs have already been through CI and review, though, so this says little about what an agent does before it says "done". That's where the agent hook works, and it isn't measured yet.
• Most findings are true but don't need action. In the 0.6.0 pilot, 71 findings were labeled: 5 were worth a reviewer acting on (7%), 57 were true but harmless, and 9 were wrong. All 5 useful ones said that nothing proved the change. 0.7.0 fixed the causes of the 9 wrong findings and made that one its headline, change-not-proven. The labels are drafts by Claude that no person has confirmed yet, and the labeler could see the code, so they weren't fully blind.
• trustbut ran on about half of the PRs. Of the 92 it couldn't run, 78 failed to install the project's dependencies or toolchain in the study's runner, 13 had no supported test runner, and 1 result is missing.

Not measured yet: whether change-not-proven findings are worth acting on, and whether the agent hook leads to better agent work.

On my own repos (DogBase), receipts found a fix commit with no test, 7 new tests that passed with or without their change, two new eslint-disable lines, and a dependency missing from the local install, which became the deps-not-installed warning.

Have a run of your own? Open an issue with what the receipt found, or missed.

Install

Requirements: Node.js 22 or newer, git, and a repository with tests in one of the supported languages and test runners, or any runner that can write JUnit XML. The toolchain of your project (Go, Cargo, Maven, .NET, Composer, …) must be installed, as it is when you run the tests yourself.

Run without installing (recommended):

npx trustbut verify

Install globally:

npm i -g trustbut
trustbut verify

From source:

git clone https://github.com/isAlmogK/trustbut.git
cd trustbut
npm ci
npm link

To have your agent verify its own work after every turn, see Use it with your agent.

Usage

Run it in your repository after an agent finishes:

trustbut verify            # the committed work on your branch
trustbut verify --working  # uncommitted work too (tracked edits plus untracked files)

By default trustbut compares your branch with the point where it split off main (or master), the same way a GitHub pull request shows its diff.

Common recipes

# Compare against a specific branch or commit
trustbut verify --base develop
trustbut verify --base HEAD~3

# Verify uncommitted work (the agent hasn't committed yet)
trustbut verify --working

# Faster: rerun only the new and changed test files at head
trustbut verify --quick

# The tests live in a subfolder, or the config lives outside the repo
trustbut verify --root services/api
trustbut verify --config ~/configs/myapp.trustbut.json

# Gate a script or CI job on warnings as well as failures
trustbut verify --strict

# Machine-readable output (an agent receipt v1)
trustbut verify --json > receipt.json

Options

Verdicts and exit codes

[!TIP] By default trustbut only looks at committed changes. If you have uncommitted work, the receipt says so. Commit first, or pass --working to include it.

Use it with your agent

trustbut can run itself at the end of every agent turn, so a FAIL receipt never reaches you as "done".

[!NOTE] The commands below assume trustbut is installed globally (npm i -g trustbut).

What happens

1. When the agent finishes a turn, trustbut quickly verifies its work: your uncommitted changes plus the commits on your branch. By default it reruns only the new and changed test files (the whole suite when none changed).
2. FAIL: the agent gets the receipt (failing tests with their messages, focused tests, and so on) and keeps fixing. This repeats up to hook.maxRetries times in a row (default 3), then the agent is allowed to stop and you see the FAIL.
3. WARN or PASS: you get a one-line note and the agent finishes.
4. trustbut never blocks on its own errors: a timeout, a crash or a missing tool lets the agent finish with a "couldn't verify" note.
5. A tree already verified isn't rerun. A FAIL is sent back once per tree: if the agent stops again without changing anything, or you start a new turn with nothing changed, it is allowed to finish and you see the FAIL with the receipt path.

After each completed check, the full Markdown receipt is at .git/trustbut/last-receipt.md (per worktree; inside .git, so it is never committed).

Things to know:

• The hooks run on macOS and Linux (POSIX) for now; Windows isn't supported yet.
• Use either the Claude Code plugin or trustbut init claude, not both: with both, every turn is checked twice.
• The hook runs your tests outside the agent's sandbox; see Is it safe to run?.

Claude Code

Install the plugin from this repository's marketplace:

/plugin marketplace add isAlmogK/trustbut
/plugin install trustbut@trustbut

Or write the hook into your project without the plugin: trustbut init claude (use one or the other, not both). The plugin uses the trustbut on your PATH when it is recent enough, and otherwise runs npx trustbut@<version>. Both run the hook in exec form (args), which needs a recent Claude Code (2.1 or later).

When trustbut sends a FAIL back, Claude Code labels it "Stop hook error occurred". This is expected: it is how Claude Code shows a blocking Stop hook, and the receipt is still delivered to the agent.

Codex

trustbut init codex

Codex asks you to approve new hooks: open /hooks and approve trustbut verify (needs Codex 0.129 or later). Project hooks load only in trusted projects.

Gemini CLI

trustbut init gemini

Project hooks run only in trusted folders, so trust the folder in Gemini CLI if the hook doesn't fire. Needs Gemini CLI 0.35 or later.

OpenCode

trustbut init opencode

This writes a plugin for the OpenCode V1 plugin API (opencode-ai 1.x; V2 support is a follow-up). OpenCode can't block "done", so on a FAIL the receipt is sent back to the agent as a follow-up prompt instead.

Init options

trustbut init <claude|codex|gemini|opencode> copies a small runner script next to the agent's config and, for Claude Code, Codex and Gemini CLI, merges one hook entry into it, leaving your other hooks alone. For OpenCode it writes a plugin file instead. It is safe to rerun: it updates its own entry in place.

Project files are written at the git root, and agents read project settings from the directory they were started in, so start the agent from the repo root. A settings file that contains comments is refused (exit 2) with the snippet to add by hand.

Skip it or tune it

Set TRUSTBUT_SKIP=1 in the environment to skip the check for a session (for example TRUSTBUT_SKIP=1 claude). To change how it behaves, add a hook object to .trustbut.json. It is read as committed at HEAD, so an uncommitted edit by the agent can't switch it off.

{ "hook": { "mode": "full", "maxRetries": 2 } }

The hook protocols, exit codes and environment variables are in the spec.

Supported languages and test runners

trustbut picks the runner by looking at the package folder (the repo root, or root / --root) as committed at head (with --working, your working tree). If several match, the most specific one wins; force one with --runner <name> or "runner" in the config.

Each runner's own report (JUnit XML, JSON events or TRX, depending on the tool) is turned into the same per-test results, so each language gets the same fail-before / pass-after proof.

Any other runner: the command runner

If your tests run through something not listed, point trustbut at a command that writes JUnit XML:

{
  "command": "./run-tests.sh {files}",
  "junit": "test/reports/junit.xml"
}

{files} is replaced with the quoted changed test files (and with nothing for a full run). Without junit, trustbut reads the command's stdout if it is JUnit XML. Setting command selects the command runner automatically.

Configuration (.trustbut.json)

Everything is optional; most projects need no config at all. Put .trustbut.json in the repo root. trustbut reads it as committed at head (so the version under review is the one used), then falls back to the working-tree file. --config <file> uses a different file instead, which can live outside the repo. Unknown keys and wrong types are errors (exit code 2) that list the allowed keys; a $schema key is ignored.

Examples

1. A Go backend with a Firebase emulator, in a subfolder

{
  "root": "backend",
  "args": ["-p", "1"],
  "setup": "firebase emulators:start --only firestore,auth",
  "setupReadyPattern": "All emulators ready",
  "setupWaitSec": 90,
  "env": { "FIRESTORE_EMULATOR_HOST": "127.0.0.1:8080" }
}

-p 1 runs packages one at a time so they don't fight over the emulator. The emulator is started before the head run and again before the base run, and always shut down.

2. An Expo / React Native app with Jest

{
  "runner": "jest",
  "args": ["--silent"]
}

Usually you don't need any config: jest-expo and react-native presets are detected. Dependencies are linked from your checkout, so there's no reinstall.

3. Vitest in a functions/ subfolder

{ "root": "functions" }

Or skip the file and run trustbut verify --root functions.

4. Any other tool, through the command runner

{
  "command": "./scripts/run-tests.sh {files}",
  "junit": "build/test-results/junit.xml",
  "timeoutSec": 1800
}

[!WARNING] command and setup are shell commands that run as part of verification, from the config at head, just like a changed test script in package.json would. Only verify code you'd be willing to run.

What it checks

1. A clean rerun

Your test suite runs in a fresh, temporary git worktree at head, not in your working directory and not by trusting the agent's own run. A red suite is a FAIL.

2. Fail-before / pass-after proof

This is the core idea. Every new or changed test also runs against the base commit:

3. Test integrity

These findings and the escapes below are things to scan, not verdicts: in the pilot, most of them were true but harmless.

4. Escapes

New lines that silence the type checker, linter or coverage:

Checks look at real code and comments only, so a string or regex that merely mentions it.only doesn't trigger a false alarm. Removed tests and assertions are counted per language too (@Test, func Test…, #[test], assert_eq!, Assert.*, …).

5. Review these first

A ranked list of the files that deserve your attention: auth and permissions, database migrations, CI pipelines, dependencies, secrets and env files, infrastructure, agent config (.claude/, AGENTS.md, …), files with findings, and very large changes.

The agent receipts format

trustbut verify --json prints an agent receipt: an open, versioned JSON format for evidence about a code change. Any tool can produce one and any tool can read one, and trustbut is just the reference implementation. The spec defines a small required core (specVersion, producer, subject, verdict, findings), optional standard sections (tests, proofs, risks), 21 core finding ids with fixed meanings, namespaced ids like acme/secret-leak for everything else, and x- keys for tool-specific data.

Check any receipt, from trustbut or anything else, against the spec:

$ trustbut verify --json | trustbut validate
valid

The schema is spec/agent-receipt-v1.schema.json, and examples are in spec/examples/. Building a tool that emits receipts? Open an issue and I'll link it here.

Use it in CI

The trustbut GitHub Action adds a receipt to your pull requests: a sticky comment (updated in place on each push), the receipt in the job summary, and a failed job on FAIL.

# .github/workflows/trustbut.yml
name: trustbut
on: pull_request
permissions:
  contents: read
  pull-requests: write   # for the sticky comment
jobs:
  receipt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # trustbut needs history to find the merge-base
      # add your language's setup and dependency install here (setup-node + npm ci, setup-go, ...)
      - uses: isAlmogK/trustbut@v0

The action installs Node 22 if the runner has none or one older than 22; that Node stays on the PATH for later steps in the job. Add your own language's setup-* step and dependency install before it, and a .trustbut.json in the repo if your tests need setup.

Inputs (all optional)

Outputs

Pull requests from forks get the receipt in the job summary only: GitHub gives them a read-only token, so no comment is posted. Always use the pull_request event, never pull_request_target: trustbut runs the PR's tests, and pull_request_target would run them with a write token and your secrets (the action prints a warning if you do).

How it works

flowchart LR
    A[base..head] --> B[git diff]
    B --> C[Classify files<br/>test · source · snapshot · config]
    C --> D[Static checks<br/>integrity · escapes]
    C --> E[Proof engine<br/>clean rerun at head<br/>new tests at base]
    C --> F[Risk ranking]
    D --> G[🧾 Receipt<br/>PASS · WARN · FAIL]
    E --> G
    F --> G

• Throwaway worktrees. Both runs happen in temporary git worktrees that are always removed afterwards. Your working directory is never touched. Dependencies (node_modules, .venv) are linked from your checkout instead of reinstalled; other toolchains (Go, Cargo, Maven, …) use their usual caches.
• Runner and language registries. Each runner is a small adapter that turns a test command into per-test results, and each language declares how to read its comments, strings and test markers. Adding either is one file.
• Deterministic. The same commits give the same receipt. There's no model, no sampling and no network call in the core.

FAQ

Does it send my code anywhere or use an LLM?

No. trustbut runs entirely on your machine with git and your own test runner. There is no AI model, no telemetry and no account.

Which agents does it work with?

All of them. trustbut verifies the result (your git history and your tests), so it doesn't matter whether Claude Code, Codex, Cursor, Gemini CLI, OpenCode, Copilot or a human wrote the change. Agents with a turn-end hook (Claude Code, Codex, Gemini CLI, OpenCode) can also run trustbut automatically: see Use it with your agent. Deeper integrations that read agent transcripts are on the roadmap.

Is it a replacement for code review?

No. It's evidence to have before a review: did the tests really run, does any new or changed test prove the change, and what got silenced. The "Review these first" list points at the files that deserve attention. It doesn't judge whether the change is correct or well designed.

Do agents actually cheat on tests?

Rarely, in merged pull requests. On 201 real PRs, no agent PR made a passing test fail, and 97% of agents' new or changed tests failed without the change, against 88% for people's (see What we've measured). Lab reports show models do game tests during agentic work, which is what the hook checks before the agent says "done"; how often that catches something isn't measured yet.

Isn't running the tests twice slow?

The base run only executes the new and changed test files, not the whole suite. Use --quick to also limit the head run to those files.

Is it safe to run?

trustbut runs your repository's tests, which means running your code, the same trust as running npm test or pytest. It does this in temporary worktrees and never modifies your checkout. In CI, use pull_request (not pull_request_target), as in the example above.

With an agent hook, keep in mind that the check runs outside the agent's sandbox: at the end of each turn your tests, including ones the agent just wrote, and the command/setup in .trustbut.json run with your permissions and network access. Where that matters (an untrusted repo, an agent you run sandboxed), turn the hook off with "hook": { "enabled": false } in .trustbut.json or TRUSTBUT_SKIP=1, and run trustbut verify yourself in a sandbox.

Why "trustbut"?

From "trust, but verify." Agents are genuinely useful, so keep trusting them, but check the evidence before you merge. The command reads as the idea: trustbut verify.

Roadmap

• v0.2 (done): every language, a runner registry and .trustbut.json, --working
• v0.3 (done, released as 0.3.x on npm): the GitHub Action with a sticky PR comment, hooks that stop an agent from saying "done" on a red receipt (Claude Code, Codex, Gemini CLI, OpenCode), and the npm release
• v0.4 (done): the receipt format as an open spec (agent receipts v1) with stable finding ids, and trustbut validate
• 0.5 to 0.7 (released): test-regressed confirmed by a rerun, less noise from the pilot, and change-not-proven as the headline finding
• Agent test-honesty index (study in progress): measuring trustbut on real agent and human pull requests in trustbut-index; first results are in What we've measured
• Later: transcripts and claim checks, mutation testing on changed lines, trustbut compare, a local web UI

See the full roadmap and the design spec.

Contributing

Contributions are welcome, especially real agent mistakes the receipt should have caught. Open an issue with a minimal repro; it becomes a test scenario before it gets a fix.

npm ci
npm test           # unit and end-to-end tests (real git repos, real test runs)
npm run typecheck
npm run build
npm run receipt   # trustbut's receipt for your branch; run before opening a PR

Every pull request gets a trustbut receipt in the CI job summary, and pull requests from this repository also get it as a sticky comment (fork PRs get the job summary only, since GitHub gives them a read-only token). Run npm run receipt locally first so there are no surprises.

trustbut is built with AI coding agents and verified by itself: every pull request gets a trustbut receipt in CI (a sticky comment for branches in this repo; the job summary for forks). The rules agents follow in this repo are in AGENTS.md.

Sponsor

trustbut is free and open source, and I build it in my own time. If it saves you or your team from a bad merge, sponsoring me on GitHub keeps it maintained and moving down the roadmap.

About me

I'm Almog Koren, a software engineer in Tallinn with more than 15 years of shipping production software. I founded and built DogBase, a bootstrapped SaaS for working-dog teams that won the main pitch competition at Latitude59 2026, and before that I was co-founder and CTO of Platfarm, which was acquired. I took both companies through Techstars. These days I mostly build AI agents and the systems around them, and trustbut came out of that work: I needed to know what my agents had actually done.

I'm open to work: full-time roles, fractional CTO work and AI agent projects. My CV and contact details are at almog.io.

License

Apache-2.0

---