agentcheck

Regression testing for AI agents. Define what your agent should do in plain YAML, run it against your actual agent (any CLI command or HTTP endpoint), and let an LLM judge score every response against your stated criteria — pass/fail, with a reason. Wire it into CI so a prompt change, a tool swap, or a model upgrade can't silently break behavior your users depend on.

This is deliberately narrow: it is not a production-observability platform (that's Langfuse / Braintrust / Arize territory, and they're well-funded — don't compete head-on). It's the thing almost nobody has built well yet: a fast, dev-friendly pre-deployment check that fits in a GitHub Actions step the same way pytest does.

Why this, why now

• Full observability/eval platforms have raised $50–80M rounds in the last year and are actively crowding the "log + monitor production agent traffic" space.
• Almost none of them are built as a lightweight, git-native regression suite a solo developer can add to CI in five minutes — that gap is the wedge.
• Distribution is self-serve/PLG (open-source CLI, developer audience) rather than enterprise sales — the thing solo AI founders are consistently worst at.

60-minute quickstart

pip install -e .
export ANTHROPIC_API_KEY=sk-...
agentcheck run examples/tests.yaml

Test file format

See examples/tests.yaml. Each test case specifies an input, a plain-English description of what a correct response looks like, and one of two ways to reach your agent:

• command: "..." — run it as a subprocess; input is piped to stdin, stdout is captured as the output. Works with any language.
• agent: "module.path:function_name" — import that module and call the function in-process with input as its only argument; its return value is the output. Useful for LangGraph/CrewAI/Claude-Agent-SDK-style agents that are Python callables rather than standalone CLI scripts — see examples/inprocess_agent.py.

Exactly one of the two is required per case. Either way, the output is scored pass/fail with a one-line reason by an LLM judge — no brittle string matching.

CI reporting

agentcheck run tests.yaml --json-out results.json writes a JSON report you can upload as a build artifact (see examples/.github/workflows/agentcheck.yml).

Add --post-pr-comment and, on a pull-request run with GITHUB_TOKEN set (the job needs permissions: pull-requests: write), agentcheck posts a markdown summary table as a PR comment, updating the same comment on repeat runs instead of piling up new ones. It's a silent no-op everywhere else (pushes, local runs), so it's safe to leave on in every CI invocation.

Comparing against a baseline ("did this change make it better or worse")

A flat pass count ("18/20 passed") doesn't tell you whether a change helped or hurt — you have to go read the table. --baseline fixes that by diffing the current run against a previous --json-out report, keyed by test name:

agentcheck run tests.yaml --json-out results.json --baseline baseline.json

Try it locally against the bundled example:

agentcheck run examples/tests.yaml --baseline examples/baseline.json

Every test lands in one bucket: unchanged (same pass/fail as the baseline), regressed (baseline passed, now fails — this is the one you care about), improved (baseline failed, now passes), new (not in the baseline), or removed (in the baseline but not in this run — probably a deleted test case, worth a glance). The console prints a one-line summary plus a table of regressions and improvements; a missing or unreadable baseline (there's no baseline yet on a repo's first run) prints a warning and falls back to the plain pass/fail report instead of failing the whole run.

--post-pr-comment picks this up automatically when --baseline is also set, so the PR comment leads with "vs baseline: 2 unchanged, 1 improved, 1 regressed" and calls out the regressions specifically, instead of just restating the full results table.

To actually wire this into CI you need somewhere for the baseline to come from — the usual pattern is: on every push to your default branch, run agentcheck with --json-out results/baseline.json and commit that file back to the repo; on every PR, read the base branch's copy of that file (git show origin/main:results/baseline.json) and pass it as --baseline. See examples/.github/workflows/agentcheck.yml for a full working version of that.

Development

pip install -e ".[dev]"
pytest

Roadmap (do this in order, don't skip ahead)

1. This week: get this CLI working end to end against your own toy agent. Dogfood it.
2. Week 2: open-source it. Post it where agent builders actually hang out (r/LocalLLaMA, the LangChain/LlamaIndex Discords, Hacker News "Show HN", relevant X threads). The goal isn't virality — it's finding 10-20 people who hit this exact pain point and will tell you what's missing.
3. Week 3-4: add the thing they ask for most. Likely candidates: a hosted dashboard for run history, Slack/GitHub PR-comment reporting, or support for a specific popular agent framework (LangGraph, CrewAI, the Claude Agent SDK) as a first-class integration.
4. Only once people are using the free CLI regularly: introduce a paid hosted tier (run history, team sharing, trend charts) — don't build monetization before you have free users who'd miss it if it disappeared.

What NOT to do

Don't try to become a general observability platform — that's the crowded, well-funded lane. Stay the "fast, git-native regression check" tool. Depth in one narrow job beats breadth against funded competitors.

License

MIT — see LICENSE.