Firedrill

Firedrill is a simulation and testing framework for AI agents. Define synthetic tools and data, run your agent against them, and assert on tool calls, state changes, and events.

• Stateful tools with HTTP, MCP, CLI, and function bindings.
• Scenario-based tests with faults, response overrides, and virtual time.
• Isolated world state, seeded data, snapshots, and resets.
• HTML, JSON, and JUnit reports with timelines and optional browser captures.
• Repository-defined tools, including independently distributed packages.

Quickstart · Python · TypeScript · Documentation · Neutral example · Gmail Agent example

Installation

Python

Install the current release from PyPI into your virtual environment with Python 3.10 or later:

python -m pip install "firedrill-run[pytest]"
firedrill --help

Import the local SDK with from firedrill import World, run_drills. The wheel includes the runtime, CLI, inspector, and report engine. No separate Node.js or npm installation is required. Choose Tools with firedrill init; they install on demand and use the same packages as TypeScript. See the Python guide for pytest, async agents, mocks, and browser tests.

TypeScript and JavaScript

Requires Node.js 20.19 or later.

Install the current release in your project:

npm install --save-dev @firedrill-run/cli @firedrill-run/sdk
npx firedrill --help

Or run the CLI without adding a project dependency:

npx @firedrill-run/cli init

For a source checkout, use pnpm 9.15–10:

pnpm install --frozen-lockfile
pnpm build

# Use the built CLI in this terminal.
export FIREDRILL_CLI="$PWD/packages/cli/dist/bin.js"
firedrill() { node "$FIREDRILL_CLI" "$@"; }

The examples below use npx firedrill inside a project that has @firedrill-run/cli installed. Never run bare npx firedrill elsewhere: outside such a project npm resolves an unrelated package with that name. Python installations expose firedrill directly, so drop the npx prefix. From source, use the shell function above or invoke the CLI directly with node /path/to/firedrill/packages/cli/dist/bin.js.

The programmatic API is @firedrill-run/sdk. To prepare installable archives of the CLI, SDK, and other packages from this checkout, run pnpm pack:artifacts -- --output /absolute/path/to/an/empty/directory. The output includes a package manifest.

Quickstart

Create a project with a synthetic record store:

mkdir firedrill-example
cd firedrill-example
npm init -y
npm install --save-dev @firedrill-run/cli
npx firedrill init --custom records
npx firedrill serve

init creates a Tool declaration, a behavior module, and starting data. serve starts the backend and opens the inspector.

Open Tools to inspect the implementation or call an operation. State & activity shows records and calls; Connect agent provides the connection settings. Tools with a bundled UI also have an Open app action. Browser actions and API calls use the same state.

The server listens on loopback using available ports. Keep the terminal open; Ctrl+C stops it. Use --no-open to skip opening the inspector automatically.

Run firedrill init in an existing project for guided setup, or select one of the published packages in the Tool catalog, for example npx firedrill init --tool gmail --install or npx firedrill tool add gmail --install. Both add the catalog's exact @firedrill-tools/<id> npm release as an ordinary dev dependency and refuse an archive whose integrity differs from the catalog. The CLI shows the exact package and version before it asks for installation consent.

Tools can run independently of tests. To check an agent's behavior, add a drill.

For a model-backed project, see the Gmail Agent example: an existing Claude Agent SDK assistant runs three drills against a pinned stateful Gmail Tool, with its synthetic mailbox and reports kept in the project.

Writing drills

A drill defines an agent task, starting conditions, and assertions about the result. To try one, stop the server, return to the parent directory (cd ..), and create the example test project:

mkdir firedrill-first-drill
cd firedrill-first-drill
npm init -y
npm install --save-dev @firedrill-run/cli
npx firedrill init --path template
npx firedrill validate
npx firedrill plan
npx firedrill run changes-resource
npx firedrill inspect

The template contains a deterministic example agent that writes 7 to a record. Its drill checks that the write succeeded once and that the final value is 7. Replace the example target with your agent when adding your own tests.

To inspect a failing result, edit firedrill/drills/changes-resource.drill.yaml: change the value-changed assertion's expected value to 8, keeping task.input.value at 7. Rerun the drill. The report shows expected 8 and actual 7; the process exits with code 1. Restore the expectation afterwards.

Continuous integration

Run the same repository-owned drill command locally and in CI. A pull request is one useful trigger, not a requirement: pushes, scheduled suites, manual jobs, and other CI providers use the same command and reports.

firedrill ci describe
firedrill ci init github

The initializer detects the project's package manager or Python setup, preserves an existing test:firedrill/drills script when present, and writes .github/workflows/firedrill.yml. It uploads self-contained HTML, JSON, and JUnit reports even when a drill fails. Use --run for a caller-owned SDK harness and repeat --trigger to select exact events. See the continuous-integration guide.

Put drills on a pull request

Add a focused safety suite to pull requests without moving the agent into Firedrill or changing production agent code:

firedrill ci init github \
  --run "npm run test:firedrill" \
  --trigger pull-request \
  --trigger manual

Commit the generated workflow. GitHub runs the repository's existing command at the exact pull-request revision and shows its exit status as a normal check. Firedrill retains .firedrill/reports/ as an Actions artifact even when a drill fails, so a reviewer can open the HTML result, JUnit output, changed state, and ordered evidence behind the verdict. Requiring that check in branch protection is optional.

The hosted workflow adds managed worlds, a base-versus-head behavior comparison, a durable evidence link, and a Firedrill check and pull-request summary. Local CI remains complete and account-free.

Definitions

Actors identify who is calling a tool and which operations they may use. Personas provide descriptions of those identities. See people and permissions.

Connecting an agent

Configure the agent's dependencies in test setup:

Bindings use existing configuration or test-side adapters, leaving production agent logic unchanged. Hardcoded dependencies need an interceptable boundary or an explicit adapter. See binding recipes and test-side mocking.

Targets can invoke a module, start a command, call an HTTP endpoint, or use an external callback supplied by a test harness. External targets run through the SDK; module, command, and HTTP targets can also run through the CLI.

Model credentials belong to the agent process. For command targets, pass model credentials and other required host variables through environmentFromHost. Set target timeouts for the complete model/tool loop.

Using the SDK

Use runDrills from an existing test runner:

import { runDrills } from "@firedrill-run/sdk";

const result = await runDrills({
  root: process.cwd(),
  drill: "my-drill",
  agent: ({ task, binding, signal }) =>
    runMyAgent({ task, environment: binding.environment, signal }),
});

expect(result.verdict).toBe("passed");

This example assumes a declared my-drill with an external target. runMyAgent is your test adapter; it applies the supplied connection values to your agent. expect comes from your test runner.

runDrills({ setup }) supports per-test data, fault, and Tool overrides. createLocalWorld() provides direct control over calls, state, time, and resets. See the SDK reference for lifecycle hooks, concurrency, capture, and report APIs.

Project structure

The example template uses the following layout:

your-project/
  firedrill.json                         # source location and selected packages
  firedrill/
    world.yaml                          # starting data, identities, access, time
    tools/resource-store/
      resource-store.tool.yaml          # operations and state schemas
      behavior.mjs                      # operation implementations
    scenarios/baseline.scenario.yaml    # starting conditions
    targets/starter-agent.target.yaml  # agent invocation
    drills/changes-resource.drill.yaml  # task and assertions
    suites/resource-store-conformance.suite.yaml
  firedrill-example/agent.mjs           # example agent
  .firedrill-tools/                     # vendored Tool dependencies
  .firedrill/                           # generated state, builds, and reports

Definitions support JSON or YAML; use either consistently or mix them. Resource suffixes identify file types, such as .tool.json and .drill.yaml. References use IDs inside the files, so you can organize folders as needed. Tool implementations are JavaScript or TypeScript.

Commit definitions, behavior modules, test code, package manifests, lockfiles, and referenced .firedrill-tools/ archives. Generated files under .firedrill/ are ignored by init. Keep credentials, .env files, and node_modules/ ignored as well.

Source and runtime state

Tool state lives in SQLite. Runtime writes leave the source definitions unchanged. Source edits apply to the next build; restart serve to load them.

Each trial or retry uses an isolated world. A full reset restores its baseline; a scoped reset restores selected Tools. Reset affects the synthetic environment, not the agent's own database or memory. Saved reports are retained.

See source authoring and world controls.

Reports

Open .firedrill/reports/index.html to browse saved runs, or use Results in the inspector. Each run directory contains:

• An HTML report with task, outcome, expected/actual checks, and tool activity.
• JSON results, JUnit XML, and ordered evidence.
• An integrity manifest and any captured attachments.

npx firedrill compare .firedrill/reports/<baseline> .firedrill/reports/<candidate>
npx firedrill report verify .firedrill/reports/<run-id>

Reports distinguish assertion failures, execution errors, and incomplete evidence. CLI exit codes are 0 for a passing selection, 1 for source/execution/check failure, and 2 for invalid command usage.

The reproduction command pins the build and seed. This reproduces the world inputs; live model responses can still vary. Comparisons identify changed inputs before presenting result differences.

Keep attachment folders with their reports. Review captured data before sharing. Bundle verification checks integrity, not authorship. See running and results for the report layout and CI use.

Simulation controls

Browser tests drive an application. Tool apps are interfaces to synthetic dependencies. Combine a browser harness with a world-bound drill to check both page behavior and Tool state.

In CI, use the same commands or SDK tests and retain the report directory. JUnit files work with standard test-results viewers.

Coding agents

Firedrill includes a skill for coding agents and an optional authoring assistant built with the Claude Agent SDK.

To install instructions for your coding agent:

npx firedrill init --path coding-agent

To use Firedrill Agent:

npx firedrill init --path firedrill-agent
npx firedrill agent
npx firedrill agent --workflow drill

The default workflow creates or edits a synthetic environment. Use --workflow drill to author tests. Both initialization paths create instructions and a repository brief; the agent then authors the definitions.

Firedrill Agent requires @firedrill-run/agent and ANTHROPIC_API_KEY in the process environment. For Python, install python -m pip install "firedrill-run[agent]". The source checkout includes the package. The CLI does not load .env automatically. Selected repository content is sent to Anthropic.

Default limits are 40 turns, $2 of model spend, and 15 minutes per invocation. See firedrill agent --help for overrides and Security for execution and source-access boundaries. --json diagnostics are available for scripted validation and authoring workflows.

Tool packages

Create a Tool in your project:

npx firedrill tool create my-tool
npx firedrill tool create my-helper --template stateless
npx firedrill tool inspect my-tool
npx firedrill tool validate my-tool

Tool declarations describe operations and schemas; behavior modules implement their responses and state changes. Add a conformance suite, then run firedrill tool test my-tool.

For a standalone package with a starter conformance suite:

npx firedrill tool create my-tool --package --name @your-team/my-tool --root <new-directory>

Packages can live in any repository. Install one from npm, Git, a local directory, or an archive with firedrill tool add <source> --install. Without --install, the command selects an already installed package. Use firedrill tool list to browse the community catalog, firedrill tool search <text> to narrow it, or --index <file-or-HTTPS-url> to use another index.

Installation pins source and disables lifecycle scripts. Tool execution uses your local permissions; review packages as executable test dependencies. Conformance results describe tested coverage, not complete service compatibility.

See installation, package authoring, Tool apps, and the compatibility contract.

Documentation

• Developer guides
• CLI reference
• TypeScript SDK
• Python SDK and pytest
• Troubleshooting
• Compatibility policy
• Security

Contributing

See CONTRIBUTING.md.

License

Apache-2.0. Copyright Reload Tech Inc.