autoresearch-claude-code

A measured experiment loop for Claude Code and Codex. Define a goal, a fixed benchmark, and files to modify. The agent tries changes, measures results, commits supported improvements, and records failed ideas within a budget.

Inspired by pi-autoresearch. The experiment protocol is a shared skill with standard-library Python helpers; no MCP server is required.

Install

Requires Python 3.10+, Bash, Git, and your chosen agent client. Linux and macOS are supported; use WSL on Windows. The optional ML example has its own dependencies.

gh repo clone drivelineresearch/autoresearch-claude-code
cd autoresearch-claude-code
./install.sh --codex      # Codex skill in ~/.agents/skills/autoresearch
./install.sh --claude     # Claude skill, command, and four hooks
# Or: ./install.sh --all

No argument defaults to Claude for compatibility. Installations use symlinks, so keep this checkout in place. Existing foreign files/directories are preserved and conflicts are reported. Claude settings are validated before changes, updated atomically, and backed up when modified. Codex installation does not edit its configuration or register Claude hooks.

Claude also supports a session-local plugin load:

claude --plugin-dir /absolute/path/to/autoresearch-claude-code

Choose plugin loading or manual Claude installation; installing both can duplicate hooks. For removal, use ./uninstall.sh --codex, --claude, or --all. Removal only deletes symlinks owned by this checkout and corresponding exact hook entries; manually copied files and other installations remain untouched. See installation/review notes for migration limits.

Quick start

In Codex CLI/IDE, invoke the installed skill:

$autoresearch optimize test suite runtime with at most 20 runs
$autoresearch status
$autoresearch report
$autoresearch off

In the Codex app, select the skill or ask to use autoresearch by name. Codex discovers symlinked skills in ~/.agents/skills. For Claude Code, manual installation uses /autoresearch; plugin loading uses the namespaced /autoresearch:autoresearch, with the same goal or subcommands.

The agent establishes scope, creates an experiment branch, prepares a fixed benchmark, calibrates noise, and logs a baseline. It preserves existing work; use a dedicated worktree when another task is in progress. Artifacts must be ignored in the target project too; this plugin's ignores do not transfer.

For unattended Codex execution, first initialize and pause the session with the skill, review the baseline, then explicitly resume it and run:

# Run from the initialized experiment worktree after reviewing/removing its pause sentinel.
python3 ~/.agents/skills/autoresearch/scripts/codex_loop.py --dry-run
python3 ~/.agents/skills/autoresearch/scripts/codex_loop.py \
  --max-turns 10 --max-seconds 3600 --turn-timeout 300 \
  --protect path/to/scorer.py

Replace path/to/scorer.py with your real scorer file; repeat --protect for other locked inputs. See the Codex guide for initialization, flags, logs, pause/cancellation, and recovery.

What is enforced

Run caps count logged experiments (including baseline and failures), not tokens or dollars. Setup/calibration and seed confirmations have additional cost. Claude's Stop hook checks at turn boundaries; a benchmark needs its own timeout. A budget extension or scorer change must be deliberate; removing a pause sentinel alone never resets the budget.

Keep/discard uses strict improvement beyond the noise floor and passing correctness checks. Fix dataset splits, training-only preprocessing, and scoring before optimizing; changing the evaluation definition starts a new segment.

The exact workflow is in SKILL.md, with state commands/schema. The full review and roadmap records findings, changes, validation, and remaining limitations.

Example: fastball velocity prediction

The OpenBiomechanics example demonstrates the model interface, athlete-grouped evaluation, and METRIC name=number output.

cd examples
uv sync
mkdir -p third_party
gh repo clone drivelineresearch/openbiomechanics third_party/openbiomechanics -- --depth 1
./autoresearch.sh 42

Run in examples/; do not copy training files out of their uv project. Data paths are relative to the script. Optional backends: uv sync --extra torch, --extra boost, --extra tabpfn, --extra tabnet, or --extra all. Some optional backends need compatible GPU builds, credentials/model downloads, or extra setup.

The 19 registered models cover boosting, neural/tabular models, linear/Bayesian regression, SVR/KNN, and stacking. They load optional dependencies only when selected; GPU support depends on the backend and installed build. Start with a CPU backend to verify your data and evaluation contract.

Historical result caveat: the archived worklog reports R² 0.440 → 0.783, but it used feature selection informed by held-out data and changed aggregation/CV during the search. Those scores are not a validated like-for-like improvement or an independent new-player accuracy estimate. The original narrative is preserved with that qualification. Current code uses training-fold feature selection and keeps held-out labels out of fitting; rerun a fixed protocol to establish a new baseline. No replacement accuracy claim is supplied by this review.

Development

python3 -m unittest discover -s tests -v
shellcheck install.sh uninstall.sh hooks/*.sh skills/autoresearch/scripts/*.sh examples/autoresearch.sh

Core tests need only the standard library; example tests skip if their scientific packages are absent. To exercise them: uv sync --project examples, then uv run --project examples python -m unittest discover -s tests -p test_examples.py -v. CI runs core tests on Linux/macOS and the example suite with core dependencies. Fake-CLI tests validate supervisor behavior; authenticated Codex runs and optional GPU/model backends require separate integration checks. The real Codex keep/commit/log cycle is now verified after repairing this host's Ubuntu AppArmor profile and granting scoped Git metadata writes. Sandbox boundary checks also passed. See the sandbox repair and verification report.

Contributor guidance: AGENTS.md. License: MIT.