beagle

Setup · Evaluate · Evolve

Beagle is a framework that allows users to evaluate and evolve agent harnesses at scale.

As it's initial relase, Beagle ships the official implementation of DarwinX as the core evolution algorithm: with an evolvable agent harness and verifier-backed tasks,DarwinX searches for harness changes that improve capability while guarding against regressions.

More features will be added to Beagle in the future. The mission of Beagle is to allow users to bring their favorite agent harness and tasks; then Beagle evolves it into a stronger agent harness that generalizes robustly to new challenges.

🔈 News

• 2026-09-02 — We open-sourced Beagle, the framework supporting the DarwinX project.
• 2026-07-31 — We released the preprint DarwinX: Evolving Agent Harnesses Through Natural Selection.
  • Preprint link | Project page

⚙️ Setup

Installation

Install Beagle and configure the environment variables.

git clone https://github.com/SalesforceAIResearch/Beagle
# run the installation script; we use uv to manage the dependencies
cd Beagle &&  bash scripts/install.sh

Then fill in the environment in .env:

cp .env.example .env

Runtime prerequisite

Evaluation & evolution requires docker container runtime:

• Local Docker (default): install Docker Engine or Docker Desktop based on your platform; start the daemon, and verify with docker info.
• xrlenv cluster: for large-scale evaluation & evolution, you can use the xrlenv rollout infrastructure. Refer to xrlenv sphinx documentation for more details.

The generated example configs default to local Docker.

Prepare your agent harnesses experiment copies

This step is required for evolution; recommended for evaluation.

The design philosophy of Beagle is to treat the agent-harness as the data. To evolve the agent-harness, we need to onboard the agent-harness into your own repository so that we can evolve it.

The script below onboards the sample agent-harnesses into your own GitHub repository.

bash scripts/onboard_all_agents.sh

For users who want transpaency, we encourage you to read the script for more details and read the manual onboarding section below.

# mini-swe — SWE-agent/mini-swe-agent
python -m beagle.tools.onboard \
    --upstream https://github.com/SWE-agent/mini-swe-agent --ref a83fcae82d2a08f0ee0c688f9d137b3566c097f8 \
    --repo $YOUR_ORG/mini_swe_agent_v2.4.6 --private --branch-name baseline \
    --dir ../beagle-experiments/mini_swe_agent_v2.4.6 \
    --version v2.4.6 --profile-name mini_swe_agent_v2.4.6

# opencode — anomalyco/opencode  (--prune shrinks clones ~79 MB → ~12 MB)
python -m beagle.tools.onboard \
    --upstream https://github.com/anomalyco/opencode --ref a3647eb025c7615159d417dcc49fc39fdaeba65b \
    --repo $YOUR_ORG/opencode_v1.18.16 --private --branch-name baseline \
    --dir ../beagle-experiments/opencode_v1.18.16 \
    --version 1.18.16 --profile-name opencode_v1.18.16 \
    --prune opencode

Quick Start

Step 0: Populate the benchmark cache

Populate every cache-capable registered benchmark. The copied .env.example defaults to the project-local .cache/xrlenv_benchmark_cache; override it in .env or pass --dest when you need a shared location:

# Downloads the four currently registered cache-backed corpora; reruns are idempotent.
.venv/bin/python scripts/populate_benchmarks_cache.py

# Or populate only what you need (repeat --benchmark to select several).
.venv/bin/python scripts/populate_benchmarks_cache.py \
  --benchmark terminal_bench_2_1

The population command and beagle evaluate both load .env; they must point at the same root. Population retries transient HTTP failures three times and hides per-request HTTP logs by default; use --show-http-logs for network diagnostics. This is a one-time operation and is required for cache-backed benchmarks.

Step 1: Generate experiment configuration files

Experiment runs are configuration-file driven. The generation scripts read the agent repository, baseline commit, and local checkout from the manifests created during onboarding. Modify the generated configs for your use case; field guidance is in Evaluate and Evolve.

The scripts below generate 2-task smoke samples under tests/smoke/ for both evaluation and evolution:

source .venv/bin/activate
python scripts/generate_eval_configs.py       # → tests/smoke/<bench>/<harness>-<version>_smoke2.yaml
python scripts/generate_evolution_config.py   # → tests/smoke/<bench>/<harness>-<version>_darwinx_evolution_smoke2.yaml

There are additional examples provided in the examples/evaluation folder on advanced use cases, like evaluating a subset of tasks from a benchmark, excluding some tasks, running a mixture of benchmarks, and more.

Step 2: Run a smoke experiment

Evaluation

# Preview the run plan without spending.
beagle evaluate --config tests/smoke/terminal_bench_2_1/opencode-1.18.16_smoke2.yaml --dry-run
# Launch the two-task evaluation.
beagle evaluate --config tests/smoke/terminal_bench_2_1/opencode-1.18.16_smoke2.yaml

By default, evaluation artifacts go to ./tmp/<run.name>/ and preserve the original benchmark runner's artifact shape.

Evolution

# Preview one generated two-task DarwinX smoke without spending.
beagle evolve --config tests/smoke/terminal_bench_2_1/opencode-1.18.16_darwinx_evolution_smoke2.yaml --dry-run
# Launch it (uses evolver and evolvee model credits).
beagle evolve --config tests/smoke/terminal_bench_2_1/opencode-1.18.16_darwinx_evolution_smoke2.yaml

The generator writes the same evolution smoke for every onboarded reference harness; select the corresponding <harness>-<version> file to use another one. All run choices are visible in its YAML. See the evolution quick start for prerequisites and outputs, then Configure DarwinX before expanding the task set or enabling additional selection and verification mechanisms.

📊 Evaluate

Score an agent harness on a benchmark. Each task rolls through the benchmark's own runner (harbor, pier, raw container path, etc.) → graded → summarized results; beagle does not reimplement scoring process.

Supported benchmarks

More benchmarks are to be added soon.

beagle runs whatever the benchmark's corpus contains. It does not filter tasks for you: a task's oracle can give you zero reward if upstream dependencies are missing, changed or broken docs/benchmark-remarks.md lists them per benchmark with the measured evidence; exclude them per run with exclude_task_ids. See the documentation for more details.

Generate eval configs

Three set of config files are provided for different purposes.

• The examples/evaluation/*.yaml is provided for user reference.

• The experiments/configs/eval_baseline/… and ``tests/smoke//-_.yamlare generated by scripts since the remaining two ties to your own agents's mainfest generated after the onboarding in.beagle/agents/.json`.

# both read your onboarded manifests in .beagle/agents/<profile>.json
# every copy × benchmark on 2 seeded-sample tasks for smoke testing
python scripts/generate_eval_configs.py              # → tests/smoke/<bench>/<harness>-<version>_smoke2.yaml
# full-benchmark sweeps 
python experiments/scripts/generate_eval_configs.py  # → experiments/configs/eval_baseline/<harness>-<version>_<bench>_<model>_<effort>_<turns>.yaml


# advanced — generate for one agent/benchmark, or at different knobs (each combination gets its own file)
python experiments/scripts/generate_eval_configs.py --check                                   # list, write nothing
python experiments/scripts/generate_eval_configs.py --agents opencode-1.18.16 --benches swe-rebench
python experiments/scripts/generate_eval_configs.py --model gpt-5.6 --effort high --max-turns 150

By default, both generators include mini-swe and OpenCode, use gpt-5.6-sol with medium effort, run with the local Docker daemon, route directly to the model provider, and forward OPENAI_API_KEY. Every generated config includes an explicit provider block. Use --model, --effort, --runtime, --provider-type, --provider-name, and --forward-env to override those defaults. Gateway overrides additionally use --provider-api-base, --provider-api-key-env, and optionally --provider-auth-header.

run:
  timeout_multiplier: 1.0  # scales task-defined agent and verifier time limits

agent:
  harness:
    name: opencode          # which adapter runs it
    version: 1.18.16        # WHICH COPY — the `--version` you onboarded with
    source:                 # filled in for you, from that file
      repo: https://github.com/<you>/opencode_v1.18.16
      ref: a3647eb0…
  timeout: 1800             # fallback only when the benchmark defines no agent time limit

Whatever you generate or hand-write, beagle evaluate --config <file> --dry-run resolves it and prints the plan without spending anything.

You are encourged to generate more configs yourself or handwrite them for your own use cases.

Run an evaluation

# plan only, no spend
beagle evaluate --config tests/smoke/terminal_bench_2_1/opencode-1.18.16_smoke2.yaml --dry-run   
# real run
beagle evaluate --config tests/smoke/terminal_bench_2_1/opencode-1.18.16_smoke2.yaml

It will generate a folder structure like below

<dir-to-run-results>/
    <run-name>/
        benchmark-name/
            <per-benchmark-artifacts>
        run.json # the run summary, collect success/failure/error, token usage, etc.

Resume & retry

beagle evaluate reads each harness's native tree, so runs are resumable. Categories split on one signal: did the trial record an error?

Flags are independent — combine to union. Resolved tasks are never re-run.

Rollout Infrastructure

In vendor/xrlenv is a rollout infrastructure for managing the containerized runtime environment at scale. It also help to manage TB scale of the docker images and job scheduling at scale. It requires a cluster of CPU-nodes to be set up. The user can also use the local docker daemon for development purposes.

Details about the rollout infrastructure xrlenv setup can be found in xrlenv sphinx documentation. Use uv run sphinx-autobuild docs docs/_build/html --open-browser --port 0 to self host the documentation.

To use the xrlenv rollout infrastructure, you need to set the following environment variables after the xrlenv cluster is set up.

XRLENV_GRPC_HOST=
XRLENV_GRPC_PORT=
XRLENV_CONSUMER_TOKEN=

and change the runtime to xrlenv-cluster in the configuration file.

run:
  runtime: xrlenv-cluster        

🧬 Evolve

Optimize the harness itself: an evolver agent edits the evolvee's source, candidates are scored on the same benchmark surface as above, and what survives the gate lands on your experiment copy. Everything in Evaluate applies — evolution scores candidates the same way.

Config shape

run:      {dir, name, runtime, parallelism}
evolvee:                                    # θ — harness under evolution
  harness: {name, version, source}           # type + version + INLINE source
  model:  {name}
  provider: {type, name, extra_args?}       # name required except on inferred direct routes
  forward_env                               # independent host→container env forwarding
  effort / max_turns / timeout / extra_args # agent knobs (extra_args = CLI)
evolver:  {harness: {name, version}, model} # proposer (e.g. cursor-agent)
algorithm: {name: darwinx, hparams: {…}}    # optimizer + typed knobs
data:     [{benchmark, tasks}]              # benchmark + tasks

Generate an evolution config

Do not hand-copy an evolvee's repository and commit. Generate the two-task DarwinX smokes from the same versioned agent/benchmark matrix used by evaluation:

python scripts/generate_evolution_config.py
# writes tests/smoke/<bench>/<harness>-<version>_darwinx_evolution_smoke2.yaml

The generator joins each harness version to its onboarded manifest, fills evolvee.harness.source, and validates the result through the same typed build path used by beagle evolve. Use a generated file directly for smoke testing. For a real run, copy it under experiments/configs/, expand the task and DarwinX settings, and give the campaign a new run.name.

Run a campaign

beagle evolve runs the loop named by the algorithm block — DarwinX unless you swap it. One config is one campaign: every node it scores is recorded in a genealogy DB, so re-launching the same config continues the tree instead of starting over.

beagle evolve --config tests/smoke/terminal_bench_2_1/opencode-1.18.16_darwinx_evolution_smoke2.yaml --dry-run
beagle evolve --config tests/smoke/terminal_bench_2_1/opencode-1.18.16_darwinx_evolution_smoke2.yaml

The evolution quick start covers provider, benchmark-cache, Docker, GitHub, and evolver prerequisites. The Python API remains available as an advanced interface in quick_start_inline.py.

Each pipeline (one proposal attempt against one parent) walks these phases:

Nodes end as completed, no_change, rejected, or failed. no_change is a normal outcome — the proposer found nothing that survived the gate — not an error.

Configure DarwinX

DarwinX currently evolves on data[0]; tasks is the pool that supplies improvement targets and preservation canaries. Measure the unevolved harness first, include tasks with observed headroom, retain reliably passing tasks so regressions are visible, and keep final validation tasks outside the evolution pool.

Every DarwinX knob lives under algorithm.hparams and is validated by DarwinXConfig. The main groups correspond to the paper:

max_loop_iters is attempts inside one pipeline, not a generation count. One pipeline produces one node; total_steps is the campaign-wide node budget (1 when omitted). Change run.name to start a new campaign.

Read Configure DarwinX before moving beyond the generated smoke. It maps each group to the paper, gives smoke and first-campaign starting points, documents coupled settings such as defer_node_full_eval + fixed_eval_panel, and explains which advanced mechanisms need separately calibrated baselines or model credentials.

What you get. <run.dir>/<run.name>/ is the run home. Beneath it, DarwinX's campaign-state directory contains state.db (the genealogy), nodes/<node-id>/, and pipelines/<pipeline-id>/; sibling directories hold worktrees and _evals/ with raw benchmark-native artifacts. Kept branches land on your experiment copy, and the CLI prints the best node when the pipeline finishes.

More

Acknowledgments

During the development of DarwinX, we are grateful to the following open sourced projects.

• Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents
• agentic-harness-engineering
• BrowserCode

Cite Us

If you find Beagle and DarwinX useful in your research, please cite us using the following BibTeX entry:

@misc{zhang2026darwinxevolvingagentharnesses,
      title={DarwinX: Evolving Agent Harnesses Through Natural Selection}, 
      author={Yifan Zhang and Yutong Dai and Juntao Tan and Luyu Yang and Rishi Mullur and Thai Hoang and Zhiyuan Hu and James Zhu and Phil Mui and Silvio Savarese and Ran Xu and Zeyuan Chen},
      year={2026},
      eprint={2608.07545},
      archivePrefix={arXiv},
      primaryClass={cs.NE},
      url={https://arxiv.org/abs/2608.07545}, 
}