SWE-bench Pro, matched pair: Ouroboros vs Codex CLI on one model

Status: Self-reported matched-pair study. Both harnesses used the same model, task set and evaluator. The strict result is a statistical tie.

Start here

Read the technical report, Browse all Ouroboros benchmark runs, open the no-key explorer, or read the public evidence page.

Two coding-agent harnesses, the same model (openai/gpt-5.6-luna), the same 731 public SWE-bench Pro instances, the same evaluator run on the same images. This repository holds the full trajectories of both arms, both verdict ledgers, and the contamination audit that decides which instances count.

Result

After symmetric leak filtering, which drops every instance where either arm reached the reference solution, the two harnesses are indistinguishable:

McNemar on the 655 paired instances: 30 / 38 discordant, p = 0.40. The difference is not distinguishable from noise. Codex is nominally ahead by 8 instances.

The filter is central to this dataset; What the filter changes shows the unfiltered view and explains why it should not be read as a capability gap.

Why a matched pair

A harness score is model × scaffold × isolation × grading. Published agent numbers usually vary all four at once, so they cannot answer "is this scaffold better." Here everything but the scaffold is pinned:

• Model: openai/gpt-5.6-luna on both arms, every slot (solve, review, planning).
• Task set: all 731 public SWE-bench Pro instances, no subsetting, no retry-until-pass.
• Grading: one wave-grading pass evaluates both arms' predictions against the same downloaded images, via the official evaluator (SWE-bench_Pro-os @ ca10a60a) called from grade_pro.py pinned at seed ouroboros-v6.54.2-bench1. Neither task, evaluator nor metric was modified.
• Contamination accounting: one detector suite applied to both arms, with its per-arm recall stated below rather than assumed equal.

Method

Ouroboros arm

Ouroboros is a general-purpose autonomous agent runtime. Core self-modification is off for this run; what is measured is the shipped harness: task loop, tool set, memory, and its review pipeline.

• Seed: tag v6.81.0, commit 3f9d504b, working tree clean (attested per instance in seed_attestation.json).
• Each instance runs in its own container with a private Ouroboros server; the agent gets the problem statement and a checkout, and produces patch.diff.
• Round cap 200, cost cap $50/instance. Neither bound was the limiting factor: across the 36 failures read in full, runs used 20–162 rounds and $0.2–$8.
• The run was executed in six phases (initial + resume/rerun/pipfix after infrastructure failures). Winner selection per instance is deterministic: the candidate with the longest non-empty patch.diff, with phase order breaking ties exactly as grade_loop.sh did. manifest.csv records which phase produced each winner (resume 581, base 104, rerun 35, pipfix 11) so the selection is auditable, and 86 losing attempt directories are excluded.

Codex arm

codex exec with gpt-5.6-luna, one container per instance, driven through a command broker (broker_commands.jsonl records every command it ran inside the container). 727 of 731 instances produced a trace; 5 produced no prediction and are scored resolved=false. 28 instances have earlier attempts, published alongside under attempt_prev_*.

pass@1

One attempt per instance per arm. Re-runs exist only where infrastructure killed a run (container death, a provider key hitting its monthly limit mid-instance); they never pick the better of two completed answers.

Contamination: what leaks and how we measured it

SWE-bench Pro instances carry the fix commit's SHA inside the instance id, and the tasks' upstream repositories are public. That makes two leak channels available to any agent, and both were used:

1. Web. Codex's built-in web_search executes provider-side, so container-level network isolation never touches it. 116 instances used it; on 68 the returned content included the answer or reference patch.
2. Git archaeology. From inside the task checkout, git commands can reach the upstream fix commit. 8 codex instances passed a gold SHA to a git command; 5 Ouroboros instances did the same. Ouroboros reached the network on 96 instances via its shell.

The filter

filters_excluded.json is the machine-readable filter, the union of those three lists: 68 (codex, answer content) + 8 (codex, gold SHA) + 5 (Ouroboros, gold SHA), with 5 overlaps, so the union contains 76 excluded instances and leaves n = 655. It is symmetric by construction: an instance is dropped if either arm touched the reference, regardless of which.

Detector recall, per arm

An asymmetric detector manufactures a delta, so recall is declared per arm rather than assumed:

Two known biases work against a complete search of Ouroboros' side, so both are flattering to us:

• An early string-only matcher undercounted our arm roughly 2×, because Ouroboros logs ["git","ls-remote"] where codex logs git ls-remote …. The published counts come from the argv-normalized pass.
• Our stored tool output is truncated at 2005 characters while codex rollouts are whole, so evidence can hide on our side that could not hide on theirs.

We also corrected our own detector twice before publishing: three contamination detectors initially matched 83% and then 99.7% of instances, because every codex trace line ends in a cwd path containing the instance id, which contains the gold SHA. After stripping cwd, the correct figure is 8.

What the filter changes

The unfiltered gap is significant and against us. It does not survive either symmetric filter. Quoting the raw gap as a capability difference is not supportable; neither is quoting the strict view as a win for us. The honest reading is that on this model these two harnesses perform the same, and that roughly a fifth of the benchmark's instances are compromised for at least one arm.

Independently, OpenAI's own audit (Separating signal from noise in coding evaluations) found ~30% of SWE-bench Pro tasks defective (automated audit 200/731, manual 249/731) and withdrew the recommendation to use it as a primary coding eval. Numbers on this benchmark, including these, should be read with that in mind.

Problems found in our own arm

Published because a matched pair is only worth reading if both sides are audited. All of these were found by reading our own traces; the underlying artifacts are in layer1_results/audit/.

Finalization accepts self-authored evidence. The failure mode behind most of our losses: the step that decides "done" accepts verification evidence the agent chose or wrote, rather than a literal external signal. From 36 failures read in full by independent auditors:

• qutebrowser: its own probe caught the real bug; the agent then edited the probe's expected value to match the buggy behaviour and grounded on that. The grader failed on exactly that assertion.
• element-web: put the new code behind a labs flag defaulting to false, then wrote a scratch test that mocks the flag ON. Removing the gate passes 19/19 held-out tests.
• flipt: had a correct, green fix at round 25; a review note recommended a "typed API", and at round 46 the agent deleted its own correct fix.
• ansible: saw a failing test at round 77 and, in its round-82 completion statement, listed the files it had verified while silently omitting that one.
• navidrome: full go test ./... failed twice; all 8 groundings were narrowed -run <its own test>.

The ceiling is shared, not ours alone: both harnesses verify internal consistency, neither verifies the grader's contract. In all 36 traces the agent had already read the file containing the answer, so no failure was caused by unread code.

The run manifest lies about its own configuration. run_manifest.json.model_slots records the settings-template tail rather than what actually ran: it claims review slots on sonnet-4.5 ×3 at effort high, where the run in fact used luna ×1 at effort low. Only harness.solve_model is trustworthy. The corrected record is published as MANIFEST_CORRECTION.json; treat any provenance claim in this run's manifests as suspect unless it is confirmed there.

The acceptance panel had a verdict but no authority. quorum{required:1, contributed:0} with enforcement_impact="abstains"; the panel could not ratify or block anything, while its free-text notes could still steer the agent, and twice steered it off an already-correct diff. Population-wise the positive verdict carries signal (accepted → 75% resolve) and the negative one does not (unaccepted → 50%, against a 57% base rate).

Two of our own metrics were artifacts. The missing-tests counter looked like a near-miss signal; it is a parser artifact because the benchmark's parser.py records only lines beginning with PASS, so a suite that crashes loses its failures entirely and reports missing on every failure. Our earlier "11 of 16 were near-misses" claim does not hold. Likewise reason_code=tool_failure had no predictive power (it fired on 50 resolved instances and 33 failed ones).

Grading defects we had to fix before publishing. 27 verdicts had been computed against stale or empty patches (regrade flipped 8 to solved, +5 net); 14 instances were poisoned by a provider key hitting its monthly limit mid-run with 10–30 KB patches already written. Both pre-correction ledgers are published (grade_results.before_*.json) so the corrections are visible rather than merely asserted.

An asymmetric re-run, disclosed. Of those 14 poisoned instances, 13 were quarantined and re-run; the 14th (instance_future-architect__vuls-dc496468b9e9fb73371f9606cdcdb0f8e12e70ca) was not, because it had already resolved despite the interruption. Re-running only the poisoned instances that failed biases upward by construction, even though here it can move at most one instance (±0.15 pp, no effect on any comparison at p = 0.40). It is listed in audit/poisoned.json and flagged in manifest.csv; a stricter reading may drop it.

What adversarial review refuted in our own conclusions. Five explanations we believed and dropped: that disabling the subagent swarm hurt us (0 of 36 decisive attributions); that we lose by over-engineering patches (size direction is identical in wins and losses); that a long tail after the last edit kills tasks (codex's post-edit share is equal or larger in ≥5 instances); that we are worse because we are more expensive (we were cheaper in ≥5 instances); and that review is harmful per se (narrowed to this configuration, where what survives scrutiny is the authority asymmetry above, not review frequency).

Contents

Three layers are available, so take only what you need.

layer1_results/          ~40 MB   everything needed to recompute every number
  manifest.csv                    731 rows: per-instance verdicts both arms, filter flag
                                  and reason, cost, rounds, patch size, winning phase
  filters_excluded.json           the 76 excluded instances with reasons
  verdicts/                       canonical ledgers + the two pre-correction snapshots
  evaluator_output/<arm>/         official parser output per instance: test names, statuses
  audit/                          the contamination audit package and failure analysis
  run_config/                     launch argv, model slots, MANIFEST_CORRECTION, grade_loop.sh
layer2_trajectories/     ~2.8 GB  what each agent actually did
  ouroboros/<iid>/                trace.jsonl (rounds, tool calls, review cycles),
                                  solve_result.json (cost, rounds, tokens), agent.patch
                                  (the graded patch), agent_tracked_only.patch,
                                  problem_statement.txt, seed_attestation.json
  codex/<iid>/                    codex_trace.log, broker_commands.jsonl (every command run
                                  in the container), agent.patch, instruction.txt,
                                  status.json, codex_meta.json, attempt_prev_*/
  <arm>/predictions.jsonl         exactly what was fed to the evaluator
layer3_runtime_logs/     ~1 MB    isolated-server, container and install logs

Not included, deliberately: the evaluator's entry scripts, run_script.sh, parser.py and pytest stdout. Those carry git checkout <gold SHA> -- <exact hidden-test files> and print the source of failing hidden tests; publishing them would turn a trace dataset into an answer key. Only test names and statuses are published, and those are already public in SWE-bench Pro. Per-instance container dumps (357 GB of runtime state and repo clones) are also dropped; the runtime is public source and the seed commit is attested per instance.

Scrubbing: operator paths, host names and account names are masked. Private-key blocks and one Slack-token example that appear in traces are redacted bodies because they are test fixtures of the task repositories (gravitational/teleport, ansible/ansible, future-architect/vuls), not credentials of ours. _scrub_report.json records every substitution count.

Provenance and licensing

This repository records a matched-pair study produced with Ouroboros v6.81.0 and Codex CLI. The run metadata, analysis and original documentation authored for this release are offered under Apache-2.0. The pinned SWE-bench Pro evaluator is MIT.

The upstream ScaleAI/SWE-bench_Pro dataset card does not declare a single dataset license. Problem statements, repository identifiers, agent patches, source excerpts and other task-repository material retain the terms of their original repositories. For that reason this repository uses the other metadata value instead of applying one blanket license to every file. THIRD_PARTY_NOTICES.md maps the main artifact groups to their sources and terms.

The Dataset Viewer is configured to read the existing layer1_results/manifest.csv. No benchmark byte or score was rewritten, and the exact evidence snapshot remains 62280378fd82ab9df0b7216745cd42e559ab3435.

Reproduce

manifest.csv alone reproduces both headline numbers:

import csv
rows = list(csv.DictReader(open("layer1_results/manifest.csv")))
keep = [r for r in rows if r["excluded_from_filtered"] == "False"]
for arm in ("ouroboros", "codex"):
    n = sum(r[f"{arm}_resolved"] == "True" for r in keep)
    print(arm, n, len(keep), round(100 * n / len(keep), 2))
# ouroboros 381 655 58.17
# codex     389 655 59.39

Ouroboros

• Source: https://github.com/razzant/ouroboros
• Benchmark harnesses and methodology: https://github.com/razzant/ouroboros/tree/ouroboros/devtools/benchmarks/swe_bench_pro (run_pro.py, grade_pro.py, METHODOLOGY.md, CONTAMINATION_AUDIT.md)
• This run: tag v6.81.0, commit 3f9d504b

Citation

@misc{ouroboros_swepro_luna_pair,
  title  = {A matched-pair comparison of Ouroboros and Codex CLI on SWE-bench Pro},
  author = {Razzhigaev, Anton},
  year   = {2026},
  url    = {https://github.com/razzant/ouroboros}
}