five-bugs

26 agents, from a 4-bit quantized 7B model up to Opus 5, were given the same one-line bug and its test suite. All 26 made the tests pass. All 26 stayed broken.

They wrote four different regexes to do it:

re.sub(r"<[^>]*>", "", html)   # 14x  Codex CLI, Opus 5, 4 repeats, 8 local
re.sub(r"<[^>]+>", "", html)   #  8x  Claude Code, Sonnet 5, 1 repeat, 5 local
re.sub(r"<.*?>",   "", html)   #  3x  local
re.sub(r"<.+?>",   "", html)   #  1x  Haiku 4.5

All four turn the sentence a < b and c > d into a d.

Nobody was wrong to write them. The test suite they were handed contained only well-formed HTML, so that was the entire specification, and every one of them satisfied it exactly. Scaling the model across two orders of magnitude did not help, because the model was never the binding constraint.

Handing one agent a better test fixed it on the first try.

The full account — what was ruled out, and what it does and does not support — is in docs/CONVERGENCE.md.

Verify it in two seconds, without running an agent or spending anything:

git clone https://github.com/vyang472/five-bugs
cd five-bugs
python3 scripts/verify.py

---

What this repo is

Five single-file Python bugs. Each ships with the checker the agent is given, and a second checker, written afterwards, that no agent ever sees. The gap between those two is the whole experiment.

Everything was run on one laptop: the tasks, the patch every agent actually wrote, raw timings, token counts and costs, and the scripts that produced them. Nothing here asks you to trust a number I typed.

---

How the blind spot showed up

Claude Code and Codex CLI both fixed all five bugs, 5/5 on the checker they were given. It looked like a tie and a clean sweep.

Then I wrote six new cases per bug — after the patches were in, never shown to either agent — and ran them again.

task agent    visible   hidden
t1   claude   PASS      PASS
t1   codex    PASS      PASS
t2   claude   PASS      PASS
t2   codex    PASS      PASS
t3   claude   PASS      PASS
t3   codex    PASS      PASS
t4   claude   PASS      FAIL
t4   codex    PASS      FAIL
t5   claude   PASS      PASS
t5   codex    PASS      PASS

Four of five held. On the fifth, both agents failed in exactly the same way. The bug was a greedy regex:

re.sub(r"<.+>", "", html)      # the seeded bug

So I ran it again on three more agents — Claude Haiku 4.5, Sonnet 5 and Opus 5. Five agent configurations, two labs, three model tiers. They wrote three different patterns:

Three distinct fixes. One identical wrong answer:

strip_tags("a < b and c > d")   # -> "a  d"    (all five)

None of them invented the blind spot. The visible checker contained only well-formed HTML, so that was the entire specification any of them was given, and all five satisfied it exactly.

So I ran two more experiments to find out whether the model or the spec was the binding constraint.

Is it reproducible, or did I get unlucky? Same task, same weak checker, same prompt, five more independent runs:

run  visible  hidden  pattern
1    PASS     FAIL    <[^>]+>
2    PASS     FAIL    <[^>]*>
3    PASS     FAIL    <[^>]*>
4    PASS     FAIL    <[^>]*>
5    PASS     FAIL    <[^>]*>

Ten runs on this task tonight across two labs and three model tiers. Ten hidden failures. Not variance — a fixed point.

What happens if I fix the spec instead of the model? I handed the agent the hidden checker as its visible one. Nothing else changed: same default model, same prompt, same one attempt.

All five tasks passed, t4 included. It wrote:

re.sub(r"</?[A-Za-z!?][^>]*>", "", html)

and documented why, unprompted: "A < not followed by a name character (as in a < b) is literal text and is left alone."

Ten runs could not buy that fix. One better test did. The test suite is the spec, and the agent optimises for exactly what you wrote down.

Reproduce all of it with python3 scripts/verify.py — no agents run, nothing is billed.

---

What else the runs showed

Wall time: quote a range, not a median. Same five tasks, one run each, timed end to end including CLI startup.

The 58s outlier was the easiest task in the set. With n=5 that is an anecdote about variance, not a property of either tool — which is the point: the median hid it completely.

That spread is across tasks. Repeating one task five times, run-to-run variance was almost nil: wall time within 3%, cost within 1%, output tokens within 9% (results/repeat.csv). The unpredictability lives in which task you hand the agent, not in rerunning the same one.

The patches converged. Two of five came back byte-identical between the two agents. On money rounding both independently reached for the same non-obvious Decimal(str(x)).quantize(..., ROUND_HALF_UP). For bugs this standard, model choice looks like it matters less than the arguing suggests. See patches/.

Output compression, measured. The caveman skill advertises a 65% token cut. On these five agentic-coding tasks I measured 21.7% (8,299 → 6,494 output tokens), at no cost to quality — 5/5 either way, and the same 4/5 on the hidden tests, failing the same case. Not a false claim; a different workload. It compresses prose, and in agentic coding most output tokens are tool calls and code, which it deliberately leaves alone. results/caveman.csv.

Paying more did not buy a better answer

Same five bugs, same prompt, three Claude model tiers, one run each.

Identical scores on both test sets, for 5.4x the cost. Two things surprised me:

• The cheap tier was the slow one. Haiku took 1.37x the total wall time of Opus, because it wrote 2.2x more output tokens to get to the same patch. Cheaper per token is not the same as cheaper per task, and it is not faster.
• The overhead dwarfs the prompt. Across all 15 runs I typed 328 tokens of prompt. The runs read 3,216,012 tokens of cached context to answer them — roughly 9,800 read for every 1 written. The cheapest possible run in the whole set still cost $0.04. You are not paying for your question.

On this workload the tier decision is a latency decision, not a quality one. Five bugs is nowhere near enough to generalise that — but it is enough to stop assuming the expensive tier is automatically the right default. results/models.csv.

Quantization changed nothing, and everything underneath

A reader (@bnjmn_marie) reported that on DeepSWE, two runs of a quantized model scored an identical 31.86% while disagreeing on 42 of 113 tasks. This is the smallest version of that experiment I could run.

Same model (Qwen2.5-Coder 7B), same prompt, two quantizations, 8 runs per task, 80 runs total. Single-turn code generation, not agentic — a 7B model cannot drive the tool loop the other agents used.

The scores are identical. The answers are not. On 2 of the 5 tasks the two quantizations produced completely different answer sets. Q8 took 2.3x longer to score the same.

Distinct-answer count tracks difficulty better than the score does:

t2  1 answer    8/8 pass     the model knows this one
t1  2 answers   8/8 pass
t4  3 answers   8/8 pass     ← and 0/16 on the hidden test
t3  4 answers   0/8 pass
t5  7 answers   0/8 pass     the model is guessing

t4 is the dangerous row: low spread, perfect score, uniformly wrong. The local model converged on <[^>]*> (8 runs), <[^>]+> (5) and <.*?> (3). Two of those are exactly what the frontier agents wrote; the third differs from Haiku's <.+?> by one character, which is the kind of difference this whole repo exists to notice.

Counting last night's runs, 26 independent attempts spanning a 4-bit quantized 7B model and five frontier agents produced four patterns and zero hidden-test passes. All four collapse a < b and c > d to a d.

On t3 the model either left the bug untouched (10 of 16 runs returned round(amount, 2) unchanged) or invented API that does not exist (round(amount, 2, rounding_method=ROUND_HALF_EVEN)). Both failure modes show up as spread before they show up as a score.

results/quantization.csv, patches/quant/ (one file per distinct answer), scripts/run_quant.py.

---

Run it yourself

The five bugs matter less than the shape. Pick five mistakes you already understand, write the obvious checker, then write the mean one afterwards.

scripts/run_agents.sh     # Claude Code vs Codex CLI      -> results/tools.csv
scripts/run_models.sh     # haiku vs sonnet vs opus       -> results/models.csv
scripts/run_caveman.sh    # output-compression A/B        -> results/caveman.csv
scripts/run_spec.sh       # hidden tests handed over as the spec -> results/spec.csv
scripts/run_repeat.sh     # one task, five times          -> results/repeat.csv
scripts/run_quant.py      # local model, two quantizations -> results/quantization.csv
scripts/verify.py         # replay archived patches, no agents run

The runner scripts invoke real agents and consume real quota; verify.py does not.

Layout

tasks/      five buggy files, each with the checker the agent is given
hidden/     the checkers no agent ever saw
patches/    what each agent actually wrote, including all five t4 attempts
results/    raw csv: timings, token counts, cost, pass/fail on both test sets
scripts/    the runners, and verify.py to replay the archive
docs/       METHOD.md — how the runs were made and what they cannot tell you
            CONVERGENCE.md — the 26-run result in full

Limits

n = 5, one machine, one attempt per cell, textbook bugs already localised to a single function. Sampling is non-deterministic; close cells are ties. This cannot be extrapolated to large refactors or unfamiliar codebases, and it never tests the hard part of real debugging, which is finding the bug. Full caveats in docs/METHOD.md.

MIT licensed. Built by @vyang472.