Goldset

Verified bug-fix records for evaluating coding agents.

Every record is a real bug in public software, the fix its author wrote, and the test that fails before the fix and passes after it. A record is kept only once both runs have been observed, so what is published here is a reproduction rather than a claim.

goldset.dev · Datasheet · Attribution

Verify it yourself

The point of this corpus is that you do not have to take our word for it.

git clone https://github.com/andysalvo/goldset
cd goldset
python3 verify.py --sample 20

Each record names a public project. Verifying one clones that project, checks out the commit before the fix and the commit of the fix, copies the fix commit's test file into both, and runs the named tests against each. The record holds if they fail on the older code and pass on the fix.

Needs git, python3 and pytest. Install uv as well and old projects are run under an interpreter that can actually import them, which recovers most of what is otherwise unrunnable: a project from 2016 often cannot be imported on a current Python at all.

Clones are cached under ~/.cache/goldset and reused between records and between runs, so a project is downloaded once rather than once per record. Set GOLDSET_NO_CACHE=1 to turn that off.

A record that cannot run on your machine is reported as SKIP, never as a failure. A FAIL means a published record did not hold up, and it is worth an issue.

Evaluating a model

pip install goldset
export OPENROUTER_API_KEY=...        # or OPENAI_API_KEY, or ANTHROPIC_API_KEY

goldset eval --model anthropic/claude-sonnet-5 --sample 50

The model is given the specification and the broken code, and asked to return the corrected code. It never sees the test or the author's fix. Its answer is written back into the project and the recorded tests are run against it. The score is the share of records where they pass.

Nothing is scored unless the author's own fix passes on the same machine first, so a record your environment cannot run is never counted against a model. Results carry a Wilson interval, because a pass rate over fifty records is not a precise number and should not be printed as though it were.

goldset eval --model gpt-5 --granularity function --since 2025-01-01 --out run.json
goldset verify --sample 20
goldset show 37d9ffc578a058d9

Keys are read from the environment and are never read from or written to this repository. Running an evaluation bills whoever runs it.

Both commands clone public projects and run their test suites locally. Use a container if that matters to you.

One record

{
  "fingerprint": "37d9ffc578a058d9",
  "repo": "more-itertools/more-itertools",
  "unit": "iterate",
  "subject": "Issue #707: fix ``iterate()`` to enable ``func`` to raise StopIteration",
  "committed_at": "2023-04-19",
  "license": "MIT",
  "granularity": "function",
  "spec": "Return `start`, `func(start)`, `func(func(start))`, ...",
  "source_before": "def iterate(func, start):\n    ...\n        start = func(start)",
  "source_after":  "def iterate(func, start):\n    ...\n        try:\n            start = func(start)\n        except StopIteration:\n            break",
  "test_file": "tests/test_more.py",
  "fail_to_pass": ["tests/test_more.py::IterateTests::test_func_controls_iteration_stop"],
  "parent": "7c1b6482a00cb9c1f76061c12be192ed5043e5ea",
  "commit": "cd0a3a87d231033db06a3c222fe893025e398a86"
}

Fields

One JSON object per line in oracles.jsonl.

Loading it

import json

records = [json.loads(line) for line in open("oracles.jsonl")]

# Only single-function records, and only fixes a model trained to 2025 could not have seen.
subset = [
    r for r in records
    if r["granularity"] == "function" and r["committed_at"] > "2025-01-01"
]

Or with the datasets library:

from datasets import load_dataset
ds = load_dataset("json", data_files="oracles.jsonl", split="train")

What is not in here

A further 385 records from 157 projects are held back and not published. Projects are split whole, so a project's records sit entirely on one side, and nothing here can be used to work out what is in the held-out set. A public evaluation set is contaminated the moment it is public, which is the reason for keeping a disjoint one.

What is published here stays published. The CC BY 4.0 grant on this release is permanent and nothing is ever withdrawn.

Two things to know before you use it

A passing test does not mean a fix is correct. It means the recorded test passes. That is the property this corpus guarantees, and it is not the same claim.

This is not a random sample of software defects. It is the subset of real fixes that shipped a regression test in the same commit, touched a single source file, and could be reduced to one documented unit or one whole file whose test still flips. Use it as an evaluation set, not as an estimate of how bugs are distributed in the wild. The datasheet is explicit about the rest of the limits, including a difficulty claim we tested and could not replicate.

Licence and attribution

Nothing here was written by us. Every defect, every fix and every test is the work of the project that produced it, used under a permissive licence, and permanently linked to the upstream commit so anyone can re-verify it.

Each record carries the licence it came under, so attribution can be checked per record rather than taken on trust. Per-project counts are in ATTRIBUTION.md, and the full text of every project's licence, as it stood at the commit cited, is in licenses/. MIT, BSD and Apache-2.0 all condition redistribution on the notice travelling with the code rather than on a link to it, so it is included rather than pointed at. A project whose licence text could not be retrieved has its records withheld. Copyleft-licensed projects are excluded before anything is cloned.

If you maintain a project here and would rather it were not, write to andy@goldset.dev or open an issue, and it will be removed, no questions asked.

The compilation, meaning the selection and arrangement of records, is offered under the terms in LICENSE. The code inside each record remains under the licence of the project it came from.

Citing

@misc{goldset2026,
  title  = {Goldset: verified bug-fix records for evaluating coding agents},
  author = {Salvo, Andrew},
  year   = {2026},
  url    = {https://goldset.dev}
}