SWE-bench Pro V2

SWE-bench Pro is a benchmark of long-horizon software engineering tasks drawn from real pull requests in 11 open-source repositories (Go, Python, JavaScript, TypeScript). Each task gives an agent a repository at a base commit plus a PR description with explicit requirements and interfaces; the agent's patch is graded by hidden fail-to-pass and pass-to-pass tests in a pristine container.

V2 (2026-09-22) is the default config: 642 tasks (go 256, python 237, js 145, ts 4). The hard config is the 51-task HARD subset. The original 731-task release is preserved unchanged as config v1 and as git tag v1.0.

• Paper: https://static.scale.com/uploads/654197dc94d34f66c0f5184e/SWEAP_Eval_Scale%20(9).pdf
• Evaluation harness, task directories (Harbor format), and locked-protocol tooling: https://github.com/scaleapi/SWE-bench_Pro-os (see v2/README.md)
• Public leaderboard: https://scale.com/leaderboard/swe_bench_pro_public

Loading

from datasets import load_dataset
v2   = load_dataset("ScaleAI/SWE-bench_Pro", split="test")            # V2, 642 tasks (default config)
hard = load_dataset("ScaleAI/SWE-bench_Pro", "hard", split="test")    # HARD-51 subset
v1   = load_dataset("ScaleAI/SWE-bench_Pro", "v1", split="test")      # original 731 tasks (also: revision="v1.0")

What changed in V2

Starting from the 731 v1 tasks, 89 were dropped after review (invalid or unrepairable tests); the 642 that remain were re-validated end to end. Relative to v1:

• Instructions: 529 problem statements were rewritten so that every graded assertion traces back to a sentence in the text; only the problem statement, requirements, and interfaces changed. Each corrected instruction was implemented blind by a second engineer and graded by the task's own hidden tests.
• Tests and grading: test lists were regenerated against repaired output parsers (176 fail-to-pass sets, 47 pass-to-pass sets and 83 selected-file sets differ from v1); 214 test patches and 38 gold patches were revised.
• Images: every image was rebuilt from a sanitised repository bundle (no fixing commit, stray refs, stashes or hooks) and is published on GitHub Container Registry (see docker_image). 211 images received dependency fixes.
• Release gate: every task passes with the reference patch and fails with an empty patch.
• Protocol: the recommended evaluation runs the agent with network access limited to the model endpoint and re-grades the agent's diff on a pristine image. Tooling for this ships in the GitHub repository.

Fields

All list-valued fields are strict JSON strings (json.loads them). Columns 1-16 keep the v1 names and order.

The v1 config has the original 16 columns and values exactly as published on 2026-02-23.

Running an evaluation

The task directories in v2/tasks/ of the GitHub repository contain, for every task, the verifier (tests/test.sh, run_script.sh, parser.py, config.json, test_patch.patch), the reference solution and the image reference, in the Harbor task format:

harbor run -p v2/tasks -e modal -n 50 -a oracle                    # sanity: reference patch resolves every task
PYTHONPATH=v2/tooling harbor run -p v2/tasks -e modal -n 50 \
  -a locked_claude_code:LockedClaudeCode -m <model> --allow-agent-host <your model endpoint host>
PYTHONPATH=v2/tooling harbor run -p v2/tasks -e modal -n 50 \
  -a patch_replay:PatchReplayAgent -m replay --ak source_job=jobs/<run>   # fresh-sandbox re-grade = reported score

The HARD subset is the 51 tasks failed by at least two of five model families under this protocol; every one of them passes with the reference patch and fails with an empty patch. Use the hard config, or harbor run -p v2/tasks $(sed 's/^/-i /' v2/hard51_ids.txt) ....

Images

docker pull ghcr.io/scaleapi/swe-bench_pro-v2:<instance_id>

Images are public and pull anonymously. The repository inside the image is at /app (a few tasks use /testbed), checked out at base_commit, with a sanitised git history.

License

The evaluation harness and tooling are released under the MIT license (see the GitHub repository). Task content (problem statements, patches, tests) is derived from the 11 upstream open-source repositories and remains subject to their respective licenses.

Changelog

• v2.0.0 (2026-09-22): 642-task V2 becomes default; hard (51) and v1 (731) configs added; images moved to ghcr.io/scaleapi/swe-bench_pro-v2; new columns docker_image, hard, version; list fields normalised to strict JSON.
• v1.0 (2026-02-23): 731-task release (tag v1.0).

Citation

@article{deng2025swebenchpro,
  title={SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?},
  author={Deng, Xiang and Da, Jeff and Pan, Edwin and Sun, Yannis Yiming and Wu, Chen Bo Calvin and Bhutani, Sanyam and others},
  journal={arXiv preprint arXiv:2509.16941},
  year={2025}
}