README
AweAgent-Meta-SWE-Bench-Pro
This dataset provides the metadata used by AweAgent to run the SWE-Bench-Pro evaluation.
If you are looking for the underlying benchmark itself (task design, repositories, test suites), please refer to the original project: scaleapi/SWE-bench_Pro-os and the accompanying paper SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (arXiv:2509.16941).
Files
swe_bench_pro_aweagent.jsonl— one JSON object per SWE-Bench-Pro instance (731 instances).
Schema
| Field | Type | Description |
|---|---|---|
instance_id | str | Unique identifier for the instance. |
repo | str | The GitHub repository the task is drawn from (e.g. NodeBB/NodeBB). |
repo_language | str | Primary programming language of the repo (e.g. js, python). |
base_commit | str | Commit SHA to check out as the starting state. |
problem_statement | str | Natural-language issue description shown to the agent as the task. |
requirements | str | Detailed functional requirements derived from the issue. |
interface | str | Required method / function signatures the solution must conform to. |
patch | str | Gold reference patch (the human-written fix), used as ground truth. |
test_patch | str | Gold test patch — the new or modified tests that encode the expected behavior. |
fail_to_pass | str (JSON list) | Tests that must transition from failing to passing after the agent's patch is applied. |
pass_to_pass | str (JSON list) | Tests that must remain passing after the agent's patch is applied (regression guard). |
selected_test_files_to_run | str (JSON list) | Test files actually executed during evaluation. |
before_repo_set_cmd | str | Shell commands run inside the container to reset the repo and stage the gold test patch before the agent starts. |
issue_specificity | str (JSON list) | Tags describing the issue type (e.g. major_bug, data_bug). |
issue_categories | str (JSON list) | Tags describing the knowledge domains involved (e.g. back_end_knowledge, database_knowledge). |
tag | str | Short tag name corresponding to the image build. |
source_image | str | Upstream source image the per-instance image was derived from. |
Acknowledgements
This dataset is built on top of, and would not exist without, the excellent SWE-Bench-Pro benchmark by the Scale AI team. All benchmark instances, problem statements, gold patches, and test suites originate from their work; this dataset only repackages the per-instance metadata in the form AweAgent's evaluation harness expects. Huge thanks to the SWE-Bench-Pro authors for releasing such a high-quality, long-horizon software-engineering benchmark — please cite their paper if you use this dataset:
License
Released under CC BY 4.0. When using this dataset, please also cite and credit the upstream SWE-Bench-Pro project and paper.
Contributors
4 commits
Similar repos
README
AweAgent-Meta-SWE-Bench-Pro
This dataset provides the metadata used by AweAgent to run the SWE-Bench-Pro evaluation.
If you are looking for the underlying benchmark itself (task design, repositories, test suites), please refer to the original project: scaleapi/SWE-bench_Pro-os and the accompanying paper SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (arXiv:2509.16941).
Files
swe_bench_pro_aweagent.jsonl— one JSON object per SWE-Bench-Pro instance (731 instances).
Schema
| Field | Type | Description |
|---|---|---|
instance_id | str | Unique identifier for the instance. |
repo | str | The GitHub repository the task is drawn from (e.g. NodeBB/NodeBB). |
repo_language | str | Primary programming language of the repo (e.g. js, python). |
base_commit | str | Commit SHA to check out as the starting state. |
problem_statement | str | Natural-language issue description shown to the agent as the task. |
requirements | str | Detailed functional requirements derived from the issue. |
interface | str | Required method / function signatures the solution must conform to. |
patch | str | Gold reference patch (the human-written fix), used as ground truth. |
test_patch | str | Gold test patch — the new or modified tests that encode the expected behavior. |
fail_to_pass | str (JSON list) | Tests that must transition from failing to passing after the agent's patch is applied. |
pass_to_pass | str (JSON list) | Tests that must remain passing after the agent's patch is applied (regression guard). |
selected_test_files_to_run | str (JSON list) | Test files actually executed during evaluation. |
before_repo_set_cmd | str | Shell commands run inside the container to reset the repo and stage the gold test patch before the agent starts. |
issue_specificity | str (JSON list) | Tags describing the issue type (e.g. major_bug, data_bug). |
issue_categories | str (JSON list) | Tags describing the knowledge domains involved (e.g. back_end_knowledge, database_knowledge). |
tag | str | Short tag name corresponding to the image build. |
source_image | str | Upstream source image the per-instance image was derived from. |
Acknowledgements
This dataset is built on top of, and would not exist without, the excellent SWE-Bench-Pro benchmark by the Scale AI team. All benchmark instances, problem statements, gold patches, and test suites originate from their work; this dataset only repackages the per-instance metadata in the form AweAgent's evaluation harness expects. Huge thanks to the SWE-Bench-Pro authors for releasing such a high-quality, long-horizon software-engineering benchmark — please cite their paper if you use this dataset:
License
Released under CC BY 4.0. When using this dataset, please also cite and credit the upstream SWE-Bench-Pro project and paper.
Similar repos
Linked in READMEs
Contributors
4 commits