This is the multi - language version of SWE-bench-Live (repository_link ; paper_link).
0
44 commits
7 linked in READMEs
updated Sep 20, 2026
README
This is the multi - language version of SWE-bench-Live (repository_link ; paper_link).
We crawled GitHub issues as SWE tasks for coding agents to solve.
We used RepoLaunch (repository_link) to setup docker images as the problem-solving sandboxes for each instance.
Each split is task set of one language.
Each split is filtered by prompting LLM -- a task instance is discarded if LLM cannot infer test patch from problem statement or problem statement contains solution (in NL or code).
Dataset Schema
| Field | Type |
|---|---|
| instance_id | str |
| repo | str |
| pull_number | str |
| issue_numbers | list[str] |
| base_commit | str |
| problem_statement | str |
| patch | str |
| test_patch | str |
| hints_text | Optional[str] from the discussions of the issue |
| all_hints_text | Optional[str] |
| commit_urls | list[str] |
| created_at | str in ISO datetime format |
| commit_url | str |
| rebuild_cmds | list[str] |
| test_cmds | list[str] |
| log_parser | str |
| FAIL_TO_PASS | list[str] |
| PASS_TO_PASS | list[str] |
| docker_image | str |
Evaluation Protocol
[!NOTE] SWE-bench-Live evaluation strictly follows the original
SWE-benchprotocol:
- During a rollout, the agent may access only the
problem_statementfield of the Hugging Face dataset and the docker image of the task instance. It must not access any other fields, such ashint,FAIL_TO_PASS, ortest_patch. Thetest_patchmust not be applied to the repository before or during the rollout. The agent must perform a single rollout based solely on theproblem_statementon the docker container started from the image of the task instance.- Prompts, skills, and workflow instructions provided to the agent must not contain solutions specific to any task instance. They may contain only general instructions for the entire benchmark or, at most, for a specific repository. The agent must not use results from the ground-truth evaluation script to refine its solution.
Compliant prompts should follow the SWE-agent prompt and the OpenHands prompt, which contain only the problem statement and general workflow instructions.
When submitting results to our submissions repository, you must include your agent's raw rollout trajectories so that the maintainers can verify compliance with the SWE-bench protocol. A trajectory consists of the complete sequence of inputs to and outputs from your agent across all rollout rounds for a given task instance, including the initial prompt provided to the agent. Please follow this SWE-agent compliant trajectory example when submitting your result. If your organization's policies prohibit sharing the complete set of trajectories, you must provide at least some representative samples for verification. There is a checklist when submitting a PR to help you check whether you meet the protocol requirements again.
Contributors
41 commits
3 commits
Similar repos
This is the multi - language version of SWE-bench-Live (repository_link ; paper_link).
0
44 commits
7 linked in READMEs
updated Sep 20, 2026
README
This is the multi - language version of SWE-bench-Live (repository_link ; paper_link).
We crawled GitHub issues as SWE tasks for coding agents to solve.
We used RepoLaunch (repository_link) to setup docker images as the problem-solving sandboxes for each instance.
Each split is task set of one language.
Each split is filtered by prompting LLM -- a task instance is discarded if LLM cannot infer test patch from problem statement or problem statement contains solution (in NL or code).
Dataset Schema
| Field | Type |
|---|---|
| instance_id | str |
| repo | str |
| pull_number | str |
| issue_numbers | list[str] |
| base_commit | str |
| problem_statement | str |
| patch | str |
| test_patch | str |
| hints_text | Optional[str] from the discussions of the issue |
| all_hints_text | Optional[str] |
| commit_urls | list[str] |
| created_at | str in ISO datetime format |
| commit_url | str |
| rebuild_cmds | list[str] |
| test_cmds | list[str] |
| log_parser | str |
| FAIL_TO_PASS | list[str] |
| PASS_TO_PASS | list[str] |
| docker_image | str |
Evaluation Protocol
[!NOTE] SWE-bench-Live evaluation strictly follows the original
SWE-benchprotocol:
- During a rollout, the agent may access only the
problem_statementfield of the Hugging Face dataset and the docker image of the task instance. It must not access any other fields, such ashint,FAIL_TO_PASS, ortest_patch. Thetest_patchmust not be applied to the repository before or during the rollout. The agent must perform a single rollout based solely on theproblem_statementon the docker container started from the image of the task instance.- Prompts, skills, and workflow instructions provided to the agent must not contain solutions specific to any task instance. They may contain only general instructions for the entire benchmark or, at most, for a specific repository. The agent must not use results from the ground-truth evaluation script to refine its solution.
Compliant prompts should follow the SWE-agent prompt and the OpenHands prompt, which contain only the problem statement and general workflow instructions.
When submitting results to our submissions repository, you must include your agent's raw rollout trajectories so that the maintainers can verify compliance with the SWE-bench protocol. A trajectory consists of the complete sequence of inputs to and outputs from your agent across all rollout rounds for a given task instance, including the initial prompt provided to the agent. Please follow this SWE-agent compliant trajectory example when submitting your result. If your organization's policies prohibit sharing the complete set of trajectories, you must provide at least some representative samples for verification. There is a checklist when submitting a PR to help you check whether you meet the protocol requirements again.
Similar repos
Linked in READMEs
Contributors
41 commits
3 commits