Submissions of Results on SWE-bench-Live

This repo hosts model results, trajectories, and evaluation logs on the LLM benchmark SWE-bench-Live. We coordinate result submissions via Pull Requests.

Trajectories & Logs

Each folder contains the rollout trajectories, prediction patches and evaluation result of one agent-model combination.

The forked source codes of the coding agents we used to run SWE-bench-Live are as follows:

SWE-agent

OpenHands

ClaudeCode

Win-Agent (for windows tasks)

Submitting Instructions

Create pull requests (PRs) to this repository to submit the success rate of your model-agent combination.

[!NOTE] Several users have raised questions about the evaluation protocol, so we would like to clarify that SWE-bench-Live evaluation strictly follows the original SWE-bench protocol:

1. During a rollout, the agent may access only the problem_statement field of the Hugging Face dataset and the docker image of the task instance. It must not access any other fields, such as hint, FAIL_TO_PASS, or test_patch. The test_patch must not be applied to the repository before or during the rollout. The agent must perform a single rollout based solely on the problem_statement on the docker container started from the image of the task instance.
2. Prompts, skills, and workflow instructions provided to the agent must not contain solutions specific to any task instance. They may contain only general instructions for the entire benchmark or, at most, for a specific repository. The agent must not use results from the ground-truth evaluation script to refine its solution.

Compliant prompts should follow the SWE-agent prompt and the OpenHands prompt, which contain only the problem statement and general workflow instructions.

When submitting results to our submissions repository, you must include your agent's raw rollout trajectories so that the maintainers can verify compliance with the SWE-bench protocol. A trajectory consists of the complete sequence of inputs to and outputs from your agent across all rollout rounds for a given task instance, including the initial prompt provided to the agent. Please follow this SWE-agent compliant trajectory example when submitting your result. If your organization's policies prohibit sharing the complete set of trajectories, you must provide at least some representative samples for verification. There is a checklist when submitting a PR to help you check whether you meet the protocol requirements again.

If you want to submit the evaluation result of your model-agent combinations on SWE-bench-Live, please follow the steps below.

1. Fork the submission branch this repository SWE-bench-Live/submission. Clone the fork's submission branch

git clone your_fork --depth 1 -b submission

2. Create the folder corresponding to your evaluated subset (submissions/{subset}), create a new folder named in the format: f"{agent_name}/{model_name}".

mkdir -p submissions/{subset}/{agent_name}/{model_name}
cd submissions/{subset}/{agent_name}/{model_name}

3. Place your predictions file in preds.json, which should include the patch for each instance. Place the evaluation report generated by the SWE-bench-Live evaluation script in results.json.

4. Create a logs / trajs folder to store agent rollout logs and trajectories that reflect how your system solved the problems. This is mandantory because we need your execution trajectories to verify your agent rollout complies with the SWE-bench protocol and does not cheat. If your organization prohibits sharing the complete set of trajectories, you must provide AT LEAST TEN representative samples for verification.

5. Create a README.md to explain the agent scaffold you used and the experimental setting, including the number of rollouts, how results were sampled, the number of iterations, and other relevant details.

The expected directory structure is:

submissions/
└── {subset}/
    └── {agent_name}/
        └── {model_name}/
            ├── README.md
            ├── preds.json
            ├── results.json
            └── {logs or trajs}/
                └── <agent execution trajectory files>

6. Now commit and push your changes.

git add .
git commit -m "add {subset}/{agent_name}/{model_name} result"
git push origin submission 

7. Create a pull request (PR) from your forked submission branch to the SWE-bench-Live/submissions repository main branch (SWE-bench-Live/submission:main <- your-repo:submission). There is a checklist appended when creating a PR to help you check again whether you meet all the above requirements. DO NOT remove this checklist in your PR submission! You must tick all the items in the checklist after submitting your PR. Only when all items in the checklist are ticked will we consider merging your PR.

8. Once your PR is approved, your model-agent result will appear on our leaderboard.

Contacting

For any issues encountered during the submission process, please open an issue in the repository SWE-bench-Live/submissions or in the repository microsoft/SWE-bench-Live.