BeyondSWE is a comprehensive benchmark that evaluates code agents along two key dimensions — resolution scope and knowledge scope — moving beyond single-repo bug fixing into the real-world deep waters of software engineering.

✨ Highlights

• 500 real-world instances across 246 GitHub repositories, spanning four distinct task settings
• Two-dimensional evaluation: simultaneously expands both resolution scope (local → global) and knowledge scope (within-repo → cross-repo / domain / web)
• 18x more complex than SWE-bench Verified: 5.6 files and 209.9 lines per instance on average (vs. 1.3 files / 11.6 lines)
• SearchSWE framework: first standardized benchmark for evaluating deep research in coding, with rigorous anti-cheating mechanisms
• Key finding: frontier models plateau below 45% on BeyondSWE, despite achieving 80%+ on SWE-bench Verified

📣 News

• 2026-02-27 🚀 We released our data on Hugging Face. This release includes all 500 tasks of the BeyondSWE benchmark.

📋 Benchmark Overview

BeyondSWE covers four task settings that span the full spectrum of real-world software engineering challenges:

📊 Data Format

Each instance in BeyondSWE is a JSON object containing the following fields:

Comparison with Existing Benchmarks

---

📈 Results

Key Findings

1. The 45% Ceiling — Even frontier models (Gemini 3 Pro, GPT-5.2, DeepSeek-V3.2, etc.) fail to exceed 45% overall on BeyondSWE, compared to 80%+ on SWE-bench Verified.

2. No Single Winner — Different models lead on different tasks — Seed-Coder on CrossRepo (44.72%), DeepSeek-V3.2 on Doc2Repo (54.99%), Gemini 3 Pro on DepMigrate (41.81%) — revealing that the four tasks test fundamentally different capabilities.

3. Search Helps, but Integration Remains Open — 6 out of 9 models improve with SearchSWE, with Gemini 3 Pro gaining +7.5% on DomainFix. However, gains are inconsistent — search and coding have matured independently, but their effective fusion is still an unsolved challenge.

4. Quality over Quantity — Gemini 3 Pro searches only 0.8–1.1 times per instance yet achieves the best overall gain (+2.0%), while DeepSeek-V3.2 searches 4.2–5.4 times but shows a slight decline (-0.2%).

🚀 Quick Start

Installation

git clone https://github.com/AweAI-Team/BeyondSWE.git
cd BeyondSWE

Sample Usage

You can download the benchmark data using the huggingface_hub library:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="AweAI-Team/BeyondSWE",
    repo_type="dataset",
    local_dir="BeyondSWE",
)

Evaluation with SearchSWE (BeyondSWE's official evaluation framework)

Please refer to AweAgent for the full evaluation pipeline, including SearchSWE setup and running instructions.

📝 Citation

If you find BeyondSWE useful in your research, please cite our paper:

@misc{beyondswe2026,
      title={BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing}, 
      author={Guoxin Chen and Fanzhe Meng and Jiale Zhao and Minghao Li and Daixuan Cheng and Huatong Song and Jie Chen and Yuzhi Lin and Hui Chen and Xin Zhao and Ruihua Song and Chang Liu and Cheng Chen and Kai Jia and Ji-Rong Wen},
      year={2026},
      eprint={2603.03194},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.03194}, 
}

📄 License

This project is licensed under the CC BY 4.0 License — see the LICENSE file for details.