Claw-SWE-Bench

Paper: Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-Style Agent Harnesses on Coding Tasks

A multilingual issue-resolving benchmark with two evaluation configs:

• full — 350 instances (300 from SWE-bench Multilingual + 50 Python from SWEBench-verified-mini's size_optimized_sample).
• lite — 80-instance calibrated subset (10 per language across 8 languages: Java, Go, Rust, JS/TS, C/C++, Ruby, PHP, Python). Designed for low-cost iteration on harness implementations, model swaps, prompt edits, and bug fixes while preserving the aggregate and per-language resolve-rate distribution of the full set under a 17-column calibration pool (9 openclaw model columns + 8 cross-claw model x harness columns).

Loading

from datasets import load_dataset

# Lite is the default config
lite = load_dataset("TokenRhythm/Claw-SWE-Bench", "lite", split="test")

# Full 350-instance set
full = load_dataset("TokenRhythm/Claw-SWE-Bench", "full", split="test")

The dataset is shipped as two parquet files (data/lite-test.parquet, data/full-test.parquet) so loading is fast and the Hugging Face Dataset Viewer works out of the box. No trust_remote_code flag is required.

How the parquet files were built

The 350 instances are sourced from two upstream datasets, both MIT:

• 300 instances from SWE-bench/SWE-bench_Multilingual (test split).
• 50 Python instances from princeton-nlp/SWE-bench_Verified, filtered to the size_optimized_sample 50-id subset of mariushobbhahn/SWEBench-verified-mini.

We added two columns (language, source_dataset) and re-emitted the merged table as parquet using build/build_full350.py. The Lite-80 parquet is produced by build/build_lite80.py, which applies data/lite80_ids.json to the merged table. To rebuild the parquet files yourself:

pip install -r build/requirements.txt
python build/build_full350.py    # writes data/full-test.parquet
python build/build_lite80.py     # writes data/lite-test.parquet

See ATTRIBUTION.md for upstream citations and license notes.

Schema

Composition

Sources & License

• SWE-bench Multilingual (Khandpur, Lieret, Jimenez, Press, Yang, 2025). MIT. https://huggingface.co/datasets/SWE-bench/SWE-bench_Multilingual. Cite via the SWE-smith paper: Yang et al., arXiv:2504.21798.
• SWEBench-verified-mini / size_optimized_sample (Hobbhahn, 2024). MIT. https://github.com/mariushobbhahn/SWEBench-verified-mini. Underlying Python data is fetched from princeton-nlp/SWE-bench_Verified (MIT).

This dataset's additions (merge specification, Lite-80 selection algorithm and instance list, evaluation scripts) are released under MIT. Underlying repository code retains its original repository license; see REPO_LICENSES.md and ATTRIBUTION.md.

A full datasheet is provided in DATASHEET.md.

Citation

@misc{clawswebench2026,
  title  = {Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-Style Agent Harnesses on Coding Tasks},
  author = {Zheng, Mengyu and Han, Kai and Tian, Yuchuan and He, Wei and Zhou, Hang and Hu, Hailin and Li, Boxun and Xu, Haiyang and Guo, Jianyuan and Ma, Lin and Xu, Chao and Wei, Yunchao and Wang, Yunhe and Wang, Yu},
  year   = {2026},
  eprint = {2606.12344},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2606.12344},
  note   = {Technical report, TokenRhythm Technologies}
}