SWE-bench Science

SWE-bench Science evaluates coding agents on software-engineering tasks drawn from scientific-computing repositories. The release contains 119 tasks across 20 scientific domains, with isolated environments and separate programmatic verifiers.

• GitHub release repository: OpenMOSS/SWE-bench-Science
• Runtime images: Docker Hub, pinned by immutable linux/amd64 digests
• Evaluation framework: Pier, compatible with Harbor task format

Dataset Summary

The science_knowledge_ablation column is true for the 91-task science-knowledge split used by the ablation experiment. Its release IDs are 002-082, 084, 086, 090, 097-101, 111, and 114; all other rows are false.

Dataset Viewer And Files

The Dataset Viewer reads the canonical data/tasks.csv table and generates its preview automatically. The release does not commit a duplicate Parquet export, so the CSV remains the single source of truth for the 119 task rows.

The repository also includes:

The environment image contains the baseline source, public fixtures, dependencies, and compilers. The separate verifier image contains held-out tests and the grader. The dataset does not contain reference-answer patches, credentials, agent trajectories, or private verifier tests.

Quick Start

The pinned datacurve-pier==0.3.0 release requires Python 3.12 or newer.

python3.12 -m pip install "huggingface_hub[cli]"
hf auth login
hf download OpenMOSS-Team/SWE-bench-Science \
  --repo-type dataset --local-dir swe-bench-science
cd swe-bench-science

uv tool install --python 3.12 "datacurve-pier==0.3.0"
docker login

Materialize the default selection:

python3 scripts/materialize.py \
  --output tasks-selected --force

Materialize one task, a comma-separated list, or inclusive ranges:

python3 scripts/materialize.py \
  --task-id 002,005-007 \
  --output tasks-selected-small --force

Materialize the complete 91-task science-knowledge ablation split. It contains restricted-license tasks, so the explicit license opt-in is required:

python3 scripts/materialize.py \
  --task-id 002-082,084,086,090,097-101,111,114 \
  --allow-restricted-licenses \
  --output tasks-science-knowledge-ablation --force

Every materialization writes selection.json with the exact task IDs used for the run.

Restricted Licenses

Twenty-three tasks are excluded from the default selection because they contain GPL/LGPL/AGPL-family code, academic non-commercial sources or materials, or restricted third-party data. Include them only after confirming that your use is permitted:

python3 scripts/materialize.py \
  --allow-restricted-licenses \
  --output tasks-selected-all --force

The GPL/LGPL/AGPL-family task IDs are 003, 020, 021, 023, 032, 057, 066, 074, 075, 082, 083, 084, 085, 096, 097, 098, 100, 118. Tasks 019, 026, 035, 101, and 102 are restricted for other reasons. There is no --allow-GPL option. The selection flag controls which task bundles are materialized; it does not replace the upstream license obligations.

Run An Evaluation

Run an infrastructure smoke with no model:

pier run -p tasks-selected-small \
  --agent nop --env docker \
  --n-concurrent 1 --n-attempts 1 \
  --no-force-build --no-delete --yes

Run a real agent by selecting a harness, model, and provider profile:

# Claude Code
pier run -p tasks-selected-small \
  --agent claude-code --env docker \
  --env-file ~/.config/swe-bench-science/claude.env \
  --model anthropic/claude-opus-4-7 --n-concurrent 1

# mini-swe-agent
pier run -p tasks-selected-small \
  --agent mini-swe-agent --env docker \
  --env-file ~/.config/swe-bench-science/mini-swe-agent.env \
  --model openai/gpt-5 --n-concurrent 1

For Codex gateway profiles, use the included wrapper:

python3 scripts/run_batch.py \
  --path tasks-selected-small \
  --agent codex \
  --env-file ~/.config/swe-bench-science/codex.env \
  --n-concurrent 2 --n-attempts 1 \
  --jobs-dir jobs --job-name codex-small

For an approximately 120-second agent-stage smoke, add --agent-timeout-multiplier 0.0223; for an approximately 30-second smoke, use --agent-timeout-multiplier 0.0055556. Verifier and scientific-build timeouts remain independent. The pinned datacurve-pier==0.3.0 release requires Python 3.12 or newer.

Provider Profiles

Create profiles outside the downloaded dataset:

mkdir -p ~/.config/swe-bench-science
cp profiles/codex.env.example ~/.config/swe-bench-science/codex.env
cp profiles/claude.env.example ~/.config/swe-bench-science/claude.env
cp profiles/mini-swe-agent.env.example ~/.config/swe-bench-science/mini-swe-agent.env
chmod 600 ~/.config/swe-bench-science/*.env

Codex profiles use MODEL, OPENAI_API_KEY, CODEX_BASE_URL, CODEX_WIRE_API, CODEX_VERSION, and CODEX_REASONING_EFFORT. Set CODEX_WIRE_API=responses for the OpenAI Responses API or chat for Chat Completions. Claude Code uses ANTHROPIC_AUTH_TOKEN, ANTHROPIC_BASE_URL, and optional ANTHROPIC_CUSTOM_HEADERS. mini-swe-agent uses the provider variables expected by its selected model adapter.

Credentials are read at runtime. They are not stored in task metadata, Dockerfiles, image layers, or result summaries.

Results

Pier writes one aggregate result and one trial directory per task and attempt:

jobs/<job-name>/result.json
jobs/<job-name>/<task>__<trial>/verifier/reward.json
jobs/<job-name>/<task>__<trial>/verifier/ctrf.json
jobs/<job-name>/<task>__<trial>/verifier/test-stdout.txt

The wrapper additionally writes jobs/summary.json and jobs/summary.csv. For a direct Pier run, generate the same summaries with:

python3 scripts/summarize_results.py --jobs-dir jobs

Use pier view jobs to inspect trajectories.

See docs/run-batch.md for the complete option reference, gateway/profile configuration, dry-run mode, retry and timeout controls, and result paths.

Licensing And Attribution

The dataset card, release metadata, and helper tools use the repository's MIT terms. Task source, papers, figures, fixtures, and other third-party materials retain their upstream licenses. The source-license field does not automatically license copied scientific materials; audited material notices and modification notes are retained in the relevant task bundles.

The dataset is independent of GitHub at runtime. After download, materialization and evaluation use the local task bundle and the Docker Hub image digests recorded in task.toml and data/tasks.csv.