OpenHands Benchmarks

This repository contains benchmark evaluation infrastructure for OpenHands agents. It provides standardized evaluation pipelines for testing agent capabilities across various real-world tasks.

⚠️ Migration in Progress: We are currently migrating the benchmarks from OpenHands V0 to work with the OpenHands Software Agent SDK infrastructure in V1.

Available Benchmarks

See the individual benchmark directories for detailed usage instructions.

Quick Start

Prerequisites

Before running any benchmarks, you need to set up the environment and ensure the local Agent SDK submodule is initialized.

make build

git submodule update --init --recursive

make build

uv sync

cd vendor/software-agent-sdk
git fetch
git checkout <new_commit_or_branch>
cd ../..
git add vendor/software-agent-sdk
git commit -m "Update software-agent-sdk submodule to <new_commit_sha>"

make build

Configure Your LLM

All benchmarks require an LLM configuration file. Define your LLM config as a JSON following the model fields in the LLM class.

Example (.llm_config/example.json):

{
  "model": "litellm_proxy/anthropic/claude-sonnet-4-20250514",
  "base_url": "https://llm-proxy.eval.all-hands.dev",
  "api_key": "YOUR_API_KEY_HERE"
}

Validate your configuration:

uv run validate-cfg .llm_config/YOUR_CONFIG_PATH.json

Running Benchmarks

After setting up the environment and configuring your LLM, see the individual benchmark directories for specific usage instructions:

• SWE-Bench: Software engineering tasks from GitHub issues
• SWE-Bench Pro: Long-horizon software engineering tasks from GitHub issues
• GAIA: General AI assistant tasks requiring multi-step reasoning
• OpenAgentSafety: AI agent safety evaluation in workplace scenarios with NPC interactions

Rich Logging

Enable enhanced console output with color-coded, structured logs:

export RICH_LOGGING=1   # Enable rich logs (default: disabled)
export NO_COLOR=1       # Disable colors if needed

Rich logging shows real-time tool calls, agent messages, and a summary at the end of each instance:

10:30:45 [django-12345]  TOOL   │ ▶ bash #1 cmd='ls -la'
10:30:46 [django-12345]  TOOL   │   └─ ok
OK patch=NONEMPTY msgs(a/u)=8/3 tool_calls=12 errors(agent/conv)=0/0 end=finish_tool

File logging (logs/instance_<id>.log) is unaffected by this setting.

Workspace Types

Benchmarks support two workspace types for running evaluations:

Docker Workspace (Default)

Uses local Docker containers to run agent evaluations. Images are built locally on-demand.

• Pros: No additional setup required, works offline
• Cons: Resource-intensive on local machine, slower for large-scale evaluations
• Use case: Development, testing, small-scale evaluations

Remote Workspace

Uses a remote runtime API to provision containers in a cloud environment, enabling massive parallelization.

• Pros: Scalable to hundreds of parallel workers, no local resource constraints
• Cons: Requires pre-built images and API access
• Use case: Large-scale evaluations, benchmarking runs

How Remote Runtime Works

1. Pre-build Agent Images: Agent-server images must be pre-built for a specific SDK commit (SHA) and pushed to a public container registry (e.g., ghcr.io/openhands/eval-agent-server)

2. Runtime API: The remote workspace connects to a runtime API service (default: https://runtime.eval.all-hands.dev) that provisions containers on-demand

3. Image Resolution: Before starting evaluation, the system verifies that the required image exists in the registry with the correct tag format: {IMAGE}:{SDK_SHA}-{CUSTOM_TAG}{SUFFIX}

4. Parallel Execution: Each evaluation instance runs in its own isolated container, allowing for massive parallelization (e.g., 32+ concurrent workers)

Prerequisites for Remote Workspace

1. Pre-built Images: Images must be built and pushed to a public registry

   • In this repository, add one of the following labels to a PR to trigger image builds:
     • build-swebench-50: Build 50 SWE-Bench images (quick testing)
     • build-swebench-200: Build 200 SWE-Bench images (medium testing)
     • build-swebench: Build all SWE-Bench images (full evaluation)
     • build-swebenchpro-50: Build 50 SWE-Bench Pro images (quick testing)
     • build-swebenchpro-200: Build 200 SWE-Bench Pro images (medium testing)
     • build-swebenchpro: Build all SWE-Bench Pro images (full evaluation)
   • Images are tagged with the SDK SHA from the vendor/software-agent-sdk submodule

2. Runtime API Key: Set the RUNTIME_API_KEY environment variable

   export RUNTIME_API_KEY="your-api-key-here"
   

3. Optional Configuration:

   • RUNTIME_API_URL: Override the default API endpoint (default: https://runtime.eval.all-hands.dev)
   • SDK_SHORT_SHA: Override the SDK SHA for image selection (default: auto-detected from submodule)

See individual benchmark READMEs for specific usage examples.

SDK Compatibility and Version Management

⚠️ Important: The benchmarks repository depends on the OpenHands Agent SDK, and not every version of the benchmarks is compatible with every version of the SDK. As the SDK evolves and introduces new features, the benchmarks code may adopt these features, creating version dependencies.

SWE-Bench image layering (docutils/roman)

Some SWE-Bench instances (notably sphinx-doc) require docutils<0.21 and roman. The build pipeline now wraps only those images that need the extra layer:

• benchmarks/swebench/build_images.py wraps images for repos in a small allowlist (currently sphinx-doc).
• Other repos (e.g., scikit-learn) keep the base image unchanged.
• Wrapped images reuse the same tag (no suffix) since they're evaluation-only.

When running or dispatching builds, no extra flags are needed—the selective wrapping is handled for you.

Evaluating Different SDK Versions

When evaluating a specific SDK version, you need to ensure the benchmarks code is compatible with that SDK version. You have two options:

1. Use the benchmarks-commit parameter in the workflow (Recommended):

   • When manually triggering the build-swebench-images workflow (builds + wraps images in-place), specify both:
     • sdk-commit: The SDK version you want to evaluate
     • benchmarks-commit: A benchmarks commit that's compatible with that SDK version

2. Manually check out compatible versions locally:

   # Check out a benchmarks commit that's compatible with your target SDK version
   git checkout <benchmarks-commit>
   
   # Update the SDK submodule to your target version
   cd vendor/software-agent-sdk
   git checkout <sdk-commit>
   cd ../..
   
   # Rebuild the environment
   make build
   

Example: SDK Critic Module

A notable example of version dependency is the SDK critic module. As of SDK commit 79868ae5 (November 17, 2025), the OpenHands Agent SDK introduced the openhands.sdk.critic module. Current benchmarks code imports CriticBase from this module, which means:

• SDK versions ≥ 79868ae5: Compatible with current benchmarks code
• SDK versions < 79868ae5: Require an older benchmarks commit (before the critic import was added)

To check if a specific benchmarks commit requires the critic module:

git show <commit>:benchmarks/utils/models.py | grep "from openhands.sdk.critic"

If this command returns output, that benchmarks commit requires an SDK version with the critic module.

Links

• Original OpenHands: https://github.com/OpenHands/OpenHands/
• Agent SDK: https://github.com/OpenHands/software-agent-sdk
• SWE-Bench: https://www.swebench.com/