SanityHarness

A lightweight evaluation harness for coding agents that runs high-signal, compact but challenging problems in isolated Docker containers. Evaluate agents across 26 tasks in 6 languages with weighted scoring, integrity verification, and detailed reporting.

Note: v1.8.x is now live, both in releases and the leaderboard. Use v1.6.1 with --legacy flag if you want to compare run data to the old legacy leaderboard.

Table of Contents

• Features
• Quick Start
• Usage
• Available Tasks
• Configuration
• Agents
• How It Works
• Output
• Architecture
• Version History
• Contributing
• License

Features

• Isolated Execution: Each task runs in a dedicated Docker container
• Multi-Language Support: Go, Rust, TypeScript, Kotlin, Dart, and Zig (26 tasks)
• 19 Built-in Agents: Gemini, Claude, OpenCode, Codex, Goose, and more
• Weighted Scoring: Empirically-derived difficulty factors for fair comparison
• BLAKE3 Verification: Cryptographic integrity checks for submissions
• Watch Mode: Automatically re-run tests on file changes
• Hidden Tests: Additional validation applied only during eval
• Parallel Eval: Run multiple tasks concurrently with --parallel
• Agent Sandboxing: Bubblewrap isolation restricts agents to their workspace
• Persistent Caches: Speed up builds with .sanity-cache/ mounts

Quick Start

Prerequisites

• Go 1.25+
• Docker (running daemon)
• bubblewrap (optional, for agent sandboxing)

Installation

git clone https://github.com/lemon07r/sanityharness.git
cd sanityharness
make tools    # Install dev tools (first-time only)
make build    # Build the CLI

Global Flags

Usage

List Tasks

./sanity list                        # List all tasks
./sanity list --json                 # JSON output
./sanity list --language go          # Filter by language
./sanity list --tier core            # Filter by tier
./sanity list --difficulty hard      # Filter by difficulty

Initialize Workspace

./sanity init go/bank-account        # Create workspace with stub files
./sanity init go/bank-account -o ./my-dir

Run a Task

./sanity run go/bank-account         # Run tests once
./sanity run go/bank-account --watch # Re-run on file changes
./sanity run go/bank-account -w ./my-impl --timeout 60

Evaluate an Agent

./sanity eval --agent gemini                          # Evaluate against core tasks
./sanity eval --agent gemini --model gemini-3-pro     # Specify model
./sanity eval --agent gemini --tier all --parallel 4  # All tasks, 4 concurrent
./sanity eval --agent gemini --dry-run                # Preview without running
./sanity eval --agent droid --reasoning high          # Set reasoning effort
./sanity eval --agent gemini --use-mcp-tools          # Enable MCP tools
./sanity eval --agent opencode --use-skills           # Enable Agent Skills mode
./sanity eval --agent opencode --disable-mcp          # Disable MCP tools / currently only supported for opencode
./sanity eval --agent opencode --keep-workspaces      # Keep workspaces for debugging
./sanity eval --agent gemini --no-sandbox             # Disable bubblewrap sandbox
./sanity eval --agent gemini --legacy                 # Legacy mode (hidden tests visible to agent)
./sanity eval --resume ./eval-results/2026-01-07T120000-gemini  # Resume interrupted eval

View Results

./sanity show sessions/go-bank-account-2026-01-15T143022-a1b2c3d4
./sanity show sessions/go-bank-account-2026-01-15T143022-a1b2c3d4 --json

Verify Submission

./sanity verify ./eval-results/2026-01-07T120000-gemini

Clean Up

./sanity clean                # Interactive cleanup
./sanity clean --all --force  # Clean everything

Version

./sanity version              # Show version, commit, build date

Task References

Tasks can be referenced as:

• Canonical: <language>/<slug> (e.g., go/bank-account) - always unambiguous
• Bare slug: bank-account - works if unique across languages

Available Tasks

26 tasks across 6 languages with varying difficulty:

See docs/TASKS.md for complete task listings and metadata.

Configuration

Create sanity.toml in your project root (see sanity.toml.example):

[harness]
max_attempts = 10
default_timeout = 60
session_dir = "sessions"

[docker]
go_image = "ghcr.io/lemon07r/sanity-go:latest"
auto_pull = true

Config files are searched in order:

1. ./sanity.toml
2. ~/.sanity.toml
3. ~/.config/sanity/config.toml

See docs/CONFIGURATION.md for all options.

Agents

Built-in Agents

Custom Agents

[agents.my-agent]
command = "/path/to/my-agent"
args = ["--auto-approve", "{prompt}"]
model_flag = "-m"
env = { API_KEY = "xxx" }

See docs/CONFIGURATION.md#agent-configuration for full schema.

Workspace isolation: During sanity eval, each agent runs in an isolated temporary workspace under /tmp rather than inside eval-results/. This prevents agents from reading other eval results, sibling task solutions, or their own agent.log. After the agent finishes, files are copied back to eval-results/ for validation. Combined with the bubblewrap sandbox (which uses --tmpfs /tmp), agents have zero visibility into other evaluations.

Sandbox note: sanity eval runs agents inside a bubblewrap sandbox where $HOME is read-only by default. A configurable allowlist is mounted read/write ([sandbox] shared_readwrite_dirs) and read-only ([sandbox] shared_readonly_dirs), with additional writable paths available via [sandbox] writable_dirs. Non-allowlisted top-level home directories are masked, and extra sensitive paths can be masked with [sandbox] readable_denylist. Use --no-sandbox to disable.

Legacy mode: Prior to v1.6.0, a bug caused hidden tests to be included in the workspace during sanity eval, making them visible to agents. The --legacy flag reproduces this behavior so that older evaluation runs can be fairly compared or resumed. When --legacy is active, hidden test files are written to the workspace at init time (instead of being overlaid just before validation), and the hidden-test overlay step is skipped. Use this flag when resuming runs that were originally executed with the buggy behavior.

How It Works

1. Container Strategy: Containers run sleep infinity; commands execute via docker exec for fast reuse
2. Workspace Mounting: Your code is mounted at /workspace in the container
3. User Permissions: Runs as your host UID:GID to avoid root-owned files
4. Cache Persistence: Language caches mount from .sanity-cache/ for faster builds
5. Embedded Tasks: Task files are compiled into the binary for zero-dependency distribution

Output

Session Output

Each sanity run creates:

sessions/<session-id>/
├── result.json      # Structured results
├── report.md        # Markdown summary
├── logs/            # Per-attempt logs
└── workspace/       # Final code

Eval Output

Each sanity eval creates:

eval-results/<timestamp>-<agent>/
├── summary.json       # Complete results with weighted scores
├── attestation.json   # BLAKE3 hashes for verification
├── report.md          # Human-readable report
├── submission.json    # Leaderboard format
├── run-config.json    # Config for resume capability
└── <task>/
    ├── agent.log      # Agent output during task execution (includes HARNESS timeout footer)
    ├── validation.log # Test runner output + HARNESS validation footer (always non-empty)
    ├── integrity.json # Present on integrity violations; forensic metadata
    ├── integrity-files/ # Present on integrity violations; expected/actual file copies
    └── integrity-diff/  # Present on integrity violations; per-file diffs

Resume interrupted evals: If interrupted (CTRL+C), the harness saves partial results and prints a resume command. Use ./sanity eval --resume <dir> to continue.

See docs/SCORING.md for scoring details and output schemas.

Architecture

sanityharness/
├── cmd/sanity/          # CLI entry point
├── internal/
│   ├── cli/             # Cobra commands
│   ├── config/          # TOML configuration
│   ├── errors/          # Error summarization
│   ├── result/          # Session/attempt types
│   ├── runner/          # Docker execution
│   └── task/            # Task loading
├── tasks/               # Embedded task files
└── containers/          # Dockerfiles

See docs/DEVELOPMENT.md for architecture details.

Version History

For a full summary of all changes since v1.6.1 (the entire 1.7.x line and further), see docs/ROAD-TO-V2-Overhaul.md. Current stable release: v1.8.10.

Contributing

Contributions are welcome! Please see docs/CONTRIBUTING.md for guidelines.

Quick start:

make pre-commit  # Run before committing
make test        # Run tests

License

MIT License