FrontierHarness Eval

Explore the live results →  ·  Read the blog →

Similar pass rate. 17.5x cost differences.

We ran the same Kimi K3 model through nine coding-agent harnesses—12 configurations in total—on the same 30 software-engineering tasks. With the model, tasks, and runtime held constant, changing the harness changed pass rate, cost, cache behavior, and speed.

Full results

The interactive report includes failed runs, total cost per task, cache behavior, speed, and task-level results. For the evaluation design and analysis, read the launch article.

What is in this repository

.
├── benchmark.json              # Public benchmark definition
├── metadata/
│   ├── difficulty.json         # Difficulty assignments and source methodology
│   └── harness-versions.json   # Harness versions used for the run
├── results/
│   └── eval-data.json          # Normalized aggregate and task-level results
└── tasks/<task>/
    ├── instruction.md          # Prompt shown to every harness
    └── task.toml               # Public task metadata and environment definition

The repository intentionally contains results and task definitions only. Internal infrastructure, credentials, runtime identifiers, private evidence, solutions, and deployment configuration are not included.

Methodology

Tested harness configurations

• FrontierHarness v1.0 focuses on software engineering contexts and terminal-based tasks. It may not generalize to other areas of knowledge work.
• Evaluated on Runta agent runtimes. For each task, all harnesses and the environment defined in task.toml are prepared once as a golden checkpoint. Every run is a fresh restore with identical vCPU, memory, disk size, disk contents, and memory state.
• Kimi K3 is served by Fireworks.

Benchmark scope

• 30 tasks: 21 Terminal-Bench tasks and 9 DeepSWE tasks
• 9 harnesses: Claude Code, Codex, DeepSeek Harness, Exo Harness, Hermes, Kimi Code, Oh My Pi, OpenCode, and Pi
• 12 configurations: one canonical result for every task and harness-configuration pair
• 360 evaluations: complete task-by-harness coverage
• Deterministic scoring: verifier-based pass/fail outcomes
• Comparable cost: first-turn cache reads repriced consistently across harnesses

See benchmark.json for the public benchmark definition and results/eval-data.json for the complete normalized result set.

Use the data

jq '.harnesses[] | {name, successful, effective_cost_per_pass}' results/eval-data.json

Every task directory contains the exact public instruction and task metadata used by the benchmark.

Sponsor

Runta provided the isolated runtimes and Golden Checkpoint restores used across all 360 evaluations.