SOAR Harness

中文说明

A file-based harness for running long-horizon coding-agent research sessions.

SOAR Harness was built during the SOAR 2026 LLM inference optimization competition, but SGLang is only the application case. The reusable artifact is the harness layer: Markdown contracts, project memory, skills, hooks, and small Python utilities that keep coding agents constrained, auditable, and resumable across many sessions.

This repository is best read as a working research artifact, not a packaged Python framework. It shows how to structure a real multi-session coding-agent project so that experiments can be resumed, reviewed, and trusted after the original chat context is gone.

At a Glance

Core Workflow

task_board.md
  -> experiment session
  -> capture.py raw logs + runs/<exp>/report.md
  -> /report
  -> experiment_history.md
  -> coordinator /review
  -> retro/, current_config.md, task_board.md
  -> next experiment

The goal is not to make one agent "smarter" in a single prompt. The goal is to make many agent sessions behave like a disciplined engineering process:

• every experiment starts from an explicit task and prediction
• benchmark and eval commands are captured into raw logs
• each run ends with a structured report and validation
• a coordinator session absorbs reports and updates shared memory
• high-risk source edits are isolated in managed Git worktrees
• GPU/server ownership is tracked so cleanup is scoped to owned processes

Harness Components

Design Principles

• Light entry, many pointers: CLAUDE.md stays small and points to memory, protocol, and operations files instead of becoming a long manual.
• Two-stage closure: experiment sessions produce run-level reports with /report; coordinator sessions absorb cross-run lessons with /review.
• File is state: tasks, reports, runtime state, locks, and retrospectives are stored on disk so any new session can cold-start from files alone.
• Programmatic guarantees first: constraints such as evidence capture, GPU locking, report validation, and worktree isolation are implemented as scripts where possible.

Agent Entry Contract

CLAUDE.md is the session entry point. It keeps only the hard rules and file navigation needed at startup: role boundaries, GPU/process safety, evidence requirements, report/review discipline, and pointers into _lcy_work/memory/.

Skills

.claude/skills/ defines the repeatable agent workflows:

• report: closes one experiment by completing report.md, updating experiment_history.md, marking the task as ready for review, and running report validation.
• review: used by the coordinator to absorb completed experiments, update retro/ knowledge, revise current_config.md, and keep task_board.md focused on active work.
• spec-worktree: provisions and finalizes isolated SGLang worktrees for risky source-code experiments.
• submit: turns a validated experiment configuration into a competition submission package.

Memory Files

_lcy_work/memory/ is the shared state layer. The most important files are:

• task_board.md: the active queue, priorities, current assumptions, and rejected directions.
• current_config.md: the current best known configuration and hardware facts.
• experiment_history.md: one-line index of completed runs and links to reports.
• harness.md: the collaboration contract between human, experiment session, and coordinator session.
• coordinator_protocol.md: the automatic dispatch loop and sub-agent contract.
• retro/: distilled lessons grouped by use case instead of by date.
• design_rationale.md: why the harness is structured this way and which heavier designs were rejected.

Programmatic Guardrails

The harness uses code for the constraints that should not rely only on prompts:

• _lcy_work/infrastructure/capture.py stores full stdout/stderr and metadata for benchmark/eval commands, while showing only a short tail to the agent.
• _lcy_work/infrastructure/validate_report.py checks report structure, raw evidence, metric consistency, and the correctness threshold.
• _lcy_work/infrastructure/coordinator/runtime.py manages GPU locks, runtime state, server launch, PID/PGID registration, and owned-process cleanup.
• _lcy_work/infrastructure/coordinator/worktree.py creates, binds, finalizes, and releases experiment worktrees.
• .claude/hooks/require_capture.sh blocks direct benchmark/eval commands that are not wrapped by capture.py.

Session Types

Human / Harness / Agent Split

The human sets high-level direction, decides when to stop, and reviews strategic tradeoffs. The harness stores state, enforces workflow, and preserves evidence. The coding agent reads the scoped context, edits code, runs experiments, writes reports, and hands results back to the coordinator.

This separation is what lets the project run across many sessions without turning the task board into a chat log or losing the reason why a direction was accepted, rejected, or deferred.

Repository Layout

.
├── CLAUDE.md                 # agent entry rules and navigation
├── .claude/
│   ├── hooks/                # prompt-time guardrails
│   └── skills/               # report/review/worktree/submit workflows
├── _lcy_work/
│   ├── docs/                 # research notes and handoffs
│   ├── external/             # vendored evaluation/training references
│   ├── infrastructure/       # runtime, capture, validation, sweep scripts
│   └── memory/               # task board, config, protocol, retrospectives
└── sglang/                   # application case submodule

What Is Not Included

This repository intentionally excludes model weights, Hugging Face artifacts, large run outputs under _lcy_work/runs/, submission tarballs, and teammate workspace contents. The checked-in files are the harness, memory, scripts, and references needed to understand the workflow.

Reusing the Pattern

For another coding-agent research project, the portable pieces are:

1. Keep a short CLAUDE.md entry file with only hard rules and navigation.
2. Split run-level closure (/report) from cross-run absorption (/review).
3. Make task_board.md, current_config.md, and experiment reports the source of truth instead of the chat transcript.
4. Wrap result-producing commands with a capture layer and validate report claims against raw artifacts.
5. Use worktree and runtime helpers for the constraints that should not rely on prompt discipline.

Case Study

The SOAR/SGLang work is the concrete testbed: agents used this harness to read code, modify inference paths, run correctness and speed gates, profile kernels, package submissions, and review failed directions. The important artifact here is not a single optimization. It is the workflow that made repeated coding-agent experiments inspectable and recoverable.