Agent Reliability Protocol (ARP)

An executable protocol for measuring and controlling reliability failures in long-horizon LLM agents.

We do not ask only whether an agent eventually succeeded. We ask when the trajectory first became invalid, what evidence supported the next action, whether the system stopped at the right time, and whether an intervention improved the outcome without creating new harm.

中文简介：这是一个以可复现实验为中心的 Agent Reliability 研究项目。第一版只解决一个清晰问题：如何用任务合同和逐轮轨迹，客观区分“该停不停”（FM-1.5）与“没做完就停”（FM-3.1），并评估 runtime intervention 是否真的提高可靠性。

The research problem

Most agent evaluations compress a stochastic, state-changing trajectory into a final score. That hides at least four distinct questions:

1. Did the agent reach a valid terminal state?
2. Did it continue after completion or stop before completion?
3. Which event first violated the task contract?
4. Did a monitor fix the failure, or merely change what a judge saw?

ARP makes these questions explicit and separates the measurement instrument from the intervention.

What is in this repository now

Executable v0.1

• a normalized termination-trace format;
• a contract-based termination evaluator;
• objective labels for clean completion, FM-1.5 over-run, FM-3.1 premature termination, and incomplete/non-convergent runs;
• four offline fixtures with expected outputs;
• dependency-free unit tests and a command-line scoring tool;
• a checked-in machine-readable fixture result generated by the scoring command;
• a machine-readable benchmark registry and provenance policy.

Run it without an API key:

python3 -m unittest discover -s tests -v
python3 tools/score_examples.py
# Emit the same fixture results as machine-readable JSON.
python3 tools/score_examples.py --json

Central object: a contract episode

ARP represents a run as an episode rather than a final answer:

task contract
  → events and state transitions
  → first valid terminal state
  → first contract violation
  → propagation / recovery
  → final outcome and cost

The contract is declared before evaluating an intervention. When an executable oracle exists, it is primary. Human review and LLM judging are calibration or residual-semantic instruments, not automatic ground truth.

Termination protocol v0.1

ARP keeps the two MAST modes separate:

• FM-1.5 / over-run: a valid terminal commitment exists, but the trajectory continues beyond the allowed verification/confirmation budget;
• FM-3.1 / premature termination: the system declares completion before the terminal contract is satisfied;
• clean: the terminal contract is satisfied and the trajectory stops within the allowed budget;
• incomplete: no valid terminal contract is reached and no valid completion is declared.

See protocols/termination/termination_protocol.md and the reference implementation in src/agent_reliability/termination.py.

What makes ARP a research artefact

The project will be judged by five artefacts, not by the size of a link list:

Research boundaries

ARP does not claim that the current evaluator covers all MAST modes. The first release is deliberately narrow. The planned extensions are:

1. verification contracts: whether the evidence actually supports a claim;
2. action-consistency contracts: whether plan, tool call, and result agree;
3. state-integrity contracts: whether memory/context remains supported and current;
4. recovery contracts: whether a fault is contained and the final state is restored;
5. paired intervention experiments across repeated runs.

Relationship to existing research

ARP builds on, but is not a replacement for, strong neighbouring work:

• MAST supplies a failure vocabulary for multi-agent systems.
• Who&When formalizes agent/step failure attribution with a public benchmark.
• AgentProcessBench evaluates step-level process quality.
• TraceElephant emphasizes full execution observability and reproducible environments.
• CAR-bench, AgentSynth, and ClawBench make repeated-run or stateful reliability measurement concrete.
• Towards a Science of AI Agent Reliability argues for a multi-metric reliability profile rather than a single success score.

ARP's research question is narrower and complementary: can task contracts and per-event oracles make reliability failures operationally measurable and intervention claims causally testable?

Repository map

Roadmap

• v0.1: offline termination evaluator and fixtures.
• v0.2: adapters for MAST/AG2 traces and executable coding/math oracles.
• v0.3: verification and action-consistency contracts.
• v0.4: repeated paired intervention protocol with confidence intervals and harm analysis.
• v1.0: multi-domain benchmark release with public baseline results.

Status

This is an early research artefact. The code in v0.1 is real and testable; generalization claims are not yet established. Every empirical claim should be tied to a versioned fixture, oracle, run configuration, and result file.

Citation

See CITATION.cff. Versioned releases should be cited rather than the moving main branch.

Contributing

See CONTRIBUTING.md. New benchmark entries and protocol changes should include provenance, an explicit oracle, a limitation, and an offline check where possible.