AIC Evals

Verified benchmark artifacts, model patches, and integrity receipts for AIC, a closed-source, host-enforced AI software engineering runtime.

This repository publishes inspectable evidence from selected AIC evaluation runs. It does not contain the AIC product, source code, internal system prompts, private role artifacts, held-out tests, or credentials.

Headline evidence

Qwen3.8-Flash + AIC passed all 17/17 canonical checks on the DeepSWE v1.1 updo-policy-alerting task for an operator-reported model/API cost of $0.31.

The comparison above establishes task-difficulty context. AIC and the official model rows used different harnesses, operator protocols, and sampling designs, so it is not a controlled leaderboard.

Open the evidence bundle · Inspect the frozen patch · Review official-result context · Read rubric notes · Read the FAQ

Published results

Timing note: The observed AIC time includes six non-passing acceptance-probe executions—three probe_error and three behavior_failed outcomes—and the associated corrective turns. These included probe-construction errors and assertions superseded by corrected evidence, not canonical-verifier failures. The exact retry-adjusted duration cannot be isolated reliably, so effective time is reported only as a conservative upper bound: < 56m 55s.

Same-task context

The official DeepSWE v1.1 data includes repeated mini-swe-agent trials for this task. Most of the models below have four trials at each of five reasoning-effort levels:

These numbers provide task-difficulty context only. They are not an apples-to-apples ranking: the AIC run used a different harness and operator protocol, and one AIC attempt cannot estimate a stable pass rate. See the effort-by-effort comparison, official source links, snapshot hashes, and derivation.

Two verifier-selected event-ordering details are more specific than the public prose. They are documented neutrally in the task's rubric notes; the delivered candidate implements both and passes 17/17.

What each run contains

• the exact public task instruction given to the runtime;
• the frozen candidate patch produced by the run;
• the verifier's aggregate reward.json;
• public-safe result metadata and integrity hashes;
• a scorecard and its deterministic local renderer;
• the upstream license applicable to the patched project.

Raw test reports, test names, held-out test source, private rubrics, model transcripts, internal AIC role artifacts, and machine-local configuration are intentionally excluded.

Cost and timing are reported with an explicit scope. Model cost is operator-reported model/API spend, not total infrastructure cost. Observed AIC time runs from prompt acceptance to the delivered terminal state; end-to-end time additionally includes the post-delivery canonical verifier. Effective time excludes known non-passing probe and correction overhead, but is shown only as a strict upper bound because an exact counterfactual duration is not recoverable.

Evidence chain

Community submissions

When the public AIC desktop and CLI clients are released, developers will be encouraged to evaluate models and tasks and submit public-safe evidence bundles through pull requests.

Maintainer-published runs and community submissions will remain separate. Community results will disclose complete comparable attempt series and carry an evidence label—self-attested, artifact-checked, or, when supported by the clients, receipt-verified. A successful single attempt will not be presented as Pass@1 or a stable pass rate.

The intended bundle, privacy rules, attempt-disclosure policy, and review process are described in CONTRIBUTING.md. The clients should generate these bundles automatically so contributors do not need to expose AIC internals or protected verifier material.

How to interpret the evidence

A published run shows that the frozen patch identified by its SHA-256 digest produced the recorded verifier result under the stated environment. AIC itself is proprietary and is not distributed here, so this repository is an artifact record—not a fully reproducible copy of the orchestration system.

Attempt numbering is scoped to a materially stable AIC baseline. The first published result is post-fix attempt 1; earlier internal development and recovery rounds used materially different harness revisions and are not counted in this series. This repository does not yet represent a complete attempt ledger or a full-benchmark score. These results are independent publications and are not official leaderboard submissions.

See METHODOLOGY.md for the evidence protocol, FAQ.md for common interpretation questions, and DISCLAIMER.md for scope and limits.

For citation metadata, see CITATION.cff. Cite the repository together with the immutable run directory used by your analysis.

Repository layout

runs/
  <benchmark>/
    <task>/
      <model>/
        attempt-<nn>/
          README.md
          task.md
          model.patch
          patch.meta.json
          reward.json
          evidence.json
          manifest.json
          scorecard.png
comparisons/
  <benchmark>/
    <task>/
      README.md
      official-results.json
      RUBRIC-NOTES.md
assets/
  social-preview.png
  social-preview.html
  render-social-preview.ps1
submissions/
  <benchmark>/
    <task>/
      <model>/
        <generated-run-id>/

Licensing

Original documentation, metadata, and renderer code in this repository are licensed under the MIT License. AIC itself is not included and is not licensed by this repository. Candidate patches remain subject to the upstream project's license, included with each run where applicable.