AgentCompiler

Does your LLM agent repeat itself enough to be worth optimizing?

This tool measures it. It reads an agent's execution traces, groups runs into patterns, and tells you whether your spend is concentrated in a few recurring shapes, which is the precondition for routing the repetitive part to something cheaper.

Spend, not run count. The two diverge, and only one of them shows up on your bill: in the example below three patterns account for 90% of the spend while being 21% of the runs, because the frequent runs are the cheap short ones.

It runs entirely on your machine. No traces leave it, no account, no endpoint to configure.

See an example report →

That page is the tool's real output on 200 synthetic runs. It is regenerated on every push from the dataset in examples/, so it always shows what the current code does.

---

Quick start

pip install -e ".[clustering]"
agentcompiler quickstart

quickstart reads the Claude Code transcripts already on your disk (~/.claude/projects), converts them into traces, groups runs into patterns, and opens a coverage report: what your agent costs, how many distinct shapes the spend is split across, and whether that distribution is concentrated enough for compilation to make sense.

Everything runs locally — no trace leaves the machine. The report is the only artifact meant to be shared, and it contains shapes, counts and costs only, never the text of the traces.

Not a Claude Code user, or just want to try it dry — the repo ships a synthetic dataset:

agentcompiler analyze examples/traces-demo.jsonl

Or generate your own, at any size:

agentcompiler demo --traces 900 --output traces.jsonl
agentcompiler analyze traces.jsonl

If something does not start, agentcompiler doctor says what is missing and how to install it.

Reporting the result

agentcompiler share

Prints a block to paste elsewhere: verdict, pattern count, runs, and the two coverage figures (share of spend and share of runs). Nothing else: not the spend itself, not the agent's name, not one line of trace content. Percentages describe how a cost is distributed without revealing what it is.

If the project was useful to you, that block is the most useful thing you can send back: every result is a data point on how much real agent workloads actually repeat.

Which input fields actually matter

agentcompiler fields traces.jsonl

Classifies every input field as decisive (changing it changed the result), inert (it varied and the result did not), or undetermined. An inert field can come out of a cache key, which widens the key and catches more traffic. No fine-tuning, no model to validate.

Attribution needs pairs of runs differing in exactly one field, because a pair differing in two fields cannot say which one caused the change. Once a field is proven inert it stops confounding the others, so the analysis eliminates it and runs again: on a ticket classifier this is what turns received_at from "undetermined" into a second field you can drop.

On agents whose input is one blob of request text, it finds nothing and says so. That is a property of the data, not a failure of the measurement.

The coverage report carries the same analysis in a Cache keys you could widen section, so you get it from quickstart without running anything else. Note that field names appear there, alongside the tool names that were already shown.

Seeing the ingestion pipeline

pip install -e ".[server,dev]"
agentcompiler demo --traces 900 --serve      # read API at http://localhost:8080
cd dashboard && npm install && npm run dev   # dashboard at http://localhost:3000

Generates synthetic traffic and pushes it through SDK → collector → queue → worker → repository, printing volume, cost and signature distribution. Traces are spread over 72 hours (--spread-hours) with a day/night profile: flat volume would make a broken time axis impossible to notice.

With full infrastructure:

docker compose up -d          # Postgres+pgvector, MinIO, ElasticMQ
cp .env.example .env
agentcompiler migrate         # apply db/migrations/
agentcompiler worker &        # consume the queue
agentcompiler collector --port 8000
agentcompiler api --port 8080
cd dashboard && npm install && npm run dev

---

Scope

The compilation itself — fine-tuning, live routing to cheaper paths — is Phase 2, and it is deliberately blocked on evidence: there is no point building it until the measurement says some real workload is concentrated enough to be worth it.

---

Instrumenting an agent

Direct use

from agentcompiler.sdk import Tracer, SpanKind

tracer = Tracer(
    tenant_id="acme",
    agent_id="ticket-classifier",
    agent_version="1.4.0",
    endpoint="https://ingest.example.com",   # or AGENTCOMPILER_ENDPOINT
    api_key="...",                            # or AGENTCOMPILER_API_KEY
)

with tracer.trace(session_id="conv-42") as trace:
    with trace.span("classify", kind=SpanKind.LLM) as span:
        span.set_input({"ticket": text})
        response = call_model(text)
        span.set_reasoning(response.thinking)
        span.set_output(response.text)
        span.set_model_usage(
            model="claude-opus-5",
            input_tokens=response.usage.input_tokens,
            output_tokens=response.usage.output_tokens,
        )

Ids, timings, the span tree, cost, structural signature and delivery are all automatic.

LangChain / LangGraph

from agentcompiler.sdk.integrations.langchain import AgentCompilerCallbackHandler

handler = AgentCompilerCallbackHandler(tracer)
chain.invoke(payload, config={"callbacks": [handler]})

from agentcompiler.sdk.integrations.langgraph import graph_config

graph.invoke(state, config=graph_config(tracer, thread_id="conv-42"))

OpenAI Agents SDK

from agentcompiler.sdk.integrations.openai_agents import install

install(tracer)   # registers a TracingProcessor alongside existing ones

Recording which path served a request

from agentcompiler.sdk.integrations.routing import route_in_span

outcome = route_in_span(trace, router, payload, llm_runtime=call_the_agent,
                        llm_cost_usd=measured_llm_cost)

The span then carries execution_path, the pattern and artifact that served it, and the reason — including when the reason was "not compiled", because without the non-decisions there is no denominator and coverage cannot be computed.

It also carries the counterfactual cost: what that request would have cost on the full LLM path. Observed spend goes down when routing works, so spend alone cannot tell a saving from a quiet week. The counterfactual has to be recorded as it happens, since once the compiled path served the request the LLM call never occurred and the number can only be estimated afterwards.

Neither cost is written to span.cost. Trace.recompute_rollups sums that field, and the router's figure would be counted twice whenever the path was the LLM. They live in attributes, readable without entering the totals.

Two guarantees from the SDK

1. It never breaks your agent. Every error path is caught and downgraded to a log; no SDK exception propagates into the application. An unreachable collector costs at most one lost trace (counted in tracer.exporter.stats).
2. Redaction before egress. Sensitive keys, emails, cards and tokens are replaced inside the client process, before any network I/O. Payloads over 64 KB are truncated, and truncation stays on even with redaction disabled: it protects the pipeline, not just privacy.

For clients who cannot let text leave their perimeter, capture_payloads=False keeps payload shape and size — enough for volume, cost and structural signature — without the content.

---

Architecture

   client agent
        │  SDK (redaction, batching, non-invasive)
        ▼
   HTTP collector ──202──▶ SQS queue ──▶ Worker ──▶ Postgres  (metadata)
   (authenticates, validates,                   └─▶ S3        (payloads)
    offloads large payloads)                    └─▶ pgvector  (embeddings)
                                                       │
                                    Read API ◀─────────┘
                                        │
                                    Dashboard (read-only)

Decoupling through a queue is the central choice: the collector answers in milliseconds even when the database is slow, so storage latency never propagates back to the client's agent.

The response is 202 Accepted, not 200: the trace has been taken in charge, not yet persisted.

Error handling in the worker

Past max_receives attempts even a transient error goes to the dead-letter queue: a message that always fails is indistinguishable from a malformed one, and leaving it in circulation blocks the queue.

---

Layout

src/agentcompiler/
├── schema/          shared data contract (ids, records, patterns, artifacts)
├── pricing.py       cost model (verified Anthropic price table)
├── sdk/             tracer, redaction, exporters, framework integrations
├── ingestion/       collector, queue (SQS/in-memory), worker
├── storage/         repository (Postgres/in-memory), blobstore, embeddings
├── api/             dashboard read API
├── compile/         Clusy bridge: bundle import, divergence export
├── integrations/    trace conversion from other systems (Claude Code)
├── clustering/      grouping into patterns, locally (numpy only)
├── serving/         model serving, declarative transforms, artifact cache
├── promotion.py     control plane: human approval and audit trail
├── harness/         shadow-mode validation harness
├── routing/         routing layer
├── fields.py        per-field attribution: which inputs change the result
├── report.py        coverage report, self-contained HTML page
├── firstrun.py      first run: quickstart, analyze, doctor, share
├── demo.py          synthetic traffic generator
└── cli.py
db/migrations/       Postgres schema (5 migrations)
examples/            synthetic example dataset
dashboard/           Next.js, read-only
docs/data-schema.md  data contract documentation
tests/               488 tests, no infrastructure required

---

Declared boundaries

The scope lives in the code rather than being left implicit.

What is complete: shadow-mode harness with persistence, model serving, promotion control plane, routing layer with its gates.

What is not implemented, and raises instead of degrading silently:

• EmbeddingMatcher — it would send traffic to the compiled path based on a confidence nobody has computed.
• A model-serving client against a real endpoint. deterministic_code artifacts used to be listed here too, waiting for a sandbox; they now carry a declarative rule document instead of code, so there is no execution path to contain.

EmbeddingMatcher is wired now, but its distance threshold has no default: calibrate_threshold derives it from a human-labelled set, and without that measurement you do not have a matcher. Choosing the threshold by eye is choosing the rate of confidently wrong answers by eye.

Compiled paths that are not code

An artifact of kind deterministic_code is a pure function from input to output. The obvious way to ship one is to put code in the bundle and run it, and the equally obvious conclusion is that you then need a sandbox.

There is no sandbox here because there is nothing to contain. The artifact is a document of rules that the runtime interprets, with no eval, no exec, and no reach beyond the request payload:

{"version": 1,
 "output": {"object": {"queue": {"template": "{category}-{priority}"}}}}

The rule set is deliberately small and not Turing complete: const, field, object, template, lookup, when. It covers what compiled paths exist for — routing, mapping fields, composing strings, table lookups — and rejects the rest rather than growing.

Three reasons this beats a sandbox. A portable Python sandbox is not achievable with the standard library, since Windows has no seccomp, namespaces or resource limits, and a guarantee that holds on one platform and degrades on the other is worse than none. The control plane requires a named human to approve every promotion, and you can only approve what you can read. And the hash still only proves the bundle has not changed, so imperfect containment would move the false confidence rather than remove it.

If a pattern needs more than this, it is not a deterministic_code artifact: it is a model, and it goes through the model door with the shadow validation that door implies.

The promotion thresholds (divergence ≤1%, ≥1000 samples, savings ≥30%) are a declared starting point, not a validated one: the roadmap says "to be agreed with the partner".

Three invariants covered by tests:

1. Safe by default. No pattern, no deployed artifact, no measured divergence, low confidence, matcher error, compiled runtime error → LLM path.
2. Every decision is recorded, including the "no". Without the denominator neither coverage nor savings can be computed.
3. No automatic promotion. A named human actor is required; system, auto and empty strings are rejected.

---

Costs

The built-in price table covers Anthropic models (verified as of 2026-08-07). For other providers no rates are invented: an unknown model yields cost 0 with pricing_source="unknown", so a missing price stays visible in the dashboard instead of being silently wrong.

agentcompiler pricing                                    # loaded price table
export AGENTCOMPILER_PRICING_FILE=./pricing.json         # add other providers

[{"model": "gpt-x", "provider": "openai", "input_per_mtok": 1.0, "output_per_mtok": 4.0}]

If the provider reports the real cost, that always beats the estimate (Cost.estimated = False); only a reported cost can be used for billing.

---

Tests

python -m pytest          # 488 tests, ~4s, no infrastructure

The whole suite runs on in-memory backends — InMemoryQueue reproduces SQS visibility timeouts and at-least-once delivery, so a worker that passes here behaves the same way in production. This is deliberate: a suite that needs Postgres and MinIO gets run rarely.

CI covers Linux and Windows on Python 3.10–3.12. That is not redundancy: two of the defects found during development existed only on Windows — a CLI that died in cp1252 while printing its results, and a timeout that never expired because of clock resolution. Both would have been green on Linux alone.

Coverage by area: data contract and id determinism, span structure and cost, redaction, SDK non-invasiveness, collector authentication and offload, worker idempotency and error classification, read API aggregations, harness and routing safety gates.

---

Real traces without a partner

agentcompiler ingest-transcripts --project C--Users-you-Desktop-Project

Converts Claude Code transcripts into traces. Granularity is a decision: a turn is huge and unrepeatable, whereas one model request with its tools is the unit that recurs. On this project's own session — 588 traces, 16 signatures, 4 signatures covering 80% of traffic.

It does not replace a partner's traces for the go/no-go verdict: a development agent has a different distribution from a production one. It replaces synthetic data for calibrating everything else.

agentcompiler cluster traces.jsonl --output ./clustering

Clustering runs locally, with numpy only: traces never leave the machine. It is semantic within each signature bucket, not instead of the signature — the signature captures how the agent worked, not what on, and alone it does not identify a task. Across 666 real traces, 6 of 16 buckets split into several semantic clusters: that is the empirical measure of how much the distinction was needed.

Deterministic by construction: pattern_ids derive from the clustering, and without determinism the same group would produce different ids on every run.

The full chain, with no external services:

agentcompiler ingest-transcripts --project <project> --output traces.jsonl
agentcompiler cluster traces.jsonl --output ./clustering
agentcompiler bundle ./clustering --output ./run-003 --run-id run-003
agentcompiler import-clusy ./run-003          # patterns → candidate

The bundle produced here goes through the same importer that would validate an external producer's: no privileged path, and pattern_ids are recomputed and verified exactly as for any other bundle.

---

The compilation cycle

# 1. Clusy delivers a bundle
agentcompiler import-clusy ./run-001 --dry-run   # validate without writing
agentcompiler import-clusy ./run-001             # patterns → candidate, artifacts → draft

# 2. Shadow validation, then promotion — with a named human actor
agentcompiler artifact assess   art_xxx
agentcompiler artifact validate art_xxx --actor first.last
agentcompiler artifact approve  art_xxx --actor first.last --reason "1200 samples, 0.2%"
agentcompiler artifact deploy   art_xxx --actor first.last

# 3. Immediate rollback, no thresholds
agentcompiler artifact rollback art_xxx --actor first.last --reason "quality degraded"

# 4. Divergences go back to Clusy
agentcompiler export-feedback --tenant acme --output divergences.jsonl

No path leads from draft to deployed: skipping validation is the mistake the control plane exists to make impossible. The invariant "no production without measured divergence" is enforced at four independent levels — a Pydantic validator, a database CHECK, the control plane, and a re-check by the router before every routing decision.

The bundle manifest carries the inputs the pattern_id derives from, not just the id: the importer recomputes it and rejects the bundle if they disagree. A mismatch between the two sides becomes an import error instead of a production incident. Full contract in docs/clusy-handoff.md.

---

Comparators: the error you cannot see

The divergence rate rests entirely on how equivalence is defined. The two possible errors do not cost the same: a false DIVERGENT discards a good pattern — visible, recoverable. A false MATCH lowers the measured rate, lets an artifact clear thresholds it should not have, and ships it to production while the measurement claims everything is fine.

Hence the rule running through the module: when in doubt, DIVERGENT. Judge unreachable, answer uninterpretable, no level willing to decide — all fall back to "different".

SemanticComparator is a cascade, the same principle as the product itself:

exact → normalized → lexical → [embedding] → LLM judge

Every level may abstain; the judge costs a call and is consulted only on the cases no free check resolves.

agentcompiler calibrate pairs.jsonl --compare-all

An uncalibrated comparator is a number without a unit. The report ranks by false MATCH, not accuracy — a comparator that calls everything equivalent can have the same accuracy as a cautious one and be unusable.

---

Documentation

• docs/data-schema.md — full data contract
• docs/clusy-handoff.md — Clusy ↔ infrastructure handoff
• docs/phase1-status.md — Phase 1 status and checks
• docs/phase2-status.md — Phase 2 status and checks
• Claude.md — roadmap and vision

Some documents under docs/ are still in Italian: they are working notes for the Clusy handoff, not part of the path a new user follows.

License

Apache License 2.0. The LICENSE file is the Apache Software Foundation's original text, unmodified; copyright attribution lives in NOTICE.