software-factory-reliability

Coding agents fail in ways that look strange until you treat the software factory around them as a distributed system. Then the strangeness resolves into a short list of familiar failures: stale workers, duplicate effects, lost work, false completion, retry storms, and publication nobody can verify.

This repository is the executable companion to Software Factories Are Distributed Systems. It holds small examples and fault-injection drills for the failure modes the essay describes, plus a checker that reads a description of your factory and tells you which of those boundaries you have left open.

Want to see the core problem? Run this:

git clone https://github.com/sjarmak/software-factory-reliability
cd software-factory-reliability
make demo

See a software factory fail in five minutes

make demo builds two factories that differ in exactly one place: where the ownership fence is checked. It runs the same fault against both, and narrates the two event logs the runs actually produced.

One work item, one lease expiry, two factories. The factories differ in
exactly one place: where the ownership fence is checked.

UNSAFE  (fence checked by the writer, then an unconditional write)
  generation 7 holds the claim and has prepared artifact-g7
  generation 7 loses ownership
  generation 8 becomes current
  generation 8 writes artifact-g8
  generation 8 records the work complete
  generation 7 writes artifact-g7 anyway
  destination applies it, nothing checks the generation
  generation 7 records the work complete
  destination holds artifact-g7                               FAIL

PROTECTED  (fence checked at the destination, atomically with the write)
  generation 7 holds the claim and has prepared artifact-g7
  generation 7 loses ownership
  generation 8 becomes current
  generation 8 publishes artifact-g8
  generation 8 records the work complete
  generation 7 attempts to publish artifact-g7
  destination rejects the stale writer, current is 8
  generation 7 attempts to record completion
  destination rejects the stale completion
  destination holds artifact-g8                               PASS

The unsafe factory lost generation 8's work and published a superseded artifact under a completion record that says everything went fine. This is the failure the essay describes in "Authority has to expire cleanly".

Nothing in that output is a canned transcript. Every line is rendered from an event in out/evidence/stale-writer-completes-{unsafe,protected}.json, an unrecognized event kind prints raw, and a run whose oracle verdict is not the expected one fails the demo. Requirements are Python 3.10 or newer and pip install -r requirements-dev.txt.

The ideas from the essay, and where to watch each one break

Every MODE=unsafe command above is expected to exit 2. An unsafe control that passes means the fault never reached the boundary, which makes it a broken test rather than a safe system. Swap in MODE=protected and the same drill exits 0.

The shape of the thing

flowchart LR
  WI["work item<br/>work_id, stable for its whole life"] --> SCHED["scheduler"]
  SCHED -->|"claim: work_id at generation 7"| W["agent worker<br/>attempt_id, session_id"]

  subgraph AUTH["authority boundary: the destination checks, the writer does not"]
    direction TB
    LEDGER[("work ledger<br/>claims, generations, artifact facts")]
    DEST["external effect<br/>code host, deploy, ticket"]
  end

  W -->|"write under generation 7"| LEDGER
  W -->|"one effect per effect_id"| DEST
  LEDGER -->|"intended state"| RECON["reconcile loop<br/>level-triggered, rereads truth"]
  DEST -->|"observed state"| RECON
  RECON -->|"repair or escalate"| LEDGER

Everything inside the boundary evaluates the generation atomically with the write. A worker that lost its claim still holds credentials and still believes it is current, so the thing that reliably stops it is a destination that refuses. The full authority-plane, identity, and campaign diagrams render in docs/diagrams/.

Three ways to use this

Five minutes: break a factory. Run make demo, then run one drill in both modes and diff the two evidence files. The drill directory holds the fault placement, the oracle, and what the evidence must contain.

make drill DRILL=effect-commits-ack-is-lost MODE=unsafe     # exits 2
make drill DRILL=effect-commits-ack-is-lost MODE=protected  # exits 0
diff out/evidence/effect-commits-ack-is-lost-{unsafe,protected}.json

Fifteen minutes: read the five patterns that carry the rest. Each pattern page opens with a compact box (problem, rule, required property, the wrong shape, the right shape) and then goes deep: the invariant, the enforcement boundary, the falsifying test, and the evidence retained.

1. Stable work identity: one logical item, one id, for life.
2. Fenced authority: ownership is not authority.
3. Effect identity: unbounded attempts, one physical effect.
4. Verify before publish: the verdict binds to an immutable artifact.
5. Reconciliation: every event path has a level-triggered twin.

The other eleven pages are indexed in patterns/.

Use it on your own factory. Read the installation first, then decide. The scaffold finds the effects your factory performs on something outside itself and writes a probe pack naming them and the files they were found in; the derivation reports, per effect, whether the call sites carry anything a destination could use to tell a repeat from a new request.

python3 src/factory_check.py probes-init /path/to/your/factory --write probes.yaml
python3 src/factory_check.py infer       /path/to/your/factory --probes probes.yaml
python3 src/factory_check.py review      out/factory.derived.yaml

Every identity comes back unknown on the first pass, with the reason on the line above it, because a scaffolded pack declares none and no scan of your code can establish what your destination does with a repeat. Deciding those is the part that needs you, and the scaffold offers the flags it saw at the call sites as candidates rather than applying any of them.

That order matters. init writes a blank contract, and a blank contract can be edited to a decided value while the factory stays exactly as it was -- findings go green and nothing is fixed. Write one when you want to state the intent independently, then hold the two against each other:

python3 src/factory_check.py init factory.yaml
python3 src/factory_check.py reconcile factory.yaml /path/to/your/factory --probes probes.yaml

Reconcile reports DRIFT where the contract claims an identity the call sites do not carry, and UNDECLARED for an effect your factory performs and your contract never mentions. DRIFT is what a contract edited ahead of the code looks like: editing the claim back down turns it into OPEN, an undecided boundary, and only the call sites can turn it into CONFIRMED.

An identity worth a sentence stays a sentence. Write the composite in effect_identity, and name the token that carries it through the code in effect_identity_key; reconcile compares the token when one is named and prints the line to add when one is not. A confirmation on a named key says the token is at every call site, and does not say the token's runtime value is the identity the sentence describes -- reconcile prints that limit on those rows rather than letting the word CONFIRMED carry a claim no scanner can make.

A contract that cites the code is worth more than one that describes it, and those citations rot silently. cites resolves each path/to/file.go:120-130 against your source roots:

python3 src/factory_check.py cites factory.yaml /path/to/your/factory /path/to/a/second/repo

Ours carried a refuted citation for a day and contradicted itself on the same line number in two places. It reports the file that no longer exists and the range that runs off the end of one that does; it deliberately does not claim the cited line still says what the contract says it says, because a line-pinned claim invalidates its own refutation as soon as anything above it moves.

QUICKSTART.md walks the full first session under an hour, and docs/contract-reference.md documents every section and all twenty-five rules.

What is in here

Start with examples/README.md if you would rather find the defects yourself before the checker names them.

Going deeper

• docs/design.md: the premise, why work identity and authority move in opposite directions, and how the pieces fit.
• docs/evidence-methodology.md: what declared, enforced, and fault-tested mean; the four basis labels; and why this kit computes no maturity score.
• docs/contract-reference.md: every contract section and every rule the review can emit.
• docs/recipes/: four factory shapes worked end to end, plus a recovery path for a factory that is already broken.
• docs/observability/: event conventions, latency expectations, sample events, and queries.

Contributing

make check runs the schema checker, the prose checker, the test suite, and all nine executable drills. Prose rules are in docs/style.md; they are enforced, not advisory, and the em dash rule has no exemptions anywhere in the repository. A new pattern page must name an invariant, an enforcement boundary, a falsifying test, and the evidence retained. A new drill must fail in unsafe mode.