Why this isn't a web scraper

A scraper reads a fixed page and pulls fields. Ops Pilot operates a browser like a careful human would — it looks at the live page, decides what to do next, does it, and checks the result, step after step, across states it has never seen.

It's a framework for building reliable web automation that improves itself — perception, decision, action, and verification as separate, swappable, individually-trainable stages.

The per-step loop

Every step runs the same five stages. Each is the cheapest tool that's still confident; anything uncertain stops for a human.

flowchart LR
    O([Observe page]) --> C{classify<br/>stop-state?}
    C -- captcha / 2FA --> H[[🧑 Human]]
    C -- ok --> P[propose<br/>AX + vision candidates]
    P --> S{select<br/>cheapest-first cascade}
    S -- low confidence --> H
    S -- confident --> A[act<br/>pluggable executor]
    A --> V{verify<br/>did the page change<br/>as predicted?}
    V -- no, retry --> S
    V -- gave up --> H
    V -- yes --> O

The cheapest-first cascade (the "inner loop")

The whole cost story lives here. A decision falls through layers until one is confident — and expensive layers exist only as a catchall.

flowchart TD
    L1[1 · Deterministic rules<br/><i>FREE</i>] -->|miss| L2[2 · Fingerprint cache<br/><i>FREE</i>]
    L2 -->|miss| L3[3 · Tiny page-state classifier<br/><i>local, ~free</i>]
    L3 -->|miss| L4[4 · Micro-model selector<br/><i>local, cheap</i>]
    L4 -->|miss| L5[5 · Claude Haiku Set-of-Marks<br/><i>~$0.0026, budget-gated</i>]
    L5 -->|AX-blind| L6[6 · Vision-native / Human<br/><i>catchall</i>]
    style L1 fill:#1f8a4c,color:#fff
    style L2 fill:#1f8a4c,color:#fff
    style L5 fill:#d97757,color:#fff
    style L6 fill:#7a4ad9,color:#fff

Today, work is done by Layer 2 (cache) and Layer 5 (Haiku). Layers 1/3/4 are deliberately empty — they get earned from data once the logs show Haiku is being reached too often. Don't build models ahead of evidence.

The flywheel 🛞 — why it gets cheaper

This is the thesis. Haiku isn't the product; it's the teacher.

flowchart LR
    R[Run a task] --> L[Log every decision<br/>+ escalation]
    L --> D[(Corpora<br/>per layer)]
    D --> T[Train cheap<br/>local models]
    T --> P[Promote into<br/>the cascade]
    P --> R
    P -.->|Haiku reached less| $[💰 cost / task ↓]

Each run logs the expensive model's picks and the human's corrections. Those become a distillation corpus for tiny local models that slot into the cascade and answer for free — so the same task gets cheaper and more autonomous the longer it runs. A hard $5/week autonomous-spend cap keeps the teacher honest.

Inside the app

Control plane — operating posture at a glance: active runs, blocked runs needing a human, system health.	Structured capture — every data point is scoped to a domain · goal · scenario session with its own isolated Chrome.
Flywheel metrics — cost/day, layer mix, and reason codes over the live corpus. Cache-hit ↑ + cost flat = the wheel is turning.	Movement Playground — record real cursor paths vs. the model's motion, growing the corpus for a diffusion-based human-like input model.

The UI also includes a per-page-state Coverage tracker (drive data collection to the gaps), a live Model Test bench (run the SELECT cascade against any capture), and an Eval Runs / Model Registry for the grounding models.

The model roster

Nothing in the loop is a trained model yet — that's by design (collect data first, train later). Each has its own corpus, trainer, and a shared eval contract so it can be retrained independently as data accumulates.

Tech stack

Project status

Phase: multi-domain data collection — building the corpora that the cheap local layers will train on.

• ✅ Per-step loop, cheapest-first cascade, verifier, and all guardrails — built & tested
• ✅ Capture → corpus → coverage pipeline across multiple domains (Facebook, Indeed, …)
• ✅ Stop-state escalation verified on real reCAPTCHA & 2FA
• 🔜 First trainable model: the page-state classifier (L3)
• 🔜 Continuous retraining → the full self-improving flywheel

make dev

make setup     # first-time env setup only
make dev-stop  # stop everything
make doctor    # health check