checkpoint-run

Checkpoint-and-resume runner for long multi-step jobs on flaky connections.

New market (2026-08-18 round): not an AI wrapper, not a prompt pack, not a narration pack. Replacement product after the critical audit marked nz-swms-builder NO-GO (false legal premise — see ../AUDIT-REPORT.md). This product makes no legal, safety, or regulatory claims: it is a purely functional developer utility.

Hardened (2026-08-19, round 2): step timeouts (--timeout / --step-timeout), --continue-on-error, checksum-verified step completion (--step-artifact), --dry-run preview, and a shell-execution safety banner.

Hardened again (2026-08-19, round 3): per-attempt retry with exponential backoff (--retry N / --backoff), a failure cap for --continue-on-error (--max-failures), per-step output capture (--step-log / --logs-dir), and a GitHub Actions CI workflow (.github/workflows/ci.yml). See the flags below.

Problem

Rural NZ broadband is a documented crisis (Federated Farmers: "rural connectivity crisis demands urgent action" — scoop.co.nz; "Northland farmers losing time and money to poor internet" — ruralnewsgroup). On a dropped hotspot connection, long jobs (LLM batch, video render, big download) die mid-run and restart from zero. This machine's own AGENTS.md notes: "drops kill long API runs."

Value proposition

• Buyer: NZ rural/dev users running long jobs; any developer with flaky local networking.
• Value: split a long job into named steps (id:command); completed steps are persisted to a local JSON state file; re-run resumes where the job died instead of starting over.

What's inside

Run instructions

cd checkpoint-run

# Run a 3-step job; if the hotspot drops mid-run, just re-run the same command.
uv run python -m checkpoint_runner.cli \
  --job render-batch \
  --state-dir . \
  --step download:"curl -L -o model.bin https://example.com/model.bin" \
  --step convert:"ffmpeg -i in.mov out.mp4" \
  --step upload:"rsync -P out.mp4 backup:render/"

# Inspect progress / clear state
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . --status
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . --reset

# Preview what would run WITHOUT executing anything (shell-safety check)
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step download:"curl -L -o model.bin URL" --step render:"ffmpeg -i in.mov out.mp4" --dry-run

# Per-step time budget (a step exceeding it is recorded 'timeout' and stops the run)
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step download:"curl -L -o model.bin URL" --step render:"ffmpeg -i in.mov out.mp4" \
  --timeout 3600 --step-timeout download:600

# Keep running later steps even if one fails
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step a:"cmd" --step b:"cmd2" --continue-on-error

# Checksum-verified completion: re-runs a 'done' step if its output is missing/changed
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step render:"ffmpeg -i in.mov out.mp4" --step-artifact render:out.mp4

# Retry a flaky step up to 2 extra times with exponential backoff
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step download:"curl -L -o model.bin URL" --retry 2 --backoff 1.0

# Stop after 2 recorded failures even with --continue-on-error
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step a:"cmd" --step b:"cmd2" --continue-on-error --max-failures 2

# Capture every step's stdout+stderr to logs/<id>.log (or --step-log id:path per step)
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step render:"ffmpeg -i in.mov out.mp4" --logs-dir logs

Or install it (from the v0.1.0 release):

python -m venv .venv && .venv/bin/pip install \
  https://github.com/agenticaotearoa/checkpoint-run/releases/download/v0.1.0/checkpoint_run-0.1.0-py3-none-any.whl
checkpoint-run --job demo --step "a:echo hi" --step "b:echo bye"

No runtime dependencies: the wheel is pure Python (3.10+) on the standard library.

How it behaves

• Resume: re-running skips steps already done (same id + same command) and starts at the first incomplete step.
• Failure: a failing step is recorded as failed (exit code stored) and later steps do not run (unless --continue-on-error); re-running retries from that step.
• Timeout: with --timeout SECONDS (or --step-timeout id:SECONDS), a step that exceeds its budget is killed (default runner) and recorded as timeout; later steps stop unless --continue-on-error. Re-running retries the timed-out step. The watchdog only observes a single budget; it does not bound grandchild processes (documented limitation).
• Artifact checksums: with --step-artifact id:PATH, the step's output file hash (sha256) is recorded on completion. On resume, a done step is trusted only if the artifact still exists with the same content — missing or changed output (deleted file, restored snapshot, corrupted write) forces the step to re-run. A step that claims success but produces no declared artifact is recorded failed (exit -2) and retried.
• Dry run: --dry-run prints which steps would run and which would be skipped, executing nothing — a safe way to inspect the plan before letting commands touch the machine.
• Retry with backoff: --retry N re-attempts a failing/timeout step up to N extra times within the same run — a transient hotspot drop often succeeds on a second try. Between attempts the runner waits --backoff * 2^(attempt-1) seconds (default base 1.0 s). Only the final attempt's result is persisted; the state file records the attempt count. A step that succeeds but still lacks its declared artifact is also retried.
• Failure cap: with --continue-on-error, --max-failures N stops the run once N failed/timeout steps have been recorded (later steps stay pending and run on the next resume). Without --continue-on-error the run already stops at the first failure.
• Step logs: --step-log id:path (or --logs-dir dir for all steps) captures a step's stdout+stderr into a file instead of the terminal. The file is truncated per attempt, so a retried step's log shows its final attempt. (Output capture applies to the built-in shell runner; a custom injected runner controls its own I/O.)
• Changed command: if a step's command text changes, it is treated as pending again.
• Force: --force re-runs everything from scratch (fresh state).
• State: .checkpoint-run-<job>.json next to the job (or in --state-dir), written atomically (temp file + rename) immediately after each step exits — a kill/drop never loses completed work. No cloud, no network, no LLM, no secrets.

Safety note

Steps are executed through the shell exactly as you write them (subprocess.run(shell=True)) — the CLI prints a reminder of this on every run, and --dry-run previews the plan without executing. You are responsible for the commands you pass. This is a resume orchestrator, not a sandbox.

Tests (passing)

python -m pytest tests/ -v
# 60 passed  (2026-08-19, round 4 — full re-verify on this machine)
# note: the suite invokes `python` through the shell, so run it with the
# interpreter on PATH (e.g. `.venv/bin`).

Paid pilot / willingness to pay

checkpoint-run is free and open source (MIT). This is a genuine demand test, not a storefront:

• If a maintained, supported edition (priority bug fixes, team/CI onboarding, or a packaged retry service for unattended jobs) at ~US$49 one-time per seat would help you, say so in Discussions — start or upvote a thread titled "Paid pilot interest".
• If you'd pay a different amount, that's even more useful: state the number.
• Responses decide whether a paid pilot ships. No payment is taken here; nothing is collected, and there is no hidden paywall.

Issues & Discussions

• Bugs / feature requests → Issues (use the templates: bug report or feature request).
• Questions, show-and-tell, demand votes → Discussions.
• Both are watched; the fastest way to shape the roadmap is a Discussion.

Claims discipline (from the audit)

This product's claims are deliberately low-risk: it records which steps completed and resumes there. It does not assert legal requirements, safety authority, or regulatory compliance — the failure class that killed nz-swms-builder (see ../AUDIT-REPORT.md).