Brittle

Renamed from claude-api-guard on 2026-09-12, once "guards your Claude calls" stopped describing what it actually does — see "Multi-provider support" below for why.

A GitHub Action that scans your codebase for usage of the Claude/Anthropic, OpenAI, and Gemini APIs that's broken, or about to break, because of a known, dated API change — and auto-fixes the mechanical ones. It keeps its own rule set current by reading each provider's official release notes on a schedule and extracting new breaking changes with an LLM, so it doesn't go stale the way a hand-maintained list would.

Why this exists

LLM provider SDKs change fast, and "it worked last month" is not the same as "it still works." A sampling parameter gets removed, an HTTP client gets swapped, a response shape gets renamed — and the first anyone hears about it is a production error, not a changelog. This tool is meant to be the thing that catches that in CI, before it ships.

Where it's strongest right now, and where it's headed: Brittle started as, and is still deepest on, Anthropic's Claude API — every rule is validated against real downstream code (not just written and assumed correct; see the engineering log below for the actual false positives found and fixed), and its rule set updates itself from Anthropic's live release notes. OpenAI support followed the same bar: hand-extracted from OpenAI's own changelog and migration guides, then fully triaged against a large real codebase (litellm) until every finding checked out. Gemini is the third provider (see "Multi-provider support (Gemini)" below) — deliberately narrower on day one than the other two (1 rule, not 4-18), because that's what actually survived reading the real changelog and testing against real code, not a lower bar applied to a newer provider. The plan from here is to keep expanding provider coverage outward from that same foundation — this is meant to grow into a broader "breaking-change guard for every API your project depends on" tool, not stay a single-provider niche script. The "provider" field already built into every rule, and the per-provider PROVIDERS config in sync_rules.py, exist specifically so adding the next provider is a matter of writing its rules and its changelog parser, not restructuring the tool.

No API key needed to use it. Scanning your code costs nothing and calls no LLM at runtime — every rule ships pre-baked in this repo. An Anthropic API key is only used on this repo's own maintenance side, to power the weekly job that reads provider release notes and proposes new rules (every proposed rule still goes through a human-reviewed PR before it's live — see "Rule sync" below).

Quick start

Add this to a workflow file in the repo you want to protect (e.g. .github/workflows/brittle.yml):

name: Brittle check
on:
  pull_request:
  push:
    branches: [main]

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: MarkMoneyMan/brittle@master
        with:
          path: .
          fail-on: HIGH        # MEDIUM/LOW findings are reported but won't fail the job
          # scan-js: "true"    # also scan JS/TS files for Anthropic/OpenAI/Gemini SDK usage

That's it — no secrets, no config file, no signup. It fails the job only on HIGH-severity findings by default, so a heads-up doesn't block a merge the way a real break should. See examples/consumer-workflows/ for a weekly auto-fix variant that opens a PR for the mechanical fixes on its own.

Covers Python (Anthropic + OpenAI + Gemini SDKs) and, as a first pass, JS/TS (Anthropic + 1 OpenAI rule + 1 Gemini rule so far) — see "Known limitations" below for exactly what is and isn't covered yet.

License

Business Source License 1.1 (see LICENSE) — free to read, run, self-host, modify, and build on for your own use, including commercial use. The one thing it reserves is standing up Brittle itself as a competing paid hosted service before 2030-09-01, at which point it converts automatically to the MIT License. This is not a restriction on using the tool to protect your own project — that's unrestricted from day one.

---

Engineering log

Everything below is the detailed, dated record of how this was actually built and validated — real bugs found, real repos tested against, real CI runs, kept as running documentation rather than cleaned up after the fact. It's here for anyone who wants to verify the claims above rather than take them on faith.

Two versions in this repo

• scan.py — v1, regex over the whole file. Fast to build, but tested against 5 real public repos (anthropic-cookbook, anthropic-sdk-python, llm, aider, OpenHands, litellm) and produced 1,576 findings total, the vast majority false positives: comments, docstrings, string literals, and — in multi-provider codebases like litellm — other vendors' API calls that happened to share a method name with Anthropic's.

• ast_scan.py — v2, walks the real Python syntax tree instead of matching text. Started with hand-coded checks for the 9 hand-written rules only; as of 2026-08-27 it also runs a generic engine (generic_scan()) that turns any rule from rules.py — including ones extract_rules.py generates automatically — into an AST-level check, without hand-coding logic per rule. See "Rules" and step 2 below for how that engine earned its noise budget the hard way.

• js_scanner/ — JS/TS sibling, added 2026-08-27. Node + @babel/parser instead of Python's ast module (simpler than getting a tree-sitter grammar built in this environment; same "walk the real syntax tree" idea). See "JS/TS support" below.

Rules

rules.py holds the current rule set: 10 hand-extracted from Anthropic's live release notes (as of 2026-08-27), plus whatever sync_rules.py has appended automatically since. extract_rules.py is the LLM-extraction step itself (changelog text in, structured rules out); sync_rules.py is what actually runs it unattended — see "Rule sync" below. Both need an ANTHROPIC_API_KEY to run for real.

Known limitations (be honest about these before building further)

• Only catches calls written in a fairly direct style. Code that builds the call via **kwargs splatting or heavy indirection won't be seen (confirmed: this is why aider shows 0 findings — it talks to Claude through litellm, not the Anthropic SDK directly, so there's nothing for this tool to see there yet).
• Structurally blind to raw-HTTP integrations (a hand-built JSON body POSTed straight to api.anthropic.com, no official SDK in the call path at all) — every rule matches an SDK call shape, so there's nothing for the AST walk to find. Confirmed twice now, not just theorized: litellm's own Anthropic integration (HTTP-level, not SDK-level) and a legacy Node.js prototype inside oddsscanner (server.js) both produce 0 findings for this reason, not because they're actually safe. Update (2026-09-11): a narrower fix — just flag that a raw-HTTP integration exists at all, not try to replicate the full rule catalog inside it — was attempted and rejected after real testing, not left untried. See "Investigated and rejected: a raw-HTTP advisory rule" below for what was actually built, tested, and why it didn't survive contact with real code.
• Python only for detection; JS/TS added as a first pass (see below), no Go/other-language support.
• Rule sync runs on a schedule now (sync_rules.py + update-rules.yml), but every extracted rule still goes through a PR a human reviews before it's live — deliberately not fully unattended.
• OpenAI support (rules_openai.py, 4 rules) is Python-only, wired into rule sync and CI self-check (this bullet used to wrongly say otherwise — see the doc-drift note in "Multi-provider support" below). The JS/TS scanner now knows exactly 1 OpenAI rule (openai-v2-tool-call-output-type-widened, added 2026-09-11), not the other 3 — those are Python-implementation-level, not API-level, same reasoning that keeps several Python-only Anthropic rules out of rules_js.js too. The JS/TS scanner's string-literal false-positive gap (confirmed the same day this rule was added) is closed now, same day — see "JS/TS support" for what that fix found. See "JS/TS support" and "Multi-provider support" below for exactly what's been tested and what hasn't.
• Gemini support (rules_gemini.py, added 2026-09-11) is deliberately just 1 rule so far — the legacy google-generativeai SDK deprecation, both Python and JS/TS, tested for real against litellm. The much larger set of real 2026 breaking changes in Gemini's Interactions API was read and deliberately left unshipped (narrower, less-adopted API surface, not yet tested against real code) rather than guessed at — see "Multi-provider support (Gemini)" for exactly what was reviewed and why.

Validated against

Real code: /Users/markus/Desktop/oddsscanner, re-run directly (not through CI — that repo has no .git yet) against the full current rule set, Python and JS both, once the rule sync work above made the rule count grow well past the original 6.

• app.py (the live backend — start.sh/start.bat both run this, port 5000): clean. Confirmed by reading the actual call site, not just trusting the scanner: anthropic.Anthropic(...), client.messages.create(model="claude-sonnet-4-6", max_tokens=..., system=[...], messages=[...]) — no temperature/top_p/top_k, no .with_raw_response, no beta headers, no AnthropicBedrock. SDK is pinned to anthropic==0.28.0, well below v1.0, so the SDK-v1.0 rules correctly don't fire yet — this is a true negative, not a blind spot.
• server.js + index.html (an older Node.js prototype, both dated well before app.py and static/index.html, and not what start.sh/start.bat actually launch): also 0 findings, but for a reason worth stating plainly rather than taking credit for: this code never calls the Anthropic SDK at all. It hand-builds a JSON body and POSTs it to https://api.anthropic.com/v1/messages with Node's raw https module. Every rule in rules_js.js matches SDK call shapes (.messages.create(...), .beta.files, ...), so there is structurally nothing here for it to match — the same class of blind spot already documented for litellm's own Anthropic integration, now confirmed in a second, real, personally-used codebase rather than just a public one. Concretely: this dead path has a hardcoded, dated model snapshot (claude-sonnet-4-20250514) that a raw-HTTP-aware rule set would reasonably flag someday — worth knowing about even though it's not live traffic today.

Plus the 6 public repos above for false-positive testing.

Next steps, roughly in priority order

1. Handle **kwargs-style calls — done. Built and validated (example_project/bot.py has a synthetic test case for it), but it changed zero findings across the 6 real repos. Turned out litellm doesn't call the official anthropic SDK at all in its own Anthropic integration — it reimplements the API at the HTTP level (litellm/llms/anthropic/...), so there was never an SDK call site there to find. Same root cause explains aider's 0 findings: it talks to Claude through litellm, never through anthropic.Anthropic() directly. Correcting the earlier claim that kwargs-handling would "unlock" either of them — it doesn't; that's a structurally different, bigger problem (would need to understand each abstraction layer's own API, not just the official SDK's).

2. Automate extract_rules.py — ran for real on 2026-08-27, first real (small) cost in this project. Fed it Anthropic's actual release notes (last ~8 weeks, pipeline_runs/2026-08-27_changelog_input.txt) with a real ANTHROPIC_API_KEY. Result: pipeline_runs/2026-08-27_extracted_rules.json — 14 rules extracted automatically. It correctly found all 6 breaking changes that had been hand-written into rules.py earlier (SDK v1.0 sampling params, legacy Text Completions removal, Opus 5 xhigh/max thinking error, Opus 4.7 fast-mode removal, Opus 4.1 retirement, experimental prompt-tools retirement) plus 8 more that hand-extraction had missed: an httpx→httpx2 migration in SDK v1.0, compaction_control removal, an async .with_raw_response behavior change, AnthropicBedrock's dropped default AWS region, the Python 3.10 floor, a client.beta.files /client.beta.skills shape change, a Managed Agents header behavior change, and a computer-use toolset shape change. One real bug found and fixed running this live: the first version hardcoded max_tokens=4096, which silently truncated the JSON output mid-string on a real-size changelog batch and threw a parse error — fixed by raising the limit and by making extract() raise a clear error on stop_reason == "max_tokens" instead of failing on a cryptic JSON error.

   Known gap at the time, stated plainly: these 14 auto-extracted rules used the v1 rules.py schema (regex pattern field) that scan.py reads — ast_scan.py (the good, low-noise v2 scanner) didn't read rules.py at all; every check in it was hand-coded per rule type. Closed in step 3 below.

3. Teach ast_scan.py to consume auto-extracted rules — done, and it broke on the first real run, which is exactly why "run it for real" beats "looks right on paper." Added generic_scan(): for each of the 8 new (non-hand-coded) rules, it regex-matches the rule's pattern against the unparsed source of individual real AST nodes (Call, Import, Assign, ...) — never the whole file, so it structurally can't match a comment or a docstring the way v1 did. First run against the same 6 repos: litellm alone produced 1,720 findings, almost all from one rule (python-sdk-v1-httpx-to-httpx2, 1,604 hits) and a second (python-sdk-v1-async-with-raw-response, 114 hits). Both were the same class of bug as the very first chat.completions.create collision, just recurring at the pattern level instead of the file-context level:

   • httpx is a generic HTTP library. litellm imports it ~40 times for its own multi-provider handling and — confirmed by grep — never imports the actual anthropic package in any of them. The pattern alone can't tell "this httpx client feeds the Anthropic SDK" apart from "this httpx client does literally anything else."
   • .with_raw_response isn't Anthropic-specific either — it's a shared naming convention across every Stainless-generated SDK, and OpenAI's is one too. litellm's Azure/OpenAI calls (azure_client.chat.completions.with_raw_response.create(...)) matched it directly. Separately, the real breaking change only affects the async client, and the pattern had no async awareness at all — its single false-positive hit in anthropic-sdk-python itself was a sync test correctly calling response.parse() with no await, not broken code.

   Fix, in both cases: not a wider or narrower regex, but one real structural precondition per rule (GENERIC_EXTRA_CONDITIONS in ast_scan.py) — "this file actually imports anthropic" (checked correctly for absolute imports only; a second bug surfaced here too, since litellm's own from ...anthropic.chat.transformation import X is a relative import of its own same-named submodule and initially tripped the naive version of this check) and "this call site sits inside an async def." After both fixes, same 6 repos: litellm 1,720 → 2, aider 11 → 0, anthropic-cookbook's 36 remaining findings all check out on inspection (real client.beta.files / client.beta.skills calls that will genuinely need updating). One repo didn't clean up: anthropic-sdk-python still shows ~1,478, because it's not a fair test bed for these particular rules — it is the SDK, so its own source and test suite naturally define and exercise the exact strings these rules look for (e.g. the one memory-list hit inspected was the SDK's own source defining the MANAGED_AGENTS_BETA constant). That's a limitation of the test setup, not a scanner bug — but it's honest to say the generic engine has only been proven clean against real downstream consumer code, not against a library that mirrors its own rules back at itself.

   Update, found building the JS/TS scanner below: that ~1,478 number was itself inflated by a real bug, not just the self-referential-repo problem — generic_scan's candidate node types overlap (a Call is a child of the Assign that captures its result, e.g. client = AnthropicBedrock(...)), so the same real match got reported twice, once per node. Confirmed: 427 of the 1,478 were exact-duplicate (file, line, rule_id) triples. Fixed with a dedupe_findings() pass that also prefers the more informative duplicate (a model-scoped rule can only confirm the model on the Call node itself, never on the wrapping Assign — naive dedup could keep the less-informative "unconfirmed" copy). Real count for anthropic-sdk-python: 1,051, still mostly the self-referential-repo effect, not noise.

4. Multi-language support (start with JS/TS) — done as a first pass, see "JS/TS support" below.

5. Auto-fix: generate the actual code patch — done for a small, deliberately mechanical subset. See "Auto-fix" below. ("...and open a PR" is now just the git/gh mechanics on top of a real patch — not attempted against a real third-party repo without being asked to.)

6. Package as a CI Action — done, and it found a real bug on its first real Actions run. See "CI / GitHub Action" below.

7. Automate the rule-extraction step end-to-end (not just "ran once by hand") — done. sync_rules.py + .github/workflows/update-rules.yml run this on a schedule now instead of a human copy-pasting changelog text into a file. See "Rule sync" below.

8. Package autofix.py as something installable, instead of autofix-weekly.yml checking out this whole repo for one file — done. pyproject.toml + two console-script entry points; see "Packaging" under "Auto-fix" below for what that did and didn't fix.

Auto-fix

autofix.py generates real source patches — not suggestions in a report — for 5 of the ~19 rules, chosen because the fix is a pure deletion or a 1:1 string swap with no judgment call attached (no "which model should this migrate to," no "how should this system-prompt instruction be phrased," no "which effort level is right here"). Everything else stays detection-only on purpose: a wrong regex was already the first act of this project (scan.py); a wrong auto-fix rewrites someone's actual code, which is a worse failure than not fixing it. See the module docstring in autofix.py for the full list and the reasoning per rule, fixed and not-fixed alike.

Every patch goes through one hard gate before it's ever written: the patched file must still parse (ast.parse) or the patch is refused and logged, never applied. That gate mattered for real, immediately — first run against a real repo (a copy of anthropic-cookbook) hit a genuine bug in the edit engine: deleting the last keyword argument in a call only scanned backward through same-line whitespace looking for the separating comma, so when the previous argument was on an earlier line (the common one-arg-per-line style), it never found that comma and left the deleted argument's own trailing comma orphaned on its own line — invalid syntax. The parse gate caught it before anything was written; the practical effect was just a silently-skipped fix, not corrupted code. Fixed by mirroring the already-correct forward-scanning logic (cross one newline + its indentation, not just spaces/tabs on the same line) and re-verified.

Validated: example_project/autofix_test.py has one call per auto-fixable rule plus two calls that must NOT be touched (a deprecated model string, a manual thinking budget) — confirmed after fixing: the 5 fixable ones are gone, the 2 judgment-call ones are untouched, the file still parses. Then for real: ran --write against a full copy of anthropic-cookbook. Result: 20 edits across 9 real files, every patched file still parses, and rescanning afterward shows only the 4 assistant-prefill-removed findings left — exactly the ones this tool was never supposed to touch. Known gap: the fixer only edits a direct keyword argument on the call site itself, not one assembled in a **kwargs dict elsewhere (the same splat-resolution limitation ast_scan.py's detection side already handles for reading, but hasn't been extended to for writing) — one real temperature= finding in cookbook was left un-autofixed for exactly this reason, correctly, rather than attempting an edit somewhere else in the file it wasn't confident about.

Also fixed along the way: ast_scan.py and scan.py silently reported "no findings" when pointed at a single file instead of a directory (Path(file).rglob("*.py") returns an empty iterator, not an error) — a false "all clear" is the exact failure mode this whole project exists to prevent, so worth fixing the moment building/testing autofix.py on a single file actually hit it.

Packaging

autofix-weekly.yml used to check out this tool's whole repo into a subfolder next to the consumer project, just to reach one file (claude-api-guard-tool/autofix.py, back when the project was still named claude-api-guard — see the top of this README for the 2026-09-12 rename) — noted at the time as a known gap. Closed now: pyproject.toml packages rules.py, ast_scan.py, and autofix.py as an installable package with two console-script entry points — named claude-api-guard/ claude-api-guard-scan/claude-api-guard-autofix at the time, renamed to brittle/brittle-scan/brittle-autofix in the same commit as everything else. autofix-weekly.yml now does pip install "git+https://github.com/MarkMoneyMan/brittle.git@master" (no token needed — the repo is public) and runs brittle-autofix repo --write — one step instead of two, and no more reaching into a sibling checkout's file path by hand.

One deliberate tradeoff, stated plainly rather than hidden: the package is flat top-level modules (rules, ast_scan, autofix), not a brittle/ namespace package. That's not an oversight — those three files already import each other with bare names (from rules import RULES, from ast_scan import ...), and action.yml

• self-check.yml + sync_rules.py all already run them as plain top-level scripts by path. Packaging them as-is meant zero import changes and zero risk to any of that already-working, already-tested machinery — the actual cost is that "rules", "ast_scan", and "autofix" are generic names that could collide with something else in a shared Python environment. Acceptable here because the only realistic install path is a fresh, ephemeral CI job installing straight from this repo, not a shared environment — but a real brittle/ layout (with relative imports, and action.yml/sync_rules.py updated to match) would be the right fix before this goes anywhere wider than that.

Validated: installed into a clean virtualenv from this checkout (pip install -e . first, then pip install . to mirror what CI actually does) and run from a directory with no copy of this repo in it at all — both console scripts produced byte-identical results to running the scripts directly (brittle-scan found the same 4 known example_project/ findings and exited 1; brittle-autofix produced the same 7 edits against a copy of autofix_test.py, and the patched file still parsed). self-check.yml gained a third job, package-installs-and-runs, that runs this exact same check on every push — so a future change that breaks the installed package (not just the scripts run directly) fails CI immediately instead of only showing up the next time autofix-weekly.yml happens to fire.

Update: ran for real on GitHub Actions — self-check #9 (commit d8dbe6e) passed all three jobs, confirming pip install . (the build

• entry-point registration this sandbox couldn't test, no network path to github.com from here) works correctly on a real Ubuntu runner, not just in this sandbox's virtualenv. Being precise about what that does and doesn't cover: self-check.yml installs from the already-checked- out local directory (pip install .), which proves the package itself is sound. It does not exercise autofix-weekly.yml's specific pip install "git+https://x-access-token:...@github.com/..." line — that only fires on the Monday schedule or a manual workflow_dispatch, neither of which has happened yet. pip's git-URL install and token-in-URL auth are both extremely well-trodden mechanisms, so this is a small remaining gap, not an unknown one — but per this project's own rule of not calling something proven until it's run for real, it stays open until autofix-weekly.yml actually fires once.

JS/TS support

js_scanner/ast_scan.js — same "walk the real tree, match node-by-node, never the whole file" idea as ast_scan.py, ported to JS/TS. Built the generic engine directly from the start this time (no separate hand-coded phase first) — there was no reason to relearn the lesson from the Python side about testing against real repos before trusting a rule set.

Rule set is deliberately smaller than Python's. Went back through the same raw changelog text looking specifically for what's confirmed to touch the TypeScript SDK, rather than assuming every Python-flagged change applies by analogy. Included: API/request-level changes that don't care which language calls them (model deprecations, Opus 4.7 fast-mode removal, Opus 5 effort+thinking rejection, assistant-prefill removal, experimental endpoint retirement), plus the two changes the changelog explicitly names "Python SDK X, TypeScript SDK Y, ...": the beta.files/beta.skills shape change and the memory-list header change. Excluded: every rule whose own title says "Python SDK v1.0" (httpx→httpx2, compaction_control, async .with_raw_response, Bedrock's default region, the Python 3.10 floor) — those are Python-package-internal, and there's no changelog evidence the TypeScript SDK did the same thing. Left as an open question rather than guessed.

First live run found 4 real bugs, same pattern as every other "test it for real" pass in this project:

1. A crash, not just noise: @babel/traverse's scope-crawling threw an uncaught error on one real file in vercel/ai (a valid-but-unusual TS type/value naming collision) and killed the entire batch scan, losing every finding already collected. Fixed with a per-file try/catch, same principle as ast.parse's SyntaxError being caught per-file in Python, just a different failure mode (traverse-time, not parse-time).
2. The same cross-node duplicate-finding bug described in step 3 above — found in Python first, then confirmed live here too by literally translating the same fix and watching it matter immediately.
3. assistant-prefill-removed regex-matched 1,619 times in vercel/ai alone: role: "assistant" near content: is the shape of any code representing an assistant chat message at all (rendering history, type defs, test fixtures), not specifically "the last message of an outgoing request." Fixed by pulling this one rule out of the generic engine entirely and porting the precise version of the check from ast_scan.py's extract_messages_prefill() — only the literal last element of an actual messages.create() call's messages array counts.
4. Two variations on "a candidate node's span can be bigger than it looks": a JS test-framework call like describe('X', () => { ...whole rest of the file... }) is itself one CallExpression, so anything anywhere in that block counted as a "match" on the outer call; a large expect(x).toMatchObject({ ...huge mock... }) has the same problem without being a callback. Fixed the first with a structural check (skip a call whose argument is a function with a real body — traversal still walks into it, so a real Anthropic call nested inside still gets checked on its own node) and the second with a blunter 2000-character snippet cap, documented as a safety valve rather than a precise fix.

Net result, tested against anthropic-sdk-typescript (the SDK's own repo — same self-referential-test-bed caveat as the Python side applies) and vercel/ai (a real, large downstream consumer): vercel/ai went 1,734 → 67 findings across the 4 fixes above, and the 67 remaining check out on inspection (real references to computer_20251124, a real deprecated-model-string literal, etc. — see git history for the exact before/after JSON if you want to see the noise that got cut). Only tested against 2 real repos so far, not 6 like the Python side — this is explicitly a first pass, not yet hardened to the same degree.

Update (2026-09-11): one OpenAI rule added, closing the one real gap this file used to flag. Went through openai-node's real CHANGELOG.md (~344 versions) the same way the Anthropic pass above did — only 2 ⚠ BREAKING CHANGES sections exist in its whole history, the same count sync_rules.py's OpenAI parser already relies on for the Python side. One is included: openai-v2-tool-call-output-type-widened, same id as the Python rule of the same name (ResponseFunctionToolCallOutputItem / ResponseCustomToolCallOutput's .output field widening from string to string | Array<...>) — API/response-shape level, not SDK-implementation level, so the same reasoning that included the shared Anthropic rules applies here too. The other, "require Node.js 22" (SDK v7.0.0), is explicitly excluded: that's a package.json engines requirement, not JS/TS source code, and this scanner only ever parses .js/.ts files — there's structurally nothing for it to match, same honest gap as the raw-HTTP blind spot elsewhere in this README, not an oversight.

Tested for real, not just written and assumed correct: added js_scanner/example_project/openai_bot.ts (a real type import + a clean unrelated call, mirroring example_project/openai_bot.py's Python fixture) — the rule fires exactly once, on the real import, and the clean call produces nothing extra. Also built a throwaway fixture with the type name only inside a console.log string, never a real import — it false-positives, confirming (not just theorizing) that this scanner has the same string-literal-matching gap already found and fixed on the Python side (see "Closing the string-literal gap in generic_scan() itself" below). Originally left unfixed and tracked as an open gap — see the next update, same day, for why that changed almost immediately.

Also closed a second, older gap while here: scan-js: "true" (the composite action's JS/TS opt-in — Node setup, npm install, ast_scan.js) had never actually been exercised in a real GitHub Actions run before, only manually/locally. self-check.yml now has a job that runs it for real against js_scanner/example_project/ and checks the new OpenAI rule fires by id — the first CI-covered proof the whole scan-js path works end-to-end, not just the rule content.

Update (2026-09-11, later the same day): the string-literal gap above closed for JS/TS too — and porting the fix directly uncovered a real, separate bug in the process. ast_scan.js now has its own masking pass (buildStructuralSnippet / collectStringLiteralSpans), same idea as Python's _mask_string_literals: blank every string/template literal's contents before regex-matching, so a rule can't match text that only lives inside an unrelated string. Deliberately not implemented via @babel/traverse (even though it's already a dependency) — a plain recursive walk over own-enumerable-properties needs no scope tracking, so it can't hit the same scope-crawling crash @babel/traverse already caused once on valid-but-unusual TS (bug #1 above). Also deliberately not using @babel/generator to re-emit code from the masked tree — the existing code.slice(node.start, node.end) snippet is spliced directly at each string/template literal's own start/end offsets instead, avoiding a new dependency entirely.

The real bug: this was not a straight port of Python's exemption set, and assuming it was one would have shipped a silent regression. The first version carried over only Python's 2 non-hand-coded exemptions (memory-list-managed-agents-header-behavior-change, computer-use-toolset-new-shape). Running it against a real-usage fixture immediately broke a true positive: model-deprecated-sonnet4-opus4 stopped firing on bot.ts:33's real deprecated-model call. Root cause is architectural, not a copy-paste slip — on the Python side, every model-name/config-value rule (sdk-v1-sampling-params-removed and friends) is hand-coded in scan_source() via find_calls()/get_kwargs() — real AST field access to the actual keyword argument value, never regex-on-text, so masking is irrelevant to it. rules_js.js has no equivalent hand-coded path for any of its model-name/config rules — all of them go through the same generic regex-on-snippet loop Python's generic_scan() uses, so several rules whose real signal is a plain string value (a model name literal, a "fast"/"xhigh"/"disabled" config string, a URL path string) needed exemptions with no Python-side counterpart to copy from at all.

Re-derived the exemption set empirically instead of guessing further: a synthetic real-usage snippet per candidate rule, run through the scanner before and after masking. Confirmed 5 more rules needed exemption (model-retired-opus-4-1, model-deprecated-sonnet4-opus4, fast-mode-removed-opus-4-7, opus5-effort-xhigh-thinking-disabled, experimental-endpoint-retiring) — GENERIC_RULES_MATCH_INSIDE_STRINGS in ast_scan.js is 7 entries now, not 2. Confirmed the remaining 3 generic-path rules (manual-thinking-budget, beta-files-skills-sdk-shape-change, openai-v2-tool-call-output-type-widened) are genuinely code-shape and safe to mask — the last of those is the exact rule this whole fix was built to protect, and it still correctly fires on openai_bot.ts's real import while no longer false-positiving on the log-string case.

Promoted the throwaway fixtures into permanent regression coverage: js_scanner/example_project/more_rules_bot.ts (real usage for all 4 newly-discovered-vulnerable rules, plus the 2 previously-untested exempted rules — closing a real, separate gap: 6 of rules_js.js's 11 rules had never been exercised by any TS fixture in this repo before today) and js_scanner/example_project/string_literal_audit_fixture.ts (the 3 code-shape rules' trigger text planted purely inside unrelated console.log strings, must produce 0 findings). Net result: js_scanner/example_project/ now produces exactly 11 findings, one per RULES_JS rule, for the first time ever covering the entire JS/TS rule set against real-usage fixtures — and self-check.yml's expect-all-js-rules-fire-on-js-fixtures job checks every one of the 11 rule_ids by name, so a silent regression in any single rule can't hide behind another rule still failing the severity gate.

Multi-provider support (OpenAI)

Started as "guards your Claude API calls." The business case for going further is straightforward: almost no real project uses exactly one LLM API forever, so a tool that only watches Anthropic's SDK is watching a fraction of the code that's actually at risk. First step: rules_openai.py — 4 rules, hand-extracted the same way rules.py originally was, from OpenAI's real, live sources (httpx2.md's current migration guide, CHANGELOG.md's explicit "BREAKING CHANGES" markers, and the 2023 v1.0.0 migration guide for the still-real risk of old copy-pasted call styles). ast_scan.py merges both rule sets (RULES = anthropic rules + openai rules); a rule with no "provider" key defaults to "anthropic" so none of the 18 existing rules needed touching by hand.

One validating detail before any code was written: the httpx-to-httpx2 migration already tracked for Anthropic (python-sdk-v1-httpx-to-httpx2) turns out to be the same industry event hitting OpenAI's SDK too — both are generated by the same tool (Stainless), and httpx itself going unmaintained affects everyone built on it. Real, structural evidence this isn't a one-off, not just an assumption that "multi-provider" is worth building.

Tested against real repos immediately, not assumed correct — and found two real bugs, same pattern as every other provider/language added to this project so far:

1. file_references_openai() (the same per-file import precondition that already gates the Anthropic httpx rule) was a direct copy of file_references_anthropic() — "does this file import anything under openai.*?" Tested against litellm (170 httpx-rule hits on first run). Root cause: litellm reuses openai.types.* — OpenAI's own Pydantic response-schema submodule — as a shared return-type vocabulary across every provider it supports, including ones with nothing to do with OpenAI. Its Vertex AI (Google) image-generation handler does from openai.types.image import Image purely to borrow that shape, with zero real OpenAI-client code in the file. Fixed by excluding openai.types(.*) imports from the precondition — a bare import openai or from openai import OpenAI still counts, but borrowing a type definition doesn't. No equivalent gotcha exists on the Anthropic side (its SDK isn't reused as a cross-provider type vocabulary the same way), which is exactly why this wasn't caught by just copying the Anthropic check — it had to be tested for real.
2. The first version of openai-v2-tool-call-output-type-widened's pattern also matched a generic .output[0] shape, meant to catch code indexing into the field directly without naming the type. .output indexed at [0] turned out to be an extremely common, totally generic shape (any response wrapper, any test fixture) — 17 hits in litellm, 14 of them unrelated to this rule at all. Fixed by narrowing the pattern to the two named types themselves (ResponseFunctionToolCallOutputItem/ResponseCustomToolCallOutput), accepting under-reporting (code that reads .output without ever naming these types is missed) over noise — same trade-off this project has made every other time a pattern was too permissive.

Update: fully triaged, not just spot-checked — every one of the 140 findings above was reviewed, not a sample. That triage found three more real, structural bugs, same "test for real" pattern as everything else in this project:

3. 63 of the 129 openai-httpx-to-httpx2 hits were a bare import httpx or from httpx import ... line, with no actual httpx.Client/ Timeout/MockTransport construction anywhere else in that file — 43 files had only that. litellm/exceptions.py was typical: it uses httpx.Response/httpx.Request extensively (types this rule was never about), and the import line was the sole match. A bare import isn't actionable on its own — nothing for a developer to go change at that specific line — so this rule's Anthropic sibling (python-sdk-v1-httpx-to-httpx2) got away with matching bare imports too only because litellm barely references anthropic at all and was never stress-tested there. Fixed by dropping the bare-import alternative from the pattern entirely, keeping only the actual construction/type sites.
4. One of the 8 openai-v1-legacy-module-level-calls-removed hits wasn't real code at all: litellm's PromptLayer integration does litellm.module_level_client.post(..., json={"function_name": "openai.ChatCompletion.create", ...}) — a real Call node whose unparsed text includes a string literal that merely names the old call shape as logging metadata sent to PromptLayer's API. Nothing there is actually calling openai.ChatCompletion.create; the pattern matched inside a string value because generic_scan() regexes a node's whole unparsed text, code and any string literals it contains alike — a structural gap in the generic engine itself, not just this rule (any rule's trigger text could coincidentally appear inside some unrelated string; this is the first time it's actually been observed, not something audited across every other rule). Given every real hit for this specific rule is a genuine attribute access that's never inside quotes, fixed narrowly with a quote-adjacency guard on this rule's pattern ((?<!['"])...(?!['"])) rather than touching generic_scan() itself — safer, and doesn't risk any already-shipped rule that hasn't shown this problem.

After all three fixes: 195 → 59 OpenAI-rule findings in litellm, every one reviewed and legitimate — real httpx.Client/Timeout/ MockTransport construction or type-check sites (mostly in litellm's actual OpenAI/Azure provider code and its HTTP-mocking test fixtures), real leftover legacy openai.api_key =/openai.ChatCompletion.create(...) calls (an old cookbook example and a few of litellm's own older test setup lines), and the 3 real references to the renamed tool-call-output types. Re-confirmed clean afterward: openai-cookbook still 0 findings, ci_fixtures/known_clean.py still 0, example_project/'s own fixtures unaffected.

Also tested against openai-cookbook (OpenAI's own official examples, 224 real .py files — 0 findings throughout, a clean smoke test on actively-maintained modern code).

Update: OpenAI is now wired into rule sync and self-check too, closing the loop the same way it's closed for Anthropic. sync_rules.py is multi-provider now (--provider anthropic|openai; see "Rule sync" below for exactly how the two providers' changelogs are parsed differently), update-rules.yml runs it for both every week and opens one combined PR, and self-check.yml has a dedicated job that greps for openai-httpx-to-httpx2 and openai-v1-legacy-module-level-calls-removed by name (not just "the severity gate failed") so a silent regression in one specific OpenAI rule can't hide behind some other rule still firing. pipeline_runs/last_synced.json is now {"anthropic": {...}, "openai": {...}} (migrated automatically from the old flat one-provider shape, tested against a simulated old file, not just assumed).

Update: the string-literal false-positive class from bug #4 was audited across every other rule, not left as an open question — see "Closing the string-literal gap in generic_scan() itself" below for what that found and how it was fixed generally instead of rule-by-rule. JS/TS support for OpenAI is still untested — js_scanner/ only knows the Anthropic rule set right now — and the OpenAI side of rule sync hasn't been proven against a real new breaking change yet (unlike Anthropic's, which was — see "Rule sync" below): it's only been run in dry-run mode against real history, since there's no small, cheap way to roll OpenAI's last_synced_date back without re-processing content already reviewed by hand. It'll get its real end-to-end test whenever openai-python next ships a version with an actual ⚠ BREAKING CHANGES section and the Monday schedule (or a manual run) picks it up — same "this part waits for something real to happen" honesty already applied to Anthropic's own first automated run.

Multi-provider support (Gemini)

Third provider, after Anthropic and OpenAI, chosen deliberately rather than picked arbitrarily: researched Google Gemini against Mistral first (adoption, SDK maturity, and — most relevant to this project specifically — whether breaking changes are tracked in a structured, mechanically-parseable way or require prose-inference). Gemini's google-genai SDK won on all three: more GitHub stars than Mistral's client-python (3.9k vs 767), a real recent major-version jump (v1→v2, 2026-05-07) with explicit ### ⚠ BREAKING CHANGES markers per release the same way OpenAI's CHANGELOG.md has them (Mistral's breaking changes live in a separate, less-frequently-updated MIGRATION.md instead), and its own live SDK-deprecation story (see below).

Read all 13 real ### ⚠ BREAKING CHANGES-marked sections in python-genai's actual CHANGELOG.md by hand before writing a single rule (v0.3.0, 2024-12-17, through v2.9.0, 2026-06-19) — same "research first, write rules from what's actually there" discipline rules_openai.py was built with, not a repeat of scan.py's original mistake of guessing at what might be a breaking change. Honest finding, stated plainly rather than smoothed over: almost none of it was worth shipping as a rule yet. Roughly a dozen of the 13 are either over a year old (narrow 0.x/early-1.x method renames — generate_image → generate_images, Part.from_video_metadata removed, etc. — unlikely to still be sitting in actively-maintained code) or scoped to the Interactions API specifically, which v2.0.0's own changelog entry says outright: "the breaking changes are only in interactions. GenerateContent usage in unaffected." That's the more-2026, more-recent material, but it's a narrower, less-adopted API surface than the mainline client.models.generate_content(...) call path most real Gemini code actually uses — and unlike the rule that did ship (below), nothing about the Interactions API has been checked against a real external codebase. Rather than guess at regex precision for a part of the SDK this project hasn't tested, that's deliberately left for sync_rules.py to pick up and route through a human-reviewed PR later (see "Rule sync"), not hand-shipped speculatively. Also deliberately not added: an httpx-to-httpx2 rule matching Anthropic's and OpenAI's — checked, and v2.18.0's "Support injecting httpx2 client" is a plain Feature, not a breaking change; httpx (v1) still works today. Nothing to flag until Google actually forces that migration the way OpenAI did.

The one rule that did ship comes from a better signal than any changelog line: google-generativeai, the SDK basically every pre-2025 Gemini tutorial was written against, is a fully archived repository. Both its README and its JS sibling's (deprecated-generative-ai-js) say, word for word, "All support for this repository ended permanently on November 30, 2025." That's a stronger, more unambiguous deprecation signal than a changelog entry, and exactly the kind of thing that survives in old, unmaintained code long after — same category as rules.py's Legacy Text Completions rule and rules_openai.py's pre-v1 module-level-calls rule. Old vs. new call shape (Python, confirmed against Google's own migration guide, not guessed): import google.generativeai as genai; genai.configure(api_key=...); genai.GenerativeModel(...) → from google import genai; genai.Client(...); client.models.generate_content(...).

Tested for real against litellm, not assumed correct — found both a true positive and the exact false-positive trap this project has learned to expect by now:

1. True positive: litellm's own llms/deprecated_providers/palm.py — its own legacy PaLM/Gemini integration, still in the tree — has a real import google.generativeai as palm + palm.configure(...) + palm.generate_text(...). Confirms the pattern isn't hypothetical, and confirms it needs to be alias-independent (palm, not genai — keying on the import statement itself, not an assumed alias name, is what catches this).
2. The exact same string-literal trap as bug #4 in the OpenAI section above, found in the same repo: prompt_templates/factory.py:3268 has "google.generativeai" appearing only inside an exception message string, never a real import. Closed by construction, not by an added exemption: a Python import statement's module path is bare identifier syntax, never a string literal, so generic_scan()'s string-masking never even needs to run on it — confirmed 0 findings there.

The JS/TS port needed the opposite, deliberate handling, and a real bug of its own, caught by testing before shipping rather than after: an ES import specifier ("@google/generative-ai") is a StringLiteral node, so buildStructuralSnippet()'s masking blanks it by default — without adding gemini-legacy-sdk-deprecated to GENERIC_RULES_MATCH_INSIDE_STRINGS, the rule could never fire on a real import at all. But a first version of that exemption also matched a bare require("@google/generative-ai") substring anywhere in a node's raw text, on the assumption that only a real require() call could produce it — wrong, caught by a deliberately adversarial fixture (console.log(\...run: require("@google/generative-ai") ...`), a real live false positive, not hypothetical: one CallExpressionnode whose own text legitimately contains that substring inside a template literal's *content*). Fixed the same way every other too-broad pattern in this project has been fixed: not a cleverer regex, but dropping therequire()alternative entirely and keeping only the^import-anchored forms — a CallExpression's own unparsed text can never itself begin with the literal word "import", so a false positive would need some *other* node whose text starts with real import syntax naming this exact package, which in practice means an actual import of it. Real-world cost is low: every real hit found so far (litellm's palm.py) and every fixture in this project uses ES import/from, never CommonJS require(). Permanent regression fixture for this specific bug: js_scanner/example_project/string_literal_audit_fixture.ts`'s 4th case.

Wired into everything else the same day, not left as a standalone rule file: ast_scan.py merges rules_gemini.py into RULES (a rule with no "provider" key still defaults to "anthropic", unaffected); pyproject.toml's py-modules got rules_gemini added in the same commit it was created, specifically to not repeat the exact bug rules_openai.py hit here (ModuleNotFoundError on the installed package — see the engineering log); sync_rules.py has a gemini entry in PROVIDERS with its own section parser (parse_dated_sections_gemini, case-insensitive on the "breaking change" marker — unlike OpenAI's parser, checked and confirmed necessary: the real changelog uses at least 4 different marker spellings across its history, including a plain, lowercase-ish ### Breaking changes with no ⚠ that a case-sensitive check would miss); update-rules.yml runs it as a third step and checks all three rules files still parse; self-check.yml has a dedicated expect-gemini-rule-fires-on-gemini-fixture job (Python) and the JS job now expects 12 rule_ids instead of 11, both checked by name. pipeline_runs/ last_synced.json's "gemini" entry is seeded to 2026-09-11 (the date of this hand-seeding pass) specifically so a real sync run only processes sections after this review, not the 13 already read and deliberately left out above — same seeding logic already used for openai's entry.

Not yet done, stated plainly rather than implied: the Interactions-API breaking changes noted above haven't been turned into rules or tested against any real codebase using that API surface (may not even exist in meaningful volume yet, given how new it is); Gemini's rule sync, like OpenAI's, hasn't had a real end-to-end run against an actual new breaking change yet — it'll get one whenever python-genai next ships a version with a genuine breaking-change section after 2026-09-11 and the Monday schedule (or a manual run) picks it up.

Closing the string-literal gap in generic_scan() itself

The false positive in bug #4 above (litellm's PromptLayer integration logging "openai.ChatCompletion.create" as metadata, matched because generic_scan() regexes a node's whole unparsed text — real code and any string literal it contains, alike) was flagged at the time as a real, unaudited risk across the other rules, not something to assume was a one-off. It wasn't. Built two fixtures (pipeline_runs/string_literal_audit_fixture.py and ..._fixture2.py) that plant every generic-path rule's trigger text purely as a string literal's value — a log message, a metadata dict — structurally identical to the real bug, never as real code that actually does the thing. Ran them through generic_scan() for real rather than reasoning about it in the abstract: 10 of the 12 rules that go through the generic engine fired on text that was never real code.

Fixed generally, not one regex guard per rule. generic_scan() now builds a structural version of each candidate node's text — every string literal and f-string's contents blanked out before the node is re-unparsed — and matches every generic-path rule's pattern against that by default (_mask_string_literals() in ast_scan.py). For 7 of the 10 vulnerable rules (beta-files-skills-sdk-shape-change, python-sdk-v1-bedrock-no-default-region, python-sdk-v1-compaction-control-removed, python-sdk-v1-httpx-to-httpx2, openai-httpx-to-httpx2, openai-v2-tool-call-output-type-widened, openai-v1-error-classes-renamed), this closes the gap with zero loss of real detection — confirmed by re-running both audit fixtures (all 10 false positives gone) and every existing fixture that has known true positives (example_project/, example_project/openai_bot.py, ci_fixtures/known_clean.py) and getting byte-identical results to before the fix. That's possible because these 7 rules' real signal is always code shape — an attribute chain, a constructor call, a keyword name — never a string's value, so masking string contents away only removes the places a false positive could hide, not the places a real one lives. Masking is applied by default to every generic-path rule that isn't explicitly exempted, which also closes the previously-documented residual gap in python-sdk-v1-async-with-raw-response as a bonus (its existing GENERIC_EXTRA_CONDITIONS check only ever confirmed "this file references anthropic somewhere," not that the matched text itself was real code) — it didn't show up as one of the 10 in this specific test only because that test's fixture happened not to combine an unrelated string with a real anthropic import in the same async function, not because the gap wasn't real.

The other 3 of the 10 are the genuine exception, stated plainly rather than swept into the same fix: python-sdk-v1-min-python-version (real signal: a Programming Language :: Python :: 3.x classifier string in setup.py) and computer-use-toolset-new-shape / memory-list-managed-agents-header-behavior-change (real signal: an actual header or type-tag string value, e.g. "anthropic-beta": "managed-agents-2026-04-01") have real, legitimate matches that live inside a string literal's value, not just coincidentally. Masking string contents for these would silently turn off true detection instead of just suppressing false positives — worse than the bug it would fix. A new set, GENERIC_RULES_MATCH_INSIDE_STRINGS, opts these three out of masking, so they keep matching the raw unmasked text with the residual risk left open and documented rather than quietly patched: they can still match inside an unrelated descriptive string (confirmed live — both still fire on the audit fixtures' deliberately-unrelated log lines). A sharper future fix would check the match sits in the right structural position (the value of a dict key literally named "type" or containing "beta") rather than accepting any string on the node at all — not attempted yet, scoped out for time.

Cost of the fix, stated honestly: every candidate node now gets deep-copied and re-unparsed a second time to build the structural snippet, on top of the existing unparse. Full-repo litellm scan time went from noticeably under this to 2m39s — real, measurable, and worth knowing about before pointing this at a very large monorepo in CI, though still well within what a per-PR CI check can absorb for a repo of normal size. Re-ran the full litellm scan after the fix as a regression check too, not just the two audit fixtures: 60 findings across the merged rule set, all consistent with the shapes already documented above — no unexplained swing in either direction.

The two audit fixtures are kept in pipeline_runs/ as permanent regression fixtures, not deleted after use — a future change to generic_scan() that reopens this gap for any of the 8 fixed rules should be caught by re-running them, the same principle as ci_fixtures/known_clean.py.

Investigated and rejected: a raw-HTTP advisory rule

Same day as the string-literal fix above, went after the raw-HTTP blind spot documented in "Known limitations" — deliberately the narrow version: not trying to replicate the whole breaking-change catalog inside a raw request body (rejected upfront, after reading oddsscanner's real server.js: its request body is a plain pass-through variable, not a literal, so there's no visible shape to check even with raw-HTTP awareness), just flagging that a raw-HTTP integration to api.anthropic.com/api.openai.com exists at all, LOW severity, so a human knows to check it by hand. Built as two new rules (raw-http-anthropic-integration-detected / raw-http-openai-integration-detected) reusing the exact same generic_scan() pipeline as everything else — architecturally the cheap part. Rejected after testing, not shipped — this section documents why, the same honesty standard as the kwargs-handling correction earlier in this README.

First version (any Call/Assign/AnnAssign node containing the literal domain string, no extra precondition) against litellm: 665 of 725 total findings — almost all noise. Inspecting real hits, not just the count: the large majority were api_base = ... or "https://api.openai.com/v1"-style default-fallback constants, a completely ordinary, safe pattern for a configurable client, not a raw-HTTP call site at all.

Second version, restricted to Call nodes only: still 495 findings. Root cause this time: httpx.Request(method="POST", url="https://api.openai.com/v1") — litellm's own exception-handling code builds a placeholder Request object purely to attach to an error it's raising, never sent over the network. Checked litellm's actual real HTTP call sites directly (llms/anthropic/, llms/openai/) to see if a tighter rule would at least catch the real thing this whole feature exists for — even there, the only place the bare domain string appears is this same Request()-for-error-reporting pattern. The genuine outgoing call doesn't expose the domain as a literal at its real call site at all (almost certainly built from a base_url configured once elsewhere, exactly the same data-flow-tracking problem already out of scope for this project — see the **kwargs limitation above).

Third version, restricted to Call nodes whose trailing callee name is an actual send-shaped verb (get/post/put/patch/delete/request/ urlopen/send, explicitly excluding bare Request(...) construction): down to 14 findings. Inspected every one, not a sample — 13 of 14 were test-mocking infrastructure (respx.post(...), patch(..., return_value=...), a test HTTP client's .post(...)) verifying litellm's real behavior against these domains, not production code bypassing the SDK. The 14th, claims.get('https://api.openai.com/auth'), is a JWT claims-dict lookup — dict.get() sharing a method name with the HTTP verb GET, a real false positive of exactly the kind a narrower verb whitelist was meant to avoid, still slipping through.

The disqualifying result, checked directly rather than assumed: re-ran the same verb-whitelist idea (prototyped standalone, never merged) against oddsscanner/server.js — the actual, real, originally-confirmed motivating example. Zero matches. server.js calls the domain through a custom-named wrapper function (fetchUrl(url, options)), not a whitelisted HTTP-library method name. The same callee-name filter narrow enough to exclude the dict.get()/Request() false positives is also narrow enough to exclude the one real case this rule existed to catch — and broadening it back out reopens the noise, confirmed by testing the same idea (any Call, no verb filter) against vercel/ai and anthropic-sdk-typescript: 340 and 59 findings respectively, effectively all inside .test.ts files (mock-server base URLs for each SDK's own test suite), matching the Python results almost exactly.

Conclusion: this isn't a tuning problem, it's a structural dead end for this technique. Real code either configures a base URL once and calls relative paths after (invisible to a literal-string match, no matter how it's scoped) or wraps the raw call in an arbitrarily-named helper function (invisible to any callee-name whitelist tight enough to avoid test-mock and placeholder-object noise). Every version tried sits somewhere on that same trade-off, and none of them land in a useful spot. All of it reverted — rules.py, rules_openai.py, and ast_scan.py are back to exactly what they were before this investigation started; nothing shipped. The raw-HTTP blind spot documented in "Known limitations" stays open, now with real evidence behind why a seemingly-obvious narrow fix doesn't work, instead of just an untested idea sitting there.

Rule sync

sync_rules.py is what actually makes this project "self-maintaining" instead of "a scanner someone has to remember to update by hand." It:

1. fetches https://platform.claude.com/docs/en/release-notes/overview.md — appending .md to a platform.claude.com/docs/... URL returns raw markdown instead of the rendered page, found by trying it, not documented anywhere, and much easier to parse reliably than scraping HTML;
2. splits it into dated sections and keeps only the ones newer than pipeline_runs/last_synced.json's stored date, so a weekly run doesn't re-fetch and re-pay for the same 2+ years of history every time;
3. hands just the new text to extract_rules.py's extract() — the same extraction used for the one-off manual run that seeded rules.py;
4. drops any extracted rule whose id already exists in rules.py (defense against the same change getting described slightly differently on a re-run);
5. appends whatever's left as a new dated block (RULES_AUTO_<date> = [...] RULES = RULES + RULES_AUTO_<date>), and advances the synced-through date regardless of whether anything new was found, so a week with only additive (non-breaking) changes doesn't get re-processed forever.

Multi-provider since the OpenAI work above — this whole pipeline runs once per provider (python3 sync_rules.py --provider anthropic|openai), each with its own entry in a PROVIDERS dict: its own changelog URL, its own rules file (rules.py / rules_openai.py), and — this is the part that couldn't be shared code — its own section parser. Anthropic's release notes and openai-python's CHANGELOG.md aren't just different URLs, they're structurally different documents: Anthropic's is unstructured prose with no reliable breaking/non-breaking signal beyond what the model infers, so every new dated section has to go to it. openai-python's CHANGELOG.md explicitly marks breaking versions with a "### ⚠ BREAKING CHANGES" heading (confirmed against the real file: 344 version headers total, ever, only 2 ever marked breaking) — so its parser filters to only those sections before anything reaches the model, rather than spending tokens sending it 342 irrelevant Features/Bug Fixes/Chores sections to correctly say "nothing breaking here" over and over. pipeline_runs/last_synced.json is one file holding one entry per provider now instead of a single flat date; an old flat-shaped file (from before a second provider existed) is migrated to the new shape automatically the first time it's read.

.github/workflows/update-rules.yml runs both providers weekly (Mondays) and on workflow_dispatch, then hands off once to peter-evans/create-pull-request for whatever changed across either — same no-commit-if-nothing-changed pattern as autofix-weekly.yml, on a fixed branch name so a run before last week's PR merges updates that PR instead of opening a duplicate. Needs a repo secret, ANTHROPIC_API_KEY — the workflow fails loudly rather than silently skipping if it's missing (no OpenAI API key is needed anywhere in this: the OpenAI side only ever reads OpenAI's public changelog page, it never calls OpenAI's own API). Also needs the repo's "Allow GitHub Actions to create and approve pull requests" setting enabled (Settings → Actions → General → Workflow permissions) — without it, create-pull-request fails even with pull-requests: write declared in the workflow itself (found the hard way; see below).

What's tested and how, stated plainly: this cloud environment's own network egress blocks platform.claude.com directly (confirmed — a plain curl and urllib.request both get rejected by the sandbox's proxy, an environment restriction, not a bug in the fetch code), so the actual fetch_changelog_markdown() HTTP call hasn't run inside this box. It has been tested with the real page content, though: WebFetch (which goes through a different path) pulled the live .md page directly, and that real output — all 135 dated sections back to May 2024 — was fed through the parser and dedupe/merge logic directly. That's how a real bug got caught before this ever ran unattended: older entries use ordinal day suffixes ("April 9th, 2025", "March 31st, 2025") that strptime can't parse, while recent ones don't ("August 27, 2026") — the first version silently dropped every suffixed section instead of erroring, which would have been a quiet under-processing bug, not a crash (the exact failure shape this whole project tries to catch in other code). Fixed by stripping the suffix before parsing; re-tested against the same 135 sections, all parse correctly now. Separately verified end-to-end with synthetic candidate rules (bypassing the real API call): dedup correctly skips a rule whose id already exists, keeps a genuinely new one, appends a block that keeps rules.py parsing as valid Python, and the newly appended rule is immediately usable by ast_scan.py — it found the synthetic rule's trigger pattern in a test fixture, same as any hand-written rule would. Update: ran for real on GitHub Actions (update-rules.yml run #1, workflow_dispatch, after the ANTHROPIC_API_KEY secret was added) — succeeded in 14s. That confirms the actual fetch_changelog_markdown() HTTP call works from a real runner (this sandbox's own egress blocks it, so it had only ever been exercised with a pre-fetched copy of the page before this), and that the secret is read correctly. The 14s runtime is itself informative: too fast to have called the model, consistent with hitting the "nothing new since 2026-08-27" fast path and exiting before ever importing extract_rules. Confirmed on GitHub afterward: no PR was opened — the "nothing changed, don't bother create-pull-request" path behaves correctly for real, not just in the code reading right.

Update: the extraction call itself has now been tested for real, too — deliberately, not by waiting for Anthropic to publish something new. pipeline_runs/last_synced.json was rolled back to an earlier date on purpose (a small, disclosed, real API cost) so the next run would treat already-public content as "new" and actually exercise the model call and everything downstream of it. First attempt (update-rules.yml run #2) crashed: json.decoder.JSONDecodeError: Invalid \escape. Root cause: the extraction prompt asks the model for a "pattern" field containing a raw regex, and the model wrote single backslashes (e.g. the literal text \.) instead of the two backslashes valid JSON requires to represent one backslash character. extract_rules.py now (a) tells the model explicitly, with worked examples, that every backslash in that field must be doubled, and (b) repairs any stray single backslash before the first parse attempt regardless of whether parsing would otherwise succeed — because \b specifically is valid JSON (it decodes to a backspace control character) while meaning something unrelated in regex (word boundary), so a repair-only-on-crash design would let that one through silently: a rule that looks fine, ships fine, and then just never matches anything. Caught a bug in that repair itself during local testing, before it ever reached CI — the first version could corrupt an already-correctly-escaped \\b into \\\b — fixed and re-verified against all three cases (the crash pattern, the silent-corruption pattern, and the already-correct pattern) before redeploying. Second attempt (run #3) got past extraction cleanly but failed at a different, unrelated step: peter-evans/create-pull-request couldn't open a PR — "GitHub Actions is not permitted to create or approve pull requests," a repo-level setting (Settings → Actions → General → Workflow permissions), not a code bug, even though the workflow already declared pull-requests: write. Fixed by enabling "Allow GitHub Actions to create and approve pull requests" on the repo. Third attempt (run #4) succeeded end-to-end in 59s and opened a real PR (#1, "claude-api-guard: new rules from Anthropic's release notes"), confirmed on GitHub. That's the full loop validated for real: fetch → parse → extract via the model → dedupe → append → PR, with two real bugs found and fixed along the way instead of assumed away.

CI / GitHub Action

action.yml packages the Python (and optionally JS/TS) scanner as a composite GitHub Action, so a project can get checked on every PR instead of someone running ast_scan.py by hand and remembering to. Two pieces:

• action.yml + action_combine.py — the action itself. Runs ast_scan.py (and js_scanner/ast_scan.js if scan-js: true), merges whatever findings files actually exist, and fails the job only at or above a configurable fail-on severity (default HIGH) — a MEDIUM/LOW heads-up shouldn't block a merge the way a HIGH one should.

• .github/workflows/self-check.yml — dogfoods the action against this repo on every push: one job asserts the severity gate correctly fails against example_project/ (which has known HIGH findings by design), the other asserts it correctly passes against a dedicated known-clean fixture, ci_fixtures/known_clean.py. Both are assertions about the action's own correctness, not about this repo's code health.

  That second job originally pointed at rules.py itself, on the reasoning "it doesn't call the Anthropic API, so it should be clean." The first real run on GitHub Actions (run #1, commit fedb0e7) came back red. Reproduced locally with python3 ast_scan.py rules.py: 7 findings, several HIGH. The reasoning was wrong — "doesn't call the API" and "contains no matching text" aren't the same property, and rules.py's entire job is to store the literal trigger strings (like client.beta.files, managed-agents-2026-04-01) as rule data, so the generic engine's ast.Assign matching legitimately finds them there. Fixed by pointing the job at a small, deliberately unrelated fixture file instead of reusing a file whose actual purpose guarantees it can never be "clean." Caught by getting a real Actions run — this is exactly the class of bug local YAML validation and the unit-tested Python logic couldn't have found (see below).

• examples/consumer-workflows/ — two templates (check-on-pr.yml, a weekly autofix-weekly.yml that opens a PR via the well-established peter-evans/create-pull-request action when autofix.py finds something to fix) showing how a downstream project would wire this in. Point at the real MarkMoneyMan/brittle@master instead of a placeholder — see "Publishing" below for how that reference (and the repo's name and visibility) got there.

What's validated and what isn't, stated plainly: all 4 YAML files parse as valid YAML, and the Python logic each step actually calls (ast_scan.py's exit code, action_combine.py's severity gate and $GITHUB_OUTPUT writing) was tested directly and behaves correctly across all 3 cases that matter — findings at/above threshold, findings below fail-on, and no findings. Local testing stopped there: nektos/act (a local Actions runner) installed fine but needs a Docker daemon to spin up runner containers, and this environment doesn't have one running (docker info confirms no daemon, not just a missing CLI).

That gap got closed for real once the repo was published (see "Publishing" below): self-check.yml ran on actual GitHub Actions and immediately found a real bug — the rules.py-as-known-clean-fixture mistake described above — that no amount of local YAML validation or unit-tested Python logic could have surfaced, because the bug wasn't in the YAML wiring or the scanner logic, it was in a test's assumption about its own fixture. After swapping in ci_fixtures/known_clean.py and re-pushing, run #2 (commit 7a97f7f) went green on both jobs — confirmed end-to-end on real GitHub Actions, not just locally. That's the whole point of dogfooding this against a real remote instead of stopping at "the YAML looks right": the bug this section describes only existed to find because a real run happened.

Publishing

Published to a real GitHub repository, originally private: github.com/MarkMoneyMan/Claude-api-goat. Getting there needed two rounds of Personal Access Token permission fixes — GitHub refuses to let a token without "Workflows" scope push changes to .github/workflows/*, even if it already has "Contents: Read and write" — which isn't obvious until the push is rejected with that exact error.

Both consumer-workflow templates were pointed at the real MarkMoneyMan/Claude-api-goat@master instead of the old YOUR-GITHUB-USERNAME placeholder, but "private" wasn't free to work around at the time — two different mechanisms were involved, and they were kept separate deliberately rather than papered over:

• check-on-pr.yml's uses: MarkMoneyMan/Claude-api-goat@master (an action reference) worked for a same-account repo like OddsScanner with no extra setup — GitHub's repo Settings → Actions → General → "Access" on Claude-api-goat covered this case, and same-account repos got it by default.
• autofix-weekly.yml's actions/checkout step with repository: MarkMoneyMan/Claude-api-goat (cloning a second repo's contents, to get autofix.py itself) was a different mechanism — the default GITHUB_TOKEN a workflow run gets is scoped only to the repo it's running in, same-account or not. That step needed a token: input pointing at a PAT (read-only "Contents" scope on Claude-api-goat was enough) stored as a secret in the downstream repo.

Update (2026-09-12): both of those caveats are gone, for unrelated reasons, not because anyone went and set up the workaround above. The repo was made public at some point before this update (not tracked here exactly when — worth noting as a small process gap: a change like that should have gotten its own log entry at the time it happened, not been noticed in passing while writing an unrelated section), which makes the whole private-repo access dance above moot: uses: and git clone/ pip install against a public repo need no token and no same-account relationship at all. Separately, the repo was renamed from claude-api-guard/Claude-api-goat to Brittle (github.com/MarkMoneyMan/brittle) — see the top of this README for why. Every reference in this README, action.yml, pyproject.toml, and the consumer-workflow templates was updated to the new name and the simpler public-repo setup in the same pass; the account-token debugging story above is kept as-written because it's a real thing that happened and is useful context for anyone hitting the same "Workflows scope" error on a still-private repo of their own.