😐👍...lgtm?

Prove that your tests actually test something. Powered by Jev and your own TypeSafe API key.

Your agent wrote 40 tests. They're all green. What do they prove? lgtm reads every test block with its implementation and tells you which ones are useless.

It runs on Jev by TypeSafe, with your own TYPESAFE_API_KEY.

With this, you can prove that your agent actually wrote code that actually works, so you can say it lgtm 😐👍.

Evaluated and hill-climbed on real live apps built by Stardeck: 15,000+ real test blocks scored, 262 each read against its implementation and labelled, every check's threshold fitted so that when lgtm points at a test, the test is worth your time.

What you get

• the lgtm CLI: run it on a file, a directory or --diff, in your terminal or in CI
• a /lgtm skill for your coding agents, so the agent that wrote the tests runs the audit and fixes what it finds
• an /actually-test skill: write the tests the change needs, prove them red, then iterate with lgtm until they pass

Install

bun add -g @stardeckai/lgtm
pnpm add -g @stardeckai/lgtm
npm i -g @stardeckai/lgtm

# or per project
pnpm add -D @stardeckai/lgtm   # then: pnpm lgtm ...   (npm: npx lgtm ...)

If lgtm is not found after a global install, the package manager's global bin directory is not on your PATH. Run npm prefix -g (or pnpm bin -g, bun pm bin -g) and add its bin to PATH, then open a new shell or run rehash in zsh.

Setup

lgtm init                    # paste your API key, then install the /lgtm and /actually-test skills

It asks for your TypeSafe API key (get one at https://typesafe.ai), then asks whether to install the /lgtm and /actually-test skills with npx skills.

The key lands in ~/.config/lgtm/config.json (mode 0600); TYPESAFE_API_KEY in the environment wins over it.

Skip the prompts with --skill <where>:

If the skills CLI can't run, init falls back to writing the Claude Code skills itself.

Running lgtm before setup exits with 😐✋ No API key. Run: lgtm init.

Use

lgtm .                     # every *.test.* / *.spec.* file under a directory (a path is required)
lgtm src/user.test.ts      # one file, or a directory
lgtm --diff origin/main    # only tests changed vs a base, with the diff as evidence
lgtm --diff                # just what you're working on: changed and new tests, plus tests of changed code, vs the default branch
lgtm --dry-run src         # only the plan: files, estimated cost and runtime; no key needed

Every run starts with that plan and asks Run? [Y/n]. Outside a terminal (CI, an agent) it stops after the plan unless you pass --yes.

lgtm . --yes --format json # non-interactive

For a one-off run without installing: npx @stardeckai/lgtm --dry-run src.

test/payment.test.ts:42  "rejects expired cards"
  😐👏 mocks-seam-under-test 0.93 — The collaborator that decides this behaviour is a mock, so the test only proves the mock works.

test/refund.test.ts:17  "refunds a captured charge"
  😐🤏 swallowed-error-as-success 0.88 — The test stays green whether the error is caught, logged, or never thrown; it never pins the specific failure.

😐🫵  2 tests prove nothing.
4 contract-integration · 19 mocked-seam · 8 pure-logic

98120 input tokens used ≈ $0.0041 ($0.0001 per test)
9.8s (0.3s per test)

Four faces, one per family: 😐🤏 the assertion proves this much, 😐👏 you tested the mock, 😐🤌 what exactly are we doing here, 😐🫸 do not merge this. Colour is severity (red at 0.9 and above, yellow at or over the check's threshold, dim for --verbose suspects). The verdict is one line, 😐👍 N tests. fine. allegedly. or 😐🫵 N tests prove nothing. --format github and --format json stay plain.

/actually-test

Gets your agent to actually test the code it just wrote, and iterates on /lgtm (don't worry, it's cached) until it proves the code is actually tested.

Flags

Why

Coding agents are prolific test writers and terrible test critics. They mock whatever is inconvenient, assert that the mock was called, compute the expected value with the code under test, and hand you a suite where every line is covered and nothing is verified. A test that checks a trace's name. A test for the thing you decided not to build. Nobody reads those files. The PR says "added tests" and gets merged.

The bug that pages you lives in a seam. One side writes, the other reads, and every unit test mocked at least one of them to agree. No linter catches that. It's a judgment call, and judgment used to cost a senior engineer's afternoon per PR.

Reads like a review, runs like a linter. Every finding is one test, one smell, one probability, one sentence you can act on. 😐👏 mocks-seam-under-test means you tested the mock. 😐👏 reimplements-logic means the test and the implementation share the same bug. The summary tells you how much of your suite actually crosses a seam.

Opinionated by design. Few wide tests with real collaborators beat a hundred mocked units. Delete with confidence: a good audit shrinks the suite. And when the suite is clean, it says so.

Now it costs a cent. Jev bills $0.042 per million input tokens and answers in under a second. lgtm shows you the bill and the runtime before it spends, and caches every answer. Every check holds precision 1.00 on the held-out slice of the corpus (evals/).

What lgtm likes

A test that crosses a seam with both sides real and asserts that they agree. That test fails when the wiring breaks, which is how most things actually break.

It dislikes tests of one-line helpers (any real test of the feature covers them for free), tests that mock everything except the function name, and one-off assertions that would survive the feature being deleted. So the summary prints what your suite is made of: N contract-integration · N mocked-seam · N pure-logic. --classes lists every test with its class, which is the number to watch during an audit. Retiring three unit tests for one wider test that really fails is a win, not a coverage loss.

Checks

--format json adds a checks map with a longer explanation and the fix, once per check.

Evals

The corpus is 521 public + 262 private labelled test cases, synthetic and anonymized real-world, with positives, hard negatives and genuinely good tests. The ground truth is kept in expect.json so it never reaches the model.

The 262 private cases come from real Stardeck customer apps and from Stardeck's own codebase, each read against its implementation, labelled, and anonymized. They are scored in these numbers but not published, because anonymization removes names, not shape. The 521 public cases in evals/cases reproduce with pnpm eval alone.

Scores are at each check's own threshold. 165 of the cases are holdout, never used to fit a threshold or a prompt.

Test class accuracy: 665/783 (0.85) on all cases, 145/165 (0.88) on holdout.

A full cold run of the corpus is about 3,988,609 input tokens ≈ $0.1675 (estimated from the states; the last run spent $0.0301 after cache hits).

Every miss and false positive is listed in evals/RESULTS.md. The public cases reproduce with pnpm eval.

CI

- uses: actions/cache@v4
  with:
    path: node_modules/.cache/lgtm
    key: lgtm-${{ hashFiles('**/pnpm-lock.yaml', '**/package-lock.json') }}
    restore-keys: lgtm-
- run: npx @stardeckai/lgtm --diff origin/${{ github.base_ref }} --yes --format github
  env:
    TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}

The cache step is optional: --diff already limits a PR run to the tests it touched, and answers are keyed by the full state (test, imports, implementation, guidelines) plus the check wording and lgtm version, so a hit means nothing relevant changed. Do not commit the cache directory; it churns on every refactor and never shrinks.

lgtm is advisory by default: it prints findings and exits 0. Fetch enough history for the base ref (fetch-depth: 0) and add --fail once the findings are clean enough that you want them blocking.

Cost

TypeSafe bills $0.042 per million input tokens and nothing for output, so lgtm is cheap enough to run on every PR. One request per test block. Each state is trimmed to at most 100,000 chars (~25,000 tokens, under TypeSafe's 32k state limit): the test code, the whole test file with the block fenced, the file's imports and sibling test names, up to 60,000 chars of the imported implementation (one hop deep) and the test sections of any CLAUDE.md/AGENTS.md above it. --lean cuts that back to an 8,000-char implementation and no test file or guidelines.

Answers are cached by state hash under node_modules/.cache/lgtm, so a re-run after editing one test only pays for that test. The implementation source is the main cost lever: --no-impl cuts the bulk of each request at the price of weaker would-pass-if-broken and reimplements-logic answers. Every run prints its input tokens and the estimated cost.

Config

The key lives in ~/.config/lgtm/config.json. TYPESAFE_API_KEY in the environment always wins over it.

lgtm key <new-key>           # swap the saved key; `lgtm key` alone prompts
lgtm usage                   # cost so far: all time, last day, last week, this worktree
lgtm clear-cache             # drop this project's cached answers (node_modules/.cache/lgtm)
lgtm skill                   # (re)install the /lgtm and /actually-test skills, e.g. to add another agent
lgtm init                    # both steps again

Thresholds

Each check has its own threshold, fitted on the eval corpus to the lowest probability that still gives zero false positives (the threshold column above). That trades recall for precision on purpose: a finding should be worth your time. To see what sits just under the line, run --verbose, or --threshold 0.5 --format json on a suite you know well and pick your own number. A check that is consistently wrong for your codebase belongs in --skip.

Development

This repo uses pnpm.

Each check is one file under src/checks/<category>/; add a new one there and register it in src/checks/index.ts.

pnpm install
pnpm typecheck
pnpm test
pnpm build
node dist/cli.js --dry-run src   # no API key needed
npm link                         # expose this checkout as the global `lgtm` (symlink to dist/cli.js; rebuild to update)
pnpm gen:skill                   # regenerate skills/*/SKILL.md after changing a check or a skill (a test guards drift)
pnpm eval                        # run the labelled corpus against Jev; --offline re-scores, --fit-thresholds refits

License

MIT