Neon Ladder, the playtest-graded benchmark for local LLM stacks

Does your local LLM setup actually work for coding agents, or does it just benchmark well?

The best build this harness ever measured, 4,204 lines of polished arcade game, 17/19 on the static scorer, had a dead keyboard. The model exported its keymap as a Node module and never wired it into the browser. Syntax checks passed.

The two-minute runtime soak ran at 60fps with 2.5M draw calls. Then a human pressed Enter and nothing happened, because the first keydown threw on an undefined global and took the entire input system with it.

That's failure class #14. There are fourteen others, and every one was found the same way: by a person playing the game. Zero were found by static analysis.

That gap is the reason this repo exists.

What a run measures

Not the model. Your whole inference stack: engine, quant, drafter, speculative decoding, chat template, contract wording, thinking budget. Several of these move the result more than the model choice does, and standard benchmarks can't see any of them.

One cell:

1. A coding agent gets a fixed contract: build "Neon Overdrive", a 10-file HTML5 canvas breakout game. Exact file list, physics rules, serve behavior, pacing constants, all specified, all static-checkable.

2. Static scorer (19 checks) grades the code: structure, collision correctness, contract compliance, two different velocity-multiplication bug patterns. 3.

Runtime soak (120s, headless Chromium via raw CDP): boot errors are fatal, frames and draw calls must advance, reloads are detected. 4. Behavior probe (behavior-probe.sh): a scripted real-time pass for serve gating (the ball must not move before Space) and brick reflection (the ball must bounce, not pass through).

Catches collision-wiring classes the soak and the static scorer cannot see. 5. You play it. Two minutes with the arrow keys.

This is the grade that matters. The fifteen failure classes in the ledger all live here.

Wall clock: 5-40 minutes depending on model and effort tier. The result line is one comment's worth of data, generated for you.

The subject is validated on llama.cpp (Vulkan) + AMD Strix Halo, Qwen3.8-27B (four quants) and Qwen3.8-Flash-Next 125B-A6B, with DFlash2 and native-MTP speculative decoding. It will work on any local stack that can run a coding agent; the receipts below are from this rig.

Companion repo with the runnable pi-agent stack (server scripts, wiring, extensions): qwen38-strix-halo-harness.

Why not an existing benchmark

Model-capability benchmarks can't catch a chat template that silently ignores your effort flags, or a drafter whose acceptance collapses on reasoning-heavy turns. Those are configuration failures, and they're what this harness is built to surface.

The reference stack

Behind every number published here:

Two engine-driver findings that cost real time to learn (three-way A/B, same source commit): a native build beats the portable payload by 6-24% on decode (JSON-class 28.3 → 40.3 t/s), and the system Mesa stable beat the bundled devel snapshot on agent-relevant classes (deep decode +9.5%, emission +13%). Budget GPU memory against mem_info_vram_total + mem_info_gtt_total; both heaps count; the Vulkan allocator uses both. This nominal-128GB machine with a 16GB BIOS UMA carve exposes 16 + 109.7 ≈ 125.7 GiB total, split between a pinned VRAM heap and an evictable GTT heap.

One adaptive-drafting boundary worth knowing before you tune: the acceptance controller wins on DFlash2 block drafts and loses to fixed n4 on MTP chained drafts, measured both directions on the same harness. Use whichever wins on your stack, not whichever is newer.

The bundle

Use the bundled scorer with the bundled contract. They are calibrated as a set. Scores from modified contracts or different scorers are not comparable.

Install

Three dependencies, then clone:

# 1. System deps
sudo pacman -S nodejs chromium        # or your distro's equivalents

# 2. The pi coding agent (or build from source: github.com/earendil-works/pi)
yay -S pi-coding-agent-bin

# 3. This bundle
git clone https://github.com/aic0d3r/neon-ladder && cd neon-ladder

The agent harness is a separate repo: qwen38-strix-halo-harness holds the pi extensions, setup.sh (one command: providers, sidecar, halogen launch), start.sh (daily use), crash recovery, and the NPU retrieval toolkit. If you want the full daily-driver stack, install that too and run its ./setup.sh. This repo is the benchmark: game builds, prompts, scorers, results.

For the ladder you only need pi wired to your server. Minimal ~/.pi/agent/models.json entry (the llama.cpp path):

{
  "providers": {
    "llamacpp": {
      "baseUrl": "http://127.0.0.1:8080/v1",
      "api": "openai-completions",
      "apiKey": "dummy",
      "models": [{
        "id": "my-model",
        "reasoning": true,
        "contextWindow": 262144,
        "maxTokens": 32768,
        "compat": {
          "thinkingFormat": "chat-template",
          "chatTemplateKwargs": { "reasoning_effort": {"$var": "thinking.effort"}, "enable_thinking": {"$var": "thinking.enabled"} },
          "thinkingTokenBudgetField": "thinking_budget_tokens"
        }
      }]
    }
  }
}

Run one cell

Start your server, run the cell, play the game. Reference server command:

llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf \
  -md Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
  -mm mmproj-F16.gguf --mmproj-offload \
  --spec-type draft-dflash --spec-draft-n-max 4 \
  -ngl all -fa on -ctk f16 -ctv f16 -ctkd q8_0 -ctvd q8_0 \
  -c 262144 -b 4096 -ub 4096 --jinja \
  --chat-template-file sharp-v22.4.0.jinja \
  --host 127.0.0.1 --port 8080 --metrics
# KEY: --jinja must come BEFORE --chat-template-file, and never use
# --chat-template with a file path: it fails silently.

The cell itself:

export LLADDER=$HOME/neon-ladder-work && mkdir -p $LLADDER
bash run.sh build-run1 game-run1 medium "$(cat contract.txt)"
# ... typically 15-45 min later, on success the runner prints your result line

Open build-run1/index.html, press Enter, play two minutes. Then:

QUANT="Q4_K_XL" DRAFTER="DFlash2-Q4_M n4" PLAYTEST="Y - plays well" \
RIG="Strix Halo 395+8060S, Nathan v0.7.4.1" bash report.sh $LLADDER/build-run1 game-run1

That prints (and saves) a line like:

Q8_K_XL / DFlash2-Q4_M n4 / medium / 23min / static 17/19 / SMOKE-OK / Y - clean and polished / Strix Halo 395+8060S, 109GB, Nathan v0.7.4.1

Static score, soak verdict, effort, and wall clock come from the run artifacts automatically. You supply quant, drafter, playtest verdict, and rig, the four things only you know.

Reading the numbers (variance rules)

Divergent results are data, not contradiction. If you know the rules:

• Greedy nondeterminism was an engine bug, not sampling fate. Stale-KV between requests plus a top-k race above ~2k prompt tokens, real through v0.7.3 and upstream master, fixed in v0.7.4. Wall-time variance survives regardless (fix-loop rolls differ between builds): identical configs span 18-79 min. Treat N=1 cells as signals, not conclusions.
• About 1 in 4 cold rolls degenerates: the model answers with code in the reply instead of writing files.
• The runner wipes and retries these automatically.
• Unsloth re-uploaded quants under the same filenames (v2 → v3). Pin revisions when comparing, or your "same model" is two models.
• Power profile moves decode by a few t/s on APUs. State yours.
• Chat-template flag order is a silent trap (see the run recipe).

What to run

By available GPU memory (check mem_info_vram_total + mem_info_gtt_total, both heaps count):

By task:

• Agentic coding / build-from-scratch: Flash-Next at medium effort: best score measured (18/19) and the best speed/quality balance in playtest. The 27B is the pick under ~91GB free or for >131k single sessions.
• Deep-context analysis / RAG-style reading: Q8-27B: prefill-bound work is the one place big quants win on speed.
• Emission-heavy agent loops (tool calls, scaffolding, data transforms): adaptive draft sizing (--spec-draft-adaptive n3-7): +25% on tool-call classes. On MTP stacks, stick to fixed n4 (adaptive loses there).
• For everything else the two are within 2%.

Three knobs that matter more than they look:

• Template: on explicit contracts, decode is a wash; Sharp halves wall time on long sessions, stock produced the deepest build. Never pass a template file path to --chat-template: it fails silently (use --chat-template-file, --jinja first).
• Engine: build Nathan's line from source. The portable payload leaves 6-24% on the table.
• System Mesa stable beat the bundled devel snapshot on agent-relevant classes.
• Contract: two per-model blind spots (a serve rule, a pad physics clause) were each cured by one explicit sentence after showing up as systematic failures. When a model tier fails the same way every time, fix the contract before blaming the model.

The result ledger

Every row is a receipt from this harness. Playtest verdicts are the user's, recorded as given.

Final board (2026-09-06): every effort tier at N≥2 on both models. Static bands fully overlap (13-18). The best score is Flash-Next medium at 18/19.

Fifteen playtest-found failure classes; the last three (gameover-exit, dead keyboard, level-2 freeze) were runtime wiring that passed static score and the soak.

Five findings that only an agent workload surfaces

1. Chat templates are per-model infrastructure. Effort kwargs that work on one model's template are silently ignored by another's. 2.

Spec-decode controllers are mechanism-dependent. Acceptance-adaptive draft sizing wins on block drafters and loses on chained MTP drafts, measured both ways on the same harness. 3. Fast and healthy ≠ agent-ready. A stack can benchmark beautifully and still fail a build contract.

That gap is the reason this repo exists. 4. Per-model blind spots are usually spec gaps. Two models each failed the same subsystem on every roll.

One explicit contract sentence cured each, 2-for-2. 5. On explicit contracts, model size buys speed, not quality. A 125B MoE and a 27B, matched effort and environment: one check apart, both playable, the MoE in 7.8× less wall time with 4.3× less reasoning.

Pick by your token budget, not by a quality assumption.

Requirements

• Node (for the CDP clients and node --check), chromium (headless)
• The pi coding agent with a llama.cpp provider
• A llama.cpp server with speculative decoding (DFlash2 or MTP capable)
• Extensions and skills are not part of the protocol. If yours inject context into sessions, disable them (pi config) or your numbers carry a variable the recipe does not account for

NPU retrieval toolkit + pi harness

Both moved to qwen38-strix-halo-harness: rag-index.py / rag-query.py (NPU semantic search, ~100ms end-to-end), the codebase_search Pi extension, progress-tracker crash recovery, sidecar compaction, and the one-command setup. Measured against this repo on 2026-10-01: 8/10 top-3 retrieval hits at ~100ms.