pokertools-arena

A browser-first AI poker benchmark. Seat Jev and OpenAI-compatible models at the same no-limit Texas Hold'em table, watch every card and decision as a spectator, and let the tournament run autonomously until one model wins.

Current release: 0.18.1. The npm package now includes the shared .env parser required by its launcher, with a packed-artifact smoke test preventing future npx pokertools-arena module-resolution failures.

https://github.com/user-attachments/assets/64d63da2-d904-4b63-a327-c5fa816e6be9

---

Contents

• Why pokertools-arena
• Features
• Quick start
• Configure the table
• .env bootstrap
• Benchmark modes
• Fairness
• Methodology
• Testing
• Repository layout
• Documentation
• Privacy and security
• License

---

Why pokertools-arena

Text-model benchmarks are usually static question sets. Poker is a better stress test: adversarial, sequential, incomplete-information and unforgiving of bad reasoning. pokertools-arena turns that into an observable, reproducible benchmark.

• Same information for every seat. Poker legality, hole-card masking and public context are generated by the PokerTools engine, not by prompts.
• Any OpenAI-compatible endpoint. Hosted APIs or a local gateway; OpenRouter and direct Jev are optional presets.
• Nothing to deploy. The benchmark is fully client-side. There is no application server, database or backend API.
• Everything is inspectable. Watch live, replay any decision, export logs and replay PNGs.

Models only select from arena-generated legal action families and sizes, and every chosen move is re-validated by the engine before it is applied.

Features

Quick start

Prerequisites: Node.js ≥ 24, a modern browser with Web Crypto (Chromium for table-only recording), and an OpenAI-compatible endpoint.

Option A — npm launcher (recommended)

npx pokertools-arena

The CLI starts a tiny local static server and opens the browser. It exists only to avoid file:// restrictions. Poker state, orchestration, model calls, logs and API keys stay in the browser.

Option B — single HTML file

npm install
npm run build

Then open dist/pokertools-arena.html. The build bundles the PokerTools browser engine locally; no runtime CDN is used.

Option C — GitHub Pages

The canonical deployment is https://pokertools-arena.github.io/. Enable Settings → Pages → Source: GitHub Actions once; .github/workflows/pages.yml builds and deploys dist/ on every push to main.

Configure the table

The table is the player editor. A fresh launch shows empty seats around the felt.

1. Click a seat to add or edit a model.
2. Choose a connection, enter a model ID and pick a protocol.
3. Configure Settings → Connections (base URLs and keys) and Settings → Tournament (stack, blinds, clock, benchmark mode, architecture, representation).
4. Press Start. Only configured seats enter the tournament, and seat editing locks while it runs.

Defaults and validation limits live in src/config/arena-config.js. A host page may override any value before loading app.js by assigning a partial object to window.__POKERTOOLS_ARENA_CONFIG__; the arena deep-merges it with the documented defaults.

.env bootstrap

The launcher reads .env from the current working directory (falling back to the package root) and injects only supported fields into page memory. Disable with --no-env. A real user .env is never moved, printed or archived.

OPENAI_BASE_URL=https://openrouter.ai/api/v1
OPENAI_API_KEY="sk-or-v1-..."
OPENAI_PLAYER1=google/gemma-4-26b-a4b-it
OPENAI_PLAYER2=qwen/qwen3.8-flash
OPENAI_PLAYER3=typesafe/jev-1.13
OPENAI_STARTING_STACK=3000
OPENAI_AUTOSTART=0
OPENAI_MAX_DECISIONS=0

See .env.example.

Benchmark modes

Two tournament-wide settings keep every model on equal footing:

• Strategy (default): every seat receives a deterministic heroHand classification from @pokertools/evaluator.
• Raw cognition: no heroHand; each model infers hand strength from raw cards.

They are separate benchmark tracks. Public reports never aggregate them into a single model score.

Decision architecture is also tournament-wide. hierarchical (default) asks for an action family, then a validated size for BET/RAISE. flat is retained only as a diagnostic baseline. Representation is canonical_json, compact_json or markdown; all seats receive identical semantics.

Fairness

Every seat receives the same canonical information policy: its own hole cards, the public board/pot/stacks/positions/current bets, the same deterministic legal actions, current-hand public history, last eight public hands, and identical deterministic public opponent statistics. Opponent hole cards, other models' reasoning, prior Jev probabilities and provider metadata are never exposed.

Player masking is fail-closed. All-in players count as remaining until a completed hand eliminates them.

The full contract, action-family rules, sizing buckets, shared clock and fairness assertions are documented in docs/architecture/CONTEXT.md.

Methodology

The benchmark methodology is built around a paired fixed-state corpus:

• the exact same immutable state is used for flat and hierarchical, for every model, mode and representation;
• execution is deterministically interleaved with a recorded experimentSeed;
• family accuracy and sizing accuracy (conditional on a correct family) are reported separately from final-action accuracy;
• behavioral proportions carry sample size and Wilson 95% confidence intervals;
• fragmentation_flip_rate measures how often the broad family changes when only the number of same-family size choices changes;
• counters are standardized (pokerDecisions, familyModelCalls, sizingModelCalls, totalModelCalls, httpRequests, httpRetries, …) and one machine-readable summary is the single source of truth for every report.

A hierarchical aggressive poker decision is 1 poker decision, 2 model calls and 2+ HTTP requests if retries occur. Aggression is not a goal; the goal is symmetric representation, reproducibility, correctness and interpretable statistics.

Real tournament A/B is end-to-end behavioral validation, not a controlled architecture comparison, because actions change future states.

Testing

npm test                    # build + release checks + offline unit + integration
npm run test:unit           # corpus, hierarchy/fairness, methodology, sizing, paired, archive, report
npm run test:integration    # @pokertools/engine/browser + launcher .env bootstrap
npm run test:diagnostics:dry
npm run test:real           # real API paired corpus + diagnostics (uses .env)
npm run test:tournament-ab  # real flat-vs-hierarchical tournament (uses .env)
npm run report:diagnostics  # build logs/release-<version>/summary.json + summary.md
npm run release:archive     # source archive without secrets/logs/dist

npm test is fully offline apart from dependency installation. Real-API diagnostics are manual or secret-gated in CI. See tests/README.md.

Repository layout

pokertools-arena/
├── README.md · LICENSE · package.json · build.mjs
├── bin/pokertools-arena.mjs            local static launcher + .env bootstrap
├── src/                                browser application source
│   ├── index.html · app.js · styles.css
│   ├── config/arena-config.js           defaults, limits, presets, host overrides
│   ├── lib/decision-core.js            canonical DOM-free decision core
│   ├── benchmark/scenarios.js          in-app sanity scenarios
│   ├── env/arena-env.js                .env placeholder injected by the launcher
│   ├── shims/crypto.cjs                Web Crypto shim for the PokerTools browser entry
│   └── assets/                         favicon and social image
├── tools/
│   ├── diagnostics/                    corpus, harness, stats, counters, paired, report
│   └── release/archive.mjs             source archive builder
├── tests/
│   ├── unit/ · integration/ · real/ · analysis/
│   └── release-check.mjs
├── docs/
│   ├── architecture/CONTEXT.md
│   ├── releases/RELEASE_NOTES.md
│   ├── diagnostics/SUMMARY.md + summary.json
│   ├── verification/0.3.x.md
│   └── legal/THIRD_PARTY_NOTICES.md
└── .github/workflows/                  CI, GitHub Pages, npm publish, manual real diagnostics

Generated and gitignored: dist/, pokertools-arena.html, logs/, node_modules/.

Documentation

• Architecture and invariants → docs/architecture/CONTEXT.md
• Chronological release history → docs/releases/RELEASE_NOTES.md
• Benchmark methodology and results → docs/diagnostics/SUMMARY.md
• Machine-readable release summary → docs/diagnostics/summary.json
• Test suites → tests/README.md
• Third-party notices → docs/legal/THIRD_PARTY_NOTICES.md

Privacy and security

• API keys live only in current page memory and are excluded from saved setup state, tournament snapshots, exported logs and release archives.
• The launcher serves the .env payload only to loopback requests whenever it contains a configured API key.
• Models never receive hidden information and never construct engine commands.
• Replay images and recordings are produced locally; nothing is uploaded.

License

MIT — see LICENSE.