PokeBench: reproducible Pokemon gameplay benchmarks for AI agents
See the codePokeBench evaluates how AI agents play Pokemon Red. Models choose routes, battles, recovery and resource use in the real game. PokeSim Core decodes observations and executes controller inputs. PokeBench owns the tasks, information boundaries, budgets, scoring and replay checks.
The historical development suite has 28 active tasks. Twenty-two contribute to the development ranking and six tactical candidates remain experimental. Five trivial tasks are retired. Development results are useful evidence, but they used evolving harness versions and are not a controlled release comparison.
The exploratory leaderboard preserves completed and partial results across framework improvements. Runs record their model, task, starting save, controls and runtime so material differences remain visible. Routine backend fixes do not discard earlier results or require a full rerun.
git clone https://github.com/afk-sapien/pokebench.git
cd pokebench
uv sync --locked --extra dev
uv run pokebench --help
Python 3.12 is selected for reproducibility. The Python distribution and import remain pokeagent-bench and pokeagent_bench for compatibility. Both pokebench and the older pokeagent command work.
Supply your own Pokemon Red ROM and locally prepared fixtures. ROMs, emulator saves and generated game tables are not distributed. PyBoy RS 0.1.1 is installed from hash-pinned wheels. uv sync installs PokeSim Core 0.6.1.dev1 from the wheel in vendor/. The integration history is in the integration notes. A sibling PokeSim checkout is not a runtime dependency.
The gameplay track supplies a screenshot, readable dialogue, observed terrain and objects, party health and moves, inventory and recent action outcomes. A notebook and bounded conversation preserve working context. Eight-turn segments reset from deterministic observations and the agent's notes. There are no paid summarization calls.
Menu-operation tasks disable shortcuts and require individual button presses. Strategy and navigation tasks retain mechanical shortcuts. The task focus and actual permissions are recorded. Within a conversation segment, party updates send only changed HP, status and move fields. Full state returns at every segment start.
The agent chooses every route and tactical decision. Item, move and party shortcuts execute real controller input, preserving battle turns and item costs. Continue-only dialogue is collected until an actual decision is needed. The evaluator's hidden completion flags and future game state are not exposed. A separate visual track remains available for perception experiments.
See the information contract, bounded context design and menu shortcuts.
The subscription adapters use the official Codex and Claude Code CLIs. Existing CLI login is required. There is no automatic switch to paid API usage. Exact model IDs, requested effort, provider limitations, input/output usage and cache accounting are recorded. Effort names are not equivalent amounts of computation across providers.
uv run pokebench run \
--provider claude --model claude-sonnet-5-5 \
--codex-policy bounded --track gameplay --reasoning-effort medium \
--rom /private/red.gb --scenario /private/validated-fixture \
--game-data /private/game-data --goal challenge \
--max-total-tokens 500000 --output /private/new-run
Use --provider codex with an available exact Codex model ID for the same gameplay protocol. See Claude subscription setup and limitations.
Budgets count reported input plus output tokens, including cached input once. They measure inference workload, not the amount billed on a subscription. API-equivalent dollar estimates are estimates, not subscription charges. Admission reserves and between-decision checks limit spending. An in-flight response can exceed a threshold. Missing usage stops the release sweep for review.
The catalog includes starter acquisition, parcel delivery, navigation, gym battles, the League, and calibrated tactical battles. Six experimental tasks cover switching out of a bad matchup, healthy capture, team rescue, PP management, adaptive Champion play and a recovery detour. Their selected winning reference traces establish feasibility, not guaranteed success or independence from luck.
Each local run preserves a manifest, exact starting state, controller trace, observations, usage and results. Offline replay checks the recorded trajectory without another model call. The public website exports allowlisted results and screenshot replays. Private provider logs, model notes, session identifiers, ROMs and saves stay local.
uv run --locked --offline pytest -q
uv run --locked --offline ruff check .
The source tree retains earlier observation policies so historical traces remain interpretable. They are not interchangeable experimental conditions. See third-party notices. Pokemon is an unofficial research setting for this project, with no affiliation or endorsement.
The executable catalog is split into two benchmarks. PokeBench has 47 focused tasks plus one inventory variant across navigation, battle tactics, manual menu operations, preparation and quests. PokeBench Long is one continuous full-game campaign with 50 scored milestones worth 1,000 points.
pokebench catalog and new catalog-batch runs default to focused PokeBench. Use --benchmark pokebench-long to inspect or run the campaign separately. Each batch has its own token ceiling. Campaign progress never contributes to the focused-task ability average. The local portal has separate PokeBench and PokeBench Long pages.
See the two benchmark tracks for execution and scoring. No models are launched by inspecting the catalog.
Model input defaults to the current screenshot with observed action results and optional screenshot recall. The replay separately shows decision start and action result for human review, plus the exact image sent to the model. See the screenshot comparison for the explicit --presentation before-after experiment.
See catalog runtime and commands for setup, task controls, preparation assessment and result aggregation. Existing benchmark results remain available alongside these new definitions.
PokeBench: reproducible Pokemon gameplay benchmarks for AI agents
See the codePokeBench evaluates how AI agents play Pokemon Red. Models choose routes, battles, recovery and resource use in the real game. PokeSim Core decodes observations and executes controller inputs. PokeBench owns the tasks, information boundaries, budgets, scoring and replay checks.
The historical development suite has 28 active tasks. Twenty-two contribute to the development ranking and six tactical candidates remain experimental. Five trivial tasks are retired. Development results are useful evidence, but they used evolving harness versions and are not a controlled release comparison.
The exploratory leaderboard preserves completed and partial results across framework improvements. Runs record their model, task, starting save, controls and runtime so material differences remain visible. Routine backend fixes do not discard earlier results or require a full rerun.
git clone https://github.com/afk-sapien/pokebench.git
cd pokebench
uv sync --locked --extra dev
uv run pokebench --help
Python 3.12 is selected for reproducibility. The Python distribution and import remain pokeagent-bench and pokeagent_bench for compatibility. Both pokebench and the older pokeagent command work.
Supply your own Pokemon Red ROM and locally prepared fixtures. ROMs, emulator saves and generated game tables are not distributed. PyBoy RS 0.1.1 is installed from hash-pinned wheels. uv sync installs PokeSim Core 0.6.1.dev1 from the wheel in vendor/. The integration history is in the integration notes. A sibling PokeSim checkout is not a runtime dependency.
The gameplay track supplies a screenshot, readable dialogue, observed terrain and objects, party health and moves, inventory and recent action outcomes. A notebook and bounded conversation preserve working context. Eight-turn segments reset from deterministic observations and the agent's notes. There are no paid summarization calls.
Menu-operation tasks disable shortcuts and require individual button presses. Strategy and navigation tasks retain mechanical shortcuts. The task focus and actual permissions are recorded. Within a conversation segment, party updates send only changed HP, status and move fields. Full state returns at every segment start.
The agent chooses every route and tactical decision. Item, move and party shortcuts execute real controller input, preserving battle turns and item costs. Continue-only dialogue is collected until an actual decision is needed. The evaluator's hidden completion flags and future game state are not exposed. A separate visual track remains available for perception experiments.
See the information contract, bounded context design and menu shortcuts.
The subscription adapters use the official Codex and Claude Code CLIs. Existing CLI login is required. There is no automatic switch to paid API usage. Exact model IDs, requested effort, provider limitations, input/output usage and cache accounting are recorded. Effort names are not equivalent amounts of computation across providers.
uv run pokebench run \
--provider claude --model claude-sonnet-5-5 \
--codex-policy bounded --track gameplay --reasoning-effort medium \
--rom /private/red.gb --scenario /private/validated-fixture \
--game-data /private/game-data --goal challenge \
--max-total-tokens 500000 --output /private/new-run
Use --provider codex with an available exact Codex model ID for the same gameplay protocol. See Claude subscription setup and limitations.
Budgets count reported input plus output tokens, including cached input once. They measure inference workload, not the amount billed on a subscription. API-equivalent dollar estimates are estimates, not subscription charges. Admission reserves and between-decision checks limit spending. An in-flight response can exceed a threshold. Missing usage stops the release sweep for review.
The catalog includes starter acquisition, parcel delivery, navigation, gym battles, the League, and calibrated tactical battles. Six experimental tasks cover switching out of a bad matchup, healthy capture, team rescue, PP management, adaptive Champion play and a recovery detour. Their selected winning reference traces establish feasibility, not guaranteed success or independence from luck.
Each local run preserves a manifest, exact starting state, controller trace, observations, usage and results. Offline replay checks the recorded trajectory without another model call. The public website exports allowlisted results and screenshot replays. Private provider logs, model notes, session identifiers, ROMs and saves stay local.
uv run --locked --offline pytest -q
uv run --locked --offline ruff check .
The source tree retains earlier observation policies so historical traces remain interpretable. They are not interchangeable experimental conditions. See third-party notices. Pokemon is an unofficial research setting for this project, with no affiliation or endorsement.
The executable catalog is split into two benchmarks. PokeBench has 47 focused tasks plus one inventory variant across navigation, battle tactics, manual menu operations, preparation and quests. PokeBench Long is one continuous full-game campaign with 50 scored milestones worth 1,000 points.
pokebench catalog and new catalog-batch runs default to focused PokeBench. Use --benchmark pokebench-long to inspect or run the campaign separately. Each batch has its own token ceiling. Campaign progress never contributes to the focused-task ability average. The local portal has separate PokeBench and PokeBench Long pages.
See the two benchmark tracks for execution and scoring. No models are launched by inspecting the catalog.
Model input defaults to the current screenshot with observed action results and optional screenshot recall. The replay separately shows decision start and action result for human review, plus the exact image sent to the model. See the screenshot comparison for the explicit --presentation before-after experiment.
See catalog runtime and commands for setup, task controls, preparation assessment and result aggregation. Existing benchmark results remain available alongside these new definitions.