FightBench: a replay-verified benchmark for AI decision models.
See the code
A replay-verified benchmark for AI decision models. First game: Mortal Kombat II (Genesis), Liu Kang vs the CPU, played from game memory, not pixels.
Board: https://fightbench.aipdv.com
Setting up with a coding agent? Point it at AGENTS.md.
mk2-liukang-v1A frozen bench on this emulator: Liu Kang vs the 15 VeryHard CPU fights, 10 repeats each. Rank is mean damage dealt. Wins are reported, not ranked.
edition/mk2-liukang-v1.json holds the 8 situations, 17 moves, ladder, ROM and state SHA-1s, and seed rule. It is the clef2 prompt, word for word. A changed string is a new edition.mk2-liukang-v1|repeat|fight, every 15 frames.script (rules bot) and random (17 moves).uv run fightbench.py smoke # free, needs the ROM, skips without it
uv run fightbench.py table --policy my.py:MyPolicy # write submissions/<model>--<version>__policy_table__mk2-liukang-v1.json
uv run fightbench.py live --policy my.py:MyPolicy # write submissions/<model>--<version>__live__mk2-liukang-v1.json
uv run fightbench.py replay submissions/<file>.json # maintainer: writes results/verified/ and site/board.json
uv run fightbench.py replay script # floor row, full ladder
uv run fightbench.py replay pending # maintainer: replay every submission without a result
python3 fightbench.py board # rebuild site/board.json from results/verified (no ROM)
--policy clef is Clef-flash. table --policy clef reads ab/clef-cache.json only and never calls Workers AI. Submit by committing only your submissions/ file in a pull request. A policy that calls a paid model sets usage (input_tokens, output_tokens, usd), and table adds call count and latency. Replay checks the game outcome. It does not re-call the model, so the model name, cost, and latency are attested by the submitter.
Nobody edits the site by hand. results/verified/*.jsonl is the source of truth.
submissions/ file. The board lists it as pending.gh pr checkout <N> && uv run fightbench.py replay pending, then commits results/verified/ to the PR. The ROM stays on the maintainer's machine, so CI cannot replay.fightbench.py board and publishes site/ to https://fightbench.aipdv.com. Each PR from this repo also gets a preview URL.The board shows mean damage dealt with a 95% interval, wins, damage taken, survival time, $ per ladder, tokens, call latency, top moves, and damage per opponent.
Clef-flash (Cloudflare Workers AI) plays MK2 on the Sega Genesis. It reads game memory, not pixels.
Clef-flash beats the VeryHard CPU, replayed frame for frame from the sim. Clef answered 8 situation questions in one call before the round (6,008 tokens, $0.00054). Code reads game memory and plays a move from Clef's probabilities. Full quality: recording/veryhard-win-shang-tsung-2.mp4.
Real or luck? (Pre-edition demo, not a FightBench score.) 200 more rounds against the same VeryHard Shang Tsung, prompt clef4, fresh seeds:
| Arm | Wins | Damage dealt (of 120) | Rounds with 90+ damage | Closest loss |
|---|---|---|---|---|
| Clef v4 | 1 | 55.0 ± 3.1 | 13 | 4 health short |
| Rules bot | 1 | 43.2 ± 2.8 | 7 | 12 short |
| Random | 0 | 10.0 ± 2.1 | 0 | 37 short |
Wins are rare (about 1 in 200), and on this one fight with the clef4 prompt Clef out-damaged the rules bot. The frozen FightBench ladder is the fair test: see the board. The face in the corner is MK2's "Toasty!" Easter egg.
Other recordings: recording/demo.mp4 (2 min), recording/highlights.jpg, recording/calibration-moves.png.
attacks_in_reach (measured reach per move).choice question with a probability for each of 13 moves. Up to 3 requests are in flight. Late answers are dropped.Memory map and save states come from stable-retro's MortalKombatII-Genesis-v0 integration.
roms/.integration/ with stable-retro's memory map and save states:
uv run mk2_clef.py --rom "roms/Mortal Kombat II (W).zip"
(W) dump has a different SHA-1 and works..env holds CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN (Workers AI only). See .env.example.uv run mk2_clef.py # opens http://localhost:8000, press Play
uv run mk2_clef.py --calibrate # saves calibration/moves.png, one frame per move
| Flag | Default | Meaning |
|---|---|---|
--model | clef-flash | or clef (bigger, slower, 2.7x the price) |
--state | cycle all, easiest first | one fight, for example Level1.LiuKangVsJax |
--temperature | 0.3 | 0 = always the top move, 1 = Clef's raw distribution |
--interval | 0.12 | seconds between requests |
--max-inflight | 3 | requests in flight at once |
--frame | off | also send the game frame (about 75 more tokens per request) |
--variant | baseline | tactics or random (see A/B test) |
--ab N | off | headless A/B test, see below |
--fights N | 14 | A/B: only the first N ladder fights (saves quota) |
--arms | all variants | comma list of variants to run in A/B, e.g. tactics,v2,random |
--sim | off | headless unpaced experiment, e.g. random@0,script@24,clef2@0 |
--repeats | 3 | sim: rounds per fight per arm |
--ladder | VeryEasy.LiuKang | sim: save-state prefix; LiuKangVs = the 15 VeryHard fights (held out) |
uv run mk2_clef.py --ab 1
Plays each of the 14 VeryEasy Liu Kang fights (Rayden to Shao Kahn) once per variant, back to back, with the order flipped every fight. Writes one JSON line per round to ab/results-*.jsonl and prints a summary (wins, damage dealt and taken, neurons).
| Variant | What Clef gets |
|---|---|
baseline | 13 moves, distance, reach, health, enemy movement |
tactics | baseline + duck, high and low fireball, flying kick; "got hit / hit enemy in the last 0.75 s"; rules: no long jump-ins, fireball or duck at range, block after a hit. A code guard drops jump_in when nothing reaches. |
v2 | same 17 moves as tactics with "Use when" descriptions matching state words; state is plain-English lines; adds threat question; code reflex blocks instantly when enemy attacks close; threat override when threat > 0.7 and close. |
random | control: uniform random moves at Clef's pace (0.4 s delay), no API calls |
Sim mode (--sim) runs the same 14 fights headlessly with code-only arms (no Clef, no credentials, unpaced). Arms are random, script, or either with +reflex; append @N to set latency in frames (default 24 ≈ Clef's 400 ms). Results go to ab/sim-*.jsonl. Useful for H1 (does reflex help?), H2 (does lag hurt?), H3 (does a script beat random?).
Clef table arms (clef, clef2, clef3, clef4, prompts in TABLE_PROMPTS). Clef answers one choice question per situation (enemy jumps, enemy attacks close, 60 to 80 px, and so on) for each opponent. Code only detects the situation each frame and samples Clef's answer for it. Answers are cached in ab/clef-cache.json, so play has no model delay and repeats are free. A new prompt costs about 500 neurons for 14 opponents. Use --temperature 1 (Clef's raw distribution): it beat 0 and 0.3, because the CPU punishes predictable play.
Pre-edition experiments, not FightBench scores (2026-10-05, 10 repeats, different seeds and sampler setup):
| Arm | VeryEasy wins / 140 | dealt | VeryHard (held out) wins / 150 | dealt |
|---|---|---|---|---|
| random | 0 | 19.6 | 0 | 8.7 |
| script (rules bot) | 8 | 47.4 | 2 | 33.6 |
| clef2 (reach in px, distance bands) | 20 | 54.9 | 0 | 28.4 |
| clef4 (clef2 + matching "Use when") | 13 | 52.1 | 1 | 35.5 |
How Clef behaves: an open "what now?" question gets a near-flat answer, but a question that names the situation gets a sharp one. Counts in the state ("hit 0 of 4") barely change its answer. It matches words between the situation and the option text, so a conditional clause ("best from more than 100 px") can raise a move's score in the wrong situation.
One run costs about 8,000 neurons (28 Clef rounds), so it needs a fresh free day or the Workers Paid plan. Rounds with Clef errors (for example, quota used up) are flagged in the summary.
(W) ROM)y_position is vertical speed, not height.tactics variant. A/B results go in ab/.About 530 input tokens per decision, about 14 neurons per second. The free plan (10,000 neurons per day) is about 12 minutes of play. The cost panel counts this session only.
python3 test_mk2_clef.py
FightBench: a replay-verified benchmark for AI decision models.
See the code
A replay-verified benchmark for AI decision models. First game: Mortal Kombat II (Genesis), Liu Kang vs the CPU, played from game memory, not pixels.
Board: https://fightbench.aipdv.com
Setting up with a coding agent? Point it at AGENTS.md.
mk2-liukang-v1A frozen bench on this emulator: Liu Kang vs the 15 VeryHard CPU fights, 10 repeats each. Rank is mean damage dealt. Wins are reported, not ranked.
edition/mk2-liukang-v1.json holds the 8 situations, 17 moves, ladder, ROM and state SHA-1s, and seed rule. It is the clef2 prompt, word for word. A changed string is a new edition.mk2-liukang-v1|repeat|fight, every 15 frames.script (rules bot) and random (17 moves).uv run fightbench.py smoke # free, needs the ROM, skips without it
uv run fightbench.py table --policy my.py:MyPolicy # write submissions/<model>--<version>__policy_table__mk2-liukang-v1.json
uv run fightbench.py live --policy my.py:MyPolicy # write submissions/<model>--<version>__live__mk2-liukang-v1.json
uv run fightbench.py replay submissions/<file>.json # maintainer: writes results/verified/ and site/board.json
uv run fightbench.py replay script # floor row, full ladder
uv run fightbench.py replay pending # maintainer: replay every submission without a result
python3 fightbench.py board # rebuild site/board.json from results/verified (no ROM)
--policy clef is Clef-flash. table --policy clef reads ab/clef-cache.json only and never calls Workers AI. Submit by committing only your submissions/ file in a pull request. A policy that calls a paid model sets usage (input_tokens, output_tokens, usd), and table adds call count and latency. Replay checks the game outcome. It does not re-call the model, so the model name, cost, and latency are attested by the submitter.
Nobody edits the site by hand. results/verified/*.jsonl is the source of truth.
submissions/ file. The board lists it as pending.gh pr checkout <N> && uv run fightbench.py replay pending, then commits results/verified/ to the PR. The ROM stays on the maintainer's machine, so CI cannot replay.fightbench.py board and publishes site/ to https://fightbench.aipdv.com. Each PR from this repo also gets a preview URL.The board shows mean damage dealt with a 95% interval, wins, damage taken, survival time, $ per ladder, tokens, call latency, top moves, and damage per opponent.
Clef-flash (Cloudflare Workers AI) plays MK2 on the Sega Genesis. It reads game memory, not pixels.
Clef-flash beats the VeryHard CPU, replayed frame for frame from the sim. Clef answered 8 situation questions in one call before the round (6,008 tokens, $0.00054). Code reads game memory and plays a move from Clef's probabilities. Full quality: recording/veryhard-win-shang-tsung-2.mp4.
Real or luck? (Pre-edition demo, not a FightBench score.) 200 more rounds against the same VeryHard Shang Tsung, prompt clef4, fresh seeds:
| Arm | Wins | Damage dealt (of 120) | Rounds with 90+ damage | Closest loss |
|---|---|---|---|---|
| Clef v4 | 1 | 55.0 ± 3.1 | 13 | 4 health short |
| Rules bot | 1 | 43.2 ± 2.8 | 7 | 12 short |
| Random | 0 | 10.0 ± 2.1 | 0 | 37 short |
Wins are rare (about 1 in 200), and on this one fight with the clef4 prompt Clef out-damaged the rules bot. The frozen FightBench ladder is the fair test: see the board. The face in the corner is MK2's "Toasty!" Easter egg.
Other recordings: recording/demo.mp4 (2 min), recording/highlights.jpg, recording/calibration-moves.png.
attacks_in_reach (measured reach per move).choice question with a probability for each of 13 moves. Up to 3 requests are in flight. Late answers are dropped.Memory map and save states come from stable-retro's MortalKombatII-Genesis-v0 integration.
roms/.integration/ with stable-retro's memory map and save states:
uv run mk2_clef.py --rom "roms/Mortal Kombat II (W).zip"
(W) dump has a different SHA-1 and works..env holds CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN (Workers AI only). See .env.example.uv run mk2_clef.py # opens http://localhost:8000, press Play
uv run mk2_clef.py --calibrate # saves calibration/moves.png, one frame per move
| Flag | Default | Meaning |
|---|---|---|
--model | clef-flash | or clef (bigger, slower, 2.7x the price) |
--state | cycle all, easiest first | one fight, for example Level1.LiuKangVsJax |
--temperature | 0.3 | 0 = always the top move, 1 = Clef's raw distribution |
--interval | 0.12 | seconds between requests |
--max-inflight | 3 | requests in flight at once |
--frame | off | also send the game frame (about 75 more tokens per request) |
--variant | baseline | tactics or random (see A/B test) |
--ab N | off | headless A/B test, see below |
--fights N | 14 | A/B: only the first N ladder fights (saves quota) |
--arms | all variants | comma list of variants to run in A/B, e.g. tactics,v2,random |
--sim | off | headless unpaced experiment, e.g. random@0,script@24,clef2@0 |
--repeats | 3 | sim: rounds per fight per arm |
--ladder | VeryEasy.LiuKang | sim: save-state prefix; LiuKangVs = the 15 VeryHard fights (held out) |
uv run mk2_clef.py --ab 1
Plays each of the 14 VeryEasy Liu Kang fights (Rayden to Shao Kahn) once per variant, back to back, with the order flipped every fight. Writes one JSON line per round to ab/results-*.jsonl and prints a summary (wins, damage dealt and taken, neurons).
| Variant | What Clef gets |
|---|---|
baseline | 13 moves, distance, reach, health, enemy movement |
tactics | baseline + duck, high and low fireball, flying kick; "got hit / hit enemy in the last 0.75 s"; rules: no long jump-ins, fireball or duck at range, block after a hit. A code guard drops jump_in when nothing reaches. |
v2 | same 17 moves as tactics with "Use when" descriptions matching state words; state is plain-English lines; adds threat question; code reflex blocks instantly when enemy attacks close; threat override when threat > 0.7 and close. |
random | control: uniform random moves at Clef's pace (0.4 s delay), no API calls |
Sim mode (--sim) runs the same 14 fights headlessly with code-only arms (no Clef, no credentials, unpaced). Arms are random, script, or either with +reflex; append @N to set latency in frames (default 24 ≈ Clef's 400 ms). Results go to ab/sim-*.jsonl. Useful for H1 (does reflex help?), H2 (does lag hurt?), H3 (does a script beat random?).
Clef table arms (clef, clef2, clef3, clef4, prompts in TABLE_PROMPTS). Clef answers one choice question per situation (enemy jumps, enemy attacks close, 60 to 80 px, and so on) for each opponent. Code only detects the situation each frame and samples Clef's answer for it. Answers are cached in ab/clef-cache.json, so play has no model delay and repeats are free. A new prompt costs about 500 neurons for 14 opponents. Use --temperature 1 (Clef's raw distribution): it beat 0 and 0.3, because the CPU punishes predictable play.
Pre-edition experiments, not FightBench scores (2026-10-05, 10 repeats, different seeds and sampler setup):
| Arm | VeryEasy wins / 140 | dealt | VeryHard (held out) wins / 150 | dealt |
|---|---|---|---|---|
| random | 0 | 19.6 | 0 | 8.7 |
| script (rules bot) | 8 | 47.4 | 2 | 33.6 |
| clef2 (reach in px, distance bands) | 20 | 54.9 | 0 | 28.4 |
| clef4 (clef2 + matching "Use when") | 13 | 52.1 | 1 | 35.5 |
How Clef behaves: an open "what now?" question gets a near-flat answer, but a question that names the situation gets a sharp one. Counts in the state ("hit 0 of 4") barely change its answer. It matches words between the situation and the option text, so a conditional clause ("best from more than 100 px") can raise a move's score in the wrong situation.
One run costs about 8,000 neurons (28 Clef rounds), so it needs a fresh free day or the Workers Paid plan. Rounds with Clef errors (for example, quota used up) are flagged in the summary.
(W) ROM)y_position is vertical speed, not height.tactics variant. A/B results go in ab/.About 530 input tokens per decision, about 14 neurons per second. The free plan (10,000 neurons per day) is about 12 minutes of play. The cost panel counts this session only.
python3 test_mk2_clef.py