Chess APO Bench: Chess Puzzles for Automatic Prompt Optimization

The exact chess puzzle train/test splits used in the paper Benchmarking Prompt Optimization of Large Language Models With Chess (Lesort et al., 2026). The puzzles come from the Lichess puzzle database (CC0 1.0).

• Code: https://github.com/imec-ailabs/Automatic-Prompt-Optimization-with-Chess
• Paper: https://arxiv.org/abs/2610.00416

Quick start

from datasets import load_dataset

ds = load_dataset("imecai/chess-apo-bench")
print(ds)               # train: 559 rows, test: 559 rows
print(ds["test"][0]["FEN"], ds["test"][0]["Moves"])

test = load_dataset("imecai/chess-apo-bench", split="test")  # a single split

Splits

These files are the frozen splits used in the paper, published byte-for-byte:

The splits share no puzzle IDs, starting positions, or (FEN, Moves) lines. In the paper, optimizers see only train. In each run, train is divided into 419 training and 140 validation puzzles, keeping all positions of a puzzle together. test is held out for the final evaluation.

Composition

Both splits have the same stratified composition:

• Rating: 20 puzzles per 100-point Lichess rating bin from 500 to 2000 (19 in the 400–500 bin) and 30 per bin from 2000 to 2800.
• Mate: 280 of 559 puzzles have the mate theme.
• Solver moves (NumMoves): 1 → 159, 2 → 160, 3 → 80, 4 → 80, 5 → 80.

Task

Each puzzle is expanded into independent positions, one per solver turn. The reference line is replayed, and the model sees the position (FEN, side to move and the sorted legal moves) and must return the reference move. Unparseable or illegal answers count as wrong. The paper's main metric is puzzle-level accuracy: a puzzle counts as solved only if every one of its positions is answered correctly.

Columns

The first ten columns are the original Lichess puzzle fields. The last five record how the paper's splits were built.

Limitations

• These are fixed samples of a Lichess snapshot. The Lichess database changes over time: puzzle ratings and play counts reflect the snapshot, not current values.
• Each puzzle has a single reference line. Another winning move counts as wrong, as on Lichess.
• 559 puzzles per split is a small sample, so differences between methods should be judged over multiple seeds (the paper uses three seeds).
• The dataset contains only the puzzles. The paper's compiled prompts, model outputs and other experimental results are not included and are not covered by this card's license.

License and attribution

The puzzle records come from the Lichess puzzle database, released under CC0 1.0. This dataset is released under CC0 1.0 as well. Thanks to Lichess and its community for making the puzzle database freely available.

Citation

@article{lesort2026chessapo,
  title   = {Benchmarking Prompt Optimization of Large Language Models With Chess},
  author  = {Lesort, Timoth{\'e}e and L{\'o}pez de Aberasturi G{\'o}mez, Alejandra and Karch, Tristan and Veniat, Tom and Modard, Philippe and Tuyls, Karl and Denoyer, Ludovic},
  journal = {arXiv preprint arXiv:2610.00416},
  year    = {2026}
}