ChessFormer

This is an experimentation-focused project that aims to build a language model with native chess multimodal understanding. Rather than forcing a language model to reason about chess as text tokens, this project designs a chess-specific deep fusion architecture that enables the LLM to selectively inject learned chess representations deep within it's reasoning layers to focus on specific aspects of the position.

Main Project Details

• Native chess architecture: a specialized chess-encoder processes the position to produce 64 square latents. It implements chess relational attention mechanisms among other structured encoding mechanisms to preserve square-centric identity, supporting strong representations and interpretability analysis.

• Auxiliary objectives: auxiliary chess objectives (maia policy distillation, move evaluation, board reconstruction) use square-level prediction heads, enabling structured representation learning to support the primary task of generating commentary.

• Decoder-side fusion: chess latents are injected into the LLM's decoder layers via contextually gated cross-attention. Based on the current text latents, the model learns the kinds and amounts of chess information to inject. A decoding inspector provides visualizations of the model's attention patterns to the chess latents while decoding.

• Commentary regimes: Chess commentary is supplied by Synthetic Commentary Generation and potentially Chessvisor. Using LLM augmented tools, it's possible to curate multiple viable training curricula, from low-entropy factoids to chatbot-style Q&A or coaching.

Current State

Current efforts have aimed to train the model on low-entropy factoid-based commentary. This achieves solid state-grounded chess multimodality, setting the stage for more abstract, humanlike commentary styles. Given a board state, it generates commentary that demonstrates high accuracy on state-driven observations:

Black's knight on g3 threatens White's rook on h1. Black's pawn on a3 is attacking White's pawn on b2. White's pawn on b2 threatens Black's pawn on a3. White's queen on b3 is attacking Black's knight on b8 and Black's pawn on a3. White's pawn on f2 attacks Black's knight on g3. White's pawn on h2 is attacking Black's knight on g3. Black's knight on g3 supports Black's bishop on a6. White's pawn on d5 is attacking Black's pawn on c6. Black has a semi-open b-file. White's king on g1 guards White's rook on h1, White's pawn on f2, and White's pawn on h2. White's pawn on a2 supports White's queen on b3. Black's pawn on c6 attacks White's pawn on d5. White's king is castled kingside. White's knight on h3 is on the rim. Black's bishop on f8 guards Black's pawn on d6 and Black's pawn on g7. White's king has a weakened pawn shield. Black's queen on e5 threatens White's bishop on g5, White's pawn on e2, White's pawn on b2, and White's pawn on d5. White's pawn on e2 defends White's bishop on f3. Black has a semi-open e-file. White's knight on h3 protects White's king on g1, White's pawn on f2, and White's bishop on g5. Black's bishop on a6 is attacking White's pawn on e2. White's rook on h1 guards White's king on g1 and White's pawn on h2. Black's queen on e5 guards Black's knight on g3, Black's pawn on d6, and Black's pawn on g7. Black's king remains in the center on d7.

Architecture Overview

View diagram: ChessFormer Architecture Overview

Training Data

Training data is generated by the companion Synthetic Commentary Generation repository, which uses methods such as commentary templates, engine analysis, motif detection, importance scoring, and LLM-based commentary synthesis. Exported .pt samples are consumed directly by the training loop here.

Documentation

• Architecture Deep Dive — detailed component breakdown, CSMP modes, Perceiver regime, fusion strategies
• Training Guide — config walkthrough, training regimes, how to run
• Data Pipeline — sample contract, data flow from PGN to training
• Structured Cross-Attn Math - exact forward equations and structured square-attention regularizers
• Structured Decode Inspector - browser GUI for token-by-token commentary decoding and 64 x N_src square-attention visualization

Quick Start

pip install -r src/requirements.txt

# Full commentary training (LLM + adapter)
python src/training/train.py --config configs/chess_fusion.yaml

# Adapter-only pretraining (chess objectives only, no LLM)
# Set model.enable_lm: false in the config
python src/training/train.py --config configs/chess_fusion.yaml

# Inspect a saved structured_cross_attn checkpoint in the browser
python src/inference/decoding_inspector.py --checkpoint <checkpoint_dir>

Key Files