Retrieval-Augmented Generation for Root Board Game Rules

A hierarchical, router-driven RAG system for answering questions about the Law of Root.

Course project for Natural Language Processing, Università degli Studi di Trieste.

[!NOTE] This repository is an archived research/course-project prototype. The README documents the system and the results presented in the final project presentation. The codebase also contains exploratory scripts, intermediate datasets, cached artifacts, and historical experiment paths produced during development; it is therefore not yet a clean one-command reproduction from a fresh checkout. A clean, reproducible version is currently being worked on.

For a quick overview of the project, feel free to check the presentation :point_down:

Overview

Root is a highly asymmetric board game: every faction follows a different set of rules, while many actions are also governed by shared global rules. The resulting rulebook is hierarchical, cross-referenced, and full of faction-specific exceptions.

The goal of this project was to build a Retrieval-Augmented Generation (RAG) assistant that can answer natural language rules questions while grounding its answers in the official rulebook.

Rather than using a single dense vector search, the final retriever combines:

1. Semantic query embeddings with intfloat/e5-large-v2;
2. A learned multi-label section router over the rulebook hierarchy;
3. BM25 as a lexical fallback, should the router find no relevant section;
4. A Jina cross-encoder reranker to select the most relevant passages;
5. Mistral-7B-Instruct-v0.2 to generate the final answer from the retrieved context.

The main idea is to exploit the structure already present in the rulebook instead of treating it as a flat collection of unrelated text chunks.

Overview

---

Architecture

flowchart LR
    Q[User query] --> E[e5-large-v2 embedding]
    E --> C[Multi-label section router]

    C -->|Section hits| S[Passages from predicted sections]
    C -->|No section hits| B[BM25 top P1 = 50]

    S --> R[Jina cross-encoder reranker]
    B --> R

    R -->|Top P2 = 5| G[Mistral-7B-Instruct-v0.2]
    G --> A[Grounded natural-language answer]

This routing step is the main departure from a vanilla RAG pipeline. A query such as a faction-specific rules question can first be mapped to one or more relevant rulebook sections; only when the classifier produces no section prediction does the system fall back to global BM25 retrieval.

Data and preprocessing

The project starts from the official rulebook in PDF format. The source document is challenging for automatic retrieval because it contains:

• a deep section/subsection hierarchy;
• tables;
• faction icons that encode rule applicability;
• item icons;
• glossary entries;
• compact formatting and cross-references.

The original preprocessing pipeline therefore included:

flowchart LR
    PDF[Rulebook PDF] --> T[Text extraction]
    PDF --> TB[Table extraction]
    T --> I[Remove decorative images]
    TB --> I
    I --> F[Replace faction icons with text]
    F --> IT[Replace item icons with text]
    IT --> H[Hierarchical chunking]
    H --> P[Passages + structural metadata]

The final retrieval units follow the rulebook hierarchy. In particular:

• ordinary rules are grouped at the subsection level;
• glossary entries are preserved at a finer granularity;
• each passage retains structural metadata such as its section, faction/general type, phase, and detected actions.

A passage can therefore look conceptually like:

passage: [6. Marquise de Cat > 6.6 Evening.]
SectionType: Faction
Phase: Evening
ActionsCore: ...
ActionsNiche: ...

<rule text>

[!IMPORTANT] The archived preprocessing workflow also contains manual cleanup steps performed after PDF extraction. Reconstructing the final cleaned corpus deterministically from the raw PDF is one of the main engineering tasks planned for a future refactor.

Learned section router

The first retrieval stage predicts which high-level rulebook sections are relevant to the query.

Training data

The reported experiment started from approximately 400 manually curated query/answer or query/passage pairs. These were transformed into an augmented dataset of roughly 1,200 query-to-section training examples with multi-label targets.

[!NOTE] The repository contains additional intermediate/augmented dataset snapshots from development, so raw file counts may differ from the figures reported in the final presentation. The numbers in this README describe the reported experiment.

The router predicts among 17 rulebook sections, including global sections such as Golden Rules, Key Concepts, Key Actions, Glossary, and the individual faction sections.

Model

Queries are embedded using e5-large-v2 and fed to a One-vs-Rest logistic regression classifier. Classifier regularization and per-label probability thresholds were tuned on a held-out validation split with F1 as the target metric.

Validation results

The router was intentionally tuned toward recall: predicting an additional candidate section is usually less damaging than failing to expose a relevant section to the downstream reranker.

BM25 fallback

If the section router returns no section hits, the system falls back to BM25 over the complete passage collection.

BM25 hyperparameters were optimized by a simple grid search for several candidate-pool sizes (P1) using retrieval recall as the selection objective. The final configuration used in the reported pipeline was:

P1 = 50
k1 = 2.3
b  = 0.3

Cross-encoder reranking

Candidate passages produced either by the section router or by BM25 are scored jointly with the query using:

jinaai/jina-reranker-v1-turbo-en

The reranker is a cross-encoder, so it can model direct query-passage interactions that are lost when documents and queries are embedded independently. The final system keeps the top 5 passages (P2 = 5) as context for the generator.

Retrieval evaluation

The retrieval evaluation set contains manually curated query-positive-passage pairs. In the reported experiment, a query had on average approximately 1.35 relevant passages, so a low Precision@5 is expected even when the relevant evidence is successfully retrieved.

Final retrieval results

The most important result for the downstream QA task is the combination of high Recall@5 and high MRR@5: relevant evidence is usually present in the final context and tends to appear near the top of the reranked list.

Answer generation

The retrieved passages are passed to Mistral-7B-Instruct-v0.2. The archived implementation loads the generator with 4-bit quantization to reduce memory requirements as everything was run locally on a laptop GPU with 8GB VRAM.

The reported final prompt is deliberately conservative: the model is instructed to answer only from the retrieved rulebook context and to say when the answer cannot be found rather than relying on external knowledge.

Reported generation settings

max_new_tokens = 120
do_sample      = True
temperature    = 0.4
top_p          = 0.9
prompt         = zero-shot

Generator evaluation

Answers were evaluated on unseen questions against human-written reference answers using complementary lexical and semantic metrics.

The large gap between lexical overlap metrics and BERTScore is consistent with the model frequently producing semantically correct paraphrases rather than matching the reference wording exactly.

---

Example behavior

The system performs particularly well on questions whose answer is explicitly supported by a small number of localized rules. For example, the evaluation includes questions about:

• what happens when the Vagabond takes a hit;
• whether actions require another player's permission;
• whether the Eyrie may add two bird cards to the Decree;
• whether the Riverfolk may move along rivers.

The qualitative analysis also exposed useful failure cases. In particular, the generator can still over-interpret retrieved context or produce plausible unsupported details on ambiguous or deliberately out-of-domain questions. These examples motivated the stricter grounding and fine-tuning directions listed below.

Reproducibility status

The repository should currently be read as an experimental snapshot, not as a production library.

Preserved and inspectable

• processed rule passages and metadata;
• manually curated evaluation datasets;
• learned section-router artifact;
• BM25 search/evaluation code;
• reranking and generation code;
• plots, predictions, and the metrics reported above.

Not yet cleanly reproducible from scratch

• the complete PDF-to-final-corpus transformation, because some extraction fixes were manual;
• a single configuration-driven command for the complete experiment;
• every historical experiment script, since some were written for one-off runs and retain old paths or model variants;
• the prototype UI, which has not yet been refactored together with the final pipeline.

This is intentional transparency: the project was developed for an NLP exam, where the focus was on the retrieval/generation methodology and experimental analysis, rather than on packaging a production-ready software library.

Limitations and future work

We identified three main modeling directions for improving the system:

• create a larger and more diverse training/evaluation dataset;
• fine-tune the reranker on Root-specific query / positive / negative passage triples;
• fine-tune the generator on domain-specific question/answer pairs.

References

The project was informed by work on:

• Lewis et al. - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
• Wang et al. - Text Embeddings by Weakly-Supervised Contrastive Pre-training (E5)
• Jina AI - Jina Reranker
• Jiang et al. - Mistral 7B (2023)
• Lee Bonstra - Google prompt engineering white paper (2025)

Authors

• Nicola Cortinovis
• Marta Lucas
• Annalisa Paladino

Disclaimer

Root is a game by Leder Games. This repository is an independent academic project and is not affiliated with or endorsed by the game publisher. Third-party rules, artwork, trademarks, and other assets remain the property of their respective owners.