Legal RAG QA 🔍

Legal RAG QA by Isaacus is a challenging benchmark for evaluating the end-to-end performance of real-world legal RAG applications.

Legal RAG QA consists of 190 passages and external materials and 138 question–answer–relevent-passages triplets sourced from LibreTexts' Introduction to Criminal Law textbook.

This dataset was originally intended to serve as the Legal RAG Bench benchmark but was subsequently replaced with a much larger dataset constructed from the Victorian Criminal Charge Book. Both datasets are equally high quality, however. Accordingly, in the interests of supporting open AI evaluation, we have publicly released Legal RAG QA under the same license as the Introduction to Criminal Law textbook.

Usage 👩‍💻

Legal RAG QA may be loaded like so using the Hugging Face 🤗 datasets Python library:

import datasets

# Load passages in Legal RAG QA.
corpus = datasets.load_dataset("isaacus/legal-rag-qa", name="corpus", split="test")

# Load question-answer-passage triplets from Legal RAG QA.
qa = datasets.load_dataset("isaacus/legal-rag-qa", name="qa", split="test")

Structure 🗂️

Passages in the Legal RAG QA corpus are stored in the corpus subset, with each entry having the following fields:

• id (string): a unique identifier for the passage.
• section (string): a unique identifier for the section of the textbook from which the passage originates.
• title (string): the title of the section of the textbook from which the passage originates.
• text (string): the text of the passage, formatted in Markdown.
• is_supplemental (boolean): whether the passage comes from the textbook itself or is supplementary material.

Questions, answers, and the IDs of relevant passages are stored in the qa subset, with each entry having the following fields:

• id (string): a unique identifier for the question.
• question (string): the text of the question.
• answer (string): the text of the answer to the question.
• requires_supplemental (boolean): whether the question depends on supplementary material.
• relevant_documents (string): the unique identifiers of passages in the corpus subset that are most relevant to the question.

The corpus and qa subsets of Legal RAG QA both currently have only a single split, test.

Methodology 🧪

Legal RAG QA was constructed by downloading the Introduction to Criminal Law textbook from LibreTexts, extracting question-answer pairs, and then downloading any relevant passages from the textbook or relevant external materials to serve as a corpus.

License 📜

This dataset is licensed under CC BY NC SA 3.0.

Citation 🔖

If you've relied on Legal RAG QA for your work, please cite it alongside Legal RAG Bench:

@misc{butler2026legalragqa,
      title={Legal RAG QA},
      author={Abdur-Rahman Butler and Umar Butler},
      year={2026},
      publisher = {Isaacus},
      url={https://huggingface.co/datasets/isaacus/legal-rag-qa}
}

@misc{butler2026legalragbench,
      title={Legal RAG Bench: an end-to-end benchmark for legal RAG}, 
      author={Abdur-Rahman Butler and Umar Butler},
      year={2026},
      eprint={2603.01710},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.01710}, 
}