⚖️ Hojicha - Legal Contract Chunking Benchmark

Hojicha (HOldings JudIcial CHAllenges) is a benchmark dataset for evaluating text chunking algorithms on legal contracts. It is part of MTCB (Make That Chunker Better).

Dataset Description

Hojicha tests how well chunking algorithms handle formal legal language, including:

• Nested clauses and complex sentence structures
• Cross-references between sections
• Legal terminology and defined terms
• Structured contract sections (recitals, definitions, covenants, etc.)

Source

Derived from CUAD (Contract Understanding Atticus Dataset), which contains commercial contracts annotated by legal experts.

Statistics

Question Types

The dataset covers 41 types of contract clauses, including:

Usage

With MTCB

from mtcb import HojichaEvaluator
from chonkie import RecursiveChunker

evaluator = HojichaEvaluator(
    chunker=RecursiveChunker(chunk_size=512),
    embedding_model="voyage-3-large",
)

result = evaluator.evaluate(k=[1, 3, 5, 10])
print(result)

Direct Loading

from datasets import load_dataset

# Load corpus (479 contracts)
corpus = load_dataset("chonkie-ai/hojicha", "corpus", split="train")

# Load questions (1,982 questions)
questions = load_dataset("chonkie-ai/hojicha", "questions", split="train")

Data Format

Corpus

Each document contains:

• title: Contract identifier (e.g., "COMPANY_DATE-EX-10-AGREEMENT TYPE")
• text: Full contract text

Questions

Each question contains:

• question: The question text (asking about a specific clause type)
• document_title: Reference to the source contract
• chunk-must-contain: The passage that must appear in retrieved chunks
• question_type: Category of the clause (e.g., "Governing Law", "Cap On Liability")

Evaluation Methodology

For each question:

1. Chunk all contracts using the chunking algorithm
2. Embed all chunks and the question
3. Retrieve top-k chunks by similarity
4. Check if any retrieved chunk contains the chunk-must-contain passage
5. Calculate Recall@k and MRR@k

License

This dataset is released under CC-BY-4.0, following the original CUAD license.

Citation

@dataset{hojicha2025,
  title={Hojicha: Legal Contract Chunking Benchmark},
  author={Chonkie Team},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/chonkie-ai/hojicha}
}

@inproceedings{cuad2021,
  title={CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review},
  author={Hendrycks, Dan and Burns, Collin and Chen, Anya and Ball, Spencer},
  booktitle={NeurIPS},
  year={2021}
}

Links

• MTCB GitHub
• Chonkie Chunking Library
• Original CUAD Dataset