MTCB (Massive Text Chunking Benchmark) is a standardized evaluation framework for text chunking in RAG systems. It measures how well your chunking and embedding strategy retrieves relevant passages across 9 diverse domains, from legal contracts to scientific papers. Built on top of Chonkie.

📦 Installation

pip install mtcb

🚀 Quick Start

Run the lightweight nano benchmark to evaluate a chunking strategy in minutes:

from mtcb import NanoBenchmark
from chonkie import RecursiveChunker

benchmark = NanoBenchmark()
result = benchmark.evaluate(
    chunker=RecursiveChunker(chunk_size=512),
    embedding_model="voyage-3-large",
    k=[1, 5, 10],
)
print(result)

🧩 Available Benchmarks

Full Benchmark

The full MTCB benchmark spans 9 domains with ~17k questions across ~3k documents:

Nano Benchmark

For fast iteration during development, MTCB provides a lightweight nano benchmark with ~100 questions per dataset. Documents are selected to maximize question density:

🔧 Usage

MTCB works with Chonkie — any chunker that extends chonkie.BaseChunker is supported out of the box.

Full Benchmark

Run the complete benchmark across all 9 domains:

from mtcb import Benchmark
from chonkie import RecursiveChunker

benchmark = Benchmark()
result = benchmark.evaluate(
    chunker=RecursiveChunker(chunk_size=512),
    embedding_model="voyage-3-large",
    k=[1, 5, 10],
)
print(result)

Individual Evaluators

Run a single domain-specific evaluator:

from mtcb import GachaEvaluator
from chonkie import RecursiveChunker

evaluator = GachaEvaluator(
    chunker=RecursiveChunker(chunk_size=1000),
    embedding_model="voyage-3-large",
    cache_dir="./cache"
)

result = evaluator.evaluate(k=[1, 3, 5, 10])
print(result)

Custom Datasets

Evaluate on your own corpus using SimpleEvaluator:

from mtcb import SimpleEvaluator
from chonkie import RecursiveChunker

evaluator = SimpleEvaluator(
    corpus=["Your document text here...", "Another document..."],
    questions=["What is X?", "How does Y work?"],
    relevant_passages=["passage that must be in retrieved chunk", "another passage"],
    chunker=RecursiveChunker(chunk_size=1000),
    embedding_model="voyage-3-large",
)

result = evaluator.evaluate(k=[1, 3, 5, 10])
print(result)

Dataset Generation

Generate verified QA datasets from your own documents:

from mtcb import DatasetGenerator

generator = DatasetGenerator(deduplicate=True)
result = generator.generate(
    corpus=["Your document text..."],
    samples_per_document=10,
    output_path="./output.jsonl",
)

print(f"Generated {result.total_verified} verified samples")
for sample in result.samples:
    print(f"Q: {sample.question}")
    print(f"A: {sample.answer}")

📊 Metrics

MTCB evaluates retrieval quality using:

• Recall@k: Percentage of questions where the relevant passage appears in the top-k results
• Precision@k: Ratio of relevant chunks in the top-k results
• MRR@k: Mean Reciprocal Rank — how high the first relevant result ranks
• NDCG@k: Normalized Discounted Cumulative Gain — position-weighted relevance scoring

📚 Citation

If you use MTCB in your research, please cite:

@software{mtcb2025,
  author = {Bhavnick Minhas and Shreyash Nigam},
  title = {MTCB: Massive Text Chunking Benchmark},
  url = {https://github.com/chonkie-inc/mtcb},
  version = {0.1.0},
  year = {2025},
}