Chunkr (chunkr-rs on PyPI, import chunkr) is an ultra-high-performance document chunking and text-splitting engine written in Rust with native Python C-ABI bindings. Engineered specifically for Large Language Models (LLMs), Vector Databases (Chroma, Qdrant, Pinecone, Weaviate, Milvus), and Retrieval-Augmented Generation (RAG) pipelines, Chunkr delivers 2,000+ MB/s on prose and up to 3,200+ MB/s on source code with zero superfluous heap allocations — 2.9x to 33x faster than LangChain, Chonkie, semchunk, and the Rust text-splitter at matched chunk sizes, and up to 200x faster than LlamaIndex node parsers (BPE token chunking is the one exception; see Performance Benchmarks).

[!TIP] Quick Package Disambiguation:

• Python (pip): pip install chunkr-rs ➔ import chunkr
• Rust (cargo): cargo add chunkr ➔ use chunkr::prelude::*;
• WebAssembly (npm): npm install chunkr-wasm ➔ import { RecursiveChunker } from "chunkr-wasm" (Browser & Cloudflare Workers)
• Core Architecture: Zero heap churn, SIMD-accelerated separator scanning, and true multi-core Rayon execution that bypasses the Python GIL.

🥊 Chunkr vs. Alternatives

Why data engineers and AI developers choose Chunkr over pure-Python chunking libraries:

Throughput rows measured on Apple M4 (10 threads), Python 3.12.11, chunkr built from source; median of 15 runs on 1 MB prose with matched chunk sizes. LlamaIndex ships no fixed-width splitter. Reproduce with benchmarks/.

🧭 Which Chunker Should I Use? (Decision Guide)

📦 Installation

Python

Install chunkr-rs via pip:

pip install chunkr-rs

Pre-compiled wheels are available for Linux, Windows, and macOS (Intel & Apple Silicon) on Python 3.8 through 3.13. No Rust compiler required!

Or build from source with maturin:

maturin develop --release

Rust

Add chunkr to your Cargo.toml:

[dependencies]
chunkr = "1.2"

Or via cargo:

cargo add chunkr

WebAssembly (Browsers, Cloudflare Workers & Node.js)

Install chunkr-wasm via npm:

npm install chunkr-wasm

Or build all Wasm bindings directly from source:

# Unix / Linux / macOS
./scripts/build_wasm.sh

# Windows (PowerShell)
.\scripts\build_wasm.ps1

🚀 Supported Chunking Strategies

🐍 Python Quickstart

import chunkr

sample_text = (
    "Convolutional neural networks specialize in visual imagery. "
    "Recurrent networks process sequential text.\n\n"
    "In conclusion, deep learning powers modern vision systems."
)

# 1. Recursive Character Chunking
recursive_chunker = chunkr.RecursiveChunker(chunk_size=500, overlap=50)
docs = recursive_chunker.chunk(sample_text)
for doc in docs:
    print(doc.content, doc.metadata)

# 2. Token-Based Chunking (OpenAI cl100k_base / GPT-4)
token_chunker = chunkr.TokenChunker(chunk_size=100, overlap=20, encoding="cl100k_base")
token_docs = token_chunker.chunk(sample_text)

# 3. Query-Aware Adaptive Chunking
query_chunker = chunkr.QueryAwareChunker(query="neural networks", hotspot_sentences=1, context_sentences=2)
query_docs = query_chunker.chunk(sample_text)

# 4. Agentic Topic Chunking
agentic_chunker = chunkr.AgenticChunker(min_chars=100, max_chars=1000)
agentic_docs = agentic_chunker.chunk(sample_text)

# 5. Hierarchical Parent-Child Pairs & Tree Chunking
hier_chunker = chunkr.HierarchicalChunker(parent_size=1000, child_size=200)
pairs = hier_chunker.chunk_hierarchical(sample_text)  # List[{"parent": Document, "children": [Document, ...]}]
tree = hier_chunker.chunk_tree(sample_text)          # Nested hierarchy tree dict

# 6. Markdown Structure Chunking (with header breadcrumbs)
md_chunker = chunkr.MarkdownChunker(chunk_size=1000, overlap=100)
md_docs = md_chunker.chunk("# Title\n## Section\nContent...")

# 7. Table-Aware Chunking (Markdown / CSV / TSV with repeated headers)
table_chunker = chunkr.TableChunker(rows_per_chunk=10, overlap_rows=1)
table_docs = table_chunker.chunk("| Date | Metric | Value |\n|---|---|---|\n| 2024-01 | MRR | $50K |")

# 8. Late Chunking (Full-document context with token span snapping & pooling)
late_chunker = chunkr.LateChunker(chunk_size=300, overlap=30)
late_docs = late_chunker.chunk(sample_text)
# Pool token embeddings directly from your transformer model:
# pooled_embeddings = late_chunker.pool_embeddings(token_embeddings, late_docs)

# 9. Hugging Face Universal Token Chunking (Llama 3, Mistral, BGE, BERT)
# hf_chunker = chunkr.HFTokenChunker.from_file("path/to/tokenizer.json", chunk_size=512, overlap=50)
# hf_chunker = chunkr.HFTokenChunker.from_tokenizer(transformers_tokenizer, chunk_size=512, overlap=50)

# 10. Recursive Directory Ingestion & Auto-Routing
dir_loader = chunkr.DirectoryLoader(extensions=["pdf", "md", "csv", "py"])
dir_chunks = dir_loader.load_and_chunk("path/to/repo_or_folder")

# 11. PDF Document Loading & Chunking
loader = chunkr.PDFLoader()
pages = loader.load_pages("path/to/document.pdf")
pdf_chunks = recursive_chunker.chunk_documents(pages)

# 12. AST-Based Code Chunking (Tree-sitter syntax boundaries)
ast_chunker = chunkr.AstCodeChunker(language="python", max_chunk_size=1500)
code_chunks = ast_chunker.chunk("def calculate():\n    return 42\n\nclass Model:\n    pass")

# 13. Chunk Bin-Packing (Greedily merging small chunks into token budgets)
packer = chunkr.ChunkPacker(max_characters=1000)
packed_chunks = packer.pack(recursive_chunker.chunk(sample_text))

# 14. Post-Chunking Transformation Pipeline (Filter + Dedup + Pack + SHA-256 Enrich)
pipeline = (
    chunkr.ChunkPipeline()
    .filter_min_chars(30)
    .filter_min_alpha_ratio(0.5)
    .deduplicate(exact=True)
    .pack(max_characters=1200)
    .enrich(id_prefix="kb_doc_")
)
optimized_chunks = pipeline.process(recursive_chunker.chunk(sample_text))

# 15. Multi-Core Parallel Batch Chunking (Rayon-backed multi-threading)
batch_docs = [
    chunkr.Document(f"Document {i} content...", {"doc_id": i, "category": "AI", "score": 0.98})
    for i in range(100)
]
# Parallel processing across all available CPU cores
parallel_chunks = recursive_chunker.par_chunk_documents(batch_docs)

# 16. Streaming Chunker (Constant memory for multi-GB inputs)
streamer = chunkr.StreamChunker(chunk_size=1000, overlap=150)
stream_chunks = streamer.chunk_text(sample_text)

# 17. Ecosystem Bridges (LangChain, LlamaIndex, Hugging Face, Pandas)
langchain_docs = chunkr.to_langchain(stream_chunks)      # List[langchain_core.documents.Document]
llamaindex_nodes = chunkr.to_llamaindex(stream_chunks)  # List[llama_index.core.schema.TextNode]
records = chunkr.to_dict_list(stream_chunks)            # Direct DataFrame / Dataset input

🌐 WebAssembly Quickstart (Edge & Browser)

Chunkr's WebAssembly build brings identical ultra-fast chunking algorithms to JavaScript/TypeScript environments with zero native dependencies.

Cloudflare Workers (Edge Runtimes)

import wasm from "chunkr-wasm/wasm";
import { initSync, RecursiveChunker, PDFLoader } from "chunkr-wasm/web";

// Zero-latency synchronous initialization
initSync(wasm);

export default {
  async fetch(request) {
    const { text, chunkSize, overlap } = await request.json();
    const chunker = new RecursiveChunker(chunkSize || 500, overlap || 50);
    const chunks = chunker.chunk(text);
    return Response.json({ chunks });
  }
};

Client-Side Browser (Vite / Next.js / Vanilla ESM)

Chunk sensitive documents entirely client-side without sending text to any external server:

import { RecursiveChunker, MarkdownChunker, TokenChunker, countTokens } from "chunkr-wasm";

// 1. Recursive semantic splitting
const chunker = new RecursiveChunker(800, 100);
const chunks = chunker.chunk(documentText);

// 2. Exact OpenAI token chunking at native speed
const tokenChunker = new TokenChunker(512, 50, "cl100k_base");
const tokenChunks = tokenChunker.chunk(documentText);
const numTokens = countTokens(documentText, "cl100k_base");

In-Memory PDF Chunking at the Edge

import { PDFLoader, RecursiveChunker } from "chunkr-wasm";

const fileBytes = new Uint8Array(await request.arrayBuffer());
const loader = new PDFLoader();

// Parse and extract pages in memory
const pages = loader.loadPagesFromBytes(fileBytes);
const chunker = new RecursiveChunker(500, 50);
const chunkedPages = pages.map(p => ({
  page: p.metadata.page_number,
  chunks: chunker.chunk(p.content)
}));

🛠️ Production RAG Recipes

Recipe 1: End-to-End LangChain + ChromaDB Ingestion

Combine Chunkr's high-speed PDF loading and recursive splitting with LangChain vector stores:

import chunkr
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

# 1. High-speed native PDF parsing (16–23x faster than pypdf)
loader = chunkr.PDFLoader()
pages = loader.load_pages("annual_report.pdf")

# 2. Multi-threaded recursive chunking across all CPU cores
chunker = chunkr.RecursiveChunker(chunk_size=800, overlap=100)
chunks = chunker.par_chunk_documents(pages)

# 3. Zero-copy bridge to LangChain Document format
langchain_docs = chunkr.to_langchain(chunks)

# 4. Ingest into Chroma vector database
vectorstore = Chroma.from_documents(
    documents=langchain_docs,
    embedding=OpenAIEmbeddings(model="text-embedding-3-small")
)

Recipe 2: Small-to-Big (Parent-Child) Retrieval with LlamaIndex

Match tight child chunks for high semantic precision, then return full parent context to the LLM:

import chunkr

# 1. Generate parent-child pairs in a single native pass
hier = chunkr.HierarchicalChunker(parent_size=1500, parent_overlap=150, child_size=300, child_overlap=30)
pairs = hier.chunk_hierarchical(document_text)

# 2. Extract child chunks enriched with parent context metadata
child_docs = []
for pair in pairs:
    parent_doc = pair["parent"]
    for child in pair["children"]:
        child.metadata["parent_id"] = parent_doc.metadata.get("chunk_id")
        child.metadata["parent_content"] = parent_doc.content
        child_docs.append(child)

# 3. Export directly to LlamaIndex TextNodes
llamaindex_nodes = chunkr.to_llamaindex(child_docs)

Recipe 3: Late Chunking with SentenceTransformers & Full Document Context

Late Chunking encodes the entire document first to retain bidirectional context across all chunks:

import chunkr
import torch
from transformers import AutoTokenizer, AutoModel

# 1. Tokenize full document and obtain token embeddings
model_name = "jinaai/jina-embeddings-v2-base-en"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

inputs = tokenizer(full_text, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)
    token_embeddings = outputs.last_hidden_state[0].tolist()

# 2. Snap exact token spans and mean-pool chunk embeddings
late_chunker = chunkr.LateChunker(chunk_size=400, overlap=40)
chunks = late_chunker.chunk(full_text)
chunk_embeddings = late_chunker.pool_embeddings(token_embeddings, chunks)

Recipe 4: Universal Hugging Face Tokenizer Chunking

Chunk documents directly with open-source model tokenizers (Llama 3, Mistral, Qwen, DeepSeek):

import chunkr
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
hf_chunker = chunkr.HFTokenChunker.from_tokenizer(tok, chunk_size=512, overlap=50)

# Chunks are guaranteed to fit within exact model token limits
token_bounded_chunks = hf_chunker.chunk(document_text)

🦀 Rust Quickstart

use chunkr::prelude::*;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let text = "Convolutional neural networks specialize in visual imagery. Recurrent networks process sequential text.\n\nIn conclusion, deep learning powers modern vision.";

    // 1. Recursive Chunker
    let recursive_chunker = RecursiveChunker::new()
        .with_chunk_size(500)
        .with_overlap(50);
    let chunks = recursive_chunker.chunk(text)?;

    // 2. Query-Aware Adaptive Chunker
    let query_chunker = QueryAwareChunker::new("convolutional neural networks")
        .with_hotspot_sizing(1, 0)
        .with_context_sizing(3, 1);
    let query_chunks = query_chunker.chunk(text)?;

    // 3. Hierarchical Parent-Child Tree Chunker
    let hier_chunker = HierarchicalChunker::with_sizes(150, 20, 50, 10)?;
    let tree = hier_chunker.chunk_tree(text)?;

    // 4. Table-Aware Chunker
    let table_chunker = TableChunker::new()
        .with_rows_per_chunk(Some(5))
        .with_overlap_rows(1);
    let table_chunks = table_chunker.chunk("| Col A | Col B |\n|---|---|\n| 1 | 2 |")?;

    // 5. Late Chunking (Span Snapping & Mean-Pooling)
    let late_chunker = LateChunker::new();
    let late_chunks = late_chunker.chunk(text)?;

    // 6. Tree-sitter AST Code Chunker
    let ast_chunker = AstCodeChunker::new(AstLanguage::Rust).with_max_chunk_size(1500);
    let rust_chunks = ast_chunker.chunk("fn main() { println!(\"Hello\"); }")?;

    // 7. Chunk Bin-Packing
    let packer = ChunkPacker::new(1000);
    let packed = packer.pack(&chunks);

    // 8. Directory Ingestion & Chunker Auto-Routing
    let dir_loader = DirectoryLoader::new()
        .with_extensions(vec!["md".into(), "csv".into(), "pdf".into()]);
    // 9. Post-Chunking Pipeline (Filter + Dedup + Packing + SHA-256 Enrichment)
    let pipeline = ChunkPipeline::new()
        .filter_min_characters(30)
        .filter_min_alpha_ratio(0.5)
        .deduplicate_exact(true)
        .pack(1200)
        .enrich_metadata()
        .with_id_prefix("rust_doc_");
    let clean_chunks = pipeline.process(chunks);

    // 10. PDF Document Loading & Chunking
    let loader = PDFLoader::new();
    let pdf_pages = loader.load_pages_from_file("tests/test_files/sample_doc.pdf")?;
    let pdf_chunks = recursive_chunker.chunk_documents(&pdf_pages)?;

    // 11. Multi-Threaded Parallel Document Batch Chunking
    let parallel_chunks = recursive_chunker.par_chunk_documents(&pdf_pages)?;

    // 12. Constant-Memory Streaming Chunker (Files, Sockets, STDIN)
    let streamer = StreamChunker::new(1000, 150)?;
    let stream_iter = streamer.chunk_file("large_document.txt")?;
    for chunk_result in stream_iter {
        let chunk = chunk_result?;
        println!("Streamed chunk: {}", chunk.content.len());
    }

    Ok(())
}

⚡ Command-Line Interface (chunkr-cli)

Install or run the standalone chunkr CLI binary for fast batch processing or UNIX piping:

# Chunk any file using Markdown strategy to JSONL format
cargo run --bin chunkr -- README.md -s markdown -c 500 -f jsonl

# Stream massive multi-GB files with constant memory footprint
cargo run --bin chunkr -- large_file.txt -s stream --chunk-size 1000 -f jsonl

# Pipe from STDIN with post-chunking pipeline (dedup, filtering, packing, SHA-256 hash enrichment)
cat document.txt | chunkr -s recursive --chunk-size 800 --min-chars 30 --dedup --enrich --pack 1200 > output.jsonl

# Ingest and auto-route an entire directory
chunkr ./docs -s dir --format jsonl --out-file chunks.jsonl

📊 Performance Benchmarks

Every number below is reproducible with the harness in benchmarks/ (.venv/bin/python benchmarks/bench_chunking.py --reps 15). Measured on Apple M4 (10 threads), macOS 15.7.9, Python 3.12.11, chunkr built from source. Each implementation runs on byte-identical corpora with the same chunk-size/overlap parameters, chunker construction is excluded from timing, and the median of 15 round-robin repetitions is reported.

Text Chunking Throughput Comparison

MB/s = 10^6 bytes / wall time; higher is better. — = no equivalent splitter in that library.

Reading the table. Chunk counts matched chunkr's within 2% except where noted. LlamaIndex rows use SentenceSplitter / MarkdownNodeParser / TokenTextSplitter, which build Document/Node objects and re-tokenize candidate splits. LangChain's markdown splitter and LlamaIndex's markdown parser split on headers only and assign no size budget (113 and 130 chunks vs chunkr's 715), so their throughput is not like-for-like. Chunkr's SentenceChunker counts sentences, not characters (2124 chunks vs Chonkie's 534). semchunk emits ~26% more, smaller chunks at the same nominal size. The batch row is chunkr's Rayon path (par_chunk_texts) against each library's best available loop; it is 1.8x faster than chunkr's own sequential loop on the same data. chunkr.AstCodeChunker (tree-sitter, syntax boundaries) runs at 12 MB/s — 260x slower than the regex-based CodeChunker row above, which is the price of AST-correct splits.

Note on Token BPE: BPE is chunkr's weakest strategy: 38 MB/s vs 151 MB/s for Chonkie and 43 MB/s for LangChain's Python tiktoken wrapper, at identical chunk counts (62 each). Encoding dominates the run, and TokenChunker decodes every 512-token window separately, so per-chunk decode cost dominates its windowing overhead. Construction is cached process-wide, so repeated chunkers are cheap: first use in a fresh interpreter is 22 ms for chunkr vs 43 ms for LangChain and 457 ms for Chonkie. Use RecursiveChunker when exact token bounds are not required.

PDF Extraction & End-to-End Pipeline Latency

10-page sample (tests/test_files/sample_doc.pdf):

2,066-page textbook (19.9 MB) — page count matters, so the lead narrows on large documents:

❓ Frequently Asked Questions (FAQ)

pip install chunkr-rs

import chunkr

import chunkr

chunks = chunkr.RecursiveChunker(chunk_size=500, overlap=50).chunk("Sample text...")

# Convert to LangChain Document objects:
langchain_docs = chunkr.to_langchain(chunks)

# Convert to LlamaIndex TextNode objects:
llamaindex_nodes = chunkr.to_llamaindex(chunks)

💡 Contributing

Contributions are welcome! Please check out the Contribution Guide to get started.

📝 License

Licensed under the MIT License - see the LICENSE file for details.