OCR Benchmark: Open-Weight Models on Australian Insurance Documents

A benchmarking framework comparing 22 open-weight OCR models against 97 Australian insurance Product Disclosure Statement (PDS) PDFs (~7,500 pages). The PDFs contain native embedded text, providing free ground truth for measuring text extraction accuracy without manual annotation.

Why This Benchmark?

Most OCR benchmarks use academic datasets (receipts, handwriting, scene text). Real-world document processing — especially in regulated industries like insurance — involves dense multi-column layouts, nested tables, legal boilerplate, and mixed formatting. This benchmark fills that gap by evaluating models on genuine insurance documents that represent the kind of content organisations actually need to extract.

What We Measure

• Text accuracy: Normalized Edit Distance, Character/Word Error Rate, BLEU, fuzzy matching
• Table extraction: Tree Edit Distance Similarity (TEDS) with greedy table matching
• Coverage: percentage of pages a model successfully processes (failures scored as worst-case)
• Robustness: accuracy under image perturbations (rotation, blur, JPEG compression, downscaling)
• Confidence intervals: bootstrap CIs on all aggregate metrics

Models (22 total)

Tier 1 — T4-compatible (≤8 GB VRAM)

Tier 2 — A100 required (>8 GB VRAM)

Traditional / CPU Baselines

Architecture

PDF corpus  ──►  Data Prep (render PNGs + extract ground truth)
                        │
                        ▼
              Sample Sets (stratified page manifests)
                        │
                        ▼
              Runner (adapters load/infer/unload sequentially)
                        │
                        ▼
              Evaluator (metrics + bootstrap CIs + ranking)
                        │
                        ▼
              Dashboard (Plotly charts + qualitative HTML viewer)

All models implement a common OCRAdapter interface with load_model() / ocr_page() / unload_model(). The runner loads one model at a time, runs inference across a sample set, saves per-page markdown outputs, then unloads before the next model. A canonical inference protocol (deterministic decoding, shared prompt) ensures fair comparison.

Ground Truth

Ground truth is extracted from the PDFs' embedded text layer using pdfplumber and PyMuPDF with best-of selection per page:

• Both agree (71% of pages where PyMuPDF's column-by-column reading order is preferred)
• pdfplumber preferred when it captures more text (8% of pages)
• Both identical (21% of pages)

This dual-extraction approach handles the multi-column layouts common in insurance documents, where single-extractor approaches often scramble reading order.

Sample Sets

Getting Started

Installation

# Core (PDF processing + evaluation metrics)
pip install -e .

# Add visualisation
pip install -e ".[viz]"

# Add traditional OCR baselines (DocTR, PaddleOCR, EasyOCR, Tesseract)
pip install -e ".[traditional]"

# For Colab GPU runs
pip install -r requirements/colab_tier1.txt   # T4
pip install -r requirements/colab_tier2.txt   # A100

Verify Adapters

# Quick: import + instantiate all adapters
python scripts/smoke_adapters.py

# Test specific models
python scripts/smoke_adapters.py --models tesseract qwen25_vl_3b

# Deep: instantiate + load/unload (needs GPU for VLMs)
python scripts/smoke_adapters.py --load-model

Run Tests

pytest tests/

Notebook Workflow

The benchmark is structured as a sequence of notebooks, designed to run data prep locally and GPU inference on Colab:

Adding a New Model

1. Add an entry to config/model_registry.yaml
2. Create an adapter in src/adapters/ extending OCRAdapter
3. Implement load_model() and ocr_page() using the base class helpers (_get_instruction(), _get_generation_kwargs(), _resolve_device(), _extract_tables())
4. The runner picks it up by model key automatically

Project Status

• Data preparation (PDFs, images, ground truth, sample sets): complete
• Adapter framework (22 models): complete
• Runner, evaluator, and dashboard: complete
• GPU benchmark runs: not yet started — needs Colab T4/A100

License

This project is open source. See LICENSE for details.