OCR Benchmarking Suite

A comprehensive OCR benchmarking framework for comparing multiple state-of-the-art OCR engines and document processors on standardized HuggingFace datasets. This project enables systematic evaluation of OCR performance across various document types including invoices, receipts, handwritten text, scientific papers, and charts.

---

Overview

This benchmarking suite provides a unified pipeline to:

• Run multiple OCR engines on standardized datasets
• Compare performance metrics (CER, WER, accuracy, speed)
• Visualize results through comprehensive analysis notebooks
• Reproduce experiments with consistent evaluation protocols

Supported OCR Engines

Benchmark Datasets

Datasets are sourced from HuggingFace Hub and cover diverse document types:

Technology Stack

• Python 3.10+ (3.12 recommended)
• PyTorch with CUDA support (optional but recommended)
• Transformers for VLM-based OCR engines
• PaddleOCR for traditional OCR
• Docling for document intelligence
• Evaluation: pandas, jiwer, matplotlib, seaborn

---

Quick Start

Installation

1. Clone the repository

git clone <repository-url>
cd project-2-ocr-benchmarking

2. Set up Python environment

python -m venv my_env
source my_env/bin/activate  # On Windows: my_env\Scripts\activate

3. Install dependencies

pip install -r requirements.txt

Note: The requirements.txt contains merge conflict markers. Resolve them before installation:

• Option A (GPU support): Keep the DeepSeek/PyTorch dependencies for GPU acceleration
• Option B (Lightweight): Keep only pandas, jiwer, matplotlib, and core utilities
• For CUDA 12.1 with GPU, use: pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Running Your First Benchmark

cd scripts
python run_paddle.py  # Run PaddleOCR on default dataset

Results will be saved as CSV files in the results/ directory.

---

Project Structure

project-2-ocr-benchmarking/
├── scripts/                      # OCR engine runners & utilities
│   ├── run_paddle.py            # PaddleOCR benchmark
│   ├── run_docling.py           # Docling benchmark
│   ├── run_mmore.py             # MMoRe benchmark
│   ├── run_olmocr.py            # OlmOCR benchmark
│   ├── run_rolmocr.py           # RolmOCR benchmark
│   ├── run_dots.py              # DotsOCR benchmark
│   ├── run_folio.py             # FolioForge benchmark
│   ├── run_deepseek.py          # DeepSeek benchmark
│   ├── preprocess_data.py       # Data preprocessing & column normalization
│   └── evaluate_paddle_ocr.py   # Evaluation & metrics generation
│
├── results/                      # Benchmark outputs (organized by dataset)
│   ├── raw/                     # Raw OCR engine outputs
│   │   ├── getomni-ai_ocr-benchmark/
│   │   ├── HuggingFaceM4_ChartQA/
│   │   ├── merve_doclaynet-small/
│   │   ├── MosRat2333_ZhEn-latex/
│   │   ├── mychen76_invoices-and-receipts_ocr_v1/
│   │   ├── naver-clova-ix_cord-v2/
│   │   └── Teklia_IAM-line/
│   ├── normalized/              # Standardized metrics
│   └── final_metrics_clean/     # Comprehensive benchmark results (35+ CSV files)
│
├── utils/                        # Utility modules
│   ├── notebook_utils.py        # OCR evaluation helpers (WER, CER, plotting)
│   └── merge_data.py            # Data merging utilities
│
├── figures/                      # Generated visualizations
│
├── old/                          # Legacy scripts (archived)
│
├── evaluate_ocr.ipynb           # Main evaluation notebook
├── requirements.txt             # Python dependencies
└── README.md                    # This file

---

Running Benchmarks

Basic Usage

Each OCR engine has a dedicated runner script in the scripts/ directory. All scripts follow a consistent pattern:

cd scripts
python run_<engine>.py

Configuration

Edit the script's configuration section to customize:

# Example from run_paddle.py
DATASET_NAME = "mychen76/invoices-and-receipts_ocr_v1"  # Choose dataset
MAX_SAMPLES = None  # Set to integer for testing (e.g., 10)
OUTPUT_CSV = "invoice_benchmark_invoices.csv"  # Output filename

Engine-Specific Details

PaddleOCR

python run_paddle.py

• Type: Traditional OCR engine
• Output: CSV with columns: Index, Ground Truth, PaddleOCR Prediction, Time (s)
• Features: Fast, works on CPU, supports 80+ languages
• Best for: General text recognition, invoices, receipts

Docling

python run_docling.py

• Type: Document intelligence platform
• Output: CSV per rank for multi-process runs
• Features: Layout analysis, table extraction, GPU acceleration
• Best for: Complex documents, scientific papers

OlmOCR & RolmOCR

python run_olmocr.py
python run_rolmocr.py

• Type: Vision-Language Models (VLMs)
• Model: Based on Qwen2-VL architecture
• Features: High accuracy, context understanding
• Requirements: Significant GPU memory (~16GB+)
• Best for: Complex layouts, multilingual text

MMoRe

python run_mmore.py

• Type: Multimodal reasoning model
• Features: Integrates visual and textual understanding
• Best for: Documents requiring semantic understanding

DotsOCR

python run_dots.py

• Type: LLM-augmented OCR
• Features: Uses language models for post-processing
• Best for: Documents with complex formatting

DeepSeek

python run_deepseek.py

• Type: Vision-Language Model
• Output: Saved in scripts/deepseek_results/
• Features: High-quality OCR with context awareness

---

Evaluation & Analysis

Automatic Evaluation Script

Use evaluate_paddle_ocr.py (adaptable to other engines) to generate comprehensive metrics:

cd scripts
python evaluate_paddle_ocr.py

Computed Metrics:

• CER (Character Error Rate): Percentage of character-level errors
• WER (Word Error Rate): Percentage of word-level errors
• Exact Match: Percentage of perfectly recognized samples
• Inference Time: Processing time per image
• Text Length Analysis: Error correlation with text complexity

Generated Outputs:

• Summary statistics (mean/median CER, WER)
• Distribution plots (CER, WER, runtime)
• Scatter plots (errors vs. text length)
• Saved to figures/ directory

Interactive Notebooks

Main Evaluation Notebook (evaluate_ocr.ipynb)

Comprehensive Jupyter notebook for analyzing benchmark results:

• Comparative analysis across all 8 OCR engines
• Dataset-specific performance breakdowns and metrics
• Interactive visualizations: error distributions, performance trends, heatmaps
• Statistical analysis: mean/median metrics, confidence intervals
• Engine rankings by accuracy, speed, and consistency
• Flexible filtering: select engines, datasets, or metrics of interest

Features:

• Uses utils/notebook_utils.py for standardized metrics calculation
• Auto-detection of result file format and columns
• Batch processing of multiple result files
• Export-ready figures and summary statistics

To run:

jupyter notebook evaluate_ocr.ipynb

Result File Format

All benchmark results follow a standardized CSV format:

Index,Ground Truth,Prediction,Time (s)
0,"Sample text",Sample text (conf: 0.95),0.123
1,"Another example",Another exmple (conf: 0.87),0.145

Columns:

• Index: Sample number
• Ground Truth: Reference text from dataset
• Prediction: OCR engine output (may include confidence scores)
• Time (s): Processing time in seconds

---

Utilities & Data Processing

Preprocessing (scripts/preprocess_data.py)

Automated data preprocessing script for standardizing diverse CSV formats:

python preprocess_data.py

Features:

• Auto-detection of ground truth and prediction columns (supports multiple naming conventions)
• Schema normalization across different OCR engine outputs
• Column mapping for inconsistent data formats
• Flexible configuration for custom column names

Supported column names:

• Ground Truth: "Expected Output", "Ground Truth", "Label", "Target", "GT", "Groundtruth"
• Predictions: "Output", "Prediction", "PaddleOCR Prediction", "Docling Prediction", "MMoRe Prediction", "RolmOCR Output", "olmOCR Output", "DotsOCR Prediction", "DeepSeek Prediction", etc.

Analysis Utilities (utils/notebook_utils.py)

Comprehensive utility module for OCR evaluation notebooks:

• Text normalization and comparison
• WER/CER calculation with flexible column detection
• Statistical analysis and aggregation
• Visualization helpers (histograms, scatter plots, distribution plots)
• Label adjustment using adjustText for readable plots

Data Merging (utils/merge_data.py)

Utilities for combining results from multiple engines or datasets into unified comparison files.

Complete Benchmark Workflow

Step-by-step guide to run a full benchmark:

1. Setup & Installation

   python -m venv my_env
   source my_env/bin/activate  # Windows: my_env\Scripts\activate
   pip install -r requirements.txt
   

2. Test with small sample (5-10 images)

   cd scripts
   # Edit run_paddle.py: set MAX_SAMPLES = 10
   python run_paddle.py
   

3. Run full benchmark for all engines

   # Run each engine (can be parallelized)
   python run_paddle.py
   python run_docling.py
   python run_mmore.py
   python run_olmocr.py
   python run_rolmocr.py
   python run_dots.py
   python run_folio.py
   python run_deepseek.py
   

4. Preprocess results (optional, for column normalization)

   python preprocess_data.py
   

5. Generate metrics & plots

   python evaluate_paddle_ocr.py
   

6. Analyze in notebook

   jupyter notebook ../evaluate_ocr.ipynb
   

Configuration Best Practices

1. Test with small samples first: Set MAX_SAMPLES = 10 before full runs
2. Monitor GPU memory: Use nvidia-smi to track usage during VLM inference
3. Adjust batch sizes: For VLM-based models, reduce if out-of-memory errors occur
4. Cache models locally: Set HF_HOME environment variable to avoid re-downloads
5. Run engines in sequence: Some engines may conflict if run in parallel
6. Check results incrementally: CSV results are written row-by-row (safe to interrupt)

---

Environment Setup

GPU Acceleration

For optimal performance, GPU acceleration is recommended:

Check CUDA availability:

import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA version: {torch.version.cuda}")

Install PyTorch with CUDA:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Model Caching

To avoid disk quota issues and speed up loading:

# Set cache directory
export HF_HOME="./hf_cache"  # Linux/Mac
set HF_HOME=.\hf_cache       # Windows

This is automatically configured in most runner scripts.

Memory Optimization

For systems with limited resources:

# In runner scripts, adjust:
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"

# For VLM models, use quantization:
model = Qwen2VLForConditionalGeneration.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,  # or torch.float16
    device_map="auto",
    load_in_8bit=True  # Requires bitsandbytes
)

---

Results & Analysis

Comprehensive Benchmark Results

All benchmark results are organized in results/ with two main sections:

1. Raw Results (results/raw/)

Original OCR engine outputs by dataset:

• getomni-ai_ocr-benchmark/ - General OCR benchmarking
• HuggingFaceM4_ChartQA/ - Chart and graph understanding
• merve_doclaynet-small/ - Document layout analysis
• MosRat2333_ZhEn-latex/ - Scientific documents with LaTeX/formulas
• mychen76_invoices-and-receipts_ocr_v1/ - Financial documents
• naver-clova-ix_cord-v2/ - Receipts (Korean/English)
• Teklia_IAM-line/ - Handwritten text

2. Normalized Results (results/normalized/)

Standardized metrics across all datasets

3. Final Metrics (results/final_metrics_clean/)

Comprehensive benchmark comparison with 35+ CSV files containing:

Dataset Coverage:

• ChartQA (5 engines): Docling, MMoRe, olmOCR, PaddleOCR, RolmOCR
• IAM Handwriting (6 engines): Docling, FolioForge, MMoRe, olmOCR, PaddleOCR, RolmOCR
• Invoices & Receipts (6 engines): All engines tested
• MosRat2333 ZhEn-latex (6 engines): Multilingual scientific documents
• Receipts (6 engines): All engines tested
• Tables (2 engines): olmOCR, RolmOCR
• Text + Figures (2 engines): olmOCR, RolmOCR

Metric Columns:

• Model: OCR engine name
• Dataset: Benchmark dataset
• WER (Word Error Rate): % word-level errors
• CER (Character Error Rate): % character-level errors
• Accuracy_Score: Exact match percentage
• Inference_Time_s: Processing time per sample

Accessing Benchmark Results

Pre-computed results are available in results/final_metrics_clean/:

# View sample results
head -5 results/final_metrics_clean/ChartQA_Docling.csv
head -5 results/final_metrics_clean/IAM_Handwriting_PaddleOCR.csv
head -5 results/final_metrics_clean/Invoices_\&_Receipts_RolmOCR.csv

Quick comparison example:

# Compare all engines on ChartQA dataset
ls results/final_metrics_clean/ChartQA_*.csv

# Compare all datasets for one engine
ls results/final_metrics_clean/*_PaddleOCR.csv

Visualization Examples

The evaluation scripts generate various plots automatically:

1. Performance Distribution Plots: CER/WER histograms showing error patterns
2. Engine Comparison Plots: Box plots comparing engines across datasets
3. Runtime Comparison Charts: Bar charts showing inference speed
4. Error vs. Complexity Plots: Scatter plots correlating errors with text length
5. Heatmaps: Dataset × Engine performance matrix
6. Trend Analysis: Performance trends across document types

Figures are saved to figures/ organized by engine and dataset.

---

Performance Tips

• CPU-only systems: PaddleOCR performs best; VLM models will be very slow
• Limited GPU memory: Start with smaller models (7B parameters or less)
• Slow inference: Check if model is on GPU with model.device
• Disk space: Models can be 10-30GB; ensure sufficient space in HF_HOME

---

Third-Party Components

This project uses several third-party libraries and models:

• PaddleOCR: Apache 2.0 License
• HuggingFace Transformers: Apache 2.0 License
• PyTorch: BSD-style License
• Docling: Check respective documentation
• Datasets: Various licenses (check HuggingFace dataset pages)

Important: Each OCR engine and dataset has its own license. Review and comply with their terms, especially for commercial use.

---

Tips for Best Results

1. Start Small: Always test with MAX_SAMPLES=10 before full runs
2. Monitor Resources: Use htop (CPU) and nvidia-smi (GPU) to track usage
3. Save Incrementally: Results are written row-by-row; safe to interrupt
4. Compare Fairly: Use same dataset splits and preprocessing for all engines
5. Document Settings: Record model versions, hardware, CUDA version, and configuration
6. Validate Results: Manually inspect samples with anomalously high error rates
7. Share Findings: Contribute results back to the community through GitHub
8. Check Existing Results: Review results/final_metrics_clean/ for pre-computed benchmarks