PDF Parse Bench

This benchmark evaluates how effectively different PDF parsing solutions extract mathematical formulas and tables from documents. We generate synthetic PDFs with diverse formatting scenarios, parse them with different parsers, and score the extracted content using LLM-as-a-Judge. This semantic evaluation approach substantially outperforms traditional metrics in agreement with human judgment.

🏆 Leaderboard (2026-q1)

Results are based on two separate benchmark datasets, each containing 100 synthetic PDFs:

• 2026-q1-tables-only — PDFs with tables of varying complexity (simple, moderate, complex)
• 2026-q1-formulas-only — PDFs with inline and display-mode mathematical formulas

Legend:

• Tables: Average LLM-as-a-Judge score (0-10) across 451 tables from 2026-q1-tables-only
• Formulas: Average LLM-as-a-Judge score (0-10) across 1413 inline + 657 display formulas from 2026-q1-formulas-only
• Cost/Time: API cost (USD) for 100 pages, or GPU wall-clock time on a single NVIDIA RTX 4090
• Inference: Deployment options (CPU, GPU, API)

📊 Detailed table scores (Simple/Moderate/Complex)

Rank	Parser	Overall	Simple	Moderate	Complex
1	Gemini 3 Flash	9.44	9.64	9.23	9.45
2	LightOnOCR-2-1B	8.87	9.42	8.56	8.58
3	Mistral OCR	8.79	8.76	8.68	8.96
4	dots.ocr	8.58	8.94	8.21	8.58
5	Mathpix	8.31	9.28	7.94	7.61
6	MonkeyOCR-pro-3B	8.30	8.64	7.94	8.33
7	Chandra	8.19	8.92	7.77	7.80
8	Qwen3-VL-235B	8.12	9.31	7.53	7.40
9	GLM-4.5V	7.48	9.14	6.83	6.26
10	Claude Sonnet 4.6	6.96	6.85	7.06	6.99
11	GPT-5 mini	6.69	7.74	6.22	6.00
12	GPT-5 nano	6.44	7.54	6.23	5.41
13	Nanonets-OCR-s	6.44	8.19	5.88	5.02
14	PP-StructureV3	6.32	7.11	5.52	6.31
15	Gemini 2.5 Flash	6.23	7.71	5.80	4.94
16	MinerU2.5	5.97	6.82	5.37	5.64
17	PaddleOCR-VL	4.97	6.58	4.47	3.65
18	DeepSeek-OCR	4.83	7.18	4.18	2.83
19	PyMuPDF4LLM	4.66	6.45	4.29	2.97
20	GOT-OCR2.0	4.47	5.43	4.25	3.61
21	olmOCR-2-7B	3.51	4.35	3.10	3.00
22	GROBID	1.64	1.79	1.62	1.50

• Overall/Simple/Moderate/Complex: LLM-as-a-Judge score (0-10 scale) across 451 tables by complexity level

📊 Table score distributions

📊 Detailed formula scores (Inline/Display)

Rank	Parser	Inline	Display
1	Qwen3-VL-235B	9.82	9.86
2	Gemini 3 Flash	9.77	9.82
3	Mathpix	9.64	9.72
4	PP-StructureV3	9.49	9.80
5	LightOnOCR-2-1B	9.51	9.70
6	dots.ocr	9.44	9.77
7	MonkeyOCR-pro-3B	9.54	9.42
8	Mistral OCR	9.39	9.68
9	Chandra	9.43	9.50
10	GLM-4.5V	9.33	9.46
11	olmOCR-2-7B	9.34	9.37
12	MinerU2.5	9.36	9.25
13	Nanonets-OCR-s	9.18	9.26
14	DeepSeek-OCR	8.95	9.02
15	Claude Sonnet 4.6	8.42	8.67
16	PaddleOCR-VL	8.50	8.42
17	GOT-OCR2.0	7.77	8.53
18	GROBID	7.33	6.33
19	Gemini 2.5 Flash	6.47	6.58
20	GPT-5 mini	5.87	4.94
21	GPT-5 nano	4.88	4.57
22	PyMuPDF4LLM	6.50	0.29

• Inline: LLM-as-a-Judge score for inline formulas (1413 formulas)
• Display: LLM-as-a-Judge score for display-mode formulas (657 formulas)

Why LLM-as-a-Judge?

Rule-based metrics correlate poorly with human judgment for formula and table extraction. We validated this in two human annotation studies (Pearson r = correlation with human scores):

• formula-metric-study — 750 human ratings: text metrics reach r = 0.01, CDM r = 0.31, LLM judges r = 0.74–0.82
• table-metric-study — 1,500+ human ratings: rule-based metrics (TEDS, GriTS) top out at r = 0.70, LLM judges reach r = 0.93

Benchmark Datasets

PDFs are generated synthetically using LaTeX with randomized parameters (document class, fonts, margins, column layout, line spacing) to test parser robustness across different formatting scenarios. Since PDFs are generated from LaTeX source, we automatically obtain exact ground truth as a byproduct of the generation process.

• Formula Dataset: Each PDF contains randomly selected formulas embedded in text passages, displayed as inline or display-mode equations. Formulas are sampled from our dataset of 319,000 formulas extracted from Wikipedia, ensuring diversity in complexity and real-world relevance. Dataset: piushorn/wikipedia-latex-formulas-319k

• Table Dataset: Each PDF contains tables of varying complexity (simple, moderate, complex) with diverse content types, column layouts, and formatting. Dataset coming soon on Hugging Face.

Evaluation Pipeline

Parser outputs are assessed using a two-step pipeline:

Step 1: Extraction

Given a parser's output (the extracted text from a PDF), an LLM establishes initial correspondences between extracted elements and ground truth (formulas or tables), then fuzzy search reliably extracts exact strings from the parsed output. This achieves robust alignment even when parser outputs differ significantly from ground truth.

Step 2: Scoring with LLM-as-a-Judge

The primary metric is the LLM-as-a-Judge score (0-10 scale, default: Gemma 4 31B IT via OpenRouter). For formulas, the judge evaluates correctness, completeness, and semantic equivalence. For tables, the judge evaluates content accuracy and structure preservation. Scores are computed separately for inline/display formulas and by table complexity (simple/moderate/complex). See Why LLM-as-a-Judge? for our validation studies.

Quick Start

Benchmark Datasets: New benchmark datasets are released quarterly, each containing 100 PDFs. The current datasets are 2026-q1-tables-only (tables) and 2026-q1-formulas-only (formulas).

There are two ways to use this benchmark, depending on your needs:

Option 1: Evaluate Your Existing Parser (pip install)

Use this if: You quickly want to evaluate your PDF Parsing tool against the benchmark.

Advantage: Simple pip install, no need to integrate with the repository structure.

Installation

pip install pdf-parse-bench

Note: Set the OPENROUTER_API_KEY environment variable — the evaluation pipeline uses OpenRouter for both segment extraction (extract()) and LLM-as-a-Judge scoring (evaluate()).

Step 1: Parse the Benchmark PDFs

Get the benchmark PDFs and parse them with your parser. Choose the dataset you want to evaluate:

from pdf_parse_bench import get_benchmark_pdfs_dir
from pathlib import Path

# Available datasets: "2026-q1-tables-only", "2026-q1-formulas-only"
pdfs_dir = get_benchmark_pdfs_dir("2026-q1-tables-only")

# Parse each PDF with your parser
output_dir = Path("results/my_parser")
for pdf_path in pdfs_dir.glob("*.pdf"):
    parsed_text = your_parser.parse(pdf_path)

    # Save to expected format: {output_dir}/{pdf_name}/parsed.md
    (output_dir / pdf_path.stem / "parsed.md").parent.mkdir(parents=True, exist_ok=True)
    (output_dir / pdf_path.stem / "parsed.md").write_text(parsed_text)

Required output structure:

results/my_parser/
├── 000/
│   └── parsed.md
├── 001/
│   └── parsed.md
├── 002/
│   └── parsed.md
...

Step 2: Run Evaluation

Run the benchmark evaluation on your parsed results:

from pathlib import Path
from pdf_parse_bench import Benchmark, get_benchmark_ground_truth_dir

bench = Benchmark(
    parser_output_dir=Path("results/my_parser"),
    ground_truth_dir=get_benchmark_ground_truth_dir("2026-q1-tables-only"),
    llm_judge_models=["google/gemma-4-31b-it"],
)
bench.extract()
bench.evaluate()
bench.save_benchmark_summary()

---

Option 2: Add Parser to Repository (for reproducibility)

Use this if: You want to contribute your parser to the benchmark, reproduce published results, or ensure full reproducibility of your evaluation setup.

Advantage: Full automation with CLI, parser configuration is versioned and reproducible, easy to share exact setup with others.

Clone Repository

git clone https://github.com/phorn1/pdf-parse-bench.git
cd pdf-parse-bench

# Install with uv
uv sync

# Configure environment (copy and edit .env.example)
cp .env.example .env

Note: OPENROUTER_API_KEY is the only required key (used by the evaluation pipeline). The other keys in .env.example are only needed for the corresponding API-based parsers (see the READMEs in parsers/).

Add Your Parser Implementation

Create a new parser module in the parsers/ directory:

# parsers/my_parser/__main__.py
from pathlib import Path
from pdf_parse_bench.utilities import PDFParser
from pdf_parse_bench.pipeline import run_cli

class MyParser(PDFParser):
    @classmethod
    def display_name(cls) -> str:
        return "My Parser"

    def parse(self, pdf_path: Path, output_path: Path) -> str:
        # Your parsing logic here
        markdown = "# Parsed content"
        self._write_output(markdown, output_path)
        return markdown

if __name__ == "__main__":
    run_cli(MyParser())

Run Your Parser

uv run -m parsers.my_parser -i data/<dataset>

The benchmark infrastructure handles everything automatically:

• Loading test PDFs from data/<dataset>/pdfs/
• Parsing each PDF with your parser
• Extracting formulas from parsed output
• Running evaluation against ground truth
• Saving results in standardized format

CLI Options

The benchmark CLI provides several options to customize execution:

# Run only specific steps
uv run -m parsers.my_parser -i data/<dataset> --step parse
uv run -m parsers.my_parser -i data/<dataset> --step extract --step evaluate

# Reprocess existing results
uv run -m parsers.my_parser -i data/<dataset> --reprocess all
uv run -m parsers.my_parser -i data/<dataset> --reprocess parse --reprocess extract

# Use a different LLM judge for evaluation (OpenRouter model format)
uv run -m parsers.my_parser -i data/<dataset> --llm-judge-model openai/gpt-5-mini

Project Structure

pdf-parse-bench/
├── src/pdf_parse_bench/       # Core benchmark infrastructure
│   ├── pipeline/              # Benchmark execution pipeline
│   ├── eval/                  # Evaluation metrics and judges
│   ├── extraction/            # Formula extraction from parsed text
│   ├── utilities/             # Base classes and helpers
│   └── synth_pdf/             # Synthetic PDF generation (optional)
├── parsers/                   # Parser implementations
│   ├── pymupdf4llm/
│   ├── llamaparse/
│   ├── mathpix/
│   └── ...                    # Add your own!
├── data/                      # Benchmark datasets
│   ├── 2026-q1-tables-only/  # Table benchmark dataset
│   │   ├── pdfs/
│   │   └── ground_truth/
│   └── 2026-q1-formulas-only/ # Formula benchmark dataset
│       ├── pdfs/
│       └── ground_truth/

Contributing

Contributions are welcome!

Adding a parser implementation: See Option 2 above for instructions on adding your parser to the repository.

Bug reports and feature requests: Please open an issue on GitHub.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Citation

If you use this benchmark in your research or project, please cite our papers:

@misc{horn2025formulabench,
    title = {Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs},
    author = {Horn, Pius and Keuper, Janis},
    year = {2025},
    eprint={2512.09874},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url = {https://arxiv.org/abs/2512.09874}
}

@misc{horn2026tablebench,
    title = {Benchmarking PDF Parsers on Table Extraction with LLM-based Semantic Evaluation},
    author = {Horn, Pius and Keuper, Janis},
    year = {2026},
    eprint={2603.18652},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url = {https://arxiv.org/abs/2603.18652}
}

📄 Papers: arXiv:2512.09874 (Formulas) | arXiv:2603.18652 (Tables)

Acknowledgments

This work has been supported by the German Federal Ministry of Research, Technology and Space (BMFTR) in the program "Forschung an Fachhochschulen in Kooperation mit Unternehmen (FH-Kooperativ)" within the joint project LLMpraxis under grant 13FH622KX2.