PureDocBench

How far is document parsing from solved?
A source-traceable benchmark for OCR and document parsing across clean, digitally degraded, and real-degraded document settings.

中文说明 | Dataset | Paper | GT Review & Corrections

PureDocBench uses HTML/CSS document sources as hidden anchors: each page is rendered into images and annotated from the same structured source. This gives a benchmark where text, tables, formulas, captions, and reading order can be scored with less post-hoc annotation noise.

PureDocBench 是一个源可追踪的 OCR / 文档解析 benchmark。数据由 HTML/CSS 源文件渲染而来，GT 标注从同源结构中抽取，覆盖 clean、digital-degraded、real-degraded 三条图像轨道。

Updates

• Current GT: The stable alias is puredocbench-gt-latest; the exact revision and timestamp are recorded in Hugging Face gt/latest.json.
• 2026-06-14: Updated GT annotations and opened a GT Review app for community corrections.
• 2026-05-08: Initial public release of PureDocBench, including the paper PDF and full dataset on Hugging Face.

GT Annotation Examples

The examples below show colored coordinate boxes over clean rendered pages from an academic paper, a patent form, and a tuition invoice.

At A Glance

Main Leaderboard

The paper evaluates 40 systems across pipeline specialists, end-to-end document parsers, and general-purpose VLMs. The live leaderboard additionally tracks public post-publication results. Each track reports Overall, TextEdit, FormulaCDM, TableTEDS, and ROEdit; Avg3 averages the three track Overall scores.

Open the interactive sortable leaderboard ↗

^* OvisOCR2 scores are author-reported post-publication results from its official model card. Only per-track Overall and Avg₃ were published; unreported component metrics are shown as —.

^† NaviDC-OCR scores are author-reported post-publication results from its official repository. The source reports TextEdit, FormulaCDM, and TableTEDS for each track, but not ROEdit; unreported ROEdit metrics are shown as —.

Bold marks the best reported score in each column; underlined marks the runner-up. GitHub README tables cannot run sorting scripts, so use the interactive leaderboard to sort any metric in either direction.

Diagnostics

The diagnostic panel shows where current systems still have headroom. Formula recognition is the largest single bottleneck, and real degradation changes rankings more sharply than digital degradation.

Case Studies

The four case studies below are all taken from the paper. They show failures that aggregate scores can hide: notation loss, reading-order mistakes, annotation contamination, table-structure errors, character-level corruption, and missing visual authentication cues.

Case 1: Academic

Case 2: Business

Case 3: Finance

Case 4: Certificate

Appendix Highlights

The appendix documents the degradation design, per-category behavior, and source-validity checks used to make the benchmark reproducible.

Download

The full image/GT/HTML release is hosted on Hugging Face:

# After downloading all files from Hugging Face:
shasum -a 256 -c SHA256SUMS.txt
cat pdb_full.tar.part-* | tar -xf -

Verify the split archive and reconstructed release:

python scripts/verify_split_archive.py /path/to/downloaded/files

python scripts/validate_release_manifest.py \
  --release-root /path/to/puredocbench \
  --manifest manifests/release_manifest_candidate_1475.csv

GT Review

Stable public identifiers: puredocbench-gt-latest for the GT archive and puredocbench-review-v1 for this review UI. Exact update timestamps remain in the Hugging Face metadata and changelog rather than public URLs or directory names. Use the review app to inspect annotations and export correction patches.

• Public review app: Open GT Review App
• Repository file: review/index.html
• Correction guide: docs/ANNOTATION_CORRECTIONS.md
• Submit a correction: New GT annotation correction issue (English or Chinese)

Local launch:

mkdir -p review/assets
ln -s /path/to/puredocbench/images/clean review/assets/images
python3 -m http.server 8767 --directory review

Open:

http://127.0.0.1:8767/index.html

Static app URL:

https://zhihengli-casia.github.io/PureDocBench/review/

The GitHub repository does not include the full image release. For visual review on GitHub Pages, click Load Images and select the downloaded images/clean folder. Local launch can also use the symlink above.

GT Coordinates

If you need spatial labels, regenerate clean-render coordinates from the HTML/CSS sources:

python scripts/add_gt_coordinates.py \
  --release-root /path/to/puredocbench \
  --manifest manifests/release_manifest_candidate_1475.csv \
  --in-place \
  --include-bbox \
  --include-coordinate-system \
  --report coordinate_report.json

python scripts/validate_release_manifest.py \
  --release-root /path/to/puredocbench \
  --manifest manifests/release_manifest_candidate_1475.csv \
  --require-coordinates \
  --require-bbox

The script follows the OmniDocBench GT convention and adds a rectangular poly field to each layout_dets item. poly is a flat list of clean-image pixel coordinates in top-left, top-right, bottom-right, bottom-left order: [x1, y1, x2, y1, x2, y2, x1, y2]. A derived bbox: [x1, y1, x2, y2] can also be written with --include-bbox, but poly is the primary coordinate field. Run playwright install chromium first if the Playwright browser is not installed, or pass --browser-channel chrome to use a local Chrome installation.

Inference And Scoring

PureDocBench includes a public CLI for model-agnostic inference, lightweight scoring, and OmniDocBench export:

pip install -e .

puredocbench infer \
  --images /path/to/puredocbench/images/clean \
  --output-dir predictions/my_model_clean \
  --command-template 'python my_model_infer.py --image {image} --out {output}'

puredocbench score \
  --release-root /path/to/puredocbench \
  --manifest manifests/release_manifest_candidate_1475.csv \
  --pred-dir predictions/my_model_clean \
  --track clean \
  --out-dir scores/my_model_clean

See docs/INFERENCE_SCORING.md for the full interface and OmniDocBench export path.

Repository Contents

manifests/                         Release and sample manifests
metadata/                          Dataset card and Croissant metadata
scripts/                           Rendering, degradation, validation, leaderboard tools
puredocbench/                      Public inference, scoring, and OmniDocBench export CLI
model_inference/                   Sanitized model inference configs and runners
supplemental_inference_scoring/    API/local inference and scoring utilities
assets/figures/                    Figures from the paper
paper/                             Paper PDF

Quick Start

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
playwright install chromium

Render one HTML page:

python scripts/render_single_image.py \
  --html /path/to/page.html \
  --out /path/to/page.png \
  --dpi 300

Apply a deterministic degradation profile:

python scripts/apply_degradation_ablation.py \
  --input /path/to/clean_images \
  --output /path/to/degraded_images \
  --profile full_medium

License

• Dataset assets are released under CC BY 4.0; see LICENSE_DATA.
• Code in this repository is released under the license in LICENSE.
• Model weights are not redistributed.

Citation

@misc{puredocbench,
  title        = {How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean, Degraded, and Real-World Settings},
  author       = {Li, Zhiheng and collaborators},
  year         = {2026},
  howpublished = {\url{https://github.com/zhihengli-casia/puredocbench}},
  note         = {Dataset and benchmark release}
}