DM-LP/TP-500

Benchmark and evaluation code for Benchmarking Extraction on Arabic Title Pages: From Traditional OCR to Vision Language Models (TPDL 2026).

DM-LP/TP-500 is a 500-page subset of the Digital Maktaba corpus (DM-LP) targeting classical Arabic book title pages with a two-expert gold-standard transcription. This repository contains the per-system evaluation reports, the per-page metric outputs, the runner and evaluation code, and full documentation. The gold transcriptions and the per-page predictions of the 54 evaluated systems are distributed as a separate Release asset (see Heavy data).

Contents

DM-LP-TP-500/
├── data/reports/                     evaluation outputs (committed in repo)
│   ├── <system>.csv                  per-page CER and WER (54 files)
│   ├── <system>.json                 per-system aggregate (macro/micro/totals, 54 files)
│   ├── aggregated_summary.json
│   ├── per_model_diagnostic.json
│   ├── gold_statistics.json
│   ├── greedy_vs_sampling.json
│   ├── category_comparison_5cat.json
│   ├── full_ranking.csv              54 systems including dropped
│   └── full_ranking_published.csv    54 systems sorted by macro CER
├── docs/
│   ├── models_parameters.md          per-system decoding hyperparameters
│   ├── prompts_used.md               model-native prompts for every VLM
│   └── pages_manifest.json           canonical (doc, page) listing
└── code/
    ├── runners/                      one runner per system family (21 files)
    ├── eval/
    │   ├── evaluate_txt.py           Levenshtein metric pipeline
    │   └── collect_summaries.py
    ├── converters/
    │   └── pagexml_to_txt.py         PAGE-XML → plain-text extractor
    └── utils/
        └── nfc_mirror.py             NFC mirroring helper

Heavy data

The 500-page gold standard and the per-page predictions of all 54 systems are distributed as a Release asset to keep the repository light. Download DM-LP-TP-500-data.tar.gz from the Releases page and extract it inside the repository root:

git clone https://github.com/<owner>/DM-LP-TP-500.git
cd DM-LP-TP-500
curl -L -o data.tar.gz https://github.com/<owner>/DM-LP-TP-500/releases/latest/download/DM-LP-TP-500-data.tar.gz
tar xzf data.tar.gz --strip-components=1 -C data/

After extraction the layout under data/ is:

data/
├── gold/                 500 .txt files, one per page, Arabic-only
├── predictions/          54 systems × 500 pages, Arabic-only filtered
│   └── <system>/<doc>/<page>.txt
└── reports/              already in the repository

Asset weight: ≈7 MB compressed, ≈140 MB extracted.

Coverage

500 distinct volumes from the Giorgio La Pira Library (FSCIRE), a major Islamicate heritage collection ranging from early print to modern printed books. After Unicode NFC normalisation and Arabic-block character filtering, the gold standard contains 132,863 characters and 22,634 words; one page is empty after filtering and is reported separately. See docs/pages_manifest.json and data/reports/gold_statistics.json for the full breakdown.

The 54 evaluated configurations cover five categories: 1 commercial reference (Google Vision AI), 13 generalist VLM configurations, 24 multilingual OCR-specialist VLM configurations, 5 Arabic-specialist VLMs/OCR, and 11 traditional CNN-based OCR engines. The full taxonomy with hyperparameters is in docs/models_parameters.md; the prompt used by each VLM is in docs/prompts_used.md.

Evaluation

For each page with non-empty gold, character- and word-level Levenshtein edit distances are computed via rapidfuzz. Both micro (length-weighted global) and macro (per-page mean) error rates are reported. The full pipeline is:

gold/pred → NFC normalisation → whitespace collapse → Arabic-block filter
         → whitespace re-collapse → rapidfuzz Levenshtein

This is the same four-step pipeline applied to every system. Markup (Markdown, LaTeX, JSON keys, special tokens) falls outside the Arabic Unicode blocks and is removed automatically; no per-system parser is needed. The Arabic blocks retained are U+0600–06FF, U+0750–077F, U+08A0–08FF, U+FB50–FDFF, U+FE70–FEFF.

Quick start

After downloading and extracting the data asset (see above):

# Re-evaluate a single system against the released gold
python -m code.eval.evaluate_txt \
    --gold     data/gold \
    --pred     data/predictions/gemma4_31b_greedy \
    --json     /tmp/gemma_report.json \
    --report   /tmp/gemma_per_page.csv \
    --norm     space_collapse \
    --workers  8

The result reproduces the corresponding entry in data/reports/gemma4_31b_greedy.json exactly.

To re-aggregate the per-system JSONs after an update:

python -m code.eval.collect_summaries data/reports

code/runners/*.py contains the inference scripts for every supported system family. Each runner is self-contained and reads the model-native prompt from a top-level PROMPT constant; see docs/prompts_used.md for the exact strings.

Treatment of the commercial reference

Google Vision AI is included as a commercial reference because it produced the silver-standard layer of the parent DM-LP corpus from which the gold was derived. Independent two-expert revision and a character-by-character cross-validation diff were used to decouple the gold from the silver, but a small residual upward bias on the Google Vision AI score is possible. For this reason Google Vision AI is reported at rank 1 in full_ranking_published.csv but excluded from the per-category aggregates and from the open-source ranking discussed in the paper.

License

All contents of this repository, and the data Release asset, are distributed under the Creative Commons Attribution-NonCommercial 4.0 International license (CC BY-NC 4.0). See LICENSE.txt for the full notice and https://creativecommons.org/licenses/by-nc/4.0/ for the plain-language summary.

Citation

@unpublished{sala2026benchmarking,
  title  = {Benchmarking Extraction on {A}rabic Title Pages: From Traditional {OCR} to Vision Language Models},
  author = {Sala, Luca and Sullutrone, Giovanni and Vigliermo, Riccardo Amerigo and Aftar, Sania and Bergamaschi, Sonia},
  year   = {2026},
  note   = {Manuscript under submission}
}

The gold transcriptions of DM-LP/TP-500 were derived from the silver layer of the parent DM-LP corpus:

@inproceedings{sala2025dm,
  title        = {{DM-LP}: A Large {A}rabic Script Languages Dataset for {OCR} and Cataloguing Research},
  author       = {Sala, Luca and Vigliermo, Riccardo Amerigo and Sullutrone, Giovanni and Bergamaschi, Sonia},
  booktitle    = {International Conference on Theory and Practice of Digital Libraries (TPDL)},
  pages        = {124--134},
  year         = {2025},
  publisher    = {Springer}
}

Acknowledgements

This work was supported by the PNRR project Italian Strengthening of ESFRI RI Resilience (ITSERR) funded by the European Union — NextGenerationEU (CUP: B53C22001770006). Compute was provided by the LEONARDO supercomputer (EuroHPC JU, hosted by CINECA, Italy) and by the local DGX-A100 facility at FSCIRE. We are grateful to the Giorgio La Pira Library (FSCIRE, Bologna) for providing the source corpus and to the Arabic-language experts who produced the gold annotations.