Ten OCR systems — classical, open VLMs, specialised OCR-VLMs, and frontier closed LLMs — read the same Hindi text. On clean rendered images they are indistinguishable (chrF++ 91–98). On real printed scans nine of ten collapse, the field spreads across a 76-point range, and the English OCR ranking does not transfer to Devanagari.

[!IMPORTANT]

Scope of the claims — read this. This is an evaluation / benchmark, run honestly with its limits stated. The controlled set is 100 FLORES-Hindi sentences rendered with 5 fonts; the real set is 300 word/short-phrase printed scans (Sanskrit-typeset historical print), used as a real-image robustness probe, not a full document benchmark. The word-level format disadvantages page-oriented models — we report this confound and show the synthetic-to-real gap survives it. Numbers for the closed APIs (Gemini, Claude, GPT-5.5, Mistral) reflect the model versions at eval time and will drift. CER is code-point, not grapheme-cluster, level.

Contents

• Results
• Method
• Models
• Repository structure
• Quick start
• Reproduce
• Evaluation — the honest part
• Data & licensing
• Citation
• Acknowledgments
• License

Results

Real printed Devanagari (300 scans), sorted by chrF++. We report median CER and catastrophic-rate alongside the mean, because rare repetition blow-ups wreck the mean (see Method).

[!NOTE] On clean rendered text all ten cluster at chrF++ 91–98 — synthetic-only benchmarks do not separate these systems. The differences only appear under degradation and on real scans. Full per-condition tables and figures are in the paper and results/.

Method

Benchmark. FLORES-Hindi → images (5 Devanagari fonts) under 4 conditions: clean, Gaussian blur, sensor noise, low-DPI. Plus 300 real printed scans (Sanskrit-OCR-Typed). All scoring is Unicode NFC-normalised; metrics are CER, WER, and chrF++.

Robustness methodology. Corpus mean CER is dominated by a few catastrophic samples: under blur, DeepSeek-OCR has the best median (1.5) yet a mean of 73.7, because ~2% of outputs enter a repetition loop up to 71× the reference length. We therefore report median + catastrophic-rate, not just the mean.

Post-correction (ByT5). A byte-level corrector fine-tuned on an engine's own real OCR-noise distribution (6,000 pairs), with inference-time chunking to ≤90 chars. It improves EasyOCR (chrF++ +1.2 to +1.5 in every condition) but does not transfer to other engines — post-correction must be distribution-matched.

Models

[!NOTE] Omitted (honestly): PaddleOCR and GOT-OCR2 would not run reliably in our environment (PaddlePaddle segfault on Amazon Linux 2023; processor-instantiation error); LlamaParse returned non-Devanagari Latin output on every image. All three are excluded from the quantitative tables rather than reported with unreliable numbers.

Repository structure

.
├── scripts/          # full pipeline (run from repo root)
│   ├── render.py         # FLORES-Hindi -> images (multi-font, 4 conditions)
│   ├── make_real.py      # build the 300-image real printed-Devanagari set
│   ├── runner.py         # unified runner for all engines
│   ├── robust.py         # mean-vs-median / catastrophic analysis
│   ├── taxonomy.py       # Devanagari error taxonomy
│   ├── report*.py        # aggregate results
│   ├── pc_*.py           # post-corrector: data, train, eval
│   └── plot_*.py         # figures
├── results/          # results_matrix.json, pc_results.json (the numbers)
├── figs/             # paper figures
├── paper.tex · paper.pdf
└── README.md · SUBMISSION_PLAN.md · LICENSE

Quick start

Run any engine on a folder of images and score it against ground truth:

# score EasyOCR on the clean synthetic split
python scripts/runner.py --model easyocr --imgdir images/clean --tag clean

# score a frontier API model on the real scans (keys in ~/.ocr_keys)
python scripts/runner.py --model gemini --imgdir real/images --gt real/gt.json --tag real

API keys (closed models) are read from ~/.ocr_keys:

GEMINI_API_KEY=...
ANTHROPIC_API_KEY=...
OPENAI_API_KEY=...
MISTRAL_API_KEY=...

Reproduce

[!TIP] Run all commands from the repo root. Local models need a CUDA GPU (a single 24 GB A10G suffices — one model resident at a time). Three uv venvs are used to satisfy conflicting transformers pins: EasyOCR (4.40), Qwen/Unlimited/Qwen3-VL/olmOCR + APIs (4.57.1), DeepSeek-OCR (4.46.3).

# 1. build the benchmark
python scripts/render.py -n 100          # synthetic, 4 conditions
python scripts/make_real.py              # 300 real printed scans (from HF)

# 2. run engines (per model x condition)
python scripts/runner.py --model <m> --imgdir images/<cond> --tag <cond>

# 3. analyse
python scripts/report_real.py            # median + catastrophic stats
python scripts/robust.py                 # mean vs median
python scripts/taxonomy.py --cond clean  # error taxonomy

# 4. post-corrector
python scripts/pc_data_real.py -n 6000   # real OCR-noise pairs
python scripts/pc_train.py --data out/pc_real.jsonl --save pc_model_real
python scripts/pc_eval.py easyocr@clean easyocr@blur   # before vs after

Evaluation — the honest part

chrF measures n-gram overlap with references, not human quality, and our findings are stated with their caveats:

• The real set is word/short-phrase level and Sanskrit-typeset, which disadvantages page-oriented document parsers. We flag this confound — and show that the synthetic-to-real collapse survives it, since the gap between (e.g.) Gemini and GPT-5.5, or Qwen3-VL and olmOCR, sits within the same regime.
• Closed-model numbers reflect API versions at eval time and will move as those models update.
• Post-correction gains are small because the clean baseline CER is already low; bigger wins need harder, real, sentence-level data.

Takeaway: synthetic-only Devanagari benchmarks overstate quality; real evaluation is indispensable; and the best "general" OCR for English may be far from the best for Indic.

Data & licensing

• FLORES-200 (Hindi side) — source text, CC-BY-SA.
• Sanskrit-OCR-Typed (Hugging Face) — real printed-Devanagari scans for the real set.
• Closed models accessed via their respective APIs (Google, Anthropic, OpenAI, Mistral); open models via Hugging Face. Check each provider's terms before commercial use.

Citation

@misc{singh2026devanagari,
  title  = {Can OCR-VLMs Read Devanagari? A Stress-Test Benchmark and Post-Correction Study},
  author = {Aditya Pratap Singh},
  year   = {2026},
  eprint = {arXiv:XXXX.XXXXX},   % update once arXiv assigns the ID
  archivePrefix = {arXiv},
  primaryClass = {cs.CL}
}

Acknowledgments

• FLORES-200, Sanskrit-OCR-Typed — evaluation data.
• DeepSeek-OCR, Unlimited-OCR, Qwen2.5/3-VL, olmOCR, EasyOCR — evaluated systems.
• ByT5, sacreBLEU — post-corrector and metrics.

License