PegonKitab — Dataset and Code for Multimodal Post-OCR Correction of Pegon Manuscripts

This repository contains the dataset, code, and evaluation scripts for the paper:

When Arabic OCR Fails: Benchmarking and Multimodal Post-OCR Correction for Pegon Manuscripts
Syauqie Muhammad Marier, Bayu Nadya Kusuma, Md Mehedi Hassan Dorjoy, Linan Zhu, Xiangjie Kong

---

Overview

Pegon is a modified Arabic script used in Javanese and Sundanese Islamic manuscripts. Several Pegon-specific characters differ from standard Arabic by only one or two dots, causing systematic OCR failures when using Arabic-trained models.

This work introduces:

• PegonKitab — 2,801 annotated handwritten line images from 5 manuscripts
• A 16-model OCR benchmark spanning traditional OCR, transformer HTR, vision-language models, and Arabic-specialized systems
• CASTLE-Pegon — a multimodal post-OCR correction model (ByT5 + ViT) achieving 18.6% CER reduction over GPT-4o

---

Dataset

PegonKitab Statistics

Download

Data Format

Each JSON file in data/splits/ contains a list of records:

{
  "record_id": "Kitab01_001",
  "kitab_id": "Kitab01",
  "language": "javanese",
  "image_filename": "Kitab01-001.png",
  "ground_truth": "...",
  "ocr_gpt4o": "..."
}

---

OCR Benchmark Results

Full results on PegonKitab test set (506 lines). CER and WER are macro-averaged. RC = recognition coverage (% of inputs producing non-empty output).

---

CASTLE-Pegon Model

Architecture

CASTLE-Pegon fuses visual and textual modalities for post-OCR correction:

• Text branch: ByT5 encoder (byte-level, no tokenization needed for Arabic script)
• Visual branch: ViT encoder (patch-level image features from line image)
• Fusion: Learned gating — α·text_features + (1−α)·visual_features
• Decoder: ByT5 decoder with auto-regressive correction

Results