ocr_review_models

Scanned Arabic children's books → an interactive read-along library.

A reader opens a book, sees the page, hovers or selects a passage, and hears it read aloud in Arabic fuṣḥā — with the spoken word highlighted as it plays.

Scale target: N books, templated. Currently N = 36; design point N ≥ 1,000.

---

The core problem

This is not an OCR problem and not a TTS problem. Both are solved products you can buy. The two hard parts are:

1. Arabic TTS pronounces the diacritics, not the letters. An OCR error that looks harmless on the page becomes a confidently mispronounced word.
2. Sync. Text on screen and audio in the ear must be the same object, addressable at word granularity, in two different views (page image and reflowed text), across a catalogue too large to ever proofread by hand.

Everything below follows from those two.

---

Evidence: why the naive pipeline fails

Tashkeel coverage is bimodal, not "sometimes missing"

Diacritics per Arabic letter, measured across current OCR output:

No middle ground. Books are printed either with tashkeel (~0.8) or without (~0.0). Two paths are required; "add tashkeel everywhere" would corrupt the already-correct half.

Where tashkeel exists, it can be confidently wrong

الشمس is 78% diacritized — and has 10 letter-substitution errors in 16 pages:

الفضل is the dangerous class: a valid Arabic word, so a spellchecker passes it, a diacritizer diacritizes it beautifully, and the reader hears al-faḍl where the book says al-faṣl.

Corollary: correct before diacritizing. Diacritizing a misspelled word produces fluent, confident, wrong audio — the hardest kind of bug to find, because it sounds fine.

---

The architectural enhancement: token anchoring

This is the recommendation that makes the sync requirement tractable.

The pipeline mutates text at three stages — correction (الفضل→الفصل), diacritization (الفصل→الْفَصْلِ), and chunking. Meanwhile the reader needs to highlight a box on a scanned image and a span of reflowed text and seek into audio, all for the same word. String offsets cannot survive that; they shift at every stage.

Solution: assign every word a stable ID at OCR time and never renumber it. Every later stage attaches to the ID, not the string.

flowchart LR
    subgraph T["one token, four representations"]
        ID["id: p007w042"]
        ID --> BOX["box [412,880,498,915]<br/>from OCR — image view"]
        ID --> RAW["raw: الفضل<br/>what OCR saw"]
        ID --> TXT["txt: الْفَصْلِ<br/>corrected + diacritized"]
        ID --> TIME["t: [1240,1610] ms<br/>from TTS — audio seek"]
    end

Given that, all three surfaces are the same lookup:

Without stable IDs, correction changing مريي→مرئي (different length) silently breaks every downstream offset — and the bug appears as a highlight drifting one word off, three pages later.

---

Pipeline

flowchart TD
    PDF["PDF library — N books"] --> OCR

    subgraph S1["1. EXTRACT — must preserve bbox"]
        OCR["OCR engines (pluggable, N-way for consensus)"]
        OCR --> TOK["tokenise: assign stable word IDs + bbox"]
    end

    TOK --> GATE

    subgraph S2["2. CORRECTNESS GATE — fix letters, keep IDs"]
        GATE["per-token text"]
        GATE --> D1["dictionary — non-words"]
        GATE --> D2["LLM proofread — valid-word-wrong-context"]
        GATE --> D3["engine consensus — disagreement"]
        D1 --> VOTE{"agree?"}
        D2 --> VOTE
        D3 --> VOTE
        VOTE -->|"yes"| FIX["auto-fix + diff log"]
        VOTE -->|"split"| ESC["escalate to human"]
    end

    FIX --> MEASURE
    ESC -.-> MEASURE

    subgraph S3["3. TASHKEEL — two paths, IDs unchanged"]
        MEASURE{"diacritic ratio"}
        MEASURE -->|"below 0.15"| ADD["diacritize"]
        MEASURE -->|"0.15 or above"| KEEP["keep existing"]
    end

    ADD --> TTS
    KEEP --> TTS

    subgraph S4["4. SPEECH + ALIGNMENT"]
        TTS["TTS engine (pluggable)"]
        TTS --> W1["native word timestamps<br/>(Hume: word + phoneme)"]
        TTS --> W2["forced alignment<br/>(SILMA/Gemini: WhisperX or MFA)"]
        W1 --> BIND["bind timings to token IDs"]
        W2 --> BIND
    end

    BIND --> IDX["page index JSON<br/>text + bbox + timings"]
    BIND --> AUD["audio → object storage / CDN"]
    IDX --> APP["reader app"]
    AUD --> APP

---

Delivery: three modes, one index

All three are supported; the index is identical in each case — only when synthesis happens changes.

On-demand synthesis returns audio for exactly the selected string, so it needs no alignment map — but it does still need token IDs to know which string to send. Token anchoring is required in all three modes.

---

Scale: everything is a function of N

Ratios measured from real artifacts in this repo (31.4 pages/book, ~400 chars/page, 0.06 s/char, 5.7 chars/word).

Consequences that changed the design:

• Audio never enters git. 12 GB vs. the 111 MB OCR commit that already strained the repo. → object storage + CDN, git holds text and timings only.
• At 293 MB the index is itself a dataset, not a side-file. → per-page files, shardable, lazily fetched by the reader.
• 14 h wall-clock at SILMA's observed 1 req/s makes serial production infeasible. → the runner must parallelise across books, or use an engine with a higher cap.
• OCR stops being free. GLM-OCR is $9 at N=1,000; Baseer is $836. → engine choice is now a budget decision, informed by the pricing dashboard already in this repo.

---

Two phases

Brute-forcing every option across the full catalogue is impossible — the sweep alone would be millions of characters, and SILMA credits were exhausted in testing at ~1,500. So: sweep on a sample, lock one path for production.

flowchart LR
    subgraph P1["PHASE 1 — bake-off, 3 sample books"]
        direction TB
        A1["every OCR engine"] --> A2["every detection method"]
        A2 --> A3["every TTS + alignment combo"]
        A3 --> A4{"score"}
        A4 -->|"OCR: 8-indicator scorecard"| A5["winner per stage"]
        A4 -->|"TTS: Elo arena, human ears"| A5
        A4 -->|"sync: highlight drift in ms"| A5
    end
    subgraph P2["PHASE 2 — production, N books"]
        direction TB
        B1["locked config"] --> B2["parallel runner"]
        B2 --> B3["index + audio → CDN"]
    end
    A5 ==>|"config"| B1

Sample set — exercises both tashkeel paths

What the sweep measures

---

Data contract

One JSON per page. Audio and rendered text derive from it; it is the only source of truth.

// index/<book>/p007.json
{
  "book": "الشمس", "page": 7,
  "engine": "lighton", "tts": "hume", "align": "native",
  "diac_ratio": 0.81, "path": "keep", "status": "tts_done",
  "audio": "https://cdn/…/الشمس/p007.mp3",
  "fixes": [
    { "id": "p007w042", "from": "الفضل", "to": "الفصل",
      "by": "consensus", "votes": "3/3" }
  ],
  "tokens": [
    { "id": "p007w042",
      "raw": "الفضل",              // what OCR saw
      "txt": "الْفَصْلِ",            // corrected + diacritized  → text view
      "box": [412, 880, 498, 915], // page-image coords         → image view
      "t":   [1240, 1610] }        // ms into audio             → playback
  ]
}

---

The code

Everything below is real, runnable, and was used to produce the two books in work/. Run from the repo root.

src/pipeline/
  main.py            single entry point           — run one PDF through the whole pipeline
  extract.py         layer 01-03, text-layer path  — read PDF text + word boxes, clean by rule
  ocr_gemini.py       layer 01, OCR path           — Gemini vision + response schema, line boxes
  ocr_lighton.py      layer 01, OCR path (alt)     — LightOn, text only, no boxes
  tashkel.py         layer 04b                     — diacritise 'diacritise'-routed pages, ids unchanged
  speak.py           layer 05-06                   — bind TTS word timings onto token ids (engine-agnostic)
  tts_hume.py        layer 05 (Hume path)          — synthesise + native word timestamps in one call
  tts_gemini.py       layer 05 (Gemini path)       — synthesise only, no native timestamps on the free tier
  align_whisper.py   layer 05b (forced alignment)  — faster-whisper transcribes Gemini's own audio for timing
  run_extract.py     driver, text-layer books      — resumable: skips a page whose record already exists
  run_ocr_gemini.py  driver, scanned books          — Gemini vision path (resumable, BOOK=/PAGES= env)
  run_ocr_lighton.py driver, scanned books (alt)    — LightOn path, no boxes (resumable, BOOK=/PAGES= env)
  run_tashkel.py     driver, diacritisation        — resumable, skips 'keep' pages and pages already done
  run_speak_hume.py  driver, TTS + alignment       — Hume native timestamps, resumable
  run_speak_gemini.py driver, TTS + alignment       — Gemini + faster-whisper, resumable
  build_reader.py    layer 08                      — emits work/<book>/reader.html; drops the image
                                                       view entirely when a book has no boxes at all

Every module above is also a small, importable library (a run(...)/build(...) function), not just a script — main.py calls them directly rather than shelling out, so the single entry point below and the per-stage commands after it are the same code, not two parallel implementations.

Running it

# the whole pipeline, one PDF in, reader.html out — default engines: Gemini OCR, Gemini TTS
python3 src/pipeline/main.py "assets/PDF Samples/النجوم.pdf"

# choose engines explicitly, and cap OCR to a few pages for a quick test
python3 src/pipeline/main.py "assets/PDF Samples/book.pdf" --ocr lighton --tts hume --pages 5

# --tts gemini --voice Kore is the default; override per run:
python3 src/pipeline/main.py "assets/PDF Samples/book.pdf" --voice Puck

Every stage main.py calls is independently resumable — re-running it after a quota stall or a crash picks the book up where it left off rather than redoing already-done pages, all the way back through extraction. Run the individual stages by hand (env-var driven, same resumability) if you want to inspect or restart one stage in isolation:

# text-layer book (no OCR needed — 25 of 36 PDFs qualify)
BOOK="النجوم" python3 src/pipeline/run_extract.py

# scanned book (OCR path)
BOOK="المومياوات ناشيونال جيوجرافيك " PAGES=5 python3 src/pipeline/run_ocr_gemini.py

# diacritise pages the router sent down 'diacritise' ('keep' pages untouched)
BOOK="المومياوات ناشيونال جيوجرافيك " python3 src/pipeline/run_tashkel.py

# synthesise + align via Gemini (default) or Hume (native timestamps, needs credit)
BOOK="النجوم" python3 src/pipeline/run_speak_gemini.py
BOOK="النجوم" VOICE="Layla, Arabic Philosopher" python3 src/pipeline/run_speak_hume.py

BOOK="النجوم" python3 src/pipeline/build_reader.py

Every driver is resumable: a page already done is skipped, so hitting a quota ceiling costs one page on retry, not the book. That was not theoretical — all three providers (Hume, SILMA, Gemini) hit their limits during this build.

---

Correcting OCR errors (tools/)

main.py above gets a book from PDF to reader. It doesn't tell you whether the OCR was right. LightOn especially can't be checked in isolation — it returns no bounding boxes at all (/parse and even /search's include_bboxes — see tmp/lighton_bbox_support_request.md), so there's no image to crop and verify a suspicious word against. The fix: OCR the same PDF a second time with Gemini (which does return boxes and saves the rendered page image), and use disagreement between the two engines — plus a text-only pass that also checks words they agree on — as the detection signal. Full design rationale in docs/ocr-correction-architecture-built.html.

tools/
  ensemble_review.py           detect              — diff LightOn vs Gemini OCR of the same PDF, page by page
  vlm_arbiter.py                resolve             — Gemini vision reads the real page per flagged page, one call not per word
  apply_corrections.py          apply (word-level)  — writes VERIFIED corrections into pXXX.json; reversible, ocr_txt kept
  replace_page.py               apply (page-level)  — wholesale-swaps a page LightOn failed almost entirely (>=50% gemini-only)
  context_detector.py           detect (correlated) — catches LightOn+Gemini agreeing on the SAME wrong word (OpenRouter/DeepSeek, text-only)
  build_context_review.py       review              — browsable HTML for context_detector.py's findings, sorted by suspicion
  apply_context_suggestions.py  apply (reviewed)     — applies only human-accepted suggestions from context_decisions.json
  build_review.py               review (deep audit)  — original-vs-corrected side by side for one page range, dual audio tracks
  run_full_pipeline.py          orchestrator         — runs the whole automatable chain in one command, stops before the one manual step

Same rule as the base pipeline: every tool is a run(...) function other tools import, not just a script, and every stage is independently resumable — a 429 mid-run costs at most the page it failed on, confirmed by repeated real quota exhaustion while building this (see Open items).

Running the correction pipeline

VX2=/Users/ahmedmostafa/miniconda3/envs/vx2lm/bin/python3

# the whole automatable chain, one command — both OCR engines through
# correlated-error detection, then it stops for the one deliberately-manual step
$VX2 tools/run_full_pipeline.py "assets/PDF Samples/book.pdf"

# scope it to specific pages (passed through to both OCR passes; same
# convention as main.py's --pages) — a plain count or an explicit range
$VX2 tools/run_full_pipeline.py "assets/PDF Samples/book.pdf" --pages 2
$VX2 tools/run_full_pipeline.py "assets/PDF Samples/book.pdf" --pages 11:15

# ...or run it stage by stage; see docs/full-pipeline-guide.html for the
# complete copy-pasteable sequence and docs/tools-reference.html for what
# each tool reads/writes and how to review its output
python3 tools/ensemble_review.py "work/<book>" "work/<book>-lighton"
$VX2  tools/vlm_arbiter.py "work/<book>-lighton/ensemble_review.json"
python3 tools/apply_corrections.py "work/<book>-lighton/ensemble_review.json"
python3 tools/context_detector.py "work/<book>-lighton/ensemble_review.json"
python3 tools/build_context_review.py "work/<book>-lighton/context_review.json"
# -> review work/<book>-lighton/context_decisions.json by hand, then:
python3 tools/apply_context_suggestions.py "work/<book>-lighton/context_review.json"

tools/*.py only need plain python3 (stdlib + requests/dotenv); the vx2lm env is needed only where Gemini or audio alignment is called (vlm_arbiter.py, build_review.py, and the base pipeline itself).

Docs

---

Two extraction paths

The corpus splits, and the split matters more than the engine choice.

Both emit the identical page record, so everything downstream is shared. The record carries source and box_level so the reader can tell the difference instead of implying precision it doesn't have.

Finding that reshaped this: OCR of السماء turned الْفَصْلِ ("classroom") into الفضل ("grace") four times. The same book's text layer has it right. For 69% of the corpus, the OCR error class is avoidable entirely.

---

Stage → implementation mapping

---

What was actually produced

---

The HTML artifacts

Four standalone files, no build step, no CDN. Open any of them directly.

About demo/pipeline_poc.html

There is no demo_pipeline.html — this is the file. It answers "what happens to a PDF, step by step, and what does the user finally see":

1. Data-integrity finding — sam_2/الشمس.txt is actually السماء.pdf, misfiled. Its cover was OCR'd as لولو and the ISBN digits disagree.
2. System flow — nine layers, PDFs in, reader out, with branch diamonds for the correctness vote and the tashkeel route. Click a node for what it emits.
3. Token anchoring — one id, four representations (box, raw, txt, t).
4. Trace — the same line at every stage, showing what each layer changes.
5. Reader POC — plays real audio; the highlight is driven by Hume's own word timestamps, not a timer.

Anything simulated in it is labelled as such — in that file, only the bounding boxes are (it predates the text-layer discovery that made real boxes free).

---

Decisions

---

Open items

Resolved during the build

• Bbox survival — solved twice. The text layer gives exact word boxes for 25 of 36 books; Gemini + a response schema gives line boxes for the rest, verified by overlaying them on the rendered page.
• Does any engine give fuṣḥā and word timestamps — Hume does emit word timestamps (93.7% coverage measured). Its Arabic voice quality is still unjudged by ear.
• No diacritiser exists — built. tashkel.py wraps arabic-diacritizer (BiLSTM + sentence cache, MIT, CPU-only), chosen per the research note (models/ibn_battuta/Tashkeel libraries and APIs - research.html) over a general LLM, which the same note measures at 3× the error rate (GPT-5.3: 20.9% DER vs. 6.6%). Diacritises full-sentence context, then re-attaches words to token ids by position rather than by re-tokenising the output — a page where the model returns a different word count than it was given is left untouched and reported, not silently misaligned. Run across both المومياوات extractions; word count matched on every page attempted.
• Gemini TTS has no word timestamps — confirmed two ways: the response carries nothing beyond raw audio, and passing audio_timestamp=True doesn't even reach the server — the SDK raises client-side (audio_timestamp parameter is only supported in Gemini Enterprise Agent Platform mode, not in Gemini Developer API mode), confirmed on the latest SDK version too. That mode exists (tested with vertexai=True on the same key — the client-side block disappears and it fails differently, on a 403 for the Agent Platform API not being enabled on the project), but it's a different Google Cloud billing surface than the free-tier key this project uses, so align_whisper.py (faster-whisper, local, free) stands in instead. Proven on a real page before wiring into the driver: 35/35 words matched in order, clean monotonic timings.
• Can LightOn's bboxes be recovered via its Search API — no, not for this account. include_bboxes: true on /api/v3/search is real and documented, but returns [] in every condition tested live (scanned PDF, text-layer PDF, tight single-chunk queries, both /search and /retrieve). Support request drafted in tmp/lighton_bbox_support_request.md.
• The correctness gate is rule-based only — built, end to end, run on two books. tools/ensemble_review.py diffs LightOn against a Gemini OCR of the same PDF (234 flagged / 17 pages on the pilot book); vlm_arbiter.py resolves each flagged page against the real image (84% of verdicts matched one of the two original reads, 7% were a genuine third answer — e.g. it stripped Gemini's own TOC dot-leader noise correctly, but also once re-guessed an ISBN digit string differently from both originals, which is exactly the class of answer nothing should auto-trust). Word-level fixes (apply_corrections.py) and whole-page replacement for pages LightOn failed almost entirely (replace_page.py, ≥50% of a page gemini-only — found by looking at the actual page image: white text reversed out of a dark background with a ghosted watermark behind it, a genuinely hard case for a 1B-parameter OCR model) cover disagreement. A separate text-only pass (context_detector.py, OpenRouter/DeepSeek so a Gemini quota day doesn't also block it) covers the one thing engine-disagreement can never see: both engines agreeing on the same wrong word — 109 and 29 cases caught on the two books tested, one confirmed by hand against the source image (الْغِنَاءُ "singing" vs. the real الْغَنَّاءِ "lush", describing a forest — and the model's own proposed fix for it was also wrong, which is exactly why nothing from this layer auto-applies). Full writeup in docs/ocr-correction-architecture-built.html, real scenarios in TESTING.md.

Blocking

• Every TTS/OCR provider tried is quota-exhausted on its free tier — Hume (mid-book on النجوم), SILMA (mid-book earlier), Gemini vision (page 18 of المومياوات), and Gemini TTS, now measured precisely across two more books built with the correction pipeline: gemini-3.1-flash-tts caps at 10 requests/day, gemini-3.6-flash (OCR + VLM arbitration) at 20 requests/day, both free-tier, both per-project-per-model. Every driver resumes, so this is a billing/scheduling question, not an engineering one — confirmed repeatedly: a run cut off by a 429 always finished cleanly the next day from wherever it stopped, at zero wasted cost for anything already done.

Known limitations of what was built

• LightOn's OCR is non-deterministic between calls on the same page. Page 10 returned clean HTML on one call and literal LaTeX artifacts ($\mathcal{S}$) on another. ocr_lighton.py's cleaner now strips any \command{...} wrapper generically (not just \text{}), not only the pattern seen first — but a future page could still surface a pattern neither call exercised.
• The binder can fail completely, not just partially, on hard pages. Two of المومياوات's 33 pages (a price/ISBN back-cover page, and a heavily over-diacritized 2-token fragment) got zero anchors — digits vs. their spoken word form, and stacked diacritics, both defeat the letter-only matching key. build_reader.py now renders those tokens as plain unhighlightable text instead of crashing on the missing timing.
• A driver resumability check clobbered downstream status once — then the same class of bug recurred twice more. run_tashkel.py originally treated status == "diacritised" as "done" — but run_speak_gemini.py advances status to "spoken", which the check didn't recognise, so re-running tashkeel after synthesis briefly reset 13 already-spoken pages back a stage (their audio and bind data were untouched on disk; the reader's status == "spoken" filter just stopped seeing them). Fixed to check for ("diacritised", "spoken"); recovered by re-aligning the existing audio locally, with no repeat API calls. The general lesson — a resume check must recognise every later state, not just the immediately-next one — was written down here but not actually applied to every driver: run_ocr_gemini.py and run_ocr_lighton.py both still checked only status == "extracted", so re-running main.py on a book the correction pipeline had already fixed would silently re-OCR every page from scratch and destroy every applied correction. Caught by building tools/run_full_pipeline.py and testing it for idempotency — fixed the same way in both files. A separate but related bug in the same family: apply_corrections.py/apply_context_suggestions.py compared corrected text against a VLM verdict with exact string equality, and vlm_arbiter.py often returns undiacritized text even when the current token is already correctly diacritized — so re-running the apply step stripped diacritics back off already-correct words and forced a pointless re-diacritize/re-TTS cycle. Fixed by comparing letters only (normalize_for_compare, the same folding ensemble_review.py already used for its own diff).
• Hume's word timestamps are not one-per-word. Page 14 returned 20 for 29 tokens, and one interval was inverted (كريم [12920, 5240]). The binder anchors what it gets and interpolates the rest, recording t_src per token.
• OCR boxes are line-level. Word boxes on the OCR path are subdivided by letter count — approximations, flagged box_level: "line".
• Pages 12–16 of النجوم were bound before raw timestamps were persisted, so their gaps were repaired from surviving anchors rather than re-bound from source. p014 is 19/29 anchored; the rest are 93–100%.
• The correctness gate is rule-based only. The dictionary/LLM/consensus design in this README is unbuilt — it wasn't needed for a text-layer book, but the OCR path has no error detection at all yet.

Still open from the plan

• Voice choice unvalidated — "Layla, Arabic Philosopher" is the only Arabic-designated voice in Hume's 160-voice library, chosen on that basis alone. Nobody has listened critically yet.
• Reference transcripts for the sample books (~1 h each, manual).
• TTS budget for N=1,000: 12.5 M characters, unpriced.
• Parallelism — 14 h serial at N=1,000; the runner must shard.
• Front matter: page 1 is title/ISBN/deposit number. Currently skipped for narration; confirm that's wanted.
• 7.9 MB of audio was committed in 702c803 before .gitignore covered it. Untracked now, but still in history.