Document-Image Machine Translation (imgtrans)

Translate the text inside an image, scan, or born-digital PDF, and render the translation back onto the page preserving the layout — like the Google Translate camera, but as a debuggable, license-clean, production-grade pipeline.

NLP in Industry — Final Assignment, Project #15. Author: Le Dinh Minh Quan (Student 23127460).

imgtrans is a cascade: an OCR front-end reads the text + boxes, a trainable machine-translation core (facebook/m2m100_418M, fine-tuned) translates each block, and a layout-preserving overlay renderer re-draws the translation into the original boxes. A deterministic agent (D1–D5) orchestrates it and gates OCR confidence, translation quality, and overlay fit. Only the MT model is trained; OCR + layout + render are pretrained/algorithmic.

image / scan / PDF ──► OCR (Tesseract) ──► layout (blocks+boxes) ──► MT (m2m100, trained)
                                                                          │
        translated image (overlay) ◄── render (fit-to-box) ◄── verify (round-trip) ◄┘

---

How this repo meets each assignment requirement

---

Repository layout

15_Document_Image_Translation/
├── src/imgtrans/
│   ├── config.py  cli.py  logging_utils.py
│   ├── data/        # samples (seed pages + dict), synth_render (generator), dataset, download
│   ├── models/      # ocr_engine (Tesseract/Seed/Stub), baseline, model_registry
│   ├── mt/          # translator (m2m100 + dictionary, reverse for back-translation)
│   ├── imaging/     # preprocess, layout (blocks+reading order), render (fit-to-box overlay)
│   ├── training/    # train_mt, train_baseline, evaluate, tune, metrics (chrF/BLEU/CER/WER)
│   ├── agent/       # state, policy (D1-D5), tools, llm_orchestrator, imgtrans_agent
│   ├── api/         # schemas, dependencies, main (FastAPI), ui (Gradio), app_combined
│   ├── analysis/    # error_analysis, latency, layout_fidelity
│   ├── autoreport/  # artifact_loader, charts, report_pdf, slides_pptx
│   ├── monitoring/  # drift_report (job-log monitor)
│   ├── automation/  # autopilot (one button)
│   └── grading/     # checklist (rubric self-check)
├── configs/         # train.yaml, infer.yaml
├── docs/            # 14 markdown docs + DESIGN_BRIEF.md
├── notebooks/       # H100 AUTOPILOT .ipynb + COLAB_GUIDE.md
├── app/  deploy/  sample_data/  scripts/  tests/
├── Dockerfile  docker-compose.yml  Makefile
├── pyproject.toml  requirements.txt  requirements_colab.txt
└── .github/workflows/ci.yml  LICENSE (MIT)  README.md

---

Models & data (all verified on the HF Hub)

Why synthetic? No public in-image / document-image translation benchmark with gold parallel text exists, so the primary data renders source sentences onto pages and embeds the gold (source, translation, boxes) spec — letting us measure OCR CER/WER, MT chrF/BLEU and end-to-end chrF on the same images. The offline SeedEngine reads that spec so the whole pipeline runs with no Tesseract, no torch, no network.

---

Quickstart

pip install -e .                 # core: runs offline (dictionary MT + SeedEngine OCR + fit-estimate)
pip install -e .[all]            # + torch/transformers, Tesseract wrapper, FastAPI/Gradio, reportlab
# system OCR (for real images): apt-get install tesseract-ocr  (Windows: install the Tesseract binary)

imgtrans demo-agent --fast                         # run the 5-decision agent on the seed pages
imgtrans translate-text --file sample_data/sample_lines_en.txt --fast
imgtrans translate-image --image sample_data/sample_document_en.png --mode overlay --out out.png --fast
imgtrans evaluate --fast                           # MT chrF/BLEU + OCR CER/WER + end-to-end + fit-rate
imgtrans autopilot --no-train                      # report.pdf + slides.pptx + grade + bundle
bash scripts/smoke.sh                              # full offline smoke

--fast uses the dictionary baseline (no model download). Drop it to use the fine-tuned m2m100.

Train on Colab (H100, auto-adapts A100/L4/T4)

Push this folder to GitHub (or upload to Drive), open notebooks/ImgTrans_Colab_Training_H100_AUTOPILOT.ipynb, set the controls in cell 0, and Runtime → Run all. It installs Tesseract + fonts, fine-tunes the MT core (resume-safe), runs the full evaluation, and writes report.pdf + slides.pptx + the submission bundle to Drive. See notebooks/COLAB_GUIDE.md.

The agent (the mandatory agentic component)

A deterministic FSM ingest → ocr → translate → verify → render with five decision points:

An optional LLM brain (anthropic) is off by default — the agent runs fully on rules with zero paid API calls. Every step is timed and traced; same input → identical output.

Serving

imgtrans serve --ui          # FastAPI on :8000 + Gradio demo at /ui
# POST /translate-image  (upload image/PDF -> translated text + base64 overlay PNG)
# POST /translate-text   (JSON {text, mode})        GET /healthz  /version
docker compose up --build    # containerized (Tesseract + Noto/DejaVu fonts + libGL baked in)

Verified offline results

On the synthetic seed pages (offline, dictionary MT + perfect-OCR SeedEngine): MT dictionary chrF 79.9 vs identity floor 22.4, OCR CER 0.0, end-to-end image-translation chrF 76.4, overlay fit-rate 1.0, grade 0.97, all 5 decision points fire, 22 tests pass. The offline floor saturates because the seed pairs overlap the dictionary; on real OPUS-100 eval pairs the fine-tuned m2m100 dominates — the honest, non-saturated comparison happens on Colab.

Tests

pytest -q        # CPU-only, no downloads (HF_HUB_OFFLINE); graceful fallbacks everywhere

Documentation

Problem · Data · Data card · Models · Architecture · Agent · Evaluation · Deployment · Continual learning & monitoring · Privacy & robustness · Ethics · Project plan · Model card · Slides outline · Design brief

Ethics & license

Document images are highly sensitive PII (IDs, contracts, medical/financial records): the default path processes images transiently, logs metadata only, and the LLM brain is off. The tool assists translation and flags low-confidence output for human review — it never asserts certainty on high-stakes documents. Code is MIT (LICENSE); the shipped model stack is permissive (m2m100 MIT + Tesseract Apache); non-commercial options (NLLB, Surya) are flagged and not shipped.