✍️ Vietnamese Handwriting Detection System

Recognize handwritten Vietnamese from images into diacritic-correct Unicode text (line level; whole pages are routed → lines → text), with a trainable TrOCR core, a CRNN+CTC baseline, and an agentic preprocess → segment → recognize → post-correct pipeline.

NLP in Industry — Final Assignment. Author: Le Dinh Minh Quan (Student 23127460).

Vietnamese is the hard case for HTR: stacked diacritics — vowel modifiers (ă â ê ô ơ ư), the consonant đ, and six tones (à á ả ã ạ). The TrOCR decoder is a language model that must learn which diacritic bytes follow which letters — an NLP problem — and an agent orchestrates the full image → text pipeline with a confidence-gated human-review fallback.

---

✅ How this repo meets every assignment requirement

---

🏗️ Pipeline

handwriting image
   │  preprocess (deskew · denoise · binarize · contrast)   ── D1 quality routing
   ▼
page vs single-line  ──────────────────────────────────────  ── D2 routing
   │  line segmentation (projection profile)
   ▼
recognize each line (TrOCR; CRNN/Tesseract/stub fallback)    ── D3 confidence gate
   │  Vietnamese post-correction (rule + optional neural)    ── D4 acceptance
   ▼
text + per-line JSON (text · confidence · flag) + manifest

📦 Models & data (ids VERIFIED on the HF Hub)

⚠️ The primary real corpus 5CD-AI/Viet-Handwriting-OCR-v2 is GATED (access must be requested + granted on the HF Hub) and has no declared license → research/educational use only. The fallback Vietnamese datasets likewise have no declared license. Any fine-tuned model inherits these restrictions. When the gated dataset is unreachable (offline / no token / access not granted) the loader gracefully skips it and training falls back to the synthetic spine + fallback datasets.

🗂️ Repository layout

src/vnhtr/
├── config.py  cli.py  logging_utils.py
├── data/         corpus.py · synth.py · dataset.py · samples.py · download_dataset.py
├── models/       text_utils.py · vocab.py · trocr_model.py · baseline_crnn.py · postcorrect.py · recognizer.py · model_registry.py
├── preprocess/   image_ops.py · segmentation.py
├── training/     train_trocr.py · train_crnn.py · evaluate.py · tune.py · metrics.py
├── agent/        state.py · policy.py · tools.py · llm_orchestrator.py · htr_agent.py
├── api/          schemas.py · dependencies.py · main.py · ui.py · app_combined.py
├── analysis/ autoreport/ monitoring/ automation/ grading/
configs/ · data/ · models/ · tests/ · docs/ · notebooks/ · app/ · deploy/ · sample_data/

---

🚀 Quickstart

pip install -e ".[ml,vision,api,report]"

vnhtr data                          # sample images + synthetic preview
vnhtr demo-agent                    # run the agent on the sample page (offline-friendly)
vnhtr recognize --image sample_data/line_sample.png

Train

vnhtr --config configs/train.yaml train         # fine-tune TrOCR (auto-resumes)
vnhtr --config configs/train.yaml train-crnn    # the CRNN baseline
vnhtr evaluate --which test                     # TrOCR vs baseline, CER/WER + diacritics

On Colab/GPU use the notebook (below) — it auto-profiles H100/A100/L4/T4.

Serve

vnhtr serve --ui --port 7860        # FastAPI /htr + /htr/batch + Gradio UI at /ui

One-button report + slides + self-grade

vnhtr autopilot --no-train          # eval → analysis → report.pdf + slides.pptx + bundle
vnhtr grade

---

🤖 The agent (mandatory agentic component)

A deterministic FSM with four decision points acting on intermediate outputs, plus an optional LLM brain (anthropic) that validates its output and falls back to rules (default = zero paid API):

• D1 preprocessing/quality routing (ok / re-preprocess / degraded)
• D2 page-vs-line routing
• D3 recognition-confidence gate (flag low-confidence lines for human review)
• D4 post-correction acceptance (rule + optional neural Vietnamese corrector)

Every step is timed + traced and a full manifest.json is written. See docs/agent_architecture.md.

☁️ Colab / H100 training

Open notebooks/VNHTR_Colab_Training_H100_AUTOPILOT.ipynb — mounts Drive, installs Colab-safe deps (never touches torch), auto-profiles the GPU, trains resume-safely, evaluates vs the baseline, runs the agent, and generates the report/slides. Step-by-step: notebooks/COLAB_GUIDE.md.

🧪 Tests

pytest -q        # CPU-only, no model/network downloads (synthetic render + stub recognizer)

📚 Docs index

docs/: problem_definition · data_description · data_card · model_selection · evaluation · agent_architecture · deployment · continual_learning_monitoring · privacy_robustness · project_plan · ethics_statement · architecture · model_card · slide_deck_outline · DESIGN_BRIEF.

📝 License

MIT — see LICENSE. Pretrained models keep their own licenses (table above). The real Vietnamese handwriting datasets are research/educational use only (no declared license); the primary corpus 5CD-AI/Viet-Handwriting-OCR-v2 is additionally GATED (Hub access must be requested + granted). Models trained on them inherit those restrictions.