Hayai OCR v2.1

Hayai is a lightweight (~150M parameter) vision-to-text OCR model designed for fast, crop-level transcription across Japanese, Chinese, Korean, and English.

By pairing Google’s SigLIP2 NaFlex vision encoder with a 12-layer custom causal transformer decoder, Hayai reads dense, stylized, horizontal, and vertical text directly from images in a single forward pass without requiring a separate text detection stage (e.g., DBNet/YOLO). (Doesn't work for full pages though. Only crops)

---

What's New in v2.1 (Joint Multimodal + Linguistic Pretraining)

In prior versions, compact OCR models struggled with visually ambiguous CJK radicals and homoglyphs (e.g., confusing 銀 vs. 高 or 校 vs. 枚) because a pure image-trained decoder lacked statistical language priors.

Hayai v2.1 introduces Joint Multi-Task Training:

• Zero-Overhead Language Prior Injection: Co-trained directly on streaming Wikipedia (JA, ZH, KO, EN) and Aozora Bunko corpora. Text-only passes enter the decoder directly with 1D RoPE (bypassing the vision backbone), teaching the decoder deep contextual CJK transition probabilities.
• Radical & Counter Disambiguation: Eliminates homograph and counter errors on complex layouts.
• Flawless Multi-Script Support: Significantly boosted English accuracy while maintaining high precision on vertical Japanese, Korean Hangul, and Chinese Hanzi.

---

Architecture

• Total Parameters: ~150M
• Vision Encoder: google/siglip2-base-patch16-naflex (~86M params)
  • Native aspect ratio preservation via NaFlex patching (no forced warping/squashing of dense characters).
• Projector: 2-layer MLP mapping visual patch features into the decoder hidden dimension.
• Decoder: 12-layer Causal Transformer (~60M params)
  • Attention: Grouped-Query Attention (8 query heads, 2 key/value heads) with RMSNorm on queries & keys.
  • FFN: SwiGLU feed-forward layers (d_model = 512, d_ffn = 2048).
  • Positional Embeddings: Dynamic 2D Multimodal Rotary Position Embeddings (2D mRoPE) over visual tokens; 1D RoPE over text tokens.
  • Attention Masking: Block-causal attention (bidirectional among visual patch tokens, causal across output text tokens).

---

Usage

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor, PreTrainedTokenizerFast

# Load Model, Processor & Tokenizer
MODEL_ID = "JustANormalTinkerer/hayai-ocr-v2"
model = AutoModel.from_pretrained(MODEL_ID, trust_remote_code=True).cuda().eval()
tokenizer = PreTrainedTokenizerFast.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-naflex")

# Load and Preprocess Image
image = Image.open("example.png").convert("RGB")
# Use max_num_patches=256 for standard lines; 384 or 512 for dense/complex panels
inputs = processor(images=[image], max_num_patches=256, return_tensors="pt").to("cuda")

with torch.no_grad():
    texts = model.generate(
        pixel_values=inputs["pixel_values"],
        pixel_attention_mask=inputs["pixel_attention_mask"],
        spatial_shapes=inputs["spatial_shapes"],
        tokenizer=tokenizer,
        max_new_tokens=128,
        repetition_penalty=1.0,   # Keep at 1.0 (disabled) for OCR accuracy
    )

print(texts[0])

Note: trust_remote_code=True is required because the model utilizes custom block-causal attention and 2D mRoPE definitions (configuration_hayai.py, modeling_hayai.py). Also the previous version of the model card recommended setting num_beams to 4, ignore that. Greedy search will give the best results for Japanese, HOWEVER, tweak it as needed.

There also exists this python library that is recommended for python apps: hayai-ocr

---

Benchmarks

JMangaBench_Mixed

Fine-tuning Dataset Train Split (ZH + JA/KO Onomatopoeia + EN)

Private Pretraining Dataset Train Split: CJK

*Throughput gain in v2.1 is due to optimized decoding batching.

Hayai OCR matches or outperforms 0.9B parameter models while delivering 10× higher throughput and operating within a ~300MB VRAM footprint in FP16.

---

Training Methodology

Training was conducted in two coordinated phases (for v2.1, for v2 refer to the older model card) using Kaggle 2× NVIDIA T4 GPUs.

1. Multi-Task Joint Base Training (~19,000 Steps)

• OCR Stream: JustANormalTinkerer/hayai-dataset-merged (~1M images) streamed and sharded across GPUs.
• Linguistic Prior Stream: Interleaved token-packed streams from Japanese, Chinese, Korean, and English Wikipedia alongside the Aozora Bunko clean literature dataset.
• Loss Objective:

  L_total = L_ocr + 0.30 * L_text
  

2. Optimization

• Optimizer: Muon for decoder 2D weight matrices (orthogonalized momentum updates); AdamW for 1D vectors, embeddings, norms, and the SigLIP2 vision encoder.
• Learning Rates: Base LR 8e-5 (Muon / Decoder AdamW) and 1e-5 (Vision AdamW), decayed via cosine schedule with a 5% linear warmup.
• Augmentation: Random affine transforms, perspective shifts, subtle rotation (±6°), color jitter, blur, and sharpness adjustment.
• Precision: Mixed Precision (FP16) with dynamic gradient scaling.

---

Text Normalization

For consistent Character Error Rate (CER) reproduction and downstream evaluation, text should be normalized as follows:

import re
import unicodedata

def normalize_text(text: str) -> str:
    if not text:
        return ""
    text = unicodedata.normalize("NFKC", str(text))
    text = re.sub(r'[\r\n\t]+', ' ', text)
    # Remove space only between CJK characters
    cjk_char = r'[\u4e00-\u9fff\u3040-\u30ff\u3400-\u4dbf\uac00-\ud7af]'
    text = re.sub(f'({cjk_char})\\s+({cjk_char})', r'\1\2', text)
    return re.sub(r'\s+', ' ', text).strip()

---

Best Practices & Limitations

• Repetition Penalty: Keep repetition_penalty = 1.0. Penalties > 1.0 force the model to avoid valid repeated characters (e.g., 2校 ... 1校 or 学校).

Citations

# current Manga109
@inproceedings{baek2026mangav26,
  title     = {{Manga109-v2026: Revisiting Manga109 Annotations for Modern Manga Understanding}},
  author    = {Baek, Jeonghun and Miyai, Atsuyuki and Onohara, Shota and Ikuta, Hikaru and Aizawa, Kiyoharu},
  booktitle = {Culture × AI Workshop at ICML 2026},
  year      = {2026},
}
# introducing Manga109 annotations
@article{multimedia_aizawa_2020,
    author={Kiyoharu Aizawa and Azuma Fujimoto and Atsushi Otsubo and Toru Ogawa and Yusuke Matsui and Koki Tsubota and Hikaru Ikuta},
    title={Building a Manga Dataset ``Manga109'' with Annotations for Multimedia Applications},
    journal={IEEE MultiMedia},
    volume={27},
    number={2},
    pages={8--18},
    doi={10.1109/mmul.2020.2987895},
    year={2020}
}
# introducing Manga109 image collection
@article{mtap_matsui_2017,
    author={Yusuke Matsui and Kota Ito and Yuji Aramaki and Azuma Fujimoto and Toru Ogawa and Toshihiko Yamasaki and Kiyoharu Aizawa},
    title={Sketch-based Manga Retrieval using Manga109 Dataset},
    journal={Multimedia Tools and Applications},
    volume={76},
    number={20},
    pages={21811--21838},
    doi={10.1007/s11042-016-4020-z},
    year={2017}
}

@inproceedings{baek2022COO,
  title={COO: Comic Onomatopoeia Dataset for Recognizing Arbitrary or Truncated Texts},
  author={Baek, Jeonghun and Matsui, Yusuke and Aizawa, Kiyoharu},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  year={2022}
}