manga-ocr-CPT

English | 中文

This is an experimental repository. We kept the architecture of mangaOCR (kha-white/manga-ocr-base) unchanged, replaced only its vocabulary, and continued pre-training it (CPT) on new training data.

The main content of this repository is the experimental findings; see the technical report for the full account. The repository also provides two models and their inference code.

The two models share the same architecture, parameter count (111M) and training settings; only the input differs. Below they are called CPT-flex and CPT-224.

Main findings

1. Training data is the main factor. With the original architecture, parameter count and 224 × 224 input, and only the training data and vocabulary replaced, EM on JMangaBench_Read v1.3 rises from 81.29% to 91.00% (both with greedy decoding, $p = 4\times10^{-110}$).
2. A 2-layer autoregressive decoder is enough. With the same data, CPT-flex ties our 54M non-autoregressive model manga-ocr-nar-small (EM 91.42% vs 91.09%, $p = 0.24$).
3. Keeping the aspect ratio helps only on extreme shapes. The overall gain is just 0.42 EM points and not significant. But on boxes with aspect ratio ≥ 8 it is 85.40% vs 76.64%, and on horizontal boxes 87.02% vs 81.78%; both are significant.
4. The original default decoding hurts on manga. The original manga-ocr package uses no_repeat_ngram_size=3. On the 301 boxes whose label contains a repeated 3-character fragment, EM is 21.59% with the default settings and 68.77% with greedy decoding.
5. The original is slow because of its implementation. With the same network architecture, per-box time at batch 1 drops from 84.9 ms (original package) to 11.0 ms. Changing only the implementation, with identical outputs, also makes two other models 2.6× faster: Hayai v2.5 Nova 54.3 → 21.2 ms, PaddleOCR-VL-For-Manga 365 → 142 ms. The code is in accel/.

See the technical report for the data, tests and limitations behind each finding.

Accuracy

All 7,789 boxes of JMangaBench_Read v1.3, RIN v1 normalisation, batch 1, RTX 2080 Ti. See the benchmark's documentation for the model settings and scoring.

• The original is shown with two decoding settings: the package defaults (4-beam search with no_repeat_ngram_size=3) and greedy decoding. Both run in fp16 on the GPU.
• Paired per box: 521 boxes are read correctly only by CPT-flex and 254 only by Hayai v2.5 Nova p384 ($p = 5\times10^{-22}$).
• The two CPT models are not added to the JMangaBench_Read leaderboard.

Speed

RTX 2080 Ti, Read v1.3 speed protocol, preprocessing included. For page latency a page is one batch; a page has 8.7 boxes on average.

• CUDA path: preprocessing runs on the GPU, and the encoder and decoding steps are replayed from CUDA graphs. Capturing the graphs at start-up takes about 1–3 s.
• Plain path: plain PyTorch; runs on CPU, Apple MPS and other devices.
• CPT-224 was timed on 2026-10-15. In the same session, the CPT-flex CUDA path measured 10.9 ms / 42.9 ms / 264 crops/s, within 2% of the table above.
• Both paths use the same weights. On all 7,789 boxes of Read v1.3, the batch-1 outputs of both paths are identical to the evaluated outputs.
• About half of the original package's slowness comes from beam search with no-repeat, and the other half from the overhead of the transformers generation loop (technical report §4.2).
• GPU down-clocking: after a few idle seconds the GPU clocks down and the next request is slower. On the CPT-flex CUDA path, the first crop after 20 s idle takes 28.1 ms, vs 9.5 ms when running continuously.

Which model to use

• For general use, choose CPT-flex. It is clearly better on elongated and horizontal boxes and ties CPT-224 elsewhere.
• CPT-224 is mainly a control. Its input is exactly that of the original mangaOCR, which makes it the right model for comparing the effect of changing only the data.

Installation and usage

pip install git+https://github.com/muscgab/manga-ocr-CPT

from manga_ocr_cpt import MangaOcrCPT

ocr = MangaOcrCPT("flex")                # or "224"; downloads the weights from Hugging Face on first use (about 444 MB)
ocr("crop.png")                          # one crop -> str
ocr(["crop1.png", "crop2.png"])          # the crops of one page -> list[str], run as one batch

• Input: file paths, PIL images or uint8 RGB arrays. Inputs must be cropped text boxes; this package does no text detection.
• Device: with an NVIDIA GPU the CUDA path is used automatically; otherwise the plain path. Choose a device with device="cpu" or "mps", and force the plain path with backend="plain".
• fp16 decoder: decoder="fp16" runs the decoder's matrix products in fp16 (CUDA only), about 1 ms faster per crop. For CPT-flex it changes 2 outputs on Read v1.3 without changing the scores. The default "fp32" reproduces the evaluated outputs exactly.
• Downloading from ModelScope: download first, then pass the local directory:

modelscope download --model muscgab/manga-ocr-base-CPT-flex --local_dir manga-ocr-base-CPT-flex

ocr = MangaOcrCPT("manga-ocr-base-CPT-flex")    # local directory

Command line:

manga-ocr-cpt crop1.png crop2.png               # one "path<TAB>text" line per crop; --model 224 switches model

Differences from the original manga-ocr

• Vocabulary: replaced by 6,817 characters. Hangul and some simplified Chinese characters of the original vocabulary are removed. The weights therefore cannot be used in the original manga-ocr package.
• Length: at most 64 characters per box; longer boxes must be split first.
• Decoding: greedy only; no beam search and no repetition blocking.
• Inference code: re-implemented; does not depend on the original manga-ocr package or transformers.

Limitations

• The benchmark JMangaBench_Read was built by the author of this repository and consists mainly of works published online. We did not evaluate on scanned manga such as Manga109-s; the original was trained on Manga109-s, so a comparison there would not be fair either.
• Parts of Read v1.2 and v1.3 were used during development. The released weights are from the last training step and were not selected on this basis.
• Each model was trained once.
• Long boxes of more than 32 characters remain clearly behind: both CPT models score 71%–73% EM there, PaddleOCR-VL-For-Manga 82.84%.
• Handwriting and stylised lettering produce clearly more errors than printed text.
• Aimed at dialogue, narration and small annotations; onomatopoeia is not a target.
• The training data and the synthetic renderer are not released.

Files

This repository contains no weights. The weights are in the two model repositories on Hugging Face and ModelScope.

License

• Code: Apache License 2.0.
• Model weights: CC BY-NC-SA 4.0; see each model repository. The training data AnimeText-OCR is CC BY-NC-SA 4.0.
• The weights are initialised from kha-white/manga-ocr-base (Apache License 2.0). See NOTICE.md for provenance and acknowledgements.