JMangaBench_Read v1.3 — a Japanese manga OCR benchmark

English | 中文 · Appendix · Interactive leaderboard · Dataset (images)

JMangaBench_Read evaluates the accuracy and speed of OCR models on text boxes cropped from Japanese manga pages. The main set contains the text a reader reads and a translation pipeline translates: speech, narration and asides. Sound effects are excluded.

• 7,789 text boxes from 1,847 pages by 1,847 artists (one page per artist).
• All pages first posted between 2025-10-01 and 2026-10-01.
• Labels from three LLM readers and human review (1,266 boxes human-confirmed).
• Speed measured per image, per page and at maximum throughput on one RTX 2080 Ti.

Found an error in a label, a score or the docs? Please open an issue. Corrections are welcome.

v1.3 changes. 4,660 new boxes from 1,116 new pages; 13 v1.2 boxes removed (rules in Box selection and labeling); scoring normalization renamed RIN v1 with two added symbol rules; all models re-measured on an RTX 2080 Ti in fp16. The authors used Read v1.2 and 1,021 of the new boxes during model development (see Development use); scores on the 4,660 new boxes are reported separately in the appendix. Full list: appendix A6.

Leaderboard

Ranked by composite score (defined in Scoring). 7,789 boxes, RTX 2080 Ti, fp16.

More models

#	Model	Arch	Params	Composite ↑	CER % ↓	EM % ↑	EM NFKC % ↑	Per-image ms ↓	Per-page ms (p50) ↓	Max img/s ↑
–	e4 L3 † §	NAR	54M	97.29	1.73	90.97	80.07	42.8	71.9	227 (b32)
–	e4 L2 † §	NAR	54M	97.28	1.75	90.92	80.06	35.5	64.3	267 (b48)
–	e4 L1 † §	NAR	54M	97.14	1.96	90.40	79.66	28.3	58.1	282 (b24)

• † Made by the authors of this benchmark (see Conflict of interest).
• ‡ Experimental model, not released. L1/L2/L3 = number of decoder loops, same weights.
• § e4: the authors' earlier mainline checkpoint (same architecture family as MangaOCR-Nar-Small, 3-loop training), to be released as a reference model. L1/L2/L3 = decoder loops at inference.
• * LLM readers. The agreed reading of all three defines 7,005 of the labels (all boxes outside hard), so these rows are listed for reference and are not ranked (appendix A4).
• Accuracy columns use batch-size-1 outputs. Under per-page batching, composite scores change by at most 0.02.
• EM NFKC: exact match after Unicode NFKC only, for reference. The gap to EM comes mostly from spelling variants that RIN merges (see Scoring).
• bN: batch size at maximum throughput.
• Params from the published weight files; mangaOCR estimated from its fp32 file size.
• Breakdowns by subset, length and role, the speed details and the NFKC results: appendix.

Accuracy vs. speed

Click a chart to open the interactive version (subsets: all / new / printed / non-printed / gold / hard; e4 under "More models").

Reading one page at a time with no wait between pages, the median time per page is 37 ms for MangaOCR-Nar-Small, 245 ms for Hayai v2.5 Nova p384 and 345 ms for Hayai v2.1 (appendix A2).

Data

Source

Black-and-white Japanese comic pages posted publicly online by independent artists, collected for research use. Source type and license follow AnimeText.

Overlap with AnimeText

All 1,847 pages were checked by image ID (dataset revisions of 2026-10-04).

• AnimeText: 0 pages. Its 735,060 image IDs (train, valid, test) range up to 8,475,567, posted on 2024-11-23. The lowest page ID in this benchmark is 10,053,471.
• animetext-ocr: 0 pages. Its 496,601 source image IDs range up to 8,475,560.

Box selection and labeling

Boxes come from one candidate pool. Pages were processed in a fixed order; v1.2 covers the pages up to a cut-off and v1.3 adds the pages after it.

1. Detection. AnimeText YOLO12x detector, confidence ≥ 0.425. For nested boxes the smaller box is kept. General pages keep all boxes; sensitive pages keep up to 8 random boxes. Result: about 21,500 candidate boxes on 2,524 pages.

2. LLM readers. Each reader worked independently and did not see the other readers' outputs. Prompts: appendix A3.

3. Exclusions (Sol screening and Astra admission):

• Scene text: shop names, station names, street signs, signboards, posters.
• Non-story text: ads, watermarks, signatures and credits, large titles.
• Large decorative sound effects (e.g. a giant ドドド).
• Damaged text: occluded, smeared, faded, blacked out, or heavily cut off.
• Non-Japanese text.

Hard-to-read, rare or deformed characters, long text, artistic lettering and low reader confidence are not exclusion criteria.

4. Main-set rules. Applied to all v1.3 boxes:

5. Labels of the new boxes.

v1.2 labels are unchanged. The review interface showed the three LLM readings only, never an OCR model's output.

Label quality. 0 errors in 300 audited new agreed boxes gives a one-sided 95% upper bound of 0.99% on their error rate (Clopper–Pearson); v1.2's audit gave 0 errors in 200 (bound 1.49%). The 6,523 labels without per-box human confirmation are verified by sampling only.

Subsets

Development use

The authors used Read v1.2 (3,142 boxes) as a development set for MangaOCR-Nar-Small and its predecessors: model selection, comparisons and error analysis. A second development set, "Sol handwriting" (2,248 boxes from the same candidate pool that GPT-6.1 Sol classified as handwritten, with LLM-reading labels), was used from 2026-10-09 for training-time validation, data and architecture comparisons and error analysis. It shares 1,021 boxes with the v1.3 new boxes: 1,004 of the 1,110 new non-printed boxes and 17 printed boxes. Aggregate statistics measured on the candidate pool, minus the v1.2 pages, informed the design of synthetic training data; those pool pages include the pages of the 4,660 new boxes. The statistics are released in docs/pool_stats/. No image from the candidate pool or the benchmark was used for training. Scores on the new boxes are in appendix A1; their non-printed part largely overlaps the Sol handwriting set.

Scoring

Predictions and labels are normalized with the same rules, RIN v1 (reading-invariant normalization). RIN merges spelling variants that do not change what a reader reads.

Characters, small kana, brackets 「」『』, 〇 / ○ and the presence of a voicing mark are kept. Labels use ♡; ♥ is not mapped. RIN v1 equals Read v1.2's r7 plus the two v1.3 symbol rules; no v1.3 label changes between the two.

Why RIN. The labels follow the readers' spelling, which varies for the same glyph: ～ in 326 boxes and 〜 in 112, … in 1,832 boxes and ・・ runs in 260, ！ in 868 and ! in 560. A model that writes each symbol one way cannot match all labels without normalization. EM after NFKC alone is listed for reference.

Metrics

• CER: total edit distance / total label characters.
• EM: fraction of boxes with an exact match.
• Composite (ranking metric), 1 − normalized edit distance as in ICDAR 2019 ReCTS, averaged equally over printed and non-printed boxes:

$$s_i = 1 - \frac{\mathrm{ED}(p_i, g_i)}{\max(|p_i|, |g_i|)}$$

$$\text{Composite} = 100 \times \tfrac{1}{2}\left(\overline{s}{\text{printed}} + \overline{s}{\text{non-printed}}\right)$$

Speed

One RTX 2080 Ti (11 GB) per model, all models in fp16 (the card has no fast bf16). Each run is pinned to 8–10 CPU cores of a 2× Xeon Gold 6148 host, with library threads capped at 8; up to four models run at the same time, one per GPU. Images are pre-decoded in memory. Timings include the model's preprocessing and decoding and are GPU-synchronized.

Read v1.2 was timed on an RTX 3080 Ti with each model's default precision. Speeds of the two versions are not comparable.

After the GPU has been idle for several seconds, the driver lowers its clock and the next request is slower (for Hayai v2.5 Nova p384 51.5 → 81.7 ms on the first crop after 20 s idle). This applies to every model; see appendix A2.

Models

fp16 against fp32 on the same inputs: identical outputs for Preview (260/260), PaddleOCR-VL-For-Manga (65/65) and MangaOCR-Nar-Small (7,788/7,789); LoopNAR-α L3 EM 83.61 (fp16) vs 83.62 (fp32). The other models were not compared.

Conflict of interest

MangaOCR-Nar-Small, e4, LoopNAR-α and Preview are made by the authors of this benchmark.

• Labels come from three third-party LLMs and human review. The review interface showed LLM readings only, no OCR model outputs.
• Read v1.2 was used during the development of the authors' models (Development use). 1,021 of the new boxes were in a development set before v1.3 was assembled. Box selection and labels do not use any OCR model output.
• All model predictions, the scoring script and the speed scripts are released.

Limitations

• 6,523 labels are agreed LLM readings verified by sampling only.
• During human review the majority reading was pre-filled.
• The hard subset is defined by LLM disagreement and flags.
• Sound effects were excluded by LLM role labels; boxes labeled dialogue were not checked one by one.
• Boxes were screened by LLMs; boxes judged unreadable are not included.
• One source type (recent fan and original works), one GPU type.
• At the max-throughput batch size, outputs match batch-size-1 outputs for 99.6–100% of boxes.

Usage

pip install regex
python eval/score.py --pred my_preds.jsonl --name my_model --out results/my_model.json

my_preds.jsonl: one line per box, {"key": "10053471_0", "pred": "作戦を続行する"}. Missing keys count as empty predictions.

gt.jsonl fields: key, file, text, text_rin, n_chars, gold, gold_hard, gold_random_audit, printed, non_printed, font, role, orientation, crop_wh, page_id, box, rating, origin (v1.2 / new), review.

For evaluation only. Do not use for training.

Corrections

If you find any error (a wrong label, a box that should not be in the set, a scoring or speed problem), please open an issue. Fixes go into the next version and are listed in appendix A6.

License

CC BY-NC-SA 4.0 for data and code.

Citation

@misc{jmangabench_read,
  title  = {JMangaBench\_Read: A Japanese Manga OCR Benchmark},
  year   = {2026},
  url    = {https://github.com/muscgab/JMangaBench_Read}
}