JMangaBench_Mixed

A reproducible mixed-domain benchmark for Japanese manga OCR.

English · 中文

---

English

JMangaBench_Mixed is a relatively easy, reproducible mixed-domain benchmark for Japanese manga OCR that simulates real-world conditions. It evaluates both real manga text and controlled synthetic text. It contains 3,286 samples and reports not only one overall score, but also results by data source, text-length group, crop type, scene type, and writing orientation.

The repository contains the code, recipes, released rendering layers, hashes, and evaluator required to reproduce the benchmark. Manga109-s pixels are not redistributed: users obtain Manga109-s separately and build the image set locally.

Dataset composition

Synthetic crop types

Both crop types are scored with OCR metrics. No detector metric is computed.

The synthetic groups are:

Every group is balanced between 250 text_detector_crop and 250 balloon_detector samples.

The synthetic set uses 28 distinct font files and 82 distinct container assets:

• 44 GPL-licensed SVG balloon assets;
• 36 project hand-drawn assets;
• 2 project procedural assets.

The repository retains 48 GPL SVG sources; 44 of them are used by the final samples. A pool of 1,000 Manga109-s background regions is used twice across the 2,000 synthetic samples.

Manga109-s source version

The version label is recorded as source metadata. A mismatch in a source asset or decoded source crop stops construction. A final generated-image SHA-256 difference does not stop construction and is recorded in the warning report.

Linux reference environment and Docker

The generated-image reference hashes were produced on the following Linux/amd64 system:

The repository also provides a pinned Linux/amd64 Docker build. Its base image is ghcr.io/astral-sh/uv@sha256:531f855bda2c73cd6ef67d56b733b357cea384185b3022bd09f05e002cd144ca. The resulting runtime uses Python 3.13.11, Pillow 12.3.0, JPEG codec ABI 6.2, libjpeg-turbo 3.1.4.1, and zlib 1.2.13.

Native builds on macOS or Windows can produce a different final PNG byte stream or slightly different generated pixels because image codecs and numerical implementations differ across platforms. Such differences may be visually and OCR-negligible, but SHA-256 alone does not measure their practical effect. The builder therefore reports every affected generated image without treating it as a source-data failure.

Reference OCR results

The following results were exported on 2026-07-28. All inference was performed on the same Linux server with one NVIDIA GeForce RTX 4090 24 GB GPU. Each model processed one image at a time in canonical benchmark order. Per-image latency does not include model loading and is hardware-dependent.

Two locally constructed image sets were evaluated:

• Linux reference: the Linux/amd64 reference construction described above;
• macOS private: the earlier private construction produced on macOS.

Both sets contain the same 3,286 sample IDs and reference strings. The accuracy metrics were identical for each model, so they are reported once:

Sequential per-image latency on the RTX 4090:

Cross-construction prediction comparison:

The single raw PaddleOCR difference for jmm_19_40_000180 consisted only of inserted line breaks; whitespace normalization made the predictions identical. Across 13,144 model-sample comparisons, there were no normalized prediction differences and no metric differences. For these two constructions and four tested models, the cross-platform image differences had no measurable effect on the reported OCR accuracy.

Quick start

Docker is the shortest reference-platform path. Docker Desktop users on macOS or Windows should keep --platform linux/amd64 in both commands:

docker build --platform linux/amd64 -t jmangabench-mixed:1.0.0 .

mkdir -p /absolute/path/to/output-parent

docker run --rm --platform linux/amd64 \
  -v "/absolute/path/to/Manga109s_released_2026_05_21:/data/Manga109s:ro" \
  -v "/absolute/path/to/output-parent:/output" \
  jmangabench-mixed:1.0.0 \
  build \
  --manga109s-root /data/Manga109s \
  --output /output/JMangaBench_Mixed \
  --workers 10

For a native installation, requirements are:

• Python 3.13;
• uv;
• a separately obtained Manga109-s image package matching the recorded hashes (reference release: Manga109-v2026).

From the repository root:

uv sync --frozen

uv run python tools/verify_release.py --release-root .

uv run python -m jmangabench build \
  --manga109s-root /path/to/Manga109s_released \
  --output ./JMangaBench_Mixed \
  --workers 10

The one-command builder verifies the repository, reconstructs the real and synthetic components, performs strict source checks, and assembles the final benchmark in canonical order. The output directory must not already exist.

Expected output:

JMangaBench_Mixed/
  annotations.jsonl
  manifest.json
  sha256_warnings.jsonl
  images/
    real_crops/
    synthetic/
  masks/
    body_text/

sha256_warnings.jsonl is empty when every generated image matches the Linux reference. Otherwise, each line identifies one image, its reference SHA-256, and its actual SHA-256. Source asset and source-crop failures remain fatal.

Connect a model

JMangaBench_Mixed does not import or manage model frameworks. Copy examples/predict_jsonl_template.py into the model's own environment and implement load_model() and predict():

cp examples/predict_jsonl_template.py /path/to/model/run_jmangabench.py

/path/to/model/python /path/to/model/run_jmangabench.py \
  --benchmark ./JMangaBench_Mixed \
  --output ./predictions.jsonl

The model must produce exactly one JSON object for every benchmark ID:

{"id":"jmm_4_8_000001","prediction":"やあ坊主","latency_seconds":0.123,"error":null}

latency_seconds and error are optional. prediction must always be a string.

Evaluate predictions

uv run python -m jmangabench validate-predictions \
  --benchmark ./JMangaBench_Mixed \
  --predictions ./predictions.jsonl

uv run python -m jmangabench evaluate \
  --benchmark ./JMangaBench_Mixed \
  --predictions ./predictions.jsonl \
  --output ./evaluation/my_model

Evaluation creates:

evaluation/my_model/
  report.json
  per_sample.jsonl

The evaluator rejects missing, extra, or duplicate IDs; invalid predictions or latencies; modified annotations; and image or mask hash mismatches.

Metrics and slices

The frozen protocols are:

jmangabench_mixed_ocr_metrics/v1
jmangabench_mixed_text_normalization/v1

Normalization applies Unicode NFKC and removes whitespace. The text-only policy removes Unicode punctuation while preserving U+30FC (ー). CER is micro-averaged; exact match is sample-averaged.

Each report contains overall results and slices by:

• source_kind: real or synthetic;
• subset: real crops or one of the four synthetic length groups;
• crop_type: real crop, text crop, or balloon crop;
• scene_type: real crop, balloon, or direct text;
• orientation: horizontal, vertical, or unspecified.

When latency is provided, the report also includes mean, p50, and p95 latency.

Reproducibility

• release_manifest.json records every distributable file's size and SHA-256.
• Source-page, source-crop, foreground, and label-layer hashes are strict construction checks.
• Generated-image reference differences are preserved in sha256_warnings.jsonl and summarized in manifest.json.
• The built image and mask hashes are frozen in annotations.jsonl and checked by the evaluator before scoring.
• Final ordering is independent of input JSONL line order.
• The assembled manifest.json freezes annotations and slice counts.
• Public repository files contain no Manga109-s pixels or local absolute paths.
• The locally built image benchmark contains Manga109-s-derived pixels and is marked redistribution_allowed: false.

Do not upload the locally built benchmark or any private verification image directory.

License and data terms

Project-authored code, documentation, recipes, metadata, and assets are licensed under GPL-3.0-only, unless a file or directory says otherwise.

Third-party terms remain in effect:

• GPLv3 speech-bubble SVG sources are preserved under notices/manga_editor_desu_gpl3/;
• font notices are preserved under notices/fonts/ under SIL Open Font License 1.1;
• Manga109-s must be obtained separately and remains governed by its own terms.

See THIRD_PARTY_NOTICES.md for details.

---

中文

JMangaBench_Mixed 是一个面向日文漫画 OCR 的模拟真实情况的较简单的可复现混合域评测集。它同时测试真实漫画文字与可控合成文字，共包含 3,286 个样本。

本仓库提供复现 benchmark 所需的代码、配方、发布渲染层、哈希和 evaluator，不重新分发 Manga109-s 像素。用户需要单独取得 Manga109-s，并在本地构建图片测试集。

数据组成

合成 crop 类型

两种 crop 均使用 OCR 指标评分，不计算检测器指标。

四个合成分组为：

合成样本实际使用 28 个不同的字体文件和 82 个不同的容器资产：

• 44 个 GPL SVG 气泡资产；
• 36 个项目自绘资产；
• 2 个项目程序生成资产。

仓库保留了 48 个 GPL SVG 源文件，其中 44 个实际进入最终样本。合成数据使用 1,000 个 Manga109-s 背景区域，每个背景在 2,000 张样本中使用两次。

Manga109-s 来源版本

版本名称作为来源元数据记录。源资产或解码后源 crop 不一致时构建会立即失败；仅最终 生成图片 SHA-256 不一致时不会中止，而是写入 Warning 报告。

Linux 参考环境与 Docker

最终生成图片的参考哈希来自以下 Linux/amd64 环境：

仓库同时提供固定的 Linux/amd64 Docker 构建，基础镜像为 ghcr.io/astral-sh/uv@sha256:531f855bda2c73cd6ef67d56b733b357cea384185b3022bd09f05e002cd144ca。 容器运行时使用 Python 3.13.11、Pillow 12.3.0、JPEG codec ABI 6.2、 libjpeg-turbo 3.1.4.1 和 zlib 1.2.13。

在 macOS 或 Windows 上原生构建时，图片编解码器和数值实现的跨平台差异可能改变最终 PNG 字节流，或造成轻微的生成像素差异。这类差异对肉眼观感和 OCR prediction 的影响 可能很小，但 SHA-256 本身不能衡量实际影响。因此构建器会列出所有受影响的生成图片， 但不会把它们判定为源数据错误。

参考 OCR 结果

以下结果导出于 2026-07-28。全部推理均在同一台 Linux 服务器上使用一张 NVIDIA GeForce RTX 4090 24 GB GPU 完成。每个模型按照 benchmark 固定顺序逐张处理图片。 单张延迟不包含模型加载时间，并且与硬件相关。

本次评测使用两套本地构建图片：

• Linux reference：使用上文 Linux/amd64 参考环境构建；
• macOS private：此前在 macOS 上构建的 private 版本。

两套图片包含相同的 3,286 个样本 ID 和 reference。每个模型在两套图片上的准确率指标 完全相同，因此只展示一次：

RTX 4090 上的逐张推理延迟：

跨构建 prediction 对比：

PaddleOCR 唯一一条原始差异出现在 jmm_19_40_000180，内容仅为插入换行；删除空白后 两条 prediction 完全相同。四个模型共 13,144 次样本对比中，规范化 prediction 差异 为 0，评测指标差异也为 0。对于本次对比的两套构建和四个模型，跨平台图片差异没有对 报告中的 OCR 准确率产生可测量影响。

快速开始

Docker 是最短的参考平台构建方式。在 macOS 或 Windows 的 Docker Desktop 上也应在 两条命令中保留 --platform linux/amd64：

docker build --platform linux/amd64 -t jmangabench-mixed:1.0.0 .

mkdir -p /absolute/path/to/output-parent

docker run --rm --platform linux/amd64 \
  -v "/absolute/path/to/Manga109s_released_2026_05_21:/data/Manga109s:ro" \
  -v "/absolute/path/to/output-parent:/output" \
  jmangabench-mixed:1.0.0 \
  build \
  --manga109s-root /data/Manga109s \
  --output /output/JMangaBench_Mixed \
  --workers 10

原生安装的环境要求：

• Python 3.13；
• uv；
• 用户单独取得且符合配方哈希的 Manga109-s 图片包（参考版本：Manga109-v2026）。

在仓库根目录执行：

uv sync --frozen

uv run python tools/verify_release.py --release-root .

uv run python -m jmangabench build \
  --manga109s-root /path/to/Manga109s_released \
  --output ./JMangaBench_Mixed \
  --workers 10

一键构建程序会验证仓库文件，恢复真实与合成数据，严格检查源数据，并按照固定顺序组装 最终 benchmark。输出目录不能预先存在。

构建结果：

JMangaBench_Mixed/
  annotations.jsonl
  manifest.json
  sha256_warnings.jsonl
  images/
    real_crops/
    synthetic/
  masks/
    body_text/

所有生成图片与 Linux 参考值一致时，sha256_warnings.jsonl 为空。否则每一行会记录 一个图片路径、参考 SHA-256 和实际 SHA-256。源资产和源 crop 校验失败仍会中止构建。

接入模型

JMangaBench_Mixed 不会导入或管理模型框架。将 examples/predict_jsonl_template.py 复制到模型自己的环境，并实现 load_model() 和 predict()：

cp examples/predict_jsonl_template.py /path/to/model/run_jmangabench.py

/path/to/model/python /path/to/model/run_jmangabench.py \
  --benchmark ./JMangaBench_Mixed \
  --output ./predictions.jsonl

模型必须为每个 benchmark ID 输出且只输出一个 JSON 对象：

{"id":"jmm_4_8_000001","prediction":"やあ坊主","latency_seconds":0.123,"error":null}

latency_seconds 和 error 可以省略，prediction 必须始终是字符串。

评测 prediction

uv run python -m jmangabench validate-predictions \
  --benchmark ./JMangaBench_Mixed \
  --predictions ./predictions.jsonl

uv run python -m jmangabench evaluate \
  --benchmark ./JMangaBench_Mixed \
  --predictions ./predictions.jsonl \
  --output ./evaluation/my_model

成功后生成：

evaluation/my_model/
  report.json
  per_sample.jsonl

如果存在缺失、额外或重复 ID，非法 prediction 或延迟，annotation 被修改，或者图片、 mask 哈希不一致，evaluator 会拒绝评分。

指标与切片

冻结的协议为：

jmangabench_mixed_ocr_metrics/v1
jmangabench_mixed_text_normalization/v1

文本规范化执行 Unicode NFKC 并删除空白。忽略标点规则删除 Unicode 标点，但保留 U+30FC（ー）。CER 使用 micro-average，Exact Match 使用样本平均。

每份报告包含总体结果，以及以下切片：

• source_kind：真实或合成；
• subset：真实 crop 或四个合成长度分组；
• crop_type：真实 crop、文字 crop 或气泡 crop；
• scene_type：真实 crop、气泡或直接文字；
• orientation：横排、竖排或 unspecified。

如果 prediction 提供延迟，报告还会计算平均、p50 和 p95 延迟。

可复现性

• release_manifest.json 记录全部发布文件的大小和 SHA-256；
• 源页面、源 crop、foreground 和 label layer 哈希均为严格构建校验；
• 生成图片参考哈希差异保存在 sha256_warnings.jsonl，并汇总到 manifest.json；
• 构建后的图片与 mask 哈希写入 annotations.jsonl，evaluator 会在评分前验证；
• 最终顺序不受输入 JSONL 行顺序影响；
• 组装后的 manifest.json 冻结 annotation 和切片数量；
• GitHub 仓库不包含 Manga109-s 像素或本机绝对路径；
• 本地构建的图片 benchmark 包含 Manga109-s 派生像素，并标记为 redistribution_allowed: false。

不得上传本地构建的 benchmark 或任何 Private 验证图片目录。

许可证与数据条款

除非文件或目录另有说明，项目原创代码、文档、配方、元数据和资产采用 GPL-3.0-only。

第三方条款继续有效：

• GPLv3 气泡 SVG 源文件保存在 notices/manga_editor_desu_gpl3/；
• 字体声明保存在 notices/fonts/，采用 SIL Open Font License 1.1；
• Manga109-s 由用户单独取得，并继续受其自身条款约束。

详情见 THIRD_PARTY_NOTICES.md。