MangaTranslator

Gradio-based web application for automating the translation of manga/comic page images using AI. Targets speech bubbles and text outside of speech bubbles. Supports 59 target languages and custom font pack usage. The current runtime profile is CUDA-only for local GPU and cloud deployments.

Table of Contents

• Features
• Requirements
• Install
• Post-Install Setup
• Run
• Documentation
• Updating
• License & Credits

Features

• Detection: Speech bubble detection & segmentation (YOLO + SAM 2.1/3)
• Cleaning: Inpaint speech bubbles and OSB text (Flux.1 Kontext or OpenCV)
• Translation: LLM-powered OCR & translation (59 languages)
• Rendering: Text rendering with alignment and custom font packs
• Upscaling: 2x-AnimeSharpV4 for enhanced output quality
• Processing: Single/batch processing with directory preservation and ZIP support
• Interfaces: Backend API (FastAPI), Web UI (Gradio), and CLI
• Automation: One-click translation; no intervention required

Requirements

• NVIDIA CUDA GPU with a working CUDA-enabled PyTorch install
• Python 3.10+
• Git (if cloning) or a ZIP extractor (if downloading the repository archive)
• Font pack with .ttf/.otf files placed under fonts/
• LLM provider/API key for translation

[!IMPORTANT] This branch is a CUDA-only build. CPU, Apple Metal (mps), and Intel XPU fallbacks are not supported in the current runtime.

Install

Clone From GitHub (Recommended)

1. Clone and enter the repo

git clone https://github.com/vtp772002/manga_toan.git
cd manga_toan

Download ZIP (No Git)

If you do not want to use Git, download a source archive from GitHub:

1. Download the latest archive from Releases or main branch ZIP
2. Extract it
3. Open a terminal in the extracted folder

Set Up Python Environment

Use this path on a CUDA-capable Windows/Linux machine or GPU cloud VM/container.

1. Create and activate a virtual environment (recommended)

python -m venv venv
# Windows PowerShell/CMD
.\venv\Scripts\activate
# Linux
source venv/bin/activate

2. Install PyTorch CUDA build (see: PyTorch Install)

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126

3. Install Nunchaku (required for Flux.1 Kontext CUDA backend)

• Nunchaku wheels are not on PyPI. Install directly from the v1.2.1 GitHub release URL, matching your OS and Python version. CUDA only.

# Example (Windows, Python 3.13, PyTorch 2.10.0, CUDA 13.0)
pip install https://github.com/nunchaku-ai/nunchaku/releases/download/v1.2.1/nunchaku-1.2.1+cu13.0torch2.10-cp313-cp313-win_amd64.whl

4. Install dependencies

pip install -r requirements.txt

Optional: Keep Local Assets Across Reinstalls

If you refresh the repo or replace it with a newer archive, you can usually keep these directories:

• fonts/
• models/
• output/

Example Font Packs

This repo currently includes packs such as:

• fonts/Comic_Neue
• fonts/font5/VNF-Comic Sans
• fonts/font5/Font TeddyBear

Post-Install Setup

Models

• The application will automatically download and use all required models

Fonts

• Put font packs as subfolders in fonts/ with .otf/.ttf files
• Prefer filenames that include italic/bold or both so variants are detected
• Example structure:

fonts/
├─ CC Wild Words/
│  ├─ CCWildWords-Regular.otf
│  ├─ CCWildWords-Italic.otf
│  ├─ CCWildWords-Bold.otf
│  └─ CCWildWords-BoldItalic.otf
└─ Komika/
   ├─ KOMIKA-HAND.ttf
   └─ KOMIKA-HANDBOLD.ttf

LLM setup

• Providers: Google, OpenAI, Anthropic, xAI, DeepSeek, Z.ai, Moonshot AI, OpenRouter, OpenAI-Compatible
• Web UI: configure provider/model/key in the Config tab (stored locally)
• CLI: pass keys/URLs as flags or via env vars
• Env vars: GOOGLE_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY, XAI_API_KEY, DEEPSEEK_API_KEY, ZAI_API_KEY, MOONSHOT_API_KEY, OPENROUTER_API_KEY, OPENAI_COMPATIBLE_API_KEY
• OpenAI-compatible default URL: http://localhost:1234/v1

[!NOTE] YanoljaNEXT-Rosetta models (e.g., yanolja/YanoljaNEXT-Rosetta-4B-2511-GGUF) are automatically detected when used via the OpenAI-Compatible provider and receive optimized prompting. These are text-only models and require two-step + local OCR model. The Special Instructions field is mapped to Rosetta's translation glossary (one entry per line, e.g., Yanolja NEXT -> 야놀자넥스트).

OSB text setup (optional)

If you want to use the OSB text pipeline, you only need a Hugging Face token for gated models:

• black-forest-labs/FLUX.1-Kontext-dev (only required if using Flux.1 Kontext with Nunchaku backend)
• facebook/sam3 (only required if using SAM 3 segmentation)

The OSB text detector itself is public and can be preloaded without a Hugging Face token.

Steps to create a token:

1. Sign in or create a Hugging Face account
2. Visit and accept the terms on:
   • FLUX.1 Kontext (dev) (optional, if using Kontext with Nunchaku)
   • SAM 3 (optional, if using SAM 3 instead of SAM 2.1)
3. Create a new access token in your Hugging Face settings with read access to gated repos ("Read access to contents of public gated repos")
4. Add the token to the app:
   • Web UI: set hf_token in Config
   • Env var (alternative): set HF_TOKEN
5. Save config to preserve the token across sessions

Run

Backend API (FastAPI)

• Default API port: 8000
• Health endpoints: /healthz, /readyz
• Main translate endpoint: POST /v1/translate
• Output download endpoint: GET /v1/outputs/{filename}

Local run:

uvicorn api:app --host 0.0.0.0 --port 8000 --workers 1

Example request:

curl -X POST http://localhost:8000/v1/translate \
  -F "image=@/path/to/page.png" \
  -F "output_language=Vietnamese" \
  -F "provider=OpenAI" \
  -F "font_pack=font5/VNF-Comic Sans"

The API auto-detects source language with PaddleOCR-VL. By default it saves the translated image under output/api/ and returns a JSON payload with a download URL.

Web UI (Gradio)

Run:

python app.py --open-browser

Or bind to all interfaces manually:

python app.py --host 0.0.0.0 --port 7676

Options: --models (default ./models), --fonts (default ./fonts), --port (default 7676). First launch can take ~1–2 minutes.

Once launched, configure your LLM provider in the Config tab, choose the target language, then upload images and click Translate. Source language is auto-detected locally with PaddleOCR-VL.

Sample images for local testing are available in one_piece/.

Docker / Kubernetes

• NVIDIA GPU required for container/cloud usage
• Docker Compose: docker compose up --build
• K8s: apply k8s/pvc.yaml, k8s/secret.example.yaml, k8s/deployment.yaml, then k8s/service.yaml
• Preload assets: python app.py --preload-model-assets
• Warm up models: python app.py --warmup-models
• API + health endpoints: :8000/v1/translate, :8000/healthz, :8000/readyz
• Required secrets in K8s: API keys for your chosen provider; HF_TOKEN is optional unless you want gated assets such as Flux.1 Kontext or SAM 3
• Mount persistent storage for models/, fonts/, and output/
• Public assets such as the OSB text detector can preload without HF_TOKEN; gated Flux/SAM 3 assets preload when HF_TOKEN is provided
• For GPU cloud nodes, keep Flux/Nunchaku enabled for best outside-text quality; use ample VRAM and longer startup probes
• If you want the full pipeline but faster startup, run the preload step once against the PVC, then start the app normally
• For production cloud use, prefer mounting a shared cache volume for /cache/huggingface and /cache/torch
• Flux = the image inpainting model used to erase/replace text regions after OCR/detection; it is the quality-heavy part of outside-text removal
• Nunchaku = an optimized CUDA backend for running Flux faster on GPU while keeping quality

CLI

Examples:

# Single image, Japanese → English, Google provider
python main.py --input <image_path> \
  --font-dir "fonts/Comic_Neue" --provider Google --google-api-key <AI...>

# Quick local smoke test with a bundled sample image
python main.py --input "one_piece/p1.jpg" \
  --font-dir "fonts/Comic_Neue" --provider Google --google-api-key <AI...>

# Batch folder, auto-detected source text, custom target language, OpenAI-Compatible provider (LM Studio)
python main.py --input <folder_path> --batch \
  --font-dir "fonts/Comic_Neue" \
  --output-language <tgt_lang> \
  --provider OpenAI-Compatible --openai-compatible-url http://localhost:1234/v1 \
  --output ./output

# Single Image, Japanese → English (Google), OSB text pipeline, custom OSB text font
python main.py --input <image_path> \
  --font-dir "fonts/Comic_Neue" --provider Google --google-api-key <AI...> \
  --osb-enable --osb-font-dir "fonts/font5/VNF-Comic Sans"

# Cleaning-only mode (no translation/text rendering)
python main.py --input <image_path> --cleaning-only

# Upscaling-only mode (no detection/translation, only upscale)
python main.py --input <image_path> --upscaling-only --image-upscale-mode final --image-upscale-factor 2.0

# Test mode (no translation; render placeholder text)
python main.py --input <image_path> --test-mode

# Full options
python main.py --help

Documentation

• Backend API Deployment
• Recommended Fonts
• Troubleshooting

Updating

Git Clone

From the repo root:

git pull
pip install -r requirements.txt  # Or activate venv first if present

ZIP Download / Extracted Copy

If you installed from a ZIP archive instead of Git:

1. Download the latest archive from the repo or releases page
2. Extract it into a new folder
3. Copy over fonts/, models/, and output/ if you want to keep local assets
4. Reinstall dependencies in your virtual environment if needed:

pip install -r requirements.txt

License & credits

• License: Apache-2.0 (see LICENSE)
• Repository: vtp772002/manga_toan
• Original author: grinnch

ML Models & Libraries

• YOLOv8m Speech Bubble Detector: kitsumed
• Manga109 Speech Bubble Detector: huyvux3005
• Comic Speech Bubble Detector YOLOv8m: ogkalu
• SAM 2.1: Segment Anything in Images and Videos: Meta AI
• SAM 3: Meta AI
• FLUX.1 Kontext: Black Forest Labs
• Nunchaku: Nunchaku AI
• 2x-AnimeSharpV4: Kim2091
• PaddleOCR-VL-1.5: PaddlePaddle
• Manga109 YOLO: deepghs
• OSB Text YOLO: armvectores