All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts

Xingsong Ye, Yongkun Du, Jiaxin Zhang, Zhixian Li, Chong Sun, Chen Li, Jing Lyu, Lianwen Jin, Zhineng Chen

---

Highlights

• 🌍 All-in-one multilingual STR across 10 writing scripts and 229 languages.
• 🧩 Script-aware Mixture-of-Experts with top-2 script-aligned routing.
• 🔗 Shared expert for cross-script knowledge transfer.
• 🏗️ Single visual encoder shared across all scripts.
• 🧪 TextMuSS-10M: 10 scripts, 229 languages, 1M synthetic samples per script.
• 📊 TextMuSS-Bench: 10 scripts and 10,899 real images.
• 🚀 End-to-end OCR integration with PP-OCRv5.
• 🔬 Complete training, inference, and evaluation code is provided in this repository.

---

Resources

---

Method

ScriptMoE consists of a shared visual encoder followed by a sparse script-aware MoE decoder.

Given an input text image, an image-level router predicts the relevant scripts and dispatches the input to the top-2 script-aligned experts. A shared expert is additionally used to absorb knowledge shared across different writing systems.

Conceptually:

                 Input Image
                     │
                     ▼
          ┌─────────────────────┐
          │ Shared Visual       │
          │ Encoder (SVTRv2)    │
          └──────────┬──────────┘
                     │
                     ▼
             Image-level Router
                     │
          ┌──────────┼──────────┐
          │          │          │
          ▼          ▼          ▼
       Expert 1   Expert 2   Shared Expert
       Script-A   Script-B   Cross-script
          │          │          │
          └──────────┼──────────┘
                     │
                     ▼
             Sparse MoE Decoder
                     │
                     ▼
              Recognized Text

---

Directory Structure

ScriptMoE/
├── OpenOCR/
│   ├── configs/
│   │   └── rec/
│   │       └── scriptmoe/
│   │           └── svtrv2_scriptmoe_mlt.yml
│   └── openrec/
│       ├── modeling/
│       ├── losses/
│       └── postprocess/
│
├── E2EOCR/
│   ├── infer.py
│   ├── modeling.py
│   ├── postprocess.py
│   ├── visualize.py
│   ├── model.safetensors
│   └── assets/
│       └── dict.txt
│
├── eval_textmussbench/
├── Eval-TextMuSS-Bench/
├── CC-OCR-MLT/
├── run_cc_ocr_mlt.py
└── README.md

Main components

---

Installation

End-to-end inference

cd E2EOCR
pip install -r requirements.txt

Core dependencies include:

torch >= 2.0
torchvision
safetensors
opencv-python
Pillow

Optional detectors:

# OpenOCR detector
pip install openocr-python

# PP-OCRv5 / PP-OCRv6
pip install "paddleocr>=3.7.0" paddlepaddle

Training

cd OpenOCR
pip install -r requirements.txt

---

End-to-End Inference

The E2EOCR/ directory provides a self-contained inference pipeline consisting of text detection followed by ScriptMoE recognition.

The recognizer weights and multilingual character dictionary are already included.

Single image

cd E2EOCR

python infer.py --image path/to/image.jpg

PP-OCRv5 detector

python infer.py \
    --image path/to/image.jpg \
    --det ppv5

Recognition only

For an already cropped text line:

python infer.py \
    --image path/to/crop.jpg \
    --det rec_only

Batch inference

python infer.py \
    --image path/to/image_directory \
    --output out.json

Main arguments

---

Training

Training is implemented on top of the OpenOCR framework.

The main ScriptMoE configuration is:

OpenOCR/configs/rec/scriptmoe/svtrv2_scriptmoe_mlt.yml

Important implementation files include:

OpenOCR/openrec/modeling/decoders/scriptmoe_decoder.py
OpenOCR/openrec/losses/scriptmoe_loss.py
OpenOCR/openrec/postprocess/scriptmoe_postprocess.py
OpenOCR/openrec/modeling/encoders/svtrv2_lnconv_two33.py

Before training, update the training and validation dataset paths in:

Train.dataset.data_dir_list
Eval.dataset.data_dir_list

Example

cd OpenOCR

python -m torch.distributed.launch \
    --nproc_per_node=8 \
    tools/train_rec.py \
    --c configs/rec/scriptmoe/svtrv2_scriptmoe_mlt.yml

Training recipe

The main training recipe follows the paper:

For standard STR evaluation:

max_text_length = 25
max_ratio = 8

For end-to-end recognition:

max_text_length = 100
max_ratio = 20

---

Evaluation

TextMuSS-Bench

The repository provides evaluation code for TextMuSS-Bench.

python eval_textmussbench/xxx.py

Please refer to the scripts under:

eval_textmussbench/
Eval-TextMuSS-Bench/

for the corresponding evaluation settings.

CC-OCR-MLT

To reproduce the end-to-end evaluation reported in the paper:

python run_cc_ocr_mlt.py \
    --det ppv5 \
    --det_thresh 0.0 \
    --rec_thresh 0.7

For a smaller subset:

python run_cc_ocr_mlt.py \
    --det ppv5 \
    --languages Korean Japanese \
    --max_per_lang 20

Inference without evaluation:

python run_cc_ocr_mlt.py \
    --det ppv5 \
    --no_eval

---

Results

TextMuSS-Bench

ScriptMoE achieves 82.06% average word accuracy on TextMuSS-Bench.

CC-OCR End-to-End

Replacing the recognizer in PP-OCRv5 with ScriptMoE gives:

---

Citation

If you find ScriptMoE, TextMuSS-10M, or TextMuSS-Bench useful in your research, please cite:

@article{ye2026scriptmoe,
  title   = {All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts},
  author  = {Ye, Xingsong and Du, Yongkun and Zhang, Jiaxin and Li, Zhixian and Sun, Chong and Li, Chen and Lyu, Jing and Jin, Lianwen and Chen, Zhineng},
  journal = {arXiv preprint arXiv:2609.24058},
  year    = {2026}
}

---

License

The code in this repository is released under the license specified in the repository.

Please check the licenses of the individual datasets and third-party components before redistribution or commercial use.