TextMuSS-Bench

TextMuSS-Bench is a multilingual real-world scene text recognition benchmark introduced in:

All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts
Xingsong Ye, Yongkun Du, Jiaxin Zhang, Zhixian Li, Chong Sun, Chen Li, Jing Lyu, Lianwen Jin, Zhineng Chen
arXiv:2609.24058 (2026)

• 📄 Paper: https://arxiv.org/abs/2609.24058
• 🤗 Hugging Face Paper: https://huggingface.co/papers/2609.24058
• 💻 Code: https://github.com/YesianRohn/ScriptMoE
• 🚀 Demo: https://huggingface.co/spaces/Yesianrohn/MultilingualOCR-Demo
• 🧪 Training Dataset: https://huggingface.co/datasets/Yesianrohn/TextMuSS-10M

Dataset Summary

TextMuSS-Bench is designed for evaluating multilingual scene text recognition (STR) on real-world images.

The benchmark covers:

• 10 writing scripts
• 10,899 real-world images
• multilingual scene text recognition scenarios
• additional evaluation coverage for scripts including Russian, Thai, and Tibetan

The benchmark is used to evaluate ScriptMoE and other multilingual OCR / STR systems.

Motivation

Most multilingual scene text recognition systems are evaluated primarily on high-resource languages and scripts.

TextMuSS-Bench provides a broader real-world evaluation setting, with multiple scripts represented in a unified benchmark.

The benchmark complements the synthetic TextMuSS-10M training dataset by measuring recognition performance on real-world scene text.

Relationship to TextMuSS-10M

The two resources serve different purposes:

TextMuSS-10M provides broad synthetic supervision, while TextMuSS-Bench is intended for measuring generalization to real scene text.

Relationship to ScriptMoE

TextMuSS-Bench is the main STR benchmark introduced with ScriptMoE.

In the paper, ScriptMoE achieves an average word accuracy of 82.06% on TextMuSS-Bench.

Selected results reported in the paper include:

For the exact evaluation protocol and complete results, please refer to the paper and the official evaluation code.

Dataset Structure

The dataset files and metadata are provided in this Hugging Face repository.

The Hugging Face repository currently contains approximately 751 MB of data.

Please refer to the repository files for the exact data organization and annotation format.

Intended Use

TextMuSS-Bench is intended for:

• multilingual scene text recognition evaluation
• OCR benchmarking
• script-aware OCR research
• multilingual recognition robustness evaluation
• comparison of scene text recognition systems
• research on low-resource and multi-script OCR

Evaluation

The official ScriptMoE repository provides evaluation code:

https://github.com/YesianRohn/ScriptMoE

In particular:

eval_textmussbench/
Eval-TextMuSS-Bench/

contain the evaluation-related code.

Please use the evaluation protocol described in the paper when reporting results.

Important Considerations

TextMuSS-Bench is intended as an evaluation benchmark.

Users should avoid training or tuning directly on the benchmark test data when reporting comparable results.

For reproducible evaluation, please follow the official evaluation protocol provided with the ScriptMoE repository.

License

This dataset is released under the Apache-2.0 license.

Please review the licenses and terms of the original data sources and third-party resources used in the benchmark before redistribution or commercial use.

Citation

If you use TextMuSS-Bench, please cite the accompanying paper:

@article{ye2026scriptmoe,
  title   = {All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts},
  author  = {Ye, Xingsong and Du, Yongkun and Zhang, Jiaxin and Li, Zhixian and Sun, Chong and Li, Chen and Lyu, Jing and Jin, Lianwen and Chen, Zhineng},
  journal = {arXiv preprint arXiv:2609.24058},
  year    = {2026}
}

Paper: https://arxiv.org/abs/2609.24058

Hugging Face Paper: https://huggingface.co/papers/2609.24058