M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models

[Webpage] [Paper] [Huggingface Dataset] [Leaderboard]

🎨 M4U-mini

To support more languages, we have constructed a tiny version of M4U with three additional languages (Japanese, Arabic, and Thai). We randomly selected 5% of the test data and follow our processing pipeline to construct these parts. We plan to expand M4U to include these medium or low-resource languages and more disciplines regrading cultural contexts later.

Evaluation on M4U-mini

The evaluation pipeline of M4U-mini is consistent with M4U. You can generate the response of GPT-4o on M4U-mini, then calculate the scores following these instructions.

python evaluate_gpt4o.py \
  --model gpt-4o \
  --field all \
  --lang zh,en,de,ar,th,ja \
  --result_folder ./result/M4U-mini/gpt4o

Evaluation results

We evaluate state-of-the-art close-sourced models (GPT-4o, Gemini-1.5-Flash) and open-sourced models (DeepSeek-VL-Chat, LLaVA-NeXT-34B, etc.) on M4U-mini. Detailed results are shown below:

📖 Dataset Usage

The format of M4U-mini is consistent with M4U. First, make sure that you have successfully setup:

pip install datasets

Then you can easily download this dataset from Huggingface.

from datasets import load_dataset

dataset = load_dataset("M4U-Benchmark/M4U-mini")

✅ Cite

If you find M4U useful for your research and applications, please kindly cite using this BibTeX:

@article{wang2024m4u,
  title={M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models},
  author={Hongyu Wang and Jiayu Xu and Senwei Xie and Ruiping Wang and Jialin Li and Zhaojie Xie and Bin Zhang and Chuyan Xiong and Xilin Chen},
  month={May},
  year={2024}
}

🧠 Acknowledgments

Some implementations in M4U are either adapted from or inspired by the MMMU repository and the MathVista repository.