M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models

Code for the Paper M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models.

[Webpage] [Paper] [Huggingface Dataset] [Leaderboard]

💥 News 💥

• [2024.05.23] Our paper, dataset and code are public aviailable.

👀 About M4U

Multilingual multimodal reasoning is a core component to achieve human-level intelligence. However, most of the existing benchmarks for multilingual multimodal reasoning struggle to differentiate between models of varying performance: even language models that don't have visual capabilities can easily achieve high scores. This leaves a comprehensive evaluation for the leading multilingual multimodal models largely unexplored.

The detailed statistics of M4U dataset.

In this work, we introduce M4U, a novel and challenging benchmark for assessing the capability of multi-discipline multilingual multimodal understanding and reasoning. M4U contains 10005 samples covering 64 disciplines of 16 subfields from Science, Engineering and Healthcare.

With M4U, we conduct extensive evaluations for 22 leading LMMs and 4 LLMs with external tools. The evaluation results show that the state-of-the-art model, GPT-4o, only achieves 47.6% average accuracy on M4U. Besides, we observe that the leading LMMs have significant language preferences.

Our in-depth analysis shows that the leading LMMs, including GPT-4o, suffer from the performance degradation when they are prompted with cross-lingual multimodal questions, e.g., the images have key textual information in Chinese, while the question is in English or German.

🏆 Leaderboard 🏆

Contributing the Leaderboard

🚨🚨 The leaderboard is continuously being updated.

If you want to upload your model's results to the Leaderboard, please send an email to Hongyu Wang and Ruiping Wang.

The evaluation instructions are available at 🔮 Evaluations on M4U.

Leaderboard

Leaderboard on M4U-mini

• Method types
  • LMM : Large Multimodal Model
  • Tool : Tool-augmented Large Language Model. The captions are generated by Gemini 1.0 Pro.

📖 Dataset Usage

Examples

An example from the Chemistry-Inorganic of M4U dataset:

Format

The dataset is in json format. We provide the detailed attributes as follows:

{
    "question": [string] The question text,
    "image_files": [list] The images stored in bytes
    "image_caption": [list] The image caption generated by Gemini Pro 1.0
    "image_type": [list] The category of the images
    "options": [list] Choices for multiple-choice problems.
    "answer": [string] The correct answer for the problem,
    "answer_index": [integer] The correct answer index of the options for the problem,
    "language": [string] Question language: "en", "zh", or "de",
    "discipline": [string] The discipline of the problem: "Geophysics" or "Optical",
    "subfield": [string] The subfield of the problem: "Math" or "Geography"
    "field": [string] The field of the problem: "science", "engineering" or "healthcare"
    "cross_lingual": [bool] Whether the visual content of the problem has the key information or concepts
}

Download

First, make sure that you have successfully setup:

pip install datasets

Then you can easily download this dataset from Huggingface.

from datasets import load_dataset

dataset = load_dataset("M4U-Benchmark/M4U")

Here are some examples of how to access the downloaded dataset:

# print the first example on the science_en split
print(dataset["science_en"][0])
print(dataset["science_en"][0]['question']) # print the question 
print(dataset["science_en"][0]['options']) # print the options
print(dataset["science_en"][0]['answer']) # print the answer

✅ Cite

If you find M4U useful for your research and applications, please kindly cite using this BibTeX:

@article{wang2024m4u,
  title={M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models},
  author={Hongyu Wang and Jiayu Xu and Senwei Xie and Ruiping Wang and Jialin Li and Zhaojie Xie and Bin Zhang and Chuyan Xiong and Xilin Chen},
  month={May},
  year={2024}
}