M5 - A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks

This repository contains the code for the M5 Benchmark paper.

News

• 2024-11: 🌴 We happily presented the M5 Benchmark at EMNLP in Miami! It is now available on the ACL Anthology!
• 2024-10: 👷 We are currently working on M5 Benchmark v2 with a cleaner and more extendable codebase, support for more models, and additional datasets.
• 2024-10: ⭐️ We publicly released the code for the M5 Benchmark.
• 2024-09: ⭐️ Our paper got accepted at EMNLP (Findings) 2024, Miami, FL, USA.
• 2024-07: 📝 We released the first preprint of the M5 Benchmark paper.

Setup

Note that all code was tested only on Debian-based systems (Ubuntu 20.04 and 22.04) with CUDA 11.8 pre-installed.

To setup the environments, run the setup.sh script:

./setup.sh

This takes a long time (!) and will:

1. Install the mamba package manager if not already installed.
2. Create the following mamba environments:
   • m5b for the majority of the models
   • m5b-cogvlm for the CogVLM models
   • m5b-yivl for the Yi-VL models. Note that this also downloads the model weights.
   • m5b-llava for the original LlaVA models from Haotian Liu
   • m5b-omnilmm for the OmniLMM models
   • m5b-qwenvl for the QwenVL models
3. Tests each environment to ensure that it is correctly setup.

You can also install single environments by running the following command:

./setup.sh mamba-env-create <env_name>

where <env_name> is one of the environments listed above.

If the installation was successful, you should see a message similar to the following for each environment:

##################### PYTHON ENV 'm5b-llava' INFO START #####################
Python version: 3.10.15 | packaged by conda-forge | (main, Jun 16 2024, 01:24:24) [GCC 13.3.0]
PyTorch version: 2.1.2
CUDA available: True
CUDA version: 11.8
CUDA devices: 5
Flash Attention 2 Support: True
Transformers version: 4.37.2
Datasets version: 3.0.1
Lightning version: 2.4.0
##################### PYTHON ENV 'm5b-llava' INFO END #####################

Getting the Data

To speed up the evaluation process, the datasets of the M5 Benchmark are stored as WebDataset archives. You can download all datasets as a single 46G tar-archive using the following command:

curl https://ltdata1.informatik.uni-hamburg.de/m5b/m5b_datasets.tar -o /path/to/the/m5b/datasets/m5b_datasets.tar \
&& tar -xvf /path/to/the/m5b/datasets/m5b_datasets.tar

Evaluate a Model

Available Models

Because different models require different environments, you have to activate the correct environment before running the benchmark.

Available Datasets

The M5 Benchmark consists of the following datasets: marvl xgqa xm3600 xvnli maxm xflickrco m5b_vgr m5b_vlod

Running the evaluation for a single model on a single dataset

First, activate the correct environment for the model you want to evaluate as described above. Then, in the project root, run the following command:

PYTHONPATH=${PWD}/src python src/m5b/scripts/eval.py \
    --model_id="<model_id>" \
    --dataset="<dataset_id>" \ 
    --data_base_path=/path/to/the/m5b/datasets

where <model_id> is the model ID from the models table above and <dataset_id> is the dataset ID from the datasets table above.

Citation

If you use this code or the M5 Benchmark in your research, please cite the following paper:

@inproceedings{
    schneider2024m5benchmark,
    title={M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks},
    author={Schneider, Florian and Sitaram, Sunayana},
    booktitle={Findings of the Association for Computational Linguistics: EMNLP 2024},
    address={Miami, Florida, USA},
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.250",
    pages={4309--4345},
    year={2024}
}