Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark

1. Introduction

MMLA is the first comprehensive multimodal language analysis benchmark for evaluating foundation models. It has the following features:

• Large Scale: 61K+ multimodal samples.
• Various Sources: 9 datasets.
• Three Modalities: text, video, and audio
• Both Acting and Real-world Scenarios: films, TV series, YouTube, Vimeo, Bilibili, TED, improvised scripts, etc.
• Six Core Dimensions in Multimodal Language Analysis: intent, emotion, sentiment, dialogue act, speaking style, and communication behavior.

We also build baselines with three evaluation methods (i.e., zero-shot inference, supervised fine-tuning, and instruction tuning) on 8 mainstream foundation models (i.e., 5 MLLMs (Qwen2-VL, VideoLLaMA2, LLaVA-Video, LLaVA-OV, MiniCPM-V-2.6), 3 LLMs (InternLM2.5, Qwen2, LLaMA3). More details can refer to our paper.

2. Datasets

2.1 Statistics

Dataset statistics for each dimension in the MMLA benchmark. #C, #U, #Train, #Val, and #Test represent the number of label classes, utterances, training, validation, and testing samples, respectively. avg. and max. refer to the average and maximum lengths.

2.2 Collection Timeline

• MIntRec: Released in 2022/10.
• MIntRec2.0: Released in 2024/01.
• MELD: Collected from TV series (released in 2019/05).
• UR-FUNNY-v2: Collected from publicly available TED talks (released in 2019/11).
• MUStARD: Collected from TV series (released in 2019/07).
• MELD-DA: Dialogue act annotations added to MELD in 2020/07. Derived from the EMOTyDA dataset, which re-annotated the original MELD training set videos without collecting new video data.
• IEMOCAP-DA: Dialogue act annotations added to IEMOCAP (released in 2020/07). Derived from the EMOTyDA dataset, which re-annotated all original IEMOCAP videos without collecting new video data.
• MOSI: Collected from YouTube opinion videos (released in 2016/06).
• IEMOCAP: Collected from scripted improvisational acting (released in 2008/12).
• Anno-MI: Collected from publicly available YouTube and Vimeo videos (released in 2023/03).

2.3 License

This benchmark uses nine datasets, each of which is employed strictly in accordance with its official license and exclusively for academic research purposes. We fully respect the datasets’ copyright policies, license requirements, and ethical standards. For those datasets whose licenses explicitly permit redistribution, we release the original video data (e.g., MIntRec, MIntRec2.0, MELD, UR-FUNNY-v2, MUStARD, MELD-DA, CH-SIMS v2.0, and Anno-MI. For datasets that restrict video redistribution, users should obtain the videos directly from their official repositories (e.g., MOSI, IEMOCAP and IEMOCAP-DA. In compliance with all relevant licenses, we also provide the original textual data unchanged, together with the specific dataset splits used in our experiments. This approach ensures reproducibility and academic transparency while strictly adhering to copyright obligations and protecting the privacy of individuals featured in the videos.

3. LeaderBoard

3.1 Rank of Zero-shot Inference

3.2 Rank of Supervised Fine-tuning (SFT) and Instruction Tuning (IT)

4. Data Integrity

All files included in the MMLA benchmark are verified using SHA-256 checksums. Please ensure the integrity of the files using the following checksums:

5. Acknowledgements

For more details, please refer to our Github repo. If our work is helpful to your research, please consider citing the following paper:

@article{zhang2025mmla,
  author={Zhang, Hanlei and Li, Zhuohang and Zhu, Yeshuang and Xu, Hua and Wang, Peiwu and Zhu, Haige and Zhou, Jie and Zhang, Jinchao},
  title={Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark},
  year={2025},
  journal={arXiv preprint arXiv:2504.16427},
}