MME-Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning

Ziliang Gan , Dong Zhang , Haohan Li , Yang Wu , Ji Liu , Haipang Wu , Chaoyou Fu , Zenglin Xu ,
Rongjunchen Zhang^♠, Yong Dai

📖Paper |🏠Homepage|🤗Huggingface

In recent years, multimodal benchmarks for general domains have guided the rapid development of multimodal models on general tasks. However, the financial field has its peculiarities. It features unique graphical images (e.g., candlestick charts, technical indicator charts) and possesses a wealth of specialized financial knowledge (e.g., futures, turnover rate).

Benchmarks from general fields often fail to measure the performance of multimodal models in the financial domain, and thus cannot effectively guide the rapid development of large financial models. To promote the development of large financial multimodal models, we introduce MME-Finance, the first comprehensive bilingual multimodal benchmark designed for financial analysis.

📢 News

• 🚀 [08/01/2026] We are pleased to announce the release of our new benchmark FinMTM! Welcome to visit the official link: FinMTM
• 🚀 [01/08/2025] Our paper has been accepted for ACM MM 2025.
• 🚀 [30/05/2025] We released MME-Finance-MT multi-turn question benchmark and MME-Finance-Binary benchmark.
• 🚀 [08/01/2025] We have released all samples in both English and Chinese.
• 🚀 [05/11/2024] We released MME-Finance benchmark, a bilingual multimodal benchmark in financial domain.

💡 Highlights

• 🔥 Bilingual multimodal financial benchmark: MME-Finance is the first Bilingual multimodal financial benchmark which comprises 1,171 English and 1,103 Chinese open-ended questions, covering diverse financial image types and various multimodal capabilities.
• 🔥 Evaluation strategy: MME-Finance proposes a Elaborate evaluation strategy that taking image into consideration, and has a high consistency with humans. It can serve as a reference for evaluating MLLMs for other works.
• 🔥 Valuable insights: We conduct extensive evaluation on 19 MLLMs based on MME-Finance, revealing critical insights about the strengths and shortcomings of the current MLLMs in financial applications.

🛠️ Usage

We have integrated MMfin into the VLMEvalKit framework. For the environment configuration and the use of API, please refer to VLMEvalKit. Regarding the data, first of all, you should download the MMfin.tsv and MMfin_CN.tsv files, as well as the relevant financial images. The folder structure is shown as follows:

├─ datasets
    ├─ images
        ├─ MMfin
            ...
        ├─ MMfin_CN
            ...
        ├─ MMfin_MT_CN
            ...      
        ├─ MMfin_Binary
            ...      
    │ MMfin.tsv
    │ MMfin_CN.tsv
    | MMfin_MT_CN.tsv
    | MMfin_Binary.tsv

The following is the process of inference and evaluation (Qwen2-VL-2B-Instruct as an example):

export LMUData="The path of the datasets"
python run.py --data MMfin --model Qwen2-VL-2B-Instruct --verbose
python run.py --data MMfin_CN --model Qwen2-VL-2B-Instruct --verbose
python run.py --data MMfin_MT_CN --model Qwen2-VL-2B-Instruct --verbose
python run.py --data MMfin_Binary --model Qwen2-VL-2B-Instruct --verbose

Results

Evaluation results on the English MME-Finance for all tasks.

Evaluation results on English MME-Finance for different types and styles of images.

Evaluation results on the Chinese MME-Finance for all tasks.

Evaluation results on Chinese MME-Finance for different types and styles of images.

Evaluation results on True/Fasle version of MME-Finance.

Evaluation results on multi-turn version of MME-Finance.

✒️Citation

@article{gan2024woodpecker,
  title={MME-Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning},
  author={Gan, Ziliang and Lu, Yu and Zang, Dong and Li, Haohan and Liu, Che and Liu, Jian and Liu, Ji and Wu, Haipang and Fu, Chaoyou and Xu, Zenglin and Zhang, Rongjunchen and Dai, Yong},
  journal={arXiv preprint arXiv:2411.03314},
  year={2024}
}

📄 License

Usage and License Notices: The data and code are intended and licensed for research use only. License: Attribution-NonCommercial 4.0 International It should abide by the policy of OpenAI: https://openai.com/policies/terms-of-use

💖 Acknowledgement

• VLMEvalKit: the codebase we built upon.