MMCBench Dataset: Benchmarking Dataset for Multimodal Model Evaluation 🚀

Overview

The MMCBench Dataset is a curated collection of data designed for the comprehensive evaluation of Large Multimodal Models (LMMs) under common corruption scenarios. This dataset supports the MMCBench framework, focusing on cross-modal interactions involving text, image, and speech. It provides essential data for generative tasks such as text-to-image, image-to-text, text-to-speech, and speech-to-text, enabling robustness and self-consistency assessments of LMMs.

Dataset Composition 📊

The MMCBench Dataset is structured to facilitate the evaluation across four key generative tasks:

• Text-to-Image: A collection of text descriptions with their corresponding corrupted versions and associated images.
• Image-to-Text: A set of images with clean and corrupted captions.
• Text-to-Speech: Text inputs with their clean and corrupted audio outputs.
• Speech-to-Text: Audio files with transcriptions before and after audio corruptions.

Each subset of the dataset has been meticulously selected and processed to represent challenging scenarios for LMMs.

Using the Dataset 🛠️

To use the MMCBench Dataset for model evaluation:

1. Access the Data: The dataset is hosted on Hugging Face and can be accessed using their dataset library or direct download.
2. Select the Task: Choose from text-to-image, image-to-text, text-to-speech, or speech-to-text tasks based on your model's capabilities.
3. Apply the Benchmark: Utilize the data for each task to test your model's performance against various corruptions. Follow the MMCBench framework for a consistent and standardized evaluation.

Dataset Structure 📁

The dataset is organized into four main directories, each corresponding to one of the generative tasks:

• text2image/: Contains text inputs and associated images.
• image2text/: Comprises images and their descriptive captions.
• text2speech/: Includes text inputs and generated speech outputs.
• speech2text/: Contains audio files and their transcriptions.

Contributing to the Dataset 🤝

Contributions to the MMCBench Dataset are welcome. If you have suggestions for additional data or improvements, please reach out through the Hugging Face platform or directly contribute via GitHub.

License 📜

The MMCBench Dataset is made available under the Apache 2.0 License, ensuring open and ethical use for research and development.

Acknowledgments and Citations 📚

When using the MMCBench Dataset in your research, please cite it appropriately. We extend our gratitude to all contributors and collaborators who have enriched this dataset, making it a valuable resource for the AI and ML community.