OmniBench

🌐 Homepage | 🏆 Leaderboard | 📖 Arxiv Paper | 🤗 Paper | 🤗 OmniBench Dataset | | 🤗 OmniInstruct_V1 Dataset | 🦜 Tweets

The project introduces OmniBench, a novel benchmark designed to rigorously evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. We define models capable of such tri-modal processing as omni-language models (OLMs).

Mini Leaderboard

This table shows the omni-language models in the full evaluation setting in OmniBench, with the "Image & Audio", "Audio", and "Image" as input contexts and accuracy as metric. More results could be found at the live leaderboard.

Dataset

The dataset consists of the following keys:

• "index": an integer suggests the question id.
• "task type": a string suggests one of the 7 task types.
• "audio type": a string suggests one of the 3 audio types (speech, sound event and music).
• "question": a string suggests the question.
• "options": a list of four strings for multi-choice questions.
• "answer": a string suggesting the correct response, must appear in "options".
• "audio_path": the basename of the audio file, need to prepend mm_data/audio before using.
• "image_path": the basename of the image file, need to prepend mm_data/image before using.
• "audio" (for HF version only): contains the numpy array for the wavfile.
• "image" (for HF version only): contains the PIL.Image() object for the image.
• "audio content": the human-annotated audio transcripts, used in text alternative experiments.
• "image content": the VLM-generated caption for the image, used in text alternative experiments.

Download from Huggingface

from datasets import load_dataset

dataset = load_dataset("m-a-p/OmniBench")

# check on the data samples
print(dataset)
print(dataset['train'][0])

# same for
dataset = load_dataset("m-a-p/OmniInstruct_v1")

Reference

@misc{li2024omnibench,
    title={OmniBench: Towards The Future of Universal Omni-Language Models}, 
    author={Yizhi Li and Ge Zhang and Yinghao Ma and Ruibin Yuan and Kang Zhu and Hangyu Guo and Yiming Liang and Jiaheng Liu and Jian Yang and Siwei Wu and Xingwei Qu and Jinjie Shi and Xinyue Zhang and Zhenzhu Yang and Xiangzhou Wang and Zhaoxiang Zhang and Zachary Liu and Emmanouil Benetos and Wenhao Huang and Chenghua Lin},
    year={2024},
    eprint={2409.15272},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2409.15272}, 
}