Inst-It Bench

Homepage | Code | Paper | arXiv

Inst-It Bench is a fine-grained multimodal benchmark for evaluating LMMs at the instance-Level, which is introduced in the paper Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning.

• Size: 1,000 image QAs and 1,000 video QAs
• Splits: Image split and Video split
• Evaluation Formats: Open-Ended and Multiple-Choice

Introduction

Existing multimodal benchmarks primarily focus on global understanding, failing to provide more in-depth insights into the instance-level comprehension capability of models. Specifically, Inst-IT Bench includes two parts: image-split and video-split, and is able to evaluate the models' ability in understanding instances in both images and videos. The image-split contains 1,036 QA pairs for 338 images, while the video-split contains 1,001 QA pairs for 206 videos. Each QA pair is available in both open-ended and multiple-choices formats. The followings are some examples from the video-split:

Click here to unfold more data examples:

Evaluate your model on Inst-IT Bench

If you want to evaluate your model on our Inst-IT Bench, please refer to our GitHub code for more instructions.

We conducted an extensive evaluation of Inst-IT Bench

We conduct extensive evaluations on our benchmark, including state-of-the-art open-source image models, video models, and cutting-edge proprietary models. The results that even state-of-the-art models struggle with fine-grained, instance-level understanding.

#IT indicates the number of training samples used during the instruction-tuning stage. N/A indicates that the number is unknown.

Contact

Feel free to contact us if you have any questions or suggestions

• Email (Wujian Peng): wjpeng24@m.fudan.edu.cn
• Email (Lingchen Meng): lcmeng20@fudan.edu.cn

Citation

If you find our work helpful, please consider citing our paper ✒️ and like our dataset ❤️ :

  @article{peng2024inst,
    title={Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning},
    author={Peng, Wujian and Meng, Lingchen and Chen, Yitong and Xie, Yiweng and Liu, Yang and Gui, Tao and Xu, Hang and Qiu, Xipeng and Wu, Zuxuan and Jiang, Yu-Gang},
    journal={arXiv preprint arXiv:2412.03565},
    year={2024}
  }