LLaVA-Interleave Bench Dataset Card

Dataset details

Dataset type: LLaVA-Interleave Bench is a comprehensive set of multi-image datasets that are collected from public datasets or generated by the GPT-4V API. It is constructed for evaluating the interleaved multi-image reaoning capbilities of LMMs.

Dataset date: LLaVA-Interleave Bench was collected in April 2024, and released in June 2024.

Paper or resources for more information: Blog: https://llava-vl.github.io/blog/2024-06-16-llava-next-interleave/

Evaluate LLaVA-NeXT Interleave Model

Preparation

Please download the evaluation data first.

Unzip eval_images.zip and there are Split1 and Split2 in it. Organize the downloaded data into the following structure:

interleave_data
├── Split1
│   ├── ...
│   └── ...
|
├── Split2
|   ├── ...
│   └── ...
├── multi_image_in_domain.json
├── multi_image_out_domain.json
└── multi_view_in_domain.json

Inference and Evaluation

Example: Please first edit /path/to/ckpt to the path of checkpoint, /path/to/images to the path of "interleave_data" in scripts/interleave/eval_all.sh and then run

bash scripts/interleave/eval_all.sh

Note that, the MMMU-mv data is currently not included in the released data. We will release it soon. To construct MMMU-mv, we sample the 789 questions with multiple input images from the official test set of MMMU.

License: Creative Commons Attribution 4.0 International; and it should abide by the policy of OpenAI: https://openai.com/policies/terms-of-use Where to send questions or comments about the model: fliay@connect.ust.hk

Intended use

Primary intended uses: The primary use of LLaVA-Next Interleave is research on large multimodal models and chatbots. Primary intended users: The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.