Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning

File Structure

├── MIR
    |── analogy.json
    │── codeu.json
    |── dataset_namex.json
    └── Images
        ├── analogy
        │    └── image_x.jpg
        └──codeu
            └── image_x.jpg

JSON Structure

{
    "questions": " What is the expected kurtosis of the sequence created by`create_number_sequence(-10, 10)`?\n\n1. -1.5\n2. -1.2002400240024003\n3. 0\n4. 1. 2\n\n",
    "answers": 2,
    "images": [
        "images/codeu/example_53_main.png",
        "images/codeu/example_53_enhanced_operations.png"
    ]
}

Value of images is supposed to be a list, where each element is images/{dataset_name}/image_name. And you can directly index the image from that path.

Code for evaluation

Our code for evaluating various vision-language models are available on GitHub.

Citation

@article{zhao2024mirb
  author    = {Bingchen Zhao, Yongshuo Zong, Letian Zhang, Timothy Hospedales},
  title     = {Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning},
  journal   = {arXiv preprint},
  year      = {2024},
}

Paper: arxiv.org/abs/2406.12742