Dataset Description

ReMI was introduced in ReMI: A Dataset for Reasoning with Multiple Images. It contains 13 tasks namely: EmojiAlgebra, FuncRead, GeomShape, GeomCost, Collisions, Clocks, Schedule, Charts, CodeEdit, Isomorphism, Maps, RefCOCO, and IQ.

Dataset Usage

Data Downloading

All the data examples were divided into two subsets: train and test.

• train: contains 2 examples per task (26 in total) to be used as fewshot examples.
• test: contains 200 examples per task (2600 examples in total) for evaluation.

To download the dataset, install Huggingface Datasets and then use the following command:

from datasets import load_dataset

dataset = load_dataset("mehrankazemi/ReMI")

For evaluation, use this colab.

Data Format

The dataset contains the following fields:

• question: Contains the text of the question. The question includes tags of the form <image1>, <image2>, etc. They should be replaced with the i-th image when feeding the input to models.
• label: Contains the answer to the question.
• task: Corresponds to one of the 13 tasks in the dataset.
• image_i: (for i in [1, 6]) contains the bytes for the i-th image.

Data Source

ReMI has been created through the following ways: 1- Using visualization libraries such as matplotlib, rendering TikZ code in latex, NetworkX, etc., 2- screenshots from Google Maps, 3- images from COCO.

• Purpose: ReMI is primarily designed for use as a test set.
• Prohibition: Using ReMI as a training set is prohibited.

Citation

@article{kazemi2024remi,
title={ReMI: A Dataset for Reasoning with Multiple Images},
author={Kazemi, Mehran and Dikkala, Nishanth and Anand, Ankit and Devic, Petar and Dasgupta, Ishita and Liu, Fangyu and Fatemi, Bahare and Awasthi, Pranjal and Guo, Dee and Gollapudi, Sreenivas and Qureshi, Ahmed},
journal={arXiv preprint arXiv:2406.09175},
year={2024}
}