• Project Page
• GitHub Repo
• Arxiv Paper

Dataset Description

This dataset repo contains the following two datasets:

• CaD-Inst: instruction-following dataset for 349K image pairs with instruction of summarizing CaD (Commonalities and Differences) on a pair of two images. This datsaet consists of Phase 1 data (278K) and Phase 2 data (71K). The dataset can be used as training data in visual instruction tuning of LMMs (Large Multimodal Models).
• CaD-QA: 7.5K open-ended QA pairs regading the CaD on a pair of two images for assessing the CaD understanding abilities of LMMs.

The subfolders in this dataset repo:

• Phase1: the Phase-1 CaD instruction-following data for 278K image pairs sourced from the Localized Narratives dataset.
• Phase2: the Phase-2 CaD instruction-following data for 71K image pairs (used in the MIMIC-IT Scene-Difference dataset, images originally sourced from COCO).
• CaD_QA_eval: 7.5K open-ended QA pairs based on CaD for 7.5K image pairs sourced from Visual Genome.

The samples in the three folders can be viewed using the Dataset Viewer by selecting the folder in "Subset".

Load the Dataset

The datasets can be loaded using the load_dataset() function, e.g.

from datasets import load_dataset
import os

main_dir = '/your/data/dir'
repo_name = 'wlin21at/CaD-Inst'
dataset_list = ['CaD_QA_eval', 'Phase1', 'Phase2']
for dataset in dataset_list:
    dataset_dir = os.path.join(main_dir, dataset)
    print(f'Downloading {dataset} dataset')
    if not os.path.exists(dataset_dir):
        os.makedirs(dataset_dir)
    ds = load_dataset(repo_name, dataset, cache_dir=dataset_dir)
    if dataset == 'CaD_QA_eval':
        ds = ds['test']
    else:
        ds = ds['train']
    for sample_id, sample in enumerate(ds):
        sample['conversations'] = eval(sample['conversations'])
        print(sample)

The samples in CaD_QA_eval have the following structure:

{
  'id': '2406825_2550', 
  'image': "['VG_100K_2/2406825.jpg', 'VG_100K_2/2550.jpg']", 
  'question': 'Image 1: <image>\nImage 2: <image>\nWhat colors dominate each image?', 
  'answer': 'Image 1 is dominated by shades of white, yellow, and orange, while Image 2 features a mix of darker colors for the roads and vehicles, contrasted by the greenery and blue sky.', 
  '__null_dask_index__': 0
}

The samples in Phase1 and Phase2 have the following structure (clip_sim_score is only available in the Phase1 data):

{
  'id': '8_97021_8_126714', 
  'image': "['d3551f94ff3911b5.jpg', '02fa557478e3a2ed.jpg']", 
  'conversations': 
    [
      {'from': 'human', 
       'value': 'Image 1: <image>\nImage 2: <image>\nSummarize the commonalities and differences about the visual content of the two images, including the object types, object attributes, counting the objects, object actions, object locations, relative positions between objects, etc.'}, 
      {'from': 'gpt', 
       'value': "Commonalities:\nThe two images have the following commonalities:\n\n1. Both images contain a clearly defined foreground and background.\n2. Both images contain some form of a pole or post (a light pole in Image 1 and a pole in Image 2).\n\nDifferences:\n\n1. The primary subject matter of the images is different. Image 1 features a car on the road, buildings, trees, and a sky in the background, while Image 2 features four fish of different sizes and colors in water, along with water plants.\n2. The number of objects in the images is different. Image 1 has a car, a light pole, trees, buildings, and a sky, while Image 2 has four fish, water plants, and a pole.\n3. The environment or setting of the images is different. Image 1 is set on a road or street, while Image 2 is set in water.\n4. The objects' actions are different. In Image 1, the car is stationary, while in Image 2, the fish are swimming.\n5. The objects' locations are different. In Image 1, the car is on the road, the light pole is behind the car, and the buildings are in the background. In Image 2, the fish are in the water, and the water plants and pole are in the same environment as the fish.\n6. The relative positions between objects are different. In Image 1, the car is in front of the light pole, and the buildings are in the background. In Image 2, the fish are swimming around the water plants and pole."}, 

      {'from': 'human', 
       'value': 'Which image suits the caption "In the image we can see four fish of different size and color. Here we can see water plants and something like a pole." better?\nA. Image 1\nB. Image 2\nAnswer with the option\'s letter from the given choices directly.'}, 
      {'from': 'gpt', 
       'value': 'B'}
    ], 
  'clip_sim_score': 0.4355500042438507, 
  '__null_dask_index__': 10
}

Image Source

Phase1 images:

• Open Images v6
• COCO 2017
• ADE20K
• Flickr30k

Phase2 images:

• COCO 2017

CaD_QA_eval images:

• Visual Genome

Visual Instruction Tuning

Our source code of visual instruction tuning is released in this GitHub repo.
The original json files that can be directly loaded in the code are uploaded here