Dataset Card for BiVLC

Dataset Description

• Homepage: https://imirandam.github.io/BiVLC_project_page/
• Repository: https://github.com/IMirandaM/BiVLC
• Paper: https://arxiv.org/abs/2406.09952
• Point of Contact: Imanol Miranda

Dataset Summary

BiVLC is a benchmark for Bidirectional Vision-Language Compositionality evaluation. Each instance consists of two images and two captions. Using each of the images and captions as a base, a model is asked to select the pair that correctly represents the base versus the hard negative distractor with minor compositional changes. Thus, we can measure image-to-text and text-to-image retrieval with hard negative pairs. To obtain good results on the dataset, it is necessary that the model performs well in both directions for the same instance.

Dataset instances

Each instance of the dataset consists of six fields:

• image: COCO 2017 validation image.
• caption: COCO 2017 validation text describing the COCO image.
• negative_caption: Negative caption generated from the COCO 2017 validation text description by SugarCrepe.
• negative_image: Negative image generated from the negative caption by BiVLC.
• type: Category of the negative instances: Replace, Swap or Add.
• subtype: Subcategory of the negative instances: Object, Attribute or Relation.

How to use

To load data with datasets:

>>> data = load_dataset("imirandam/BiVLC", split = "test")

Instance example

Each instance has the following structure:

{
    'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x332 at 0x7F9BFC0C5430>,
    'caption': 'A man throwing a ball while smiling and on a field.',
    'negative_caption': 'A man throwing a ball while a child is smiling on a field.',
    'negative_image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=512x512 at 0x7F9BE45571C0>,
    'type': 'add',
    'subtype': 'obj',
}

Dataset statistics

test: 2,933 instances formed by 2 images and 2 captions. 11,732 retrieval instances, 50% text-to-image and 50% image-to-text.

Source Data

• image and caption are from COCO 2017 validation split.
• negative_caption is a text description generated from the COCO caption by SugarCrepe.

Dataset curation

Step 1 - Uniformly format positive and hard negative captions

Step 2 - Generate hard negative images

Step 3 - Ask to human annotators to choose the best generated image

Step 4 - Filter ambiguous instances

More examples

Training Data

If you need training and validation data, you can use the datasets proposed in the paper in the following links, TROHN-Text and TORHN-Img.

Licensing Information

This work is licensed under a MIT License.

Citation Information

If you find this dataset useful, please consider citing our paper:

@misc{miranda2024bivlc,
      title={BiVLC: Extending Vision-Language Compositionality Evaluation with Text-to-Image Retrieval}, 
      author={Imanol Miranda and Ander Salaberria and Eneko Agirre and Gorka Azkune},
      year={2024},
      eprint={2406.09952},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}