Dataset Card for COCO-Counterfactuals

COCO-Counterfactuals is a high quality synthetic dataset for multimodal vision-language model evaluation and for training data augmentation.

Dataset Details

Dataset Description

COCO-Counterfactuals is a high quality synthetic dataset for multimodal vision-language model evaluation and for training data augmentation. Each COCO-Counterfactuals example includes a pair of image-text pairs; one is a counterfactual variation of the other. The two captions are identical to each other except a noun subject. The two corresponding synthetic images differ only in terms of the altered subject in the two captions. In our accompanying paper, we showed that the COCO-Counterfactuals dataset is challenging for existing pre-trained multimodal models and significantly increase the difficulty of the zero-shot image-text retrieval and image-text matching tasks. Our experiments also demonstrate that augmenting training data with COCO-Counterfactuals improves OOD generalization on multiple downstream tasks.

• License: CC-BY-4.0

Dataset Sources

• Repository: https://huggingface.co/datasets/Intel/COCO-Counterfactuals
• Paper: https://openreview.net/forum?id=7AjdHnjIHX

Data

The captions are located in data/examples.jsonl and the images are located in data/images.zip. You can load the data as follows:

from datasets import load_dataset
examples = load_dataset('Intel/COCO-Counterfactuals', use_auth_token=<YOUR USER ACCESS TOKEN>)

You can get <YOUR USER ACCESS TOKEN> by following these steps:

1. log into your Hugging Face account
2. click on your profile picture
3. click "Settings"
4. click "Access Tokens"
5. generate an access token

Dataset Structure

[More Information Needed]

Bias, Risks, and Limitations

Despite the impressive recent improvements in text-to-image generation capabilities, models such as Stable Diffusion have well-known limitations that should be considered when utilizing datasets which are derived from them. We do not foresee significant risks of security threats or human rights violations in our work. However, the automated nature of our image generation process may introduce the possibility of our COCO-Counterfactuals dataset containing images that some individuals may consider inappropriate or offensive.

Citation

https://openreview.net/forum?id=7AjdHnjIHX

Tiep Le and Phillip Howard contributed equally.

BibTeX:

@inproceedings{le2023cococounterfactuals,
  author = {Tiep Le and Vasudev Lal and Phillip Howard},
  title = {{COCO}-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs},
  booktitle = {Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year = 2023,
  url={https://openreview.net/forum?id=7AjdHnjIHX},
}

Dataset Card Authors

Tiep Le and Vasudev Lal and Phillip Howard

Dataset Card Contact

tiep.le@intel.com; vasudev.lal@intel.com; phillip.r.howard@intel.com