Dataset Description

SPRIGHT (SPatially RIGHT) is the first spatially focused, large scale vision-language dataset. It was built by re-captioning ∼6 million images from 4 widely-used datasets:

• CC12M
• Segment Anything
• COCO Validation
• LAION Aesthetics

This repository contains the re-captioned data from CC12M and Segment Anything, while the COCO data is present here. We do not release images from LAION, as the parent images are currently private.

Below are some illustrative examples from the SPRIGHT dataset, where the captions are correct in its entirety; both in capturing the spatial relationships and overall description of the image.

We also share some illustrative examples from the SPRIGHT dataset, where the captions are not completely correct.

Dataset Sources

CC-12M

We re-caption a total of 2.3 million images from the CC-12M data taset, filtering out images of resolution less than 768.

Segment Anything

We re-caption 3.5 million images as part of our process. Since SA has all human faces blurred, we filter out images which contain blurring i.e. we filter our images where humans are detected (using the Owl-V2 object detector). Since SA does not have ground-truth captions, we also generate its general captions using the CoCa captioning model.

Dataset Structure

Samples

Each tar file contains 10k samples. Each sample is composed of:

• an image - "{idx}.jpg"
• related captions (general caption and spatial caption) - "{idx}.json"
• metadata (image width and height, original dataset the image was taken from and its original id) - "{idx}.metadata.json"

How to use it

In order to load the data, you can use the load_data.py script. The metadata.json file contains the size and the split for each tar file. We also provide a script robust_upload.py used to efficiently upload the data to Hugging Face Hub.

Note: filenames inside each .tar partition do NOT contain leading zeroes, which may confound some sorting mechanism (eg: python's sort() function); users that download and extract data or filenames from the .tar partions should be aware of this and use a "natural sort" style function to accomodate this convention.

Dataset Creation

Data Generation

We leverage LLaVA-1.5-13B to produce synthetic spatial captions, and use the following prompt to create the SPRIGHT dataset:

"Using 2 sentences, describe the spatial relationships seen in the image. You can use words like left/right, above/below, front/behind, far/near/adjacent, inside/outside. Also describe relative sizes of objects seen in the image."

Dataset validation

• Using FAITHScore: We leverage a large language model to deconstruct generated captions into atomic (simple) claims that can be individually and independently verified in VQA format. The captions are on average 88.9% correct.
• Using GPT4(V): We perform a small-scale study on 100 images to evaluate our captions with GPT-4(V). Specifically, we prompt GPT-4(V) to rate each caption between a score of 1 to 10, especially focusing on the correctness of the spatial relationships captured. We achieve a mean and median rating of 6.41 and 7.0.
• Human annotation: We also annotate a total of 3000 images through a crowd-sourced human study, where each participant annotates a maximum of 30 image-text pairs. Most captions in SPRIGHT have >1 sentences. Therefore, for a fine-grained evaluation, we randomly select 1 sentence, from a caption in SPRIGHT and evaluate its correctness for a given image. Across 149 responses, we get an accuracy of 66.57%.

Acknowledgements

We thank Lucain from the Hugging Face team for helping us with the robust_upload.py script.

Citation

@misc{chatterjee2024getting,
      title={Getting it Right: Improving Spatial Consistency in Text-to-Image Models}, 
      author={Agneet Chatterjee and Gabriela Ben Melech Stan and Estelle Aflalo and Sayak Paul and Dhruba Ghosh and Tejas Gokhale and Ludwig Schmidt and Hannaneh Hajishirzi and Vasudev Lal and Chitta Baral and Yezhou Yang},
      year={2024},
      eprint={2404.01197},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

License

SPRIGHT-T2I/spright is licensed under the Intel Research License. All Rights Reserved.

Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See Intel’s Global Human Rights Principles. Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.