Dataset Card for SeeTRUE-Feedback

• Dataset Description
  • Supported Tasks and Leaderboards
  • Languages
• Dataset Structure
  • Data Fields
  • Data Splits
• Dataset Creation
• Licensing Information
• Citation Information

Dataset Description

The SeeTRUE-Feedback dataset is a diverse benchmark for the meta-evaluation of image-text matching/alignment feedback. It aims to overcome limitations in current benchmarks, which primarily focus on predicting a matching score between 0-1. SeeTRUE provides, for each row, the original caption, feedback related to text-image misalignment, and the caption+visual source of misalignments (including a bounding box for the visual misalignment).

Languages

The dataset supports English language.

Dataset Structure

Data Fields

• image_caption - Caption associated with the image.
• image_name: The name of the image file.
• dataset_source: The source/origin dataset of the image.
• id_in_source_dataset: The ID of the dataset where the row originates from.
• image_url: An S3 link from which you can download the image.
• human_feedback: Human-annotated feedbacks about image-text misalignment.
• feedback: Summary of feedback consolidated into a single entry (Generated by LLM: PaLM-2)
• feedback_clean: A parsed and "clean" version of feedback field.
• caption_misalignment: Source of misalignment in the image caption.
• visual_misalignment: Source of misalignment in the image.
• bbox_GroundingDino: Detected visual misalignment bounding-box in GroundingDino output format.
• bbox_PaLI: Detected visual misalignment bounding-box in PaLI output format.

Data Splits

SeeTRUE-Feedback contains a single split: TEST, and should not be used for training.

Dataset Creation

The dataset has been created by sourcing and matching images and text from multiple datasets. More information in the paper:

Licensing Information

The dataset is under the CC-By 4.0 license.

Citation Information

TODO