PixMo-Point-Explanations

PixMo-Point-Explanations is a dataset of images, questions, and answers with explanations that can include in-line points that refer to parts of the image. It can be used to train vison language models to respond to questions through a mixture of text and points.

PixMo-Point-Explanations is part of the PixMo dataset collection and was used to train the Molmo family of models

We consider this dataset experimental, while these explanations can be very informative we have also seen models can hallucinate more when generating outputs of this sort. For that reason, the Molmo models are trained to only generate outputs like this when specifically requested by prefixing input questions with "point_qa:". This mode can be used in the Molmo demo

Quick links:

• 📃 Paper
• 🎥 Blog with Videos

Loading

data = datasets.load_dataset("allenai/pixmo-point-explanations")

Data Format

Images are stored as URLs.

The in-line points use a format from the LLM/annotators that does not exactly match the Molmo format. The data includes some fields derived from these responses to make them easier to parse, these fields can be null if the original response was not parsed.

• parsed_response responses with the text "<|POINT|>" where the inline point annotations were
• alt_text the alt text for each point annotation in the response
• inline_text the inline text for each point annotation in the response
• points the list-of-list of points for each point annotation

Checking Image Hashes

Image hashes are included to support double-checking that the downloaded image matches the annotated image. It can be checked like this:

from hashlib import sha256
import requests

example = data[0]
image_bytes = requests.get(example["image_url"]).content
byte_hash = sha256(image_bytes).hexdigest()
assert byte_hash == example["image_sha256"]

License

This dataset is licensed under ODC-BY-1.0. It is intended for research and educational use in accordance with Ai2's Responsible Use Guidelines. This dataset includes data generated from Claude which are subject to Anthropic terms of service and usage policy.