PixMo-Points

PixMo-Points is a dataset of images paired with referring expressions and points marking the locations the referring expression refers to in the image. It was collected using human annotators and contains a diverse range of points and expressions, with many high-frequency (10+) expressions.

PixMo-Points is a part of the PixMo dataset collection and was used to provide the pointing capabilities of the Molmo family of models

Quick links:

• 📃 Paper
• 🎥 Blog with Videos

Loading

data = datasets.load_dataset("allenai/pixmo-points", split="train")

Data Format

Images are stored as URLs that will need to be downloaded separately. Note URLs can be repeated in the data.

The points field contains the x, y coordinates specified in pixels.

The label field contains the string name of what is being pointed at, this can be a simple object name or a more complex referring expression.

The collection_method field specifies whether the image was chosen to target high-frequency counting ("counting") or general pointing ("pointing").

Image Checking

Image hashes are included to support double-checking that the downloaded image matches the annotated image. It can be checked like this:

from hashlib import sha256
import requests

example = data[0]
image_bytes = requests.get(example["image_url"]).content
byte_hash = sha256(image_bytes).hexdigest()
assert byte_hash == example["image_sha256"]

License

This dataset is licensed under ODC-BY-1.0. It is intended for research and educational use in accordance with Ai2's Responsible Use Guidelines.