PixMo-CapQA

PixMo-CapQA is a synthetic dataset of question/answer pairs about images. The data was generated by using the Claude large language model to build Q/A pairs from dense captions of images (the model did not see the actual images).

PixMo-CapQA is a part of the PixMo dataset collection and was used to train the Molmo family of models

Quick links:

• 📃 Paper
• 🎥 Blog with Videos

Loading

data = datasets.load_dataset("allenai/pixmo-cap-qa", split="train")

Data Format

Images are stored as URLs that will need to be downloaded separately. The image URLs can be repeated since many of the images have multiple Q/A pairs.

• The question field contains the input text, it includes "[USER]" and "[ASSISTANT]" tags
• The answer field contains the final target output text
• The messages field contains the same data in a list-of-messages formats. The first message is from the user, then messages alternative between user and assistant. This text does not contain "[USER]" and "[ASSISTANT]" tags

License

This dataset is licensed under ODC-BY-1.0. It is intended for research and educational use in accordance with Ai2's Responsible Use Guidelines. This dataset includes data generated from Claude which are subject to Anthropic terms of service and usage policy.