PixMo-Cap

PixMo-Cap is a dataset of very long (roughly 200 words on average), detailed captions. It can be used to pre-train and fine-tune vision-language models. PixMo-Cap was created by recording annotators speaking about an image for 60-90 seconds and then using the Claude large language model to turn the audio transcripts(s) into a long caption. The audio transcripts are also included.

PixMo-Cap is part of the PixMo dataset collection and was used to train the Molmo family of models

Quick links:

• 📃 Paper
• 🎥 Blog with Videos

Loading

data = datasets.load_dataset("allenai/pixmo-cap", split="train")

Data Format

Images are stored as URLs that will need to be downloaded separately.

The transcripts fields contains one or more audio transcripts

The caption field contains the caption from the LLM.

License

This dataset is licensed by ODC-BY-1.0. It is intended for research and educational use in accordance with Ai2's Responsible Use Guidelines. This dataset includes output data generated from Claude which are subject to Anthropic terms of service and usage policy.