Cambrian-10M Dataset

Please see paper & website for more information:

• https://cambrian-mllm.github.io/
• https://arxiv.org/abs/2406.16860

Overview

Cambrian-10M is a comprehensive dataset designed for instruction tuning, particularly in multimodal settings involving visual interaction data. The dataset is crafted to address the scarcity of high-quality multimodal instruction-tuning data and to maintain the language abilities of multimodal large language models (LLMs).

Data Collection

Multimodal Data Sources

Unlike language data, multimodal instruction-tuning data is much rarer and harder to collect. To address this, we leverage existing multimodal benchmarks and datasets involving visual interaction data, such as Visual Question Answering (VQA) and Optical Character Recognition (OCR) data. This approach helps mitigate the catastrophic forgetting commonly observed when fine-tuning multimodal LLMs.

Language-Only Instruction-Following Data

To ensure the preservation of language capabilities, we also collect a small volume of high-quality language-only instruction-following data from the community.

Targeted Internet Data Collection Engine

We introduce a data engine designed to create large-scale, reliable, high-quality knowledge-based multimodal instruction tuning data. The engine works as follows:

1. Field and Subfield Selection: The engine selects a target field and subfield, such as “Physics”.
2. Topic Identification: An LLM like GPT-4 identifies topics within the field (e.g., “Newton’s Laws”).
3. Reliable Source Search: The engine searches reliable sources like Wikipedia for each topic.
4. Text-Image Association Extraction: The parser extracts image-caption-text tuples from the sources.
5. Q&A Pair Generation: The caption-text is fed to an LLM, such as GPT-3.5, to generate instruction-type Q&A pairs about the image.

These Q&A pairs, along with the images, form our VQA dataset.

GPT Rewriting

We also incorporate recent MLLMs such as GPT-4v and GPT-4o to generate extended responses and free-form instruction tuning data. To play with gpt generated data, use gpt4v_77k, Curated gpt4o_60k

• gpt4v_77k contains more extended responses from Cambrian-10M.
• gpt4o_60k contains more creative data in visual interactions.

Cambrian-10M Composition

The Cambrian-10M dataset consists of approximately 9.784 million data points, offering a diverse range of data for various research applications. The composition of the dataset is visualized in Fig. 9.

Cambrian-7M

We make an initial effort to study data curation. In particular, we find the following data ratio to perform most optimally

• Language: 21.00%
• General: 34.52%
• OCR: 27.22%
• Counting: 8.71%
• Math: 7.20%
• Code: 0.87%
• Science: 0.88%

Getting Started with Cambrian Data

Before you start, ensure you have sufficient storage space to download and process the data.

Cambrian-10M contains a total of 10 million images collected from previous datasets, an internet data engine, and GPT-generated instruction tuning data. Follow these steps to get started:

1. Download the Data Repository

   Download the data repository. Note that due to Hugging Face policy constraints, the data folder is archived into tar files. We also split the allava and data_engine data into smaller tar files because they exceed the 50 GB size limit.

2. Merge Tar Files

   To explore the Cambrian-10M dataset, first merge the different parts of allava and data_engine together:

   python merge_tars.py
   

3. Extract Tar Files

   Then, extract all the tar files into the current directory:

   python extract.py
   

4. Training with Cambrian

You can train with the raw Cambrian10M, Curated Cambrian7M. We recommend using the Curated Cambrian7M with system prompt that also alleviates 'answer machine' problem.