🐳 OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text

⭐️ NOTE: Several parquet files were marked unsafe (viruses) by official scaning of hf, while they are reported safe by ClamAV and Virustotal. We found many false positive cases of the hf automatic scanning in hf discussions and raise one discussion to ask for a re-scanning.

This is the repository of OmniCorpus-CC, which contains 988 million image-text interleaved documents collected from Common Crawl.

• Repository: https://github.com/OpenGVLab/OmniCorpus
• Paper (ICLR 2025 Spotlight): https://arxiv.org/abs/2406.08418

OmniCorpus dataset is a large-scale image-text interleaved dataset, which pushes the boundaries of scale and diversity by encompassing 8.6 billion images interleaved with 1,696 text tokens from diverse sources, significantly surpassing previous datasets. This dataset demonstrates several advantages over its counterparts:

1. Larger data scale: Our dataset is 1.7 times larger in images and 12.5 times larger in texts compared to the previously largest multimodal dataset, LAION-5B, while maintaining excellent data quality.
2. Richer data diversity: Drawing from a broader range of data sources, our dataset is more diverse than other image-text interleaved datasets. It includes bilingual multimodal data in both Chinese and English, and encompasses text-centric and vision-centric documents extracted from common websites and video platforms.
3. More flexible format: The streaming data format of our dataset offers exceptional flexibility, allowing adaptation to various data structures, including pure text corpora, image-text pairs, and interleaved data formats.

The OmniCorpus contains three sections:

• OmniCorpus-CC: processed from dumps in Common Crawl from 2013 to Nov./Dec. 2023.
• OmniCorpus-CW: sourced from Chinese internet resources, will be availiable in OpenDataLab platform.
• OmniCorpus-YT: samples Youtube video frames as images and collects subtitles as texts.

Code for pre-training, evaluating, main body extracting, and filtering have been released in the official repository. A pre-trained model is availiable here.

Data Pipeline

Our data pipeline consists of five key stages: main body extraction, preliminary text filtering, document deduplication, image downloading & filtering, and detailed text filtering. Each stage efficiently reduces the dataset to retain only high-quality data. Please refer to our paper for more details about the data pipeline.

Usages

The image-text interleaved documents are recommanded for the following usages:

• Pre-training multimodal large language model (MLLM): Recent MLLMs (such as Flamingo series, EMU series, IDEFICS series, MM1, Cambrian-1, and xGen-MM) have shown that image-text interleaved data aids multimodal in-context learning and maintains the capabilities of large language models during multimodal fine-tuning.
• Long text-image retrieval: We provide image-text similarities calculated with CLIP, which can convert the documents to image-text retrieval dataset with longer text. A retrieval model pre-trained on such data can retrieval images based on longer text, which can be used for multimodal RAG, converting pure text to multimodal sample, etc.
• Source for futher dataset research: Our data is large-scale, which can serve as the source for researches for data curation strategies. We provide many useful attributes as metadata for each document, which can enrich the filtering strategy and reduce the cost.
• ......

Data Format

Following common practices, the data is organized into Parquet file format. You might encounter errors when using pandas.read_parquet (because the data structure contains nested elements). We recommend using fastparquet to load the parquet files.

import fastparquet
df = fastparquet.ParquetFile(parquet_file_path).to_pandas()

# You can also use iter_batches
parquet_file = pq.ParquetFile(filepath)
for batch in parquet_file.iter_batches():
    df = batch.to_pandas()

You can convert the i-th document and convert it into a dictionary.

doc_dict = df.iloc[i].to_dict()

The document format is as follow:

{
    'images': [
        <str: image_1_url>,
        None,
        <str: image_2_url>,
        None,
    ],
    'texts': [
        None,
        <str: text_paragraph_1_content>
        None,
        <str: text_paragraph_2_content>,
    ]
    'metadata': [
        <dict: image_1_metadata>,
        None,
        <dict: image_2_metadata>,
        None
    ],
    'general_metadata': {
        "url": <str: document url>,
        "id": <str: document id>,
        "domain": <list[str]: domains extracted from document url>,
        "fluency_prob": <float: the probability of fluency>,
        "non_advertisement_prob": <float: the probability of non-advertisement>,
        "porn_prob": <float: the probability of porn content>,
        "politics_prob": <float: the probability of politics content>,
        "toxic_prob": <float: the probability of toxic content>,
    }
}

Each image metadata is as follow:

{
    "img_url_sha": <str: sha code of image url>,
    "width": <int: image width>,
    "height": <int: image height>,
    "bytes": <int: byte number of the image file>,
    "d_hash": <str: d_hash code of the image, used for image deduplication>,
    "p_hash": <str: p_hash code of the image, used for image deduplication>,
    "d_hash_dup_count": <int: duplicated times detected by d_hash code>,
    "p_hash_dup_count": <int: duplicated times detected by p_hash code>,
    "aesthetic prob": <float: aesthetic probility>,
    "unsafe prob": <float: NSFW probility>, 
}

License and Terms of Use

The OmniCorpus dataset is distributed under the CC BY 4.0 License. The open-source code is released under the Apache License 2.0.

The Terms of Use (ToUs) have been developed based on widely accepted standards. By accessing or using this dataset, users acknowledge their responsibility to comply with all relevant legal, regulatory, and ethical standards.

• All users, whether from academia or industry, must comply with the ToUs outlined in the CC BY 4.0 License.
• Any derived datasets or models must acknowledge the use of the OmniCorpus dataset to maintain transparency.
• The OmniCorpus must not be used in any project involving sensitive content or harmful outcomes, including but not limited to political manipulation, hate speech generation, misinformation propagation, or tasks that perpetuate harmful stereotypes or biases.
• The use of this dataset in any manner that violates rights, such as copyright infringement, privacy breaches, or misuse of sensitive information, is strictly prohibited.
• While we do not enforce jurisdiction-specific terms, we strongly recommend that users ensure compliance with applicable local laws and regulations.
• The use of specific subset must comply with the ToUs of the primary source. Specifically, the use of OmniCorpus-CC, OmniCorpus-CW, and OmniCorpus-YT must comply with the Common Crawl ToUs, the regulations on the security management of Internet data in China, and YouTube’s ToUs, respectively.
• These ToUs do not supersede the ToUs of the original content sources. Users must ensure that any use of the dataset’s content complies with the original ToUs and the rights of the data subjects.

Citation

@inproceedings{li2024omnicorpus,
  title={OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text},
  author={Li, Qingyun and Chen, Zhe and Wang, Weiyun and Wang, Wenhai and Ye, Shenglong and Jin, Zhenjiang and others},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025}
}