This dataset is also discussed in the survey paper A Survey of Deep Learning for Geometry Problem Solving. The accompanying reading list/code for the survey can be found at: https://github.com/majianz/gps-survey
InfiMM-WebMath-40B is a large-scale, open-source multimodal dataset specifically designed for mathematical reasoning tasks. It incorporates both text and images, extracted from web documents, to advance the pre-training of Multimodal Large Language Models (MLLMs). The dataset is tailored to support sophisticated reasoning tasks that involve understanding both text and visual elements like diagrams, figures, and geometric plots.
The InfiMM-WebMath-40B dataset includes:
These documents were sourced from Common Crawl data snapshots (2019–2023), filtered to focus on high-quality mathematical and scientific content in both English and Chinese.
The dataset is organized in a format that captures both text and images in their original order, ensuring accurate interleaving between the two modalities. The structure is as follows:
{
"URL": "...", # The URL of the source document.
"text_list": [...], # List of extracted text segments, None if the element is an image.
"image_list": [...], # List of image URLs, None if the element is a text segment.
"metadata": {...} # Metadata containing information about the extraction process (e.g., processing details, timestamps).
"metadata": { # Metadata containing information about the extraction process (e.g., processing details, timestamps).
"ft_lang_label", # Type of languages detected by fastText
"ft_lang_prob", # Probability of type of language detected by fastText
"math_prob", # First round math content detection with high recal FastText model
"size",
"snap", # Timestamp of Common Crawl snapshot
"text_gpt3_token_len",
"char_repetition_ratio",
"word_repetition_ratio",
"special_character_ratio",
"punctuation_ratio",
"nsfw_num_words", # Number of words which are NSFW
"has_unicode_error", # If there's any unicode error exists
"math_prob_llama3", # Probability of second round math detection with high precision FastText model
}
}
The text_list and image_list are designed as parallel arrays, maintaining the sequence of the document. This interleaving structure allows models to reconstruct the flow of the original document:
text_list[i] contains text, then image_list[i] is None, indicating that the content at this position is text.text_list[i] is None, then image_list[i] contains a URL to an image at that position in the document.This interleaving of text and images ensures that models trained on this dataset can process the content in the same way a human would, following the logical flow between text explanations and accompanying visual aids.
The InfiMM-WebMath-40B dataset was created through a comprehensive multi-stage filtering and extraction process, starting with over 120 billion web pages from the Common Crawl repository. The key steps in this pipeline are outlined below::
If you want more data with more precision, you can always use higher thresholds with math_prob and math_prob_llama3 fields in metadata.
InfiMM-WebMath-40B is made available under an ODC-By 1.0 license; users should also abide by the CommonCrawl ToU: https://commoncrawl.org/terms-of-use/. We do not alter the license of any of the underlying data.
@misc{han2024infimmwebmath40badvancingmultimodalpretraining,
title={InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning},
author={Xiaotian Han and Yiren Jian and Xuefeng Hu and Haogeng Liu and Yiqi Wang and Qihang Fan and Yuang Ai and Huaibo Huang and Ran He and Zhenheng Yang and Quanzeng You},
year={2024},
eprint={2409.12568},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2409.12568},
}
11 commits
2 commits
This dataset is also discussed in the survey paper A Survey of Deep Learning for Geometry Problem Solving. The accompanying reading list/code for the survey can be found at: https://github.com/majianz/gps-survey
InfiMM-WebMath-40B is a large-scale, open-source multimodal dataset specifically designed for mathematical reasoning tasks. It incorporates both text and images, extracted from web documents, to advance the pre-training of Multimodal Large Language Models (MLLMs). The dataset is tailored to support sophisticated reasoning tasks that involve understanding both text and visual elements like diagrams, figures, and geometric plots.
The InfiMM-WebMath-40B dataset includes:
These documents were sourced from Common Crawl data snapshots (2019–2023), filtered to focus on high-quality mathematical and scientific content in both English and Chinese.
The dataset is organized in a format that captures both text and images in their original order, ensuring accurate interleaving between the two modalities. The structure is as follows:
{
"URL": "...", # The URL of the source document.
"text_list": [...], # List of extracted text segments, None if the element is an image.
"image_list": [...], # List of image URLs, None if the element is a text segment.
"metadata": {...} # Metadata containing information about the extraction process (e.g., processing details, timestamps).
"metadata": { # Metadata containing information about the extraction process (e.g., processing details, timestamps).
"ft_lang_label", # Type of languages detected by fastText
"ft_lang_prob", # Probability of type of language detected by fastText
"math_prob", # First round math content detection with high recal FastText model
"size",
"snap", # Timestamp of Common Crawl snapshot
"text_gpt3_token_len",
"char_repetition_ratio",
"word_repetition_ratio",
"special_character_ratio",
"punctuation_ratio",
"nsfw_num_words", # Number of words which are NSFW
"has_unicode_error", # If there's any unicode error exists
"math_prob_llama3", # Probability of second round math detection with high precision FastText model
}
}
The text_list and image_list are designed as parallel arrays, maintaining the sequence of the document. This interleaving structure allows models to reconstruct the flow of the original document:
text_list[i] contains text, then image_list[i] is None, indicating that the content at this position is text.text_list[i] is None, then image_list[i] contains a URL to an image at that position in the document.This interleaving of text and images ensures that models trained on this dataset can process the content in the same way a human would, following the logical flow between text explanations and accompanying visual aids.
The InfiMM-WebMath-40B dataset was created through a comprehensive multi-stage filtering and extraction process, starting with over 120 billion web pages from the Common Crawl repository. The key steps in this pipeline are outlined below::
If you want more data with more precision, you can always use higher thresholds with math_prob and math_prob_llama3 fields in metadata.
InfiMM-WebMath-40B is made available under an ODC-By 1.0 license; users should also abide by the CommonCrawl ToU: https://commoncrawl.org/terms-of-use/. We do not alter the license of any of the underlying data.
@misc{han2024infimmwebmath40badvancingmultimodalpretraining,
title={InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning},
author={Xiaotian Han and Yiren Jian and Xuefeng Hu and Haogeng Liu and Yiqi Wang and Qihang Fan and Yuang Ai and Huaibo Huang and Ran He and Zhenheng Yang and Quanzeng You},
year={2024},
eprint={2409.12568},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2409.12568},
}
11 commits
2 commits