
📐 FineMath consists of 34B tokens (FineMath-3+) and 54B tokens (FineMath-3+ with InfiMM-WebMath-3+) of mathematical educational content filtered from CommonCrawl. To curate this dataset, we trained a mathematical content classifier using annotations generated by LLama-3.1-70B-Instruct. We used the classifier to retain only the most educational mathematics content, focusing on clear explanations and step-by-step problem solving rather than advanced academic papers.
The Dataset Curation section details the process for creating the dataset. More details in our paper: https://arxiv.org/abs/2502.02737v1.
The dataset is released in two versions:
We also release a filtered English text-only portion of the InfiMM-WebMath-40B dataset, classified using the same approach as FineMath:
Use one of the available configs: finemath-3plus, finemath-4plus, infiwebmath-3plus, or infiwebmath-4plus.
from datasets import load_dataset
# Load the high-quality subset
data = load_dataset("HuggingFaceTB/finemath", "finemath-4plus", split="train", num_proc=8)
# Or load the larger subset
data = load_dataset("HuggingFaceTB/finemath", "finemath-3plus", split="train", num_proc=8)
Recent language models like DeepSeekMath and MathStral have demonstrated strong mathematical capabilities, trained on specialized datasets that aren't publicly available. We developed a pipeline to identify and extract high-quality mathematical content from CommonCrawl, with several iterations of refinement to improve quality.
We began by re-extracting pages from CommonCrawl WARCs using URLs from the FineWeb dataset, collecting both the latest and largest versions of each page to capture the evolution of pages across the years. Unlike FineWeb which uses Trafilatura, we employed Resiliparse for text extraction as it better preserves forum discussions and QA answers that often contain crucial reasoning steps and solutions.
For initial quality assessment, we used Llama-3.1-70B-Instruct to generate annotations on a 3-point scale:
A multilingual-e5-small-based classifier finetuned on these annotations was used to score the initial corpus.
However, this first version performed below the OpenWebMath baseline, leading to several important refinements.
Analysis revealed that FineWeb's C4 filter removes pages containing '{' characters, inadvertently filtering out content with LaTeX notation. To address this and expand coverage, we:
The expanded corpus underwent a more fine-grained quality evaluation:
Once again, we used LLama-3.1-70B-Instruct to score a sample of newly extracted pages on a 5-point scale (full prompt available in here): We finetuned a new classifier on these annotations and scored the entire corpus. After leaving only pages with a score of 3 or higher, and deduplicating the samples using simple single-band MinHash-LSH, we obtained FineMath-3+ with 34B tokens.
The same classifier was applied to InfiMM-WebMath's text content, focusing more on reasoning rather than advanced mathematics.
Both datasets were additionally filtered using FineWeb's language classification pipeline to remove non-English content.
Following Qwen2.5-Math's approach, we removed samples with 13-gram overlaps against test sets from GSM8k, MATH, MMLU and ARC. Decontamination logs are available at HuggingFaceTB/finemath_contamination_report.
Our evaluations show several key findings:
{
'url': string, # Source page URL
'fetch_time': int64, # Crawler timestamp
'content_mime_type': string, # MIME type
'warc_filename': string, # Common Crawl WARC source file
'warc_record_offset': int32, # WARC record offset, in bytes
'warc_record_length': int32, # WARC record size, in bytes
'text': string, # Page content
'token_count': int32, # Number of Llama tokens
'char_count': int32, # Character count
'metadata': string, # Additional OpenWebMath metadata
'score': float64, # Raw quality score
'int_score': int64, # Integer quality score
'crawl': string, # Common Crawl crawl identifier
'snapshot_type': string, # Whether the page is the latest or the largest for this URL
'language': string, # Document language
'language_score': float64 # LangID probability
}
With the release of this dataset, we aim to make high-quality mathematical educational content more accessible to the machine learning community. While multiple language models have demonstrated strong mathematical capabilities, the datasets used to train these capabilities are often not publicly available. By releasing FineMath, we hope to:
The dataset may have certain inherent biases:
The dataset is released under the Open Data Commons Attribution License (ODC-By) v1.0 license. The use of this dataset is also subject to CommonCrawl's Terms of Use.
There are several avenues for future work:
@misc{allal2025smollm2smolgoesbig,
title={SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model},
author={Loubna Ben Allal and Anton Lozhkov and Elie Bakouch and Gabriel Martín Blázquez and Guilherme Penedo and Lewis Tunstall and Andrés Marafioti and Hynek Kydlíček and Agustín Piqueres Lajarín and Vaibhav Srivastav and Joshua Lochner and Caleb Fahlgren and Xuan-Son Nguyen and Clémentine Fourrier and Ben Burtenshaw and Hugo Larcher and Haojun Zhao and Cyril Zakka and Mathieu Morlon and Colin Raffel and Leandro von Werra and Thomas Wolf},
year={2025},
eprint={2502.02737},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.02737},
}

📐 FineMath consists of 34B tokens (FineMath-3+) and 54B tokens (FineMath-3+ with InfiMM-WebMath-3+) of mathematical educational content filtered from CommonCrawl. To curate this dataset, we trained a mathematical content classifier using annotations generated by LLama-3.1-70B-Instruct. We used the classifier to retain only the most educational mathematics content, focusing on clear explanations and step-by-step problem solving rather than advanced academic papers.
The Dataset Curation section details the process for creating the dataset. More details in our paper: https://arxiv.org/abs/2502.02737v1.
The dataset is released in two versions:
We also release a filtered English text-only portion of the InfiMM-WebMath-40B dataset, classified using the same approach as FineMath:
Use one of the available configs: finemath-3plus, finemath-4plus, infiwebmath-3plus, or infiwebmath-4plus.
from datasets import load_dataset
# Load the high-quality subset
data = load_dataset("HuggingFaceTB/finemath", "finemath-4plus", split="train", num_proc=8)
# Or load the larger subset
data = load_dataset("HuggingFaceTB/finemath", "finemath-3plus", split="train", num_proc=8)
Recent language models like DeepSeekMath and MathStral have demonstrated strong mathematical capabilities, trained on specialized datasets that aren't publicly available. We developed a pipeline to identify and extract high-quality mathematical content from CommonCrawl, with several iterations of refinement to improve quality.
We began by re-extracting pages from CommonCrawl WARCs using URLs from the FineWeb dataset, collecting both the latest and largest versions of each page to capture the evolution of pages across the years. Unlike FineWeb which uses Trafilatura, we employed Resiliparse for text extraction as it better preserves forum discussions and QA answers that often contain crucial reasoning steps and solutions.
For initial quality assessment, we used Llama-3.1-70B-Instruct to generate annotations on a 3-point scale:
A multilingual-e5-small-based classifier finetuned on these annotations was used to score the initial corpus.
However, this first version performed below the OpenWebMath baseline, leading to several important refinements.
Analysis revealed that FineWeb's C4 filter removes pages containing '{' characters, inadvertently filtering out content with LaTeX notation. To address this and expand coverage, we:
The expanded corpus underwent a more fine-grained quality evaluation:
Once again, we used LLama-3.1-70B-Instruct to score a sample of newly extracted pages on a 5-point scale (full prompt available in here): We finetuned a new classifier on these annotations and scored the entire corpus. After leaving only pages with a score of 3 or higher, and deduplicating the samples using simple single-band MinHash-LSH, we obtained FineMath-3+ with 34B tokens.
The same classifier was applied to InfiMM-WebMath's text content, focusing more on reasoning rather than advanced mathematics.
Both datasets were additionally filtered using FineWeb's language classification pipeline to remove non-English content.
Following Qwen2.5-Math's approach, we removed samples with 13-gram overlaps against test sets from GSM8k, MATH, MMLU and ARC. Decontamination logs are available at HuggingFaceTB/finemath_contamination_report.
Our evaluations show several key findings:
{
'url': string, # Source page URL
'fetch_time': int64, # Crawler timestamp
'content_mime_type': string, # MIME type
'warc_filename': string, # Common Crawl WARC source file
'warc_record_offset': int32, # WARC record offset, in bytes
'warc_record_length': int32, # WARC record size, in bytes
'text': string, # Page content
'token_count': int32, # Number of Llama tokens
'char_count': int32, # Character count
'metadata': string, # Additional OpenWebMath metadata
'score': float64, # Raw quality score
'int_score': int64, # Integer quality score
'crawl': string, # Common Crawl crawl identifier
'snapshot_type': string, # Whether the page is the latest or the largest for this URL
'language': string, # Document language
'language_score': float64 # LangID probability
}
With the release of this dataset, we aim to make high-quality mathematical educational content more accessible to the machine learning community. While multiple language models have demonstrated strong mathematical capabilities, the datasets used to train these capabilities are often not publicly available. By releasing FineMath, we hope to:
The dataset may have certain inherent biases:
The dataset is released under the Open Data Commons Attribution License (ODC-By) v1.0 license. The use of this dataset is also subject to CommonCrawl's Terms of Use.
There are several avenues for future work:
@misc{allal2025smollm2smolgoesbig,
title={SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model},
author={Loubna Ben Allal and Anton Lozhkov and Elie Bakouch and Gabriel Martín Blázquez and Guilherme Penedo and Lewis Tunstall and Andrés Marafioti and Hynek Kydlíček and Agustín Piqueres Lajarín and Vaibhav Srivastav and Joshua Lochner and Caleb Fahlgren and Xuan-Son Nguyen and Clémentine Fourrier and Ben Burtenshaw and Hugo Larcher and Haojun Zhao and Cyril Zakka and Mathieu Morlon and Colin Raffel and Leandro von Werra and Thomas Wolf},
year={2025},
eprint={2502.02737},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.02737},
}