SwallowMath

• October 21, 2025: Newer versions are available: SwallowCode-v2 and SwallowMath-v2 have been released with improved rewriting pipelines.

Resources

• 🐙 GitHub: Explore the project repository, including pipeline code and prompts at rioyokotalab/swallow-code-math.
• 📑 arXiv: Read our paper for detailed methodology and results at arXiv:2505.02881.
• 🤗 Sister Dataset: Discover SwallowCode, our companion dataset for code generation.

What is it?

SwallowMath is a high-quality mathematical dataset comprising approximately 2.3 billion tokens derived from the FineMath-4+ dataset through an LLM-driven rewriting pipeline. Using Llama-3.3-70B-Instruct, we transform the original dataset by removing boilerplate, restoring missing context, and reformatting solutions into concise, step-by-step explanations. The pipeline prioritizes educational clarity and mathematical reasoning, making SwallowMath ideal for training large language models (LLMs) for mathematical tasks.

More details are available in our paper: https://arxiv.org/abs/2505.02881.

What is being released?

The dataset is released as:

SwallowMath: Approximately 2.3 billion tokens, derived from FineMath-4+ (9.6 billion tokens, 6.7M documents), containing rewritten mathematical content with concise, step-by-step explanations formatted in Markdown and LaTeX. All data is publicly available under the Llama 3.3 Community License.

Dataset curation

SwallowMath builds on FineMath-4+, a high-quality subset of mathematical content filtered from CommonCrawl. We enhance this dataset through an LLM-driven rewriting pipeline tailored for mathematical reasoning, addressing limitations such as boilerplate, missing context, and verbose explanations.

Rewriting Pipeline

Using Llama-3.3-70B-Instruct, the pipeline performs the following steps:

1. Remove Boilerplate: Eliminates residual web headers, footers, privacy notices, and extraneous metadata (e.g., question/answer timestamps).
2. Restore Context: Fills in missing information in incomplete questions or answers to ensure clarity and completeness.
3. Rewrite Explanations: Reformats solutions into concise, comprehensive, step-by-step explanations, enhancing educational value.

The full rewriting prompt is available at https://github.com/rioyokotalab/swallow-code-math.

Decontamination

Results and Performance

Continual pre-training of Llama-3.1-8B for approximately 50 billion tokens, substituting FineMath-4+ with SwallowMath in the math subset (4.79% of the mixture), yields significant improvements:

• GSM8K: +12.4 points accuracy.
• MATH: +7.6 points accuracy. These gains demonstrate SwallowMath’s superior quality for training models in mathematical reasoning. Detailed results are available in the paper (Tables 19 and 20).

Considerations for Using the Data

Social Impact of the Dataset

SwallowMath aims to democratize access to high-quality mathematical training data, fostering advancements in LLM mathematical reasoning. By releasing an openly licensed dataset, we:

• Enhance transparency in the dataset improvement pipeline.
• Lower barriers for training mathematically proficient models.

Discussion of Biases

The dataset may inherit biases from FineMath-4+, including:

• Focus on English-language content.
• Potential over-representation of certain problem types (e.g., algebra vs. geometry).
• Influence of Llama-3.3-70B-Instruct’s preferences in solution style and formatting.

Licensing Information

SwallowMath is released under the Llama 3.3 Community License. Usage is also subject to CommonCrawl's Terms of Use.

Future work

Potential directions include:

• Expanding to non-English mathematical content.
• Exploring larger pre-training budgets to assess scalability.

Citation information

@misc{fujii2025rewritingpretrainingdataboosts,
      title={Rewriting Pre-Training Data Boosts LLM Performance in Math and Code}, 
      author={Kazuki Fujii and Yukito Tajima and Sakae Mizuki and Hinari Shimada and Taihei Shiotani and Koshiro Saito and Masanari Ohi and Masaki Kawamura and Taishi Nakamura and Takumi Okamoto and Shigeki Ishida and Kakeru Hattori and Youmi Ma and Hiroya Takamura and Rio Yokota and Naoaki Okazaki},
      year={2025},
      eprint={2505.02881},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2505.02881}, 
}