SwallowCode

Notice

• May 21, 2025: We have deleted ablation/exp1-the-stack-v2-train-smol-ids-python because it was flagged as potentially containing unsafe data collected from the Python subset of https://huggingface.co/datasets/bigcode/the-stack-v2-train-smol-ids. However, since this dataset can be reconstructed from the-stack-v2-train-smol-ids, there is no issue in terms of reproducibility.
• May 21, 2025: ClamAV has flagged “Win.Trojan.MSShellcode-88” in ablation/exp10-direct-sgcr/jsonl/train-00005-of-00005.jsonl (a dataset directly rewritten from the-stack-v2-train-smol-ids). While loading it as JSONL for LLM training poses an extremely low risk, please be aware.
• October 21, 2025: Newer versions are available: SwallowCode-v2 and SwallowMath-v2 have been released with improved rewriting pipelines.

Resources

• 🐙 GitHub: Explore the project repository, including pipeline code and prompts at rioyokotalab/swallow-code-math.
• 📑 arXiv: Read our paper for detailed methodology and results at arXiv:2505.02881.
• 🤗 Sister Dataset: Discover SwallowMath, our companion dataset for mathematical reasoning.

What is it?

💻 SwallowCode is a high-quality code dataset comprising approximately 16.1 billion tokens of Python code, derived from The-Stack-v2-train-smol-ids through a four-stage pipeline:syntax validation, pylint-based style filtering, and a two-stage LLM rewriting process using Llama-3.3-70B-Instruct. The pipeline enforces style conformity (Style-Guided Code Rewriting, SGCR) and transforms snippets into self-contained, algorithmically efficient examples (Self-Contained Optimization Rewriting, SCOR). SwallowCode is designed to enhance large language model (LLM) performance in program synthesis and code generation.

More details are available in our paper: https://arxiv.org/abs/2505.02881.

What is being released?

The dataset is released as:

SwallowCode: Approximately 16.1 billion tokens of Python code, processed through syntax validation, pylint filtering, SGCR, and SCOR, formatted as JSONL files. (ablation/exp11-scor/jsonl)

Additionally, intermediate datasets from ablation experiments are released in the ablation/ directory.

All data is publicly available under the Llama 3.3 Community License.

Dataset curation

SwallowCode refines Python code from The-Stack-v2-train-smol-ids through a four-stage pipeline to eliminate noise, ensure stylistic consistency, and enhance semantic quality.

Pipeline Overview

1. Syntax Error Filtering: Removes invalid Python code using the compile() function, reducing samples by 9.7% (from 41M to 37M).
2. Linter-Based Filtering: Applies pylint with a threshold score of 7.0 and a custom comment penalty heuristic, reducing samples by 34.3% (to 24.1M).
3. Style-Guided Code Rewriting (SGCR): Uses Llama-3.3-70B-Instruct to enforce Google Python Style Guide criteria, improving readability and consistency.
4. Self-Contained Optimization Rewriting (SCOR): Ensures self-containment, optimizes algorithms, and transforms trivial snippets into educational examples.

The full pipeline and prompts are available at https://github.com/rioyokotalab/swallow-code-math.

Ablation Experiments

The ablation/ directory contains JSONL files for intermediate datasets from ablation experiments. These datasets correspond to the experiments described in the paper:

• exp1-the-stack-v2-train-smol-ids-python: Baseline Python subset from The-Stack-v2.
• exp2-syntax-error-filtered: After syntax error filtering.
• exp3-linter-filtered: After pylint-based filtering (score ≥ 7).
• exp4-code_comment_ja_or_en: Restricted to English or Japanese comments.
• exp5-sgcr: After SGCR rewriting.
• exp6-llm-based-scoring: Filtered by LLM-based scoring (score ≥ 6).
• exp7: Mixed data (1:1 ratio of exp3 and exp5).
• exp10-direct-sgcr: Python subset from The-Stack-v2 (exp1) with direct SGCR applied, skipping syntax error and pylint-based filtering.
• exp11-scor: Final SwallowCode dataset after SCOR rewriting.

Each directory contains JSONL files with processed code samples. For details, see the paper’s Appendix (Tables 6–18) or the repository at https://github.com/rioyokotalab/swallow-code-math.

Results and Performance

Continual pre-training of Llama-3.1-8B for approximately 50 billion tokens, with SwallowCode as the code subset (16% of the mixture), yields significant improvements:

• HumanEval: +17.0 points pass@1 compared to Stack-Edu.
• HumanEval+: +17.7 points pass@1 compared to Stack-Edu. SwallowCode outperforms other datasets (e.g., CodeParrot-Clean, The-Stack-v1/v2, Stack-Edu) on code generation benchmarks, as shown in the paper (Figure 1, Tables 6–18).

Considerations for Using the Data

Social Impact of the Dataset

SwallowCode aims to advance LLM capabilities in code generation by providing a high-quality, openly licensed dataset. We:

• Promote transparency in code dataset curation.
• Reduce barriers for training code-proficient models.
• Offer a benchmark for code quality enhancement.

Discussion of Biases

The dataset may inherit biases from The-Stack-v2 or Llama-3.3-70B-Instruct, including:

• Over-representation of certain coding patterns (e.g., dynamic programming (DP)).
• Influence of Llama-3.3-70B-Instruct’s preferences in naming and style.

Other Known Limitations

• Limited to Python code in current experiments.
• May not capture all edge cases in code dependencies.
• Rewriting may introduce model-specific stylistic preferences.

Licensing Information

SwallowCode is released under the Llama 3.3 Community License.

Usage is subject to The-Stack-v2’s licensing terms.

Future work

Potential directions include:

• Extending the pipeline to other programming languages (e.g., Java, C++).
• Enhancing dependency handling for complex codebases.
• Exploring larger pre-training budgets to assess scalability.
• Integrating multilingual code comments beyond English and Japanese.

Citation information

@misc{fujii2025rewritingpretrainingdataboosts,
      title={Rewriting Pre-Training Data Boosts LLM Performance in Math and Code}, 
      author={Kazuki Fujii and Yukito Tajima and Sakae Mizuki and Hinari Shimada and Taihei Shiotani and Koshiro Saito and Masanari Ohi and Masaki Kawamura and Taishi Nakamura and Takumi Okamoto and Shigeki Ishida and Kakeru Hattori and Youmi Ma and Hiroya Takamura and Rio Yokota and Naoaki Okazaki},
      year={2025},
      eprint={2505.02881},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2505.02881}, 
}