Puro-2B Pretraining Data: The Recipe Behind a 2B Model

This is the materialized pretraining data release for Puro-2B-Base, a 2B base model trained from scratch on consumer-grade RTX 5090 GPUs.

The repository contains the component-level data pools used to construct the two Puro-2B pretraining phases, together with the tokenizer used for token accounting. It is organized for inspection, selective streaming, and recipe reconstruction rather than as a small train/test benchmark.

Dataset Summary

Token counts are computed with the bundled tokenizer and measure materialized token exposure, not unique upstream content. Dataset-family rows may aggregate overlapping configurations.

The production model consumed approximately 439B tokens in Phase 1 and 961B additional tokens in Phase 2. The Phase 2 training count is larger than the 938B stationary Phase 2 pool because the opening transition replays Phase 1 data while introducing the Phase 2 stream. Replay is a training-schedule operation, not additional independent source data.

Repository Structure

.
├── phase1/              # Component data used to build the Phase 1 mixture
├── phase2/              # Component data used to build the Phase 2 mixture
├── qwen2_tokenizer/     # Tokenizer files used for token accounting
└── README.md

The release defines two Hugging Face configurations, phase1 and phase2. Each configuration exposes one train split because both phases are pretraining corpora. Directory names such as train, validation, test, or benchmark inside individual source components preserve upstream organization; they are not Puro-2B evaluation splits.

Files are grouped by source component. Their filesystem order is not the final curriculum order used by the production run. The transition, replay, curriculum bucket construction, deterministic seeds, and global uniform reshuffle are recipe operations documented in the technical report and implemented in the data-processing pipeline.

Data Fields

Released Parquet records use a common schema:

Source-specific metadata not represented by these fields should be recovered from the upstream dataset and the Puro data-processing manifests.

Loading the Data

The full repository is hundreds of gigabytes. Streaming or selecting individual components is recommended for exploration.

from datasets import load_dataset

phase1 = load_dataset(
    "thu-pacman/Puro-2B",
    "phase1",
    split="train",
    streaming=True,
)

example = next(iter(phase1))
print(example["source_component_id"], example["token_count"])
print(example["text"][:500])

Load Phase 2 by changing the configuration name:

phase2 = load_dataset(
    "thu-pacman/Puro-2B",
    "phase2",
    split="train",
    streaming=True,
)

For a single component, load its Parquet files directly:

from datasets import load_dataset

open_web_math = load_dataset(
    "parquet",
    data_files={
        "train": (
            "hf://datasets/thu-pacman/Puro-2B/"
            "phase2/open-web-math/*.parquet"
        )
    },
    split="train",
    streaming=True,
)

For reproducible experiments, pin the dataset revision to a commit hash and record the selected component paths, filtering rules, and shuffle seed.

How the Recipe Was Built

The Puro-2B pipeline separates recipe selection from shard materialization:

1. Source acquisition. Candidate English, Chinese, mathematics, code, and instruction-formatted datasets are collected from public sources.
2. Within-source processing. Large web components are deduplicated within source. The project does not claim global cross-source deduplication.
3. Proxy benchmarking. A shared Qwen3-0.6B proxy checkpoint is continued on candidate sources or score slices and evaluated on a fixed 15-benchmark suite. These results guide source and within-source selection.
4. Materialization. Selected revisions, filters, token budgets, mixture weights, and random seeds are frozen into component-level Parquet files.
5. Training order. Phase 2 uses component-local ranks to build an ordered curriculum while approximately preserving the mixture. Scores are never compared numerically across different source datasets.

Because the proxy study has no matched base-mixture-only continuation, its candidate profiles should be interpreted as recipe signals rather than isolated causal effects.

Sources at a Glance

See the technical report source for the component-level token counts, materialization modes, upstream identifiers, proxy protocol, and license audit.

License and Upstream Terms

The dataset card uses the other license designation because the recipe combines components with different upstream terms. Apache 2.0 applies only to project-authored documentation, metadata, manifests, and processing artifacts explicitly released under that license. It does not replace the licenses, copyrights, privacy rights, or terms attached to third-party source content.

Important boundaries include:

• NVIDIA-controlled sources such as Nemotron-CC-v2 and the Synthetic-Code partition of Nemotron-Pretraining-Code-v1 have data terms that prohibit raw redistribution. Their corresponding directories provide reconstruction or provenance notices instead of republished raw samples where required.
• Web and code corpora may retain terms from Common Crawl, websites, authors, or original software repositories even when dataset metadata uses a permissive license.
• Some source cards do not declare a dataset-level license, and some mixtures contain subsets with stricter terms. Absence of a declared license should not be interpreted as permission to redistribute or use the content without restriction.

Users are responsible for checking the upstream terms of the components they download and for determining whether their intended use is permitted. The technical report's component inventory is the starting point for that review.

Known Limitations and Responsible Use

This is large-scale pretraining data collected from heterogeneous public and synthetic sources. It may contain personal information, copyrighted material, incorrect claims, duplicated content, offensive language, security-sensitive code, or social and cultural biases. Filtering reduces some problems but does not eliminate them.

The release does not claim exhaustive benchmark decontamination. Some source families provide decontaminated subsets, but that property should not be generalized to every component. Users should run task-specific contamination, privacy, safety, and licensing checks before training or deploying a model.

Related Artifacts

• Model: thu-pacman/Puro-2B-Base
• Training code: thu-pacman/Puro-Megatron
• Data processing: thu-pacman/Kaiyuan-Spark
• Technical report: pending upload to arXiv.

Citation

Please cite our technical report if you find our work useful:

@misc{luo2026puro2b,
      title={Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090}, 
      author={Kairong Luo and Jiarui Cui and Yaorui Yin and Shengqi Chen and Yiming Yang and Linxiang Gao and Yanmohan Wang and Mingzhe Zhang and Kaiyue Wen and Kaifeng Lyu and Wenguang Chen},
      year={2026},
      eprint={2608.27370},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.27370}, 
}