📑 Paper    |    🔨 fastText Classifier    |    🤗 Released Dataset    |    📦 Repo

PreSelect-100B is a curated ~100B token pretraining dataset that achieves great performance on various benchmarks. It is filtered by PreSelect-Classifier at 10% threshold, where the pool is a randomly sampled subset of DCLM-refinedweb, which is a cleaned version of Common Crawl raw data but without any model-based filtering.

Benchmark results

Trianing using PreSelect curated dataset achieve superior results than other dataset selection methods on various downstream tasks and below are comparisons.

Citation

If you find this work helpful, please kindly cite as:

@article{shum2025predictivedataselectiondata,
      title={Predictive Data Selection: The Data That Predicts Is the Data That Teaches}, 
      author={Kashun Shum and Yuzhen Huang and Hongjian Zou and Ding Qi and Yixuan Liao and Xiaoxin Chen and Qian Liu and Junxian He},
      journal={arXiv preprint arXiv:2503.00808},
      year={2025},
      eprint={2503.00808},
}