Ultra-FineWeb-L1

📜 Ultra-FineWeb Technical Report | 📦 UltraData Collection | 🌐 UltraData

English | 中文

📚 Introduction

Ultra-FineWeb-L1 is a large-scale English web corpus built from Common Crawl snapshots. Within UltraData's L0-L4 tiered data management framework, it serves as the L1 filtered layer for general web data and provides the foundation for subsequent L2 selection and L3 refinement. Building on the FineWeb processing pipeline, we perform main-text extraction, language filtering, heuristic filtering, sensitive-field replacement, MinHash deduplication, and customized data cleaning. We upgrade main-text extraction to trafilatura 2.0, and use UltraData's data quality inspection tools and data-cleaning agents to optimize cleaning rules and apply targeted processing for residual HTML, encoding corruption and mojibake, invisible characters, corrupted content, and abnormal document lengths. The first release contains 1T+ tokens (approximately 1.14 billion documents) from six Common Crawl snapshots in 2025. We also release Ultra-FineWeb, the L2 selected subset produced by the Ultra-FineWeb classifier.

• Ultra-FineWeb-L1: L1 filtered data after basic cleaning, heuristic filtering, sensitive-field replacement, and deduplication. (Current dataset)
• Ultra-FineWeb: L2 selected data produced by the Ultra-FineWeb classifier, containing approximately 1T English tokens and 120B Chinese tokens.
• Ultra-FineWeb-L3: L3 refined data built on Ultra-FineWeb via Q&A pair generation and multi-style rewriting, containing 400B+ English tokens and 200B+ Chinese tokens.

📢 What's New

• [2026.08.20] The Ultra-FineWeb-L1 dataset is released! Built from Common Crawl snapshots, it undergoes main-text extraction, language filtering, heuristic filtering, sensitive-field replacement, customized cleaning, and deduplication, yielding 1T+ tokens (approximately 1.14 billion documents). We simultaneously release the L2 selected data Ultra-FineWeb, selected by the Ultra-FineWeb classifier. To our best knowledge, this open-source web pre-training dataset covers the most recent Common Crawl snapshots, up to CC-MAIN-2025-51. 🚀🚀🚀
• [2026.05.28] The Ultra-FineWeb-L3 dataset is released! The L3 refined data built on Ultra-FineWeb via Q&A Pair Generation and Multi-style Rewriting, with 400B+ English and 200B+ Chinese tokens. To our best knowledge, it is the largest open-source Chinese pre-training synthetic corpus to date. 🚀🚀🚀
• [2026.05.25] MiniCPM5-1B is released! It is the first model in the MiniCPM5 series, a dense 1B Transformer designed for on-device, local deployment, and resource-constrained scenarios, reaching 1B-class open-source SOTA. Ultra-FineWeb serves as its core pre-training web dataset.
• [2026.02.08] The UltraData platform is now live, introducing the L0-L4 tiered data management framework. Ultra-FineWeb serves as the L2 selected layer for general web data in this framework. 🔍🔍🔍
• [2025.06.16] The Ultra-FineWeb-classifier is now available on Hugging Face: openbmb/Ultra-FineWeb-classifier.
• [2025.06.06] Ultra-FineWeb-en and Ultra-FineWeb-zh are now available on Hugging Face, released alongside the MiniCPM4 Series models.
• [2025.05.15] Ultra-FineWeb tops the Hugging Face Datasets Trending list, reaching the #1 spot! ⭐️⭐️⭐️
• [2025.05.09] The Ultra-FineWeb technical report is available on arXiv. 🔥🔥🔥

🏗️ Data Processing Pipeline

Each Common Crawl dump is processed independently:

1. Page and main-text extraction: retain valid HTML pages from Common Crawl and use trafilatura 2.0 to extract plain text while excluding comments, navigation bars, and other non-main content.
2. Language filtering: use fastText language identification to retain high-confidence English documents.
3. Heuristic filtering: build on FineWeb's heuristic filtering approach to address repetition, low-quality documents, boilerplate text, and abnormal line structures.
4. Sensitive-field replacement: replace detected email addresses, IP addresses, phone numbers, identity-card numbers, and credit-card numbers with valid placeholders.
5. MinHash deduplication: identify near-duplicate documents within each Common Crawl dump. Following FineWeb, deduplication is performed independently within each dump rather than across the full collection.
6. Customized cleaning: use UltraData's data quality inspection tools and data-cleaning agents to optimize cleaning rules and apply targeted processing for residual HTML, encoding corruption and mojibake, invisible characters, corrupted content, and abnormal document lengths.

📁 Dataset Structure

Ultra-FineWeb-L1 is organized by Common Crawl dump:

data/
└── CC-MAIN-YYYY-WW/
    ├── CC-MAIN-YYYY-WW-part-0001-of-NNNN.parquet
    ├── CC-MAIN-YYYY-WW-part-0002-of-NNNN.parquet
    └── ...

The classifier-selected L2 subset is released in Ultra-FineWeb:

data/ultrafineweb_l1_en_hq/
└── CC-MAIN-YYYY-WW/
    ├── ultrafineweb-l1-en-hq-CC-MAIN-YYYY-WW-part-0001-of-NNNN.parquet
    └── ...

Data fields

📈 Experimental Results

Evaluation Setup

We follow the FinePhrase evaluation setup and adopt the FineWeb ablation methodology: each data configuration is evaluated on 12 benchmarks across six categories using 3-shot prompting with a single random seed. For most tasks, we use the Cloze Format (CF), framing evaluation as next-token prediction. Compared with standard multiple-choice evaluation, CF reduces interference from instruction following and answer formatting for smaller models and provides a more stable evaluation signal. The benchmarks are:

• General Knowledge: ARC, MMLU Redux
• Reading Comprehension: SQuAD v2, DROP
• Reasoning: OpenBookQA, XCSQA
• Natural Language Understanding: WinoGrande, PIQA, HellaSwag
• Math: GSM8K
• Table Understanding: WikiTableQuestions, TriviaQA

Training Setup

To evaluate the effectiveness of the cleaning pipeline, we select CC-MAIN-2025-26, the latest Common Crawl snapshot covered by FineWeb v1.4, and compare data processed by the FineWeb v1.4 and Ultra-FineWeb-L1 pipelines. Both groups are trained for 20B tokens with the MiniCPM5-1B model under identical settings: 32 GPUs, a micro batch size of 16, a global batch size of 512, the Muon optimizer, an initial learning rate of 0.000522, and a minimum learning rate of 0.0000522.

Results

The figures below show the six-category macro-average and 12-task micro-average scores of four data configurations throughout training. Scores generally improve as the number of training tokens increases. At the end of training, Ultra-FineWeb-L1 reaches 9.668% macro and 9.180% micro, outperforming FineWeb by 0.635 and 0.696 percentage points, respectively, and validating the effectiveness of the optimized cleaning pipeline. Quality selection yields further gains: Ultra-FineWeb-from-FW reaches 9.954% / 9.414%, while Ultra-FineWeb-from-L1 achieves the best results at 10.379% / 9.798%, indicating that L1 cleaning and subsequent quality selection provide complementary benefits.

❤️ Acknowledgements

• Source Web Data: Common Crawl
• Data Processing Pipeline: FineWeb, DataTrove
• Text Extraction and Cleaning Tools: trafilatura, fastText, NLTK, ftfy

We thank these outstanding open-source projects—their contributions make Ultra-FineWeb-L1 possible! 🙌

📖 Citation

If you find Ultra-FineWeb-L1 useful in your research, please consider citing:

@misc{wang2025ultrafineweb,
  title={{Ultra-FineWeb}: Efficient Data Filtering and Verification for High-Quality LLM Training Data},
  author={Yudong Wang and Zixuan Fu and Jie Cai and Peijun Tang and Hongya Lyu and Yewei Fang and Zhi Zheng and Jie Zhou and Guoyang Zeng and Chaojun Xiao and Xu Han and Zhiyuan Liu},
  year={2025},
  eprint={2505.05427},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

@misc{ultra-fineweb-l1,
  title={Ultra-FineWeb-L1},
  author={Junshao Guo and Shuaikang Xue and Xiaofei Yang and Chuyue Zhou and Qiang Yin and Qingyang Yuan and Qiang Ma and Yuzhou Zhang and Jie Zhou and Chaojun Xiao and Yudong Wang and Zhiyuan Liu},
  year={2026},
  url={https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1},
  publisher={Hugging Face}
}

📜 License

This project is licensed under the Apache 2.0 license. Because the corpus is derived from web content, users must also comply with the rights, licenses, and terms associated with the original sources.

No unauthorized unchanged redistribution: Without prior written permission from the original authors (or this organization), any institution, organization, or third-party platform is strictly prohibited from directly reposting, mirroring, re-hosting, or commercially repackaging and republishing any artifacts of this project in any form.