Ultra-FineWeb-L3

📜 Ultra-FineWeb Technical Report | 📦 UltraData Collection | 🌐 UltraData | 🤗 MiniCPM5 Series

English | 中文

📚 Introduction

Ultra-FineWeb-L3 is the L3 refined data for general high-quality web data within UltraData's L0-L4 tiered data management framework. Moving beyond L2 quality selection, it transforms high-value web corpora into structured, high-learnability training data with clearer reasoning signals and richer educational styles. Built on top of Ultra-FineWeb, it leverages MiniCPM4 and Qwen3 to perform Q&A Pair Generation and Multi-style Rewriting, producing 400B+ English tokens and 200B+ Chinese tokens. To our best knowledge, the Chinese portion is the largest open-source Chinese pre-training synthetic dataset to date. As high-tier data designed for late-stage training efficiency, it serves as key training data for the decay phase of MiniCPM5-1B.

• Ultra-FineWeb-L1: L1 filtered data after basic cleaning, heuristic filtering, sensitive-field replacement, and deduplication.
• Ultra-FineWeb: Ultra-FineWeb, a large-scale, high-quality, and efficiently-filtered dataset, with 1T English tokens and 120B Chinese tokens.
• Ultra-FineWeb-classifier: Ultra-FineWeb classifier, for filtering high-quality data from web corpora.
• Ultra-FineWeb-L3: the L3 refined data built on Ultra-FineWeb via Q&A Pair Generation and Multi-style Rewriting, with 400B+ English and 200B+ Chinese tokens—to our best knowledge, the largest open-source Chinese pre-training synthetic corpus to date. (Current dataset)

📢 What's New

• [2026.08.20] The Ultra-FineWeb-L1 dataset is released! Built from Common Crawl snapshots, it undergoes main-text extraction, language filtering, heuristic filtering, sensitive-field replacement, customized cleaning, and deduplication, yielding 1T+ tokens (approximately 1.14 billion documents). We simultaneously release the L2 selected data Ultra-FineWeb, selected by the Ultra-FineWeb classifier. To our best knowledge, this open-source web pre-training dataset covers the most recent Common Crawl snapshots, up to CC-MAIN-2025-51. 🚀🚀🚀
• [2026.05.28] The Ultra-FineWeb-L3 dataset is released! The L3 refined data built on Ultra-FineWeb via Q&A Pair Generation and Multi-style Rewriting, with 400B+ English and 200B+ Chinese tokens. To our best knowledge, it is the largest open-source Chinese pre-training synthetic corpus to date. 🚀🚀🚀
• [2026.05.25] MiniCPM5-1B is released!, the first model in the MiniCPM5 series. It is a dense 1B Transformer built for on-device, local deployment, and resource-constrained scenarios, reaching 1B-class open-source SOTA. Ultra-FineWeb serves as the core pre-training web dataset for MiniCPM5-1B.
• [2026.02.08] The UltraData platform is now live, introducing the L0-L4 tiered data management framework. Ultra-FineWeb serves as the L2 selected layer for general web data in this framework. 🔍🔍🔍
• [2025.06.16] The Ultra-FineWeb-classifier is now available on Hugging Face: openbmb/Ultra-FineWeb-classifier.
• [2025.06.06] Ultra-FineWeb-en and Ultra-FineWeb-zh datasets are now available on Hugging Face, released alongside the MiniCPM4 Series models.
• [2025.05.15] Ultra-FineWeb tops the Hugging Face Datasets Trending list, reaching the #1 spot! ⭐️⭐️⭐️
• [2025.05.09] Ultra-FineWeb technical report is available on arXiv. 🔥🔥🔥

💡 Highlights

High-quality pre-training data is essential for improving the general reasoning and knowledge capabilities of large language models (LLMs). In the UltraData L0-L4 tiered data management framework, L1 filtered data addresses basic cleaning and denoising, and L2 selected data further enhances information density through model-driven quality selection. However, even after L2 selection, general web text still has inherent limitations:

• Declarative & Monotonous Training Signal: Most content is narrative or declarative, offering a relatively homogeneous training signal and lacking structured reasoning steps.
• Style Concentration: Expression styles are largely concentrated in informal web writing paradigms, with insufficient coverage of diverse formats such as textbooks, encyclopedias, abstracts, and blogs.
• Diluted Knowledge: Valuable knowledge points are often buried in redundant descriptions, navigation text, and residual noise, reducing training efficiency.

L3 refinement aims to move data from "higher quality" to "better learnability". Ultra-FineWeb-L3 takes Ultra-FineWeb (the trillion-token high-quality training fuel behind MiniCPM4) as seed data, and uses MiniCPM4 and Qwen-30B-A3 to perform two core synthesis tasks:

• Q&A Pair Generation: Transform declarative web documents into structured samples of "original text + multiple Question-Answer pairs". The synthesis model understands the document's information structure and generates questions and answers around core concepts, factual details, and logical relationships. During training, the original text is prepended to the Q&A pairs, allowing the model to learn more explicit knowledge organization while retaining contextual information.
• Multi-style Rewriting: Rewrite single-source web content into multiple expression styles (encyclopedia, textbook, blog, abstract, etc.) while summarizing and reorganizing core knowledge points. Expressing the same knowledge from multiple perspectives inherently improves the diversity and robustness of knowledge representation, helping the model generalize better across different text genres and task scenarios.

🏗️ L3 Data Refinement Pipeline

Ultra-FineWeb-L3 exemplifies L3 refined data in the UltraData tiered data management system for general web data, and serves as key training data for the decay phase of MiniCPM5-1B training.

Q&A Pair Generation

Transform declarative web documents into "original text + multiple Q&A pairs" structured samples:

1. The synthesis model deeply understands the document's information structure, identifying core concepts, factual details, and relationships.
2. It generates self-contained questions and detailed answers around the extracted information.
3. During training, the original text is concatenated before the Q&A pairs, enabling the model to learn explicit knowledge organization while preserving contextual information.

Multi-style Rewriting

Rewrite single-source web content into diverse expression styles and reorganize core knowledge points:

• Encyclopedia style: Modular, encyclopedic organization with concise, objective declarative sentences.
• Textbook style: Rigorous logical progression following "definition → theorem → proof → example" structure.
• Blog style: Flexible short paragraphs with a conversational tone, rich in analogies and real-life examples.
• Abstract style: Highly compressed content with core argument extraction, retaining key conclusions and data while removing redundancy.

📈 Experimental Results

Following the Ultra-FineWeb evaluation setup, we train from scratch for 100B tokens. For English, we compare against FinePhrase, FineWeb, DCLM, FineWeb-edu, and Ultra-FineWeb; for Chinese, we compare against Chinese-FineWeb, Chinese-FineWeb-edu, and Ultra-FineWeb-zh.

As training progresses, average benchmark scores across all datasets show a steady upward trend, indicating that models continue to benefit from data within the 100B token training range:

• English: Ultra-FineWeb-en and its L3 variant achieve the best overall performance, with Ultra-FineWeb-en-L3 reaching the highest average score in late-stage training.
• Chinese: The Ultra-FineWeb-zh series outperforms Chinese-FineWeb and Chinese-FineWeb-edu with more stable and higher gains, with Ultra-FineWeb-zh-L3 showing an increasingly widening advantage in later training stages, demonstrating the effectiveness of L3 refined data.

Overall, data quality selection and tiered filtering deliver better average performance across both English and Chinese benchmarks under the same training token budget.

🚀 Quick Start

You can load the dataset directly from Hugging Face:

from datasets import load_dataset

# Load Q&A Synthetic data (English)
ds = load_dataset("openbmb/Ultra-FineWeb-L3", "Ultra-FineWeb-L3-en-QA-Synthetic", split="train")

# Load Q&A Synthetic data (Chinese)
ds = load_dataset("openbmb/Ultra-FineWeb-L3", "Ultra-FineWeb-L3-zh-QA-Synthetic", split="train")

# Load Multi-Style Synthetic data (English)
ds = load_dataset("openbmb/Ultra-FineWeb-L3", "Ultra-FineWeb-L3-en-Multi-Style-Synthetic", split="train")

# Load Multi-Style Synthetic data (Chinese)
ds = load_dataset("openbmb/Ultra-FineWeb-L3", "Ultra-FineWeb-L3-zh-Multi-Style-Synthetic", split="train")

❤️ Acknowledgements

• L3 Refinement Models: MiniCPM4 Series, Qwen3 Series
• Seed Data: Ultra-FineWeb
• Data Framework: UltraData L0-L4 Tiered Data Management

📖 Citation

If you find Ultra-FineWeb-L3 useful in your research, please consider citing:

@misc{wang2025ultrafineweb,
  title={{Ultra-FineWeb}: Efficient Data Filtering and Verification for High-Quality LLM Training Data},
  author={Yudong Wang and Zixuan Fu and Jie Cai and Peijun Tang and Hongya Lyu and Yewei Fang and Zhi Zheng and Jie Zhou and Guoyang Zeng and Chaojun Xiao and Xu Han and Zhiyuan Liu},
  year={2025},
  eprint={2505.05427},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
}

@misc{ultra-fineweb-l3,
  title={Ultra-FineWeb-L3},
  author={Hongya Lyu and Chuyue Zhou and Shixin Ren and Xinlong Zhao and Zixuan Fu and Shuaikang Xue and Junshao Guo and Qiang Ma and Chaojun Xiao and Jie Zhou and Xu Han and Yudong Wang and Zhiyuan Liu},
  year={2026},
  url={https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L3},
  publisher={Hugging Face}
}

📜 License

This project is licensed under the Apache 2.0 license.

No unauthorized unchanged redistribution: Without prior written permission from the original authors (or this organization), any institution, organization, or third-party platform is strictly prohibited from directly reposting, mirroring, re-hosting, or commercially repackaging and republishing any artifacts of this project in any form.