UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

📜 Paper | 💻 Code | 🤖 Models | 📦 UltraData Collection

English | 中文

📚 Introduction

UltraX is a function-calling refinement framework for large-scale pre-training data that adaptively generates and executes editing functions for efficient instance-wise refinement. Unlike rule-based or end-to-end LLM rewriting methods, UltraX trains a lightweight refinement model to predict structured editing operations — including insertion, deletion, and modification — which are then deterministically executed on the original text.

This dataset collection contains five English pre-training corpora refined by UltraX, each with ~20B tokens:

📢 News

• [2026.07.17] UltraX tops the Hugging Face Datasets Trending list, reaching the #1 spot! ⭐️⭐️⭐️
• [2026.07.13] UltraX codebase, refinement model, and refined datasets are now available on GitHub and Hugging Face. 🚀🚀🚀
• [2026.07.10] UltraX technical report is available on arXiv. 🔥🔥🔥

💡 Highlights

• Function-Calling Refinement: Instead of end-to-end text rewriting, UltraX predicts structured editing operations (keep_all, remove_all, remove_lines, replace_str, add_line), enabling fine-grained instance-level editing with deterministic execution.
• LAM + DCR Pipeline: Line Alignment and Mapping (LAM) aligns original and refined text at line level, while Dynamic Context Replacement (DCR) converts character-level edits into reliable replace_str operations with unique context anchoring.
• Robust Large-Scale Execution: Sliding-window inference with overlap-aware aggregation, ambiguity filtering, same-line operation merging, and duplicate pattern detection ensure reliable execution at scale.
• State-of-the-Art Performance: UltraX achieves the best average performance across all five corpora in 1B-model pre-training experiments, with improvements exceeding 2% on multiple datasets and superior data efficiency.

🔬 Pipeline Overview

📈 Evaluation Results

We pretrain 1B-parameter MiniCPM models from scratch on 20B tokens for each corpus and evaluate on 10 benchmarks (ARC-C, ARC-E, CSQA, HellaSwag, MMLU, OBQA, PIQA, SIQA, WinoGrande, SciQ) using LightEval with zero-shot setting.

Key findings:

• UltraX achieves the highest average performance on all five corpora, winning 34 out of 50 task-corpus pairs.
• Compared to Raw and ProX-C, UltraX achieves average relative improvements of ~2.00% and ~1.53%, respectively.
• On FineWeb, UltraX at 16B tokens (45.49) already surpasses Raw and ProX-C at 20B tokens (45.08 / 45.05), demonstrating superior data efficiency.

📊 Data Format

Each parquet file contains 5 columns:

🚀 Quick Start

You can load the dataset directly from Hugging Face:

from datasets import load_dataset

# Load UltraX-FineWeb
ds = load_dataset("openbmb/UltraX", "UltraX-FineWeb", split="train")

# Load UltraX-RedPajama-V2
ds = load_dataset("openbmb/UltraX", "UltraX-RedPajama-V2", split="train")

# Load UltraX-AICC
ds = load_dataset("openbmb/UltraX", "UltraX-AICC", split="train")

# Load UltraX-Ultra-FineWeb
ds = load_dataset("openbmb/UltraX", "UltraX-Ultra-FineWeb", split="train")

# Load UltraX-FineWeb-ProX-Doc
ds = load_dataset("openbmb/UltraX", "UltraX-FineWeb-ProX-Doc", split="train")

🔧 Function Space

The UltraX refinement model predicts operations from the following function space:

❤️ Acknowledgements

• The UltraX datasets are built upon FineWeb, RedPajama-v2, AICC, Ultra-FineWeb, and FineWeb-ProX-Doc.
• The refinement model is trained using ms-swift.
• UltraX builds upon insights from ProX and RefineX.

Thanks for their awesome work! Open-source contributions make UltraX possible! 🙌

📖 Citation

If you find our work useful, please consider citing:

@misc{ultrax2026,
  title={UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing},
  author={Xinlong Zhao and Dongsheng Liu and Hengyu Zhao and Zixuan Fu and Zheng Wang and Jie Cai and Jie Zhou and Qiang Ma and Xuanhe Zhou and Xu Han and Yudong Wang and Zhiyuan Liu},
  year={2026},
  eprint={2607.08646},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
}

📜 License

This project is licensed under the Apache 2.0 license. Please note that since UltraX datasets are built upon multiple source corpora, users should check the LICENSE of each source dataset individually to ensure proper usage and compliance.

No unauthorized unchanged redistribution: Without prior written permission from the original authors (or this organization), any institution, organization, or third-party platform is strictly prohibited from directly reposting, mirroring, re-hosting, or commercially repackaging and republishing any artifacts of this project in any form.