UltraData-Stack-v3-L2

📦 UltraData Collection | 🌐 UltraData | 📖 Tech Report (Coming Soon) | 🤗 UltraData-Code | 🤗 UltraData-Code-L2 Classifier

English | 中文

📚 Introduction

UltraData-Stack-v3-L2 is a selected code corpus of approximately 240B tokens for the 11 programming languages in The Stack v3, obtained through quality-control cleaning and fine-grained selection with UltraData-Code-L2-Classifier. This dataset transfers the language-adaptive L2 selection framework proposed by UltraData-Code to another source corpus, in order to examine whether file-level algorithm-related relevance selection can generalize to data sources beyond UltraData-Code-L1.

Construction has two stages. First, source records are cleaned by removing URL-only entries, overly short records, and records containing garbled or malformed text. Second, a language-specific selector combines file-role prediction, algorithmic-relevance scoring, an EXCLUDE veto, and a shared code-quality constraint. The relevance threshold is calibrated separately for each language. The resulting dataset retains 20.10% of the cleaned 11-language candidate pool, and is intended to concentrate interpretable computation and problem-solving logic while preserving broad language coverage.

📢 What's New

• [2026.10.12] The UltraData-Stack-v3-L2 dataset is released! 🚀🚀🚀
• [2026.09.07] The UltraData-Code dataset is released! It is a complete implementation of the UltraData L0-L4 tiered data management framework. It covers four code data states from L0 through L3, with each level corresponding to a distinct construction stage. This release currently open-sources L2 (~400B tokens) and L3 (~150B tokens) across 11 programming languages. 🚀🚀🚀
• [2026.09.07] MiniCPM5-2B is released!, the second model in the MiniCPM5 series after MiniCPM5-1B. It is a dense 2B Transformer that scales up the same training recipe, built for on-device, local deployment, and resource-constrained scenarios. It reaches 2B-class open-source SOTA, remains competitive with 4B-class models, and shows particular advantages in coding, mathematics, long-context understanding, tool use, and agentic tasks. UltraData-RL-2609 serves as the core RL dataset for MiniCPM5-2B. 🚀🚀🚀
• [2026.02.08] The UltraData platform is now live, introducing the L0-L4 tiered data management framework. 🔍🔍🔍

💡 Highlights

• Transferable algorithmic-relevance selection: The same selection objective and model interfaces used for UltraData-Code-L2 are adapted to Stack-v3-train, with language-specific evidence and thresholds.
• Strong performance gains: Under the same 1B base model and 10B-token continual-pretraining setup, UltraData-Stack-v3-L2 reaches 30.00 average pass@1 on EvalPlus and 22.75 on MultiPL-E, compared with 25.37 and 17.30 for Stack-v3-train.

📊 Dataset Statistics

Token counts are estimated with the tokenizer used in the technical report.

📈 Evaluation Results

The 10B-token comparisons use the same 1B foundation model, training settings, decontamination, and evaluation protocol.

Python results: UltraData-Stack-v3-L2-py ranks second among the evaluated L2 datasets by average EvalPlus pass@1 and obtains the highest MBPP and MBPP+ scores at L2.

Multilingual results: UltraData-Stack-v3-L2 exceeds Stack-v3-train by 4.63 points on EvalPlus and 5.45 points on MultiPL-E, respectively, demonstrating effective L2 language-adaptive selection.

❤️ Acknowledgements

UltraData-Stack-v3-L2 is built from The Stack v3. The selection framework UltraData-Code-L2-Classifier is developed as part of UltraData-Code, and its file-level representations are computed with Qwen3-Embedding-0.6B.

Thanks to the authors and contributors of these open-source projects.

💳 License

The dataset is released under the Open Data Commons Attribution License (ODC-By) v1.0, consistent with the source dataset The Stack v3. Any use of all or part of the code gathered in UltraData-Stack-v3-L2 must abide by the terms of the original licenses, including attribution clauses where relevant.

📖 Citation

If you find this dataset useful in your research, please consider citing:

@misc{ultradata_code,
  title        = {{UltraData-Code}: From Raw Repositories to Algorithmically Dense and Task-Oriented Code Data},
  author       = {Chengying Tu and Hengyu Zhao and Shuaikang Xue and Zhongming Qu and Jihao Zhou and Xinle Lin and Junshao Guo and Zixuan Fu and Qiang Ma and Jie Zhou and Chaojun Xiao and Hongfei Yan and Yudong Wang and Xu Han and Zhiyuan Liu and Maosong Sun},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/openbmb/UltraData-Code}}
}