π¦ UltraData Collection | π UltraData | π€ MiniCPM5 Series | π Tech Report (Coming Soon) | π€ UltraData-Code-L2 Classifier
English | δΈζ
UltraData-Code is a complete implementation of the UltraData L0-L4 tiered data management framework. It covers four code data states from L0 through L3, with each level corresponding to a distinct construction stage. The pipeline starts from approximately 192 million public GitHub repositories at L0. This release provides UltraData-Code-L2 (~400B tokens) and UltraData-Code-L3 (~150B tokens) across 11 programming languages.
L0: Repository archival. We preserve the latest revision on the default branch of approximately 192 million public GitHub repositories, together with code, directory structure, file relationships, and provenance metadata. This archive provides complete raw data and traceability for subsequent processing.
L1: Standardized natural code. Starting from L0, we perform large-scale filtering, cleaning, format normalization, and near-duplicate deduplication. General rules remove oversized files, invalid paths, unsupported text, and obvious anomalies. Checks tailored to each language and file type then normalize and clean the remaining content. For deduplication, files are partitioned by extension. MinHash signatures are generated and LSH retrieves near-duplicate candidates. Similarity edges define connected components. For groups containing multiple near-duplicates, the highest ranked 50% are retained. Files with no detected near duplicate are all kept. The result is a standardized natural code corpus.
L2: Algorithmically relevant code. Across 11 programming languages, we select approximately 400B tokens of algorithmically relevant code from L1. We develop a language-adaptive selection framework that combines file role supervision with language-specific heuristic cues to learn algorithmic relevance beyond explicit ALGO files. The framework reuses precomputed semantic embeddings across role, relevance, and quality models, and applies role and quality constraints during selection. Under controlled 10B-token continual pre-training of a 1B model, L2 improves over L1 by 7.80 and 5.13 points on EvalPlus and MultiPL-E, and over Stack-Edu by 4.37 and 3.05 points.
L3: Task-oriented synthesis. We introduce a structured synthesis protocol that converts each algorithmically relevant implementation selected by L2 into a programming exercise, jointly generating a standalone task, analysis, solution, and test candidates from the same source implementation. This transformation preserves the source code's computational intent while adding explicit task and solution supervision for code generation. L3 covers the same 11 programming languages and contains approximately 150B tokens. Under controlled 10B-token continual pre-training of a 1B model, replacing half of the L2 training tokens with L3 further improves EvalPlus and MultiPL-E by 8.42 and 8.07 points over L2-only training.
Abstract: Code generation has become a core capability of large language models, and code data is a central part of the pre-training process that develops it. As code corpora continue to grow, their scale, diversity, and quality increasingly shape the capabilities learned during pre-training. Following a tiered data management perspective, we present UltraData-Code as a family of four connected data states, from repository archival at L0 through standardized natural code at L1, algorithmic selection at L2, and task-oriented synthesis at L3. L0 archives the latest revision on the default branch of each public GitHub repository with its file structure, relationships, and provenance. L1 applies scalable filtering, normalization, and near deduplication to obtain standardized natural code. L2 then selects algorithmically relevant files from L1 using language-adaptive signals from file roles and heuristics, together with code quality constraints, yielding approximately 400B tokens of UltraData-Code-L2, spanning 11 programming languages. L3 applies task-oriented synthesis to algorithmic files from L2, turning each implementation into a structured programming exercise, generating approximately 150B tokens of UltraData-Code-L3 in the same 11 languages. Under controlled 10B-token continual pre-training of a 1B model, training on L2 instead of L1 raises pass@1 on EvalPlus by 7.80 points and on MultiPL-E by 5.13 points, while exceeding Stack-Edu by 4.37 and 3.05 points, respectively. Replacing half of the L2 training tokens with L3 yields a further gain of 8.42 points on EvalPlus and 8.07 points on MultiPL-E over L2-only training, while exceeding the strongest synthetic data baseline by 5.57 and 7.80 points, respectively. When the training budget increases to 100B tokens, gains from L2 selection and L3 synthesis further widen on both benchmarks.
The 10B-token comparisons use the same 1B foundation model, training settings, decontamination, and evaluation protocol.
Each row represents one selected source file. The fields are:
uuid: unique file identifier.repo_name: source repository name.relative_path: file path relative to the repository.content: source code text.category: predicted file role, such as ALGO, WEB, TOOL, DATA, TEST, CONFIG.algo_rel_score: algorithmic relevance score in [0, 1].quality_score: code quality score in [0, 10].Each row is a programming exercise grounded in a real implementation:
uuid: unique file identifier.content: serialization containing task and solution.content_format: serialization format for content.raw_content: original generated record before serialization.task: generated standalone problem statement.analysis: algorithm, edge cases, and complexity discussion.solution: generated self-contained reference implementation.test: generated test candidates.full_content: serialization containing all generated fields.full_content_format: serialization format for full_content.Thanks for their awesome work! Open source contributions make UltraData-Code possible! π
This project is released under the Apache 2.0 license. UltraData-Code is built from code in multiple public repositories (L2) and from model-generated task records derived from those files (L3). Users must also comply with the LICENSE of each source repository. Apache 2.0 does not override those terms.
Public availability of a repository is not a grant of redistribution, commercial use, or training rights. Rights holders may request removal via the contact channel on the dataset page.
The dataset should not contain plaintext secrets or unauthorized personal data. Please report sensitive content or takedown requests through the same channel.
No unauthorized unchanged redistribution: Without prior written permission from the original authors (or this organization), any institution, organization, or third-party platform is strictly prohibited from directly reposting, mirroring, re-hosting, or commercially repackaging and republishing any artifacts of this project in any form.
If you find UltraData-Code useful in your research, please consider citing:
@misc{ultradata_code,
title = {{UltraData-Code}: From Raw Repositories to Algorithmically Dense and Task-Oriented Code Data},
author = {Chengying Tu and Hengyu Zhao and Shuaikang Xue and Zhongming Qu and Jihao Zhou and Xinle Lin and Junshao Guo and Zixuan Fu and Qiang Ma and Jie Zhou and Chaojun Xiao and Hongfei Yan and Yudong Wang and Xu Han and Zhiyuan Liu and Maosong Sun},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/datasets/openbmb/UltraData-Code}}
}
289 commits
π¦ UltraData Collection | π UltraData | π€ MiniCPM5 Series | π Tech Report (Coming Soon) | π€ UltraData-Code-L2 Classifier
English | δΈζ
UltraData-Code is a complete implementation of the UltraData L0-L4 tiered data management framework. It covers four code data states from L0 through L3, with each level corresponding to a distinct construction stage. The pipeline starts from approximately 192 million public GitHub repositories at L0. This release provides UltraData-Code-L2 (~400B tokens) and UltraData-Code-L3 (~150B tokens) across 11 programming languages.
L0: Repository archival. We preserve the latest revision on the default branch of approximately 192 million public GitHub repositories, together with code, directory structure, file relationships, and provenance metadata. This archive provides complete raw data and traceability for subsequent processing.
L1: Standardized natural code. Starting from L0, we perform large-scale filtering, cleaning, format normalization, and near-duplicate deduplication. General rules remove oversized files, invalid paths, unsupported text, and obvious anomalies. Checks tailored to each language and file type then normalize and clean the remaining content. For deduplication, files are partitioned by extension. MinHash signatures are generated and LSH retrieves near-duplicate candidates. Similarity edges define connected components. For groups containing multiple near-duplicates, the highest ranked 50% are retained. Files with no detected near duplicate are all kept. The result is a standardized natural code corpus.
L2: Algorithmically relevant code. Across 11 programming languages, we select approximately 400B tokens of algorithmically relevant code from L1. We develop a language-adaptive selection framework that combines file role supervision with language-specific heuristic cues to learn algorithmic relevance beyond explicit ALGO files. The framework reuses precomputed semantic embeddings across role, relevance, and quality models, and applies role and quality constraints during selection. Under controlled 10B-token continual pre-training of a 1B model, L2 improves over L1 by 7.80 and 5.13 points on EvalPlus and MultiPL-E, and over Stack-Edu by 4.37 and 3.05 points.
L3: Task-oriented synthesis. We introduce a structured synthesis protocol that converts each algorithmically relevant implementation selected by L2 into a programming exercise, jointly generating a standalone task, analysis, solution, and test candidates from the same source implementation. This transformation preserves the source code's computational intent while adding explicit task and solution supervision for code generation. L3 covers the same 11 programming languages and contains approximately 150B tokens. Under controlled 10B-token continual pre-training of a 1B model, replacing half of the L2 training tokens with L3 further improves EvalPlus and MultiPL-E by 8.42 and 8.07 points over L2-only training.
Abstract: Code generation has become a core capability of large language models, and code data is a central part of the pre-training process that develops it. As code corpora continue to grow, their scale, diversity, and quality increasingly shape the capabilities learned during pre-training. Following a tiered data management perspective, we present UltraData-Code as a family of four connected data states, from repository archival at L0 through standardized natural code at L1, algorithmic selection at L2, and task-oriented synthesis at L3. L0 archives the latest revision on the default branch of each public GitHub repository with its file structure, relationships, and provenance. L1 applies scalable filtering, normalization, and near deduplication to obtain standardized natural code. L2 then selects algorithmically relevant files from L1 using language-adaptive signals from file roles and heuristics, together with code quality constraints, yielding approximately 400B tokens of UltraData-Code-L2, spanning 11 programming languages. L3 applies task-oriented synthesis to algorithmic files from L2, turning each implementation into a structured programming exercise, generating approximately 150B tokens of UltraData-Code-L3 in the same 11 languages. Under controlled 10B-token continual pre-training of a 1B model, training on L2 instead of L1 raises pass@1 on EvalPlus by 7.80 points and on MultiPL-E by 5.13 points, while exceeding Stack-Edu by 4.37 and 3.05 points, respectively. Replacing half of the L2 training tokens with L3 yields a further gain of 8.42 points on EvalPlus and 8.07 points on MultiPL-E over L2-only training, while exceeding the strongest synthetic data baseline by 5.57 and 7.80 points, respectively. When the training budget increases to 100B tokens, gains from L2 selection and L3 synthesis further widen on both benchmarks.
The 10B-token comparisons use the same 1B foundation model, training settings, decontamination, and evaluation protocol.
Each row represents one selected source file. The fields are:
uuid: unique file identifier.repo_name: source repository name.relative_path: file path relative to the repository.content: source code text.category: predicted file role, such as ALGO, WEB, TOOL, DATA, TEST, CONFIG.algo_rel_score: algorithmic relevance score in [0, 1].quality_score: code quality score in [0, 10].Each row is a programming exercise grounded in a real implementation:
uuid: unique file identifier.content: serialization containing task and solution.content_format: serialization format for content.raw_content: original generated record before serialization.task: generated standalone problem statement.analysis: algorithm, edge cases, and complexity discussion.solution: generated self-contained reference implementation.test: generated test candidates.full_content: serialization containing all generated fields.full_content_format: serialization format for full_content.Thanks for their awesome work! Open source contributions make UltraData-Code possible! π
This project is released under the Apache 2.0 license. UltraData-Code is built from code in multiple public repositories (L2) and from model-generated task records derived from those files (L3). Users must also comply with the LICENSE of each source repository. Apache 2.0 does not override those terms.
Public availability of a repository is not a grant of redistribution, commercial use, or training rights. Rights holders may request removal via the contact channel on the dataset page.
The dataset should not contain plaintext secrets or unauthorized personal data. Please report sensitive content or takedown requests through the same channel.
No unauthorized unchanged redistribution: Without prior written permission from the original authors (or this organization), any institution, organization, or third-party platform is strictly prohibited from directly reposting, mirroring, re-hosting, or commercially repackaging and republishing any artifacts of this project in any form.
If you find UltraData-Code useful in your research, please consider citing:
@misc{ultradata_code,
title = {{UltraData-Code}: From Raw Repositories to Algorithmically Dense and Task-Oriented Code Data},
author = {Chengying Tu and Hengyu Zhao and Shuaikang Xue and Zhongming Qu and Jihao Zhou and Xinle Lin and Junshao Guo and Zixuan Fu and Qiang Ma and Jie Zhou and Chaojun Xiao and Hongfei Yan and Yudong Wang and Xu Han and Zhiyuan Liu and Maosong Sun},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/datasets/openbmb/UltraData-Code}}
}
289 commits