Awesome LLM Pre-training

中文版 | English Version

Pre-training is the first and most crucial training stage in the development of large language models. As the open-source community continues to improve in areas such as model architecture, training strategies, open-source datasets, and data methods, we are committed to continuously tracking resources available for large model pre-training to give back to developers in the open-source large language model community.

Compared to comprehensive reviews, our scope is limited to commonly used resources and cutting-edge attempts related to pre-training, aiming to help users quickly get started with large language model pre-training. We also welcome contributions and updates from the open-source community to jointly promote the development of large models.

Related project links: [LLMSurvey] [YuLan-Chat] | [YuLan-Mini]

Table of Contents

• Technical Reports
• Training Strategies
• Open-source Datasets
• Data Methods

I. Technical Reports

Technical reports often rely on hundreds or thousands of computing resources. Therefore, it is highly recommended to read some open-source technical reports.

1.1 Dense Models

1. The Llama 3 Herd of Models. [paper]
2. Qwen2.5 Technical Report. [paper]
3. Gemma 3 Technical Report. [paper]
4. Nemotron-4 340B Technical Report. [paper]
5. Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs. [paper]
6. Baichuan 2: Open Large-scale Language Models. [paper]

1.2 MoE Models

1. DeepSeek-V3 Technical Report. [paper]
2. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. [paper]
3. Mixtral of Experts. [paper]
4. Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models. [paper]
5. Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs. [paper]
6. OLMoE: Open Mixture-of-Experts Language Models. [paper]
7. Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent. [paper]

1.3 Models with Open-source Datasets

1. YuLan-Mini: An Open Data-efficient Language Model. [code] [resource] [paper]
2. MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series. [paper]
3. LLM360: Towards Fully Transparent Open-Source LLMs. [paper]
4. Nemotron-4 15B Technical Report. [paper]

1.4 Training/Data Strategies

1. Phi-4 Technical Report. [paper]
2. OLMo: Accelerating the Science of Language Models. [paper]
3. 2 OLMo 2 Furious. [paper]
4. Yi: Open Foundation Models by 01.AI. [paper]
5. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. [paper]

1.5 Hybrid/Linear Models

1. Falcon Mamba: The First Competitive Attention-free 7B Language Model. [paper]
2. MiniMax-01: Scaling Foundation Models with Lightning Attention. [paper]
3. Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models. [paper]

⬆️ Back to Table of Contents

All Technical Reports

LLaMA Series

1. LLaMA: Open and Efficient Foundation Language Models. [paper]
2. Llama 2: Open Foundation and Fine-Tuned Chat Models. [paper]
3. The Llama 3 Herd of Models. [paper]

Qwen Series

1. Qwen Technical Report. [paper]
2. Qwen2 Technical Report. [paper]
3. Qwen2.5 Technical Report. [paper]

DeepSeek Series

1. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. [paper]
2. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. [paper]
3. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. [paper]
4. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. [paper]
5. DeepSeek-V3 Technical Report. [paper]

Gemma Series

1. Gemma: Open Models Based on Gemini Research and Technology. [paper]
2. Gemma 2: Improving Open Language Models at a Practical Size. [paper]
3. Gemma 3 Technical Report. [paper]

Gemini Series

1. Gemini: A Family of Highly Capable Multimodal Models. [paper]
2. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. [paper]

Mistral Series

1. Mistral 7B. [paper]
2. Mixtral of Experts. [paper]

Phi Series

1. Textbooks Are All You Need. [paper]
2. Textbooks Are All You Need II: phi-1.5 technical report. [paper]
3. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. [paper]
4. Phi-4 Technical Report. [paper]

GLM Series

1. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. [paper]
2. GLM-130B: An Open Bilingual Pre-trained Model. [paper]
3. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. [paper]

Baichuan Series

1. Baichuan 2: Open Large-scale Language Models. [paper]
2. Baichuan-M1: Pushing the Medical Capability of Large Language Models. [paper]

Falcon Series

1. The Falcon Series of Open Language Models. [paper]
2. Falcon2-11B Technical Report. [paper]
3. Falcon Mamba: The First Competitive Attention-free 7B Language Model. [paper]

InternLM Series

1. InternLM: A Multilingual Language Model with Progressively Enhanced Capabilities. [paper]
2. InternLM2 Technical Report. [paper]

MiniCPM

1. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. [paper]

Yi Series

1. Yi: Open Foundation Models by 01.AI. [paper]
2. Yi-Lightning Technical Report. [paper]

Minimax Series

1. MiniMax-01: Scaling Foundation Models with Lightning Attention. [paper]

Reka Series

1. Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models. [paper]

Skywork Series

1. Skywork: A More Open Bilingual Foundation Model. [paper]
2. Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models. [paper]

Hunyuan Series

1. Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent. [paper]

Nemotron Series

1. Nemotron-4 15B Technical Report. [paper]
2. Nemotron-4 340B Technical Report. [paper]
3. Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models. [paper]

Ling Series

1. Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs. [paper]

OLMo Series

1. OLMo: Accelerating the Science of Language Models. [paper]
2. 2 OLMo 2 Furious. [paper]
3. OLMoE: Open Mixture-of-Experts Language Models. [paper]

YuLan Series

1. YuLan: An Open-source Large Language Model. [resource] [code] [paper]
2. YuLan-Mini: An Open Data-efficient Language Model. [code] [resource] [paper]

MAP-Neo Series

1. MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series. [paper]

LLM360 Project

1. LLM360: Towards Fully Transparent Open-Source LLMs. [paper]

⬆️ Back to Table of Contents

II. Training Strategies

We discuss training strategies from aspects such as training frameworks, training strategies, interpretability, model architecture improvements, and learning rate annealing.

2.1 Training Frameworks

The most commonly used training framework is Megatron-LM, which provides a good out-of-the-box and efficient benchmark. Combining it with other libraries can achieve better training speed.

1. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. [code] [paper]

   The most commonly used pre-training framework, with a high entry threshold but more stability.

2. Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts. [paper]

   Computation-communication overlapping for MoE.

3. DeepEP: an efficient expert-parallel communication library. [code]

   Expert parallel acceleration.

4. DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling. [code]

   Accelerating FP8 matrix multiplication using the asynchronous features of Hopper.

5. Liger Kernel: Efficient Triton Kernels for LLM Training. [code] [paper]

   Triton acceleration operator library.

All Training Frameworks

1. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. [code] [paper]

   Zero-redundancy data parallelism

2. TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining. [code] [paper]

   Torch-native parallelism based on DTensor

3. Flash Linear Attention [code]

   Efficient Triton-based implementations for state-of-the-art linear attention models

2.2 Training Strategies

Regarding hyperparameter Scaling Law, parallel strategies, initialization strategies, optimizer selection, FP8 training, etc.

1. Predictable Scale: Part I -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining. [paper] [homepage]

   About the Scaling Law of hyperparameters.

2. The Ultra-Scale Playbook: Training LLMs on GPU Clusters. [demo]

   Visualizing the memory usage of parallel strategies.

3. A Spectral Condition for Feature Learning. [paper]

   An advanced version of MuP.

4. Muon is Scalable for LLM Training. [code] [paper]

   An efficient optimizer.

5. COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training. [paper] [code]

   Training with optimizer states and activation values also in FP8.

6. Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models. [paper]

   About the Scaling Law of MoE.

2.3 Interpretability

We list some interpretability works that are inspiring for pre-training.

1. On the Biology of a Large Language Model. [blog]
2. Physics of Language Models. [homepage]
3. In-context Learning and Induction Heads. [blog] [paper]
4. Rethinking Reflection in Pre-Training. [paper]

2.4 Model Architecture Improvements

We list some recent improvements to model architectures.

1. Gated Delta Networks: Improving Mamba2 with Delta Rule. [paper]
2. RWKV-7 "Goose" with Expressive Dynamic State Evolution. [paper]
3. Mixture of Hidden-Dimensions Transformer. [paper]
4. Titans: Learning to Memorize at Test Time. [paper]
5. Ultra-Sparse Memory Network. [paper]
6. Large Language Diffusion Models. [paper]
7. Better & Faster Large Language Models via Multi-token Prediction. [paper]
8. Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing. [paper]
9. Stick-breaking Attention. [code] [paper]
10. Forgetting Transformer: Softmax Attention with a Forget Gate. [paper]
11. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. [unofficial code] [paper]
12. MoBA: Mixture of Block Attention for Long-Context LLMs. [code] [paper]
13. KV Shifting Attention Enhances Language Modeling. [paper]
14. Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models. [paper]
15. Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts. [paper]
16. ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMs. [paper]
17. μnit Scaling: Simple and Scalable FP8 LLM Training. [paper]

2.5 Learning Rate Annealing

Learning rate annealing is often combined with data quality screening.

1. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. [paper]
2. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. [paper]
3. Scaling Law with Learning Rate Annealing. [paper]

⬆️ Back to Table of Contents

III. Open-source Datasets

We discuss existing open-source datasets mainly from four aspects: web pages, mathematics, code, and general-purpose.

3.1 Web Pages

Web page data will form the core corpus in pre-training.

1. DCLM. [paper] [resource]

   An open-source web page dataset, a 3.8T dataset obtained after screening by Fasttext, etc.

2. FineWeb-Edu [paper] [resource]

   A corpus for educational quality scoring, screened and scored from FineWeb, which has certain effects on knowledge-intensive questions.

3. Nemotron-CC-HQ. [paper] [resource]

   NVIDIA's CC corpus.

4. Chinese-FineWeb-Edu. [resource]

   An open-source Chinese educational quality scoring corpus by OpenCSG, screened and scored from Map-CC, SkyPile, WuDao, Wanjuan, etc.

5. FineWeb2: A sparkling update with 1000s of languages [resource]

   A multilingual dataset.

3.2 Mathematics

Mathematical pre-training corpora can significantly improve the mathematical ability of the base model and the upper limit of post-training.

1. MegaMath: Pushing the Limits of Open Math Corpora. [resource] [paper]

   The largest open-source high-quality mathematical CC corpus.

2. JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models. [resource] [paper]

   Synthetic mathematical instruction data.

3. mlfoundations-dev/stackoverflow_math. [resource]

   Math-related questions.

4. DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning. [resource] [paper]

   A high-difficulty mathematical dataset.

5. YuLan-Mini: An Open Data-efficient Language Model. [code] [resource] [paper]

   Collecting open-source Lean theorem proving datasets.

3.3 Code

Code data can not only enhance the code generation ability of the base model but also improve its mathematical and logical abilities.

1. OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models. [resource] [paper]

   Cleaned from The-Stack-V2.

2. SmolLM-corpus. [resource]

   Python educational quality scoring.

3. The-Stack-V2. [resource]

   The largest-scale uncleaned code data.

4. YuLan-Mini: An Open Data-efficient Language Model. [code] [resource] [paper]

   Cleaning Jupyter-Notebook and Python data with educational quality.

5. HuggingFaceTB/issues-kaggle-notebooks. [resource]

   GitHub Issues and Kaggle Notebooks data.

6. mlfoundations-dev/stackoverflow. [resource]

   Programming Q&A forum.

7. Magicoder: Empowering Code Generation with OSS-Instruct. [resource] [paper]

   Training with synthetic instruction data generated from open-source code.

3.4 General-purpose (Books, Encyclopedias, Instructions, Long Contexts, etc.)

General-purpose data is often scarce long-tail data, which plays a crucial role in the usability of post-training models.

1. YuLan: An Open-source Large Language Model. [resource] [code] [paper]

   Long-tail knowledge enhancement and cleaning of various general-purpose data sources.

2. MinerU: An Open-Source Solution for Precise Document Content Extraction. [code] [paper]

   Converting PDF to Markdown with strong compatibility.

3. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. [homepage] [paper]

   arXiv, conversations, DM Math, etc.

4. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. [resource] [paper]

   Encyclopedias, books, papers, Reddit, etc.

5. WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models. [resource] [paper]

   Law, exams, news, patents, encyclopedias, etc.

6. MAmmoTH2: Scaling Instructions from the Web. [resource] [paper]

   Q&A for web pages.

7. togethercomputer/Long-Data-Collections. [resource]

   Filtered books, papers, web pages, and instructions from datasets such as RedPajama, Pile, and P3.

8. Longattn: Selecting long-context training data via token-level attention. [resource] [paper]

   Q&A for long-range dependencies.

⬆️ Back to Table of Contents

IV. Data Methods

Datasets are often paired with high-quality data methods. We elaborate on this from aspects such as tokenizers, data配比 and courses, and data synthesis.

4.1 Tokenizers

Tokenization is an important but often overlooked part of the model, which can significantly affect the model's ability in mathematics, knowledge, etc.

1. SuperBPE: Space Travel for Language Models. [code] [paper]

   A multi-word tokenizer training method.

2. Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies. [code] [demo] [paper]

   Predicting the vocabulary size.

3. Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs. [code] [paper]

   Comparing the tokenization methods of numbers.

4.2 Data Mixing and Curriculum

Multi-stage pre-training often enables the model to fully learn high-quality and small-scale data. Introducing more mathematical, code, CoT, and even long-thinking chain data in the continued pre-training (CPT) stage will form the core capabilities of the next generation of pre-trained models.

1. Nemotron-4 15B Technical Report. [paper]

   Divided into 8T pre-training + CPT with a smaller data scale.

2. YuLan-Mini: An Open Data-efficient Language Model. [code] [resource] [paper]

   Using educational scores for course data.

3. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. [code] [paper]

   Optimizing the pre-training data mixing ratio.

4. Efficient Online Data Mixing For Language Model Pre-Training. [paper]

   Online data mixing.

5. Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance. [paper]

   Data mixing laws.

6. Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?. [paper]

   Cracking the data ratio of commercial models such as GPT through the merging rules of BPE tokenizers.

7. CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training. [homepage] [paper]

   A clustering-based iterative data mixing bootstrapping framework.

8. Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens. [demo] [homepage] [paper]

   Building an index for large-scale pre-training datasets to check data quality.

4.3 Data Synthesis

In addition to the synthetic data for mathematics and code mentioned above, we summarize some general synthetic data methods and resources. Moreover, using more long-thinking data in the later stage of pre-training is also becoming a direction worthy of exploration.

1. Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems. [resource] [code] [paper]

   Imitation learning based on long-thinking chain synthetic data.

2. Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructions. [code] [paper]

   Generating synthetic instruction data rich in information to learn knowledge from a limited corpus.

3. LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs. [resource] [code] [paper]

   Constructing long-text Creative Writing.

4. Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use. [paper]

   Multi-step reasoning data synthesis, decomposing complex tasks into sub-trajectories and optimizing data generation with reinforcement learning.

5. WildChat: 1M ChatGPT Interaction Logs in the Wild. [resource] [paper]

   An open-source dataset of real user conversations.

6. Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing. [resource] [code] [paper]

   Alignment data synthesis.

⬆️ Back to Table of Contents

Contribution

If you have suggestions for the project content, please submit Issues and PRs to jointly promote the development of large language models.

Star History