awesome-llm-papers

A comprehensive repository for research papers, code snippets, and notes related to LLMs.

Summary

• awesome-llm-papers
  • Summary
  • OutStanding LLM Technical Report
    • GPT-4
    • GPT-4.5
    • Qwen2_5_1M_Technical_Report
    • KIMI-1.5
    • DeepSeek_R1
    • DeepSeek_V3
    • LLama3.1
  • Architecture
    • Attention Is All You Need
    • ERNIE - Enhanced Language Representation with Informative Entities
    • XLNet - Generalized Autoregressive Pretraining for Language Understanding
    • RoBERTa - A Robustly Optimized BERT Pretraining Approach
    • Swin Transformer - Hierarchical Vision Transformer using Shifted Windows
    • Learning Transferable Visual Models From Natural Language Supervision
  • Parallel Training
    • Training_Multi-Billion_Parameter_Language_Models_Usin_Model_Parallelism
    • GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
    • ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
    • DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters
    • Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
  • PreTraining
    • Model Merging in Pre-training of Large Language Models
    • Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
    • Improving Language Understanding by Generative Pre-Training
    • BERT
    • Language Models are Few-Shot Learners
    • OLMO2
  • Post-Training
    • LEARNING DYNAMICS OF LLM FINETUNING ICLR 2025 Oral, Outstanding Paper Award
    • DPO
    • NeurIPS-2022-training-language-models-to-follow-instructions-with-human-feedback-Paper-Conference

OutStanding LLM Technical Report

GPT-4

GPT-4 is OpenAI's large multimodal language model, capable of processing text and visual inputs.

Paper: GPT-4

GPT-4.5

GPT-4.5, is an advanced version of GPT-4 with enhanced pattern recognition and creativity. It supports 15 languages and offers improved conversational abilities.

Paper: GPT-4.5

Qwen2_5_1M_Technical_Report

The Qwen2.5-1M models extend context length to 1 million tokens. The series includes open-source models Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M, and the API-accessible Qwen2.5-Turbo.

Paper: Qwen2_5_1M_Technical_Report

KIMI-1.5

DeepSeek_R1

DeepSeek-R1-Zero and DeepSeek-R1. DeepSeek-R1-Zero, a model trained via large-scale reinforcement learning (RL) without supervised fine-tuning (SFT) .

Paper: DeepSeek-R1

DeepSeek_V3

DeepSeek_V3 introduces enhancements over its predecessors by incorporating advanced optimization techniques and expanded training data.

Paper: DeepSeek_V3

LLama3.1

The largest model is a dense Transformer with 405B parameters, processing information in a context window of up to 128K tokens.

Paper: LLama3.1

---

Architecture

Attention Is All You Need

The "Attention Is All You Need" paper introduces the Transformer architecture, which relies entirely on self-attention mechanisms.

Paper: Attention Is All You Need

ERNIE - Enhanced Language Representation with Informative Entities

ERNIE enhances language representation by integrating structured knowledge about entities.

Paper: ERNIE - Enhanced Language Representation with Informative Entities

XLNet - Generalized Autoregressive Pretraining for Language Understanding

XLNet combines autoregressive modeling with autoencoding to capture bidirectional context without the limitations of BERT.

Paper: XLNet - Generalized Autoregressive Pretraining for Language Understanding

RoBERTa - A Robustly Optimized BERT Pretraining Approach

RoBERTa optimizes the BERT pretraining process by removing the next sentence prediction objective and training with larger mini-batches and learning rates.

Paper: RoBERTa - A Robustly Optimized BERT Pretraining Approach

---

Swin Transformer - Hierarchical Vision Transformer using Shifted Windows

The Swin Transformer introduces a hierarchical structure using shifted windows for attention computation.

Paper: Swin Transformer - Hierarchical Vision Transformer using Shifted Windows

---

Learning Transferable Visual Models From Natural Language Supervision

The CLIP model learns visual representations by training on image-text pairs from the internet.

Paper: Learning Transferable Visual Models From Natural Language Supervision

---

Parallel Training

Training_Multi-Billion_Parameter_Language_Models_Usin_Model_Parallelism

This paper presents techniques for efficiently training language models with billions of parameters via model parallelism, enabling the scaling of model size beyond single GPU memory limitations.

Paper: Training Multi-Billion Parameter Language Models Using Model Parallelism

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

GPipe proposes a pipeline parallelism approach that splits models into segments and processes micro-batches in a pipeline, significantly improving memory efficiency and scalability.

Paper: GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

ZeRO (Zero Redundancy Optimizer) introduces a set of memory optimization techniques that enable the training of trillion-parameter models by partitioning optimizer states, gradients, and parameters across devices.

Paper: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters

DeepSpeed presents a deep learning optimization library that combines ZeRO and other system techniques to enable efficient training of models with over 100 billion parameters.

Paper: DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Switch Transformers leverages sparse expert models and routing to enable the scaling of language models to over a trillion parameters while maintaining computational efficiency.

Paper: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

---

PreTraining

Model Merging in Pre-training of Large Language Models

The paper introduces the Pre-trained Model Average (PMA) strategy for merging model weights during the pre-training phase of large language models (LLMs).

Paper: Model Merging in Pre-training of Large Language Models

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

This paper explores the capabilities of a unified text-to-text framework for transfer learning.

Paper: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Improving Language Understanding by Generative Pre-Training

The Generative Pre-Training (GPT) model improves language understanding by pre-training on a large corpus of text data and then fine-tuning on specific tasks.

Paper: Improving Language Understanding by Generative Pre-Training

BERT

BERT (Bidirectional Encoder Representations from Transformers) presents a novel approach to pre-training language representations.

Paper: BERT

Language Models are Few-Shot Learners

This paper demonstrates that large language models, like GPT-3, can perform tasks with little to no task-specific training.

Paper: Language Models are Few-Shot Learners

OLMO2

Pre Train 2 OLMO2 discusses the advancements in pretraining strategies for the OLMO2 model. Paper: pre_train_2_OLMO2

---

Post-Training

LEARNING DYNAMICS OF LLM FINETUNING ICLR 2025 Oral, Outstanding Paper Award

It provides a framework to analyze how the learning of specific training examples influences the model's predictions on other examples, offering insights into the behavior of deep learning systems.

Paper: DPO

DPO

DPO (Direct Preference Optimization) focuses on optimizing models based on direct human preferences. This approach enhances model alignment with user intentions by incorporating preference data directly into the training process.

Paper: DPO

NeurIPS-2022-training-language-models-to-follow-instructions-with-human-feedback-Paper-Conference

This paper presents methods for training language models to follow human instructions more effectively using reinforcement learning from human feedback.

Paper: NeurIPS-2022 Training Language Models to Follow Instructions with Human Feedback

---