Easy-LLM-Post-Training

[English | 中文]

Implement popular LLM post-training algorithms (SFT, DPO, PPO, GRPO, etc.) in PyTorch with clean, simple code!

以简洁代码，基于 PyTorch 实现主流大模型后训练算法！

Break free from complex frameworks and focus on the core logic of LLM post training algorithms.

摆脱繁杂框架束缚，专注于大模型后训练算法的核心逻辑。

📊 Getting Start

Env

conda create --name myenv python=3.10
pip install -r requirements.txt

Data

You can get started quickly using the sample data in the data, including SFT data, DPO data, etc.

Training

Taking SFT as an example:

cd sft/
bash train.sh

🚀 Supported Algorithms

SFT-Series

• Supervised Fine-Tuning (SFT) [Code]
• Dynamic Fine-Tuning (DFT) [Paper | Code]

DPO-Series

• Direct Preference Optimization (DPO) [Paper]
• Simple Preference Optimization (SimPO) [Paper | Code]
• Odds Ratio Preference Optimization (ORPO) [Paper | Code]

PPO-Series

• Proximal Policy Optimization (PPO) [Paper | Code]
• Group Relative Policy Optimization (GRPO) [Paper | Code]
• Group Sequence Policy Optimization (GSPO) [Paper | Code]
• REINFORCE++ [Paper | Code]
• Group Relative Policy Optimization Done Right (Dr.GRPO) [Paper | Code]
• Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) [Paper | Code]
• REINFORCE Leave-One-Out (RLOO) [Paper | Code]

🔄 Upcoming Methods

• TIS-GRPO
• VAPO
• More coming soon