Overview

• News
• Introduction
• Repository Structure
• Getting Started
• Usage
• Open-source Models
• Open-source Datasets
• Evaluation
• Citation

📢 News

• [2026/08/15] We update the camera-ready and arxiv version. We add more experiments and related references.
• [2026/07/09] The paper is accepted by COLM 2026!
• [2026/04/22] Environment setup is now available. You can either install dependencies with requirements.txt or use our Docker Hub image: jasonrqh/sft-generalization:v0.1.
• [2026/04/15] We are glad that our paper has helped guide practical model training of Jackrong/Gemopus-4-31B-it, which is another impressive release from the author of Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled. Check these models if you are interested.
• [2026/04/11] We release the raw dataset before filtering and random selection (44k queries, each with 32 responses): Huggingface, Modelscope. We also release the token-level log probability and entropy from the teacher model. We did not make full use of this dataset, and believe that it might be useful for future research.
• [2026/04/10] Our paper is available on Huggingface daily paper. If you enjoy our work, we warmly invite you to upvote it on Huggingface!
• [2026/04/09] Our paper is available on arXiv. We open-sourced all models and datasets in this Huggingface collection.

Introduction

The paper revisits the prevailing narrative that "SFT memorizes, RL generalizes", mainly focusing on reasoning SFT (with long-CoT supervision). Our core conclusion is that generalization in reasoning SFT is highly conditional, jointly shaped by:

• Optimization dynamics
• Training data quality and structure
• Base model capability

Main findings:

1. Cross-domain performance often follows a dip-and-recovery trajectory, so short training can underestimate final generalization.
2. Verified long-CoT supervision yields stronger transfer, while low-quality data can produce misleading non-generalization signals.
3. Stronger base models are more likely to internalize transferable procedural reasoning patterns and generalize better.
4. Generalization is asymmetric: reasoning improves, but safety can degrade.

Repository Structure

|-- training_scripts/                 # SFT scripts used for the paper experiments
|-- evaluation/                       # Unified evaluation toolkit (lm-eval-harness/evalchemy/math/alpaca/safety)
|-- verl/                             # Core training framework (based on verl)
|   |-- trainer/fsdp_sft_trainer_ours.py
|   `-- utils/dataset/sft_dataset.py

Getting Started

1) Environment setup

You can set up the environment in either of the following ways:

# Option 1: install dependencies from requirements.txt
pip install -r requirements.txt

# Option 2: use our Docker Hub image
docker pull jasonrqh/sft-generalization:v0.1

Main dependencies:

• Core training/runtime: torch, transformers, datasets, accelerate, hydra-core, ray, tensordict, peft, pyarrow, wandb, torchdata,flash-attn
• Evaluation: numpy, pandas, scikit-learn, matplotlib, fastapi, vllm

2) Required script edits before training

Update placeholders in training_scripts/*.sh:

1. ROOT_DIR=/path/to/this/repo
2. TRAIN_DATA=/path/to/{dataset_name}
3. WANDB_API_KEY=your_wandb_key

MODEL_PATH is already set in the provided scripts. Change it only if you want to use a different base model.

Distributed variables are expected from the runtime environment:

1. NODE_COUNT
2. PROC_PER_NODE
3. NODE_RANK
4. MASTER_ADDR

We trained all models on 8 H200 GPUs. If you encounter OOM errors, consider using a smaller micro_batch_size.

3) Training data schema

Current scripts call verl.trainer.fsdp_sft_trainer_ours with:

1. data.prompt_key=message
2. data.response_key=response
3. data.advantage_key=advantage

Each parquet row should include at least:

1. message (chat-message list)
2. response (string)
3. advantage (scalar, commonly just 1.0)

Usage

1) Run training (example)

bash training_scripts/Qwen3-14B_Math-CoT-20k_lr5e-5_ep8_bs256.sh

2) Representative scripts aligned with paper settings

1. Default Math-CoT setting
   training_scripts/Qwen3-14B_Math-CoT-20k_lr5e-5_ep8_bs256.sh
2. Short-training replication
   training_scripts/Qwen3-14B_Math-CoT-20k_lr5e-5_ep1_bs256.sh
3. Smaller LR setting
   training_scripts/Qwen3-14B_Math-CoT-20k_lr1e-5_ep1_bs256.sh
4. Overfitting stress-test schedules
   training_scripts/Qwen3-14B_Math-CoT-20k_lr5e-5_ep16_bs256_ConstLR.sh
training_scripts/Qwen3-14B_Math-CoT-20k_lr1e-4_ep16_bs256_ConstLR.sh
5. Data variants
   training_scripts/Qwen3-14B_Math-NoCoT-20k_lr5e-5_ep8_bs256.sh
training_scripts/Qwen3-14B_Countdown-CoT-20k_lr5e-5_ep8_bs256.sh
training_scripts/Qwen3-14B_DeepSeek-R1-20k_lr5e-5_ep8_bs256.sh training_scripts/Qwen3-14B_Numina-Math-20k_lr5e-5_ep8_bs256.sh
6. Capability scaling (Qwen3 1.7B/4B/8B/14B)
   training_scripts/Qwen3-1.7B_Math-CoT-20k_lr5e-5_ep8_bs256.sh
training_scripts/Qwen3-4B_Math-CoT-20k_lr5e-5_ep8_bs256.sh
training_scripts/Qwen3-8B_Math-CoT-20k_lr5e-5_ep8_bs256.sh
training_scripts/Qwen3-14B_Math-CoT-20k_lr5e-5_ep8_bs256.sh

3) Merge FSDP checkpoints

python -m verl.model_merger merge \
  --backend fsdp \
  --local_dir /path/to/ckpt/global_step_640 \
  --target_dir /path/to/ckpt/merged_step640 \
  --trust_remote_code

Batch merge helper:

bash training_scripts/model_merger.sh

Before using model_merger.sh, update ckpt_path_list in the script (for example /path/to/model/global_step_640).

Open-source Models

We have open-sourced ALL models trained in our experiments, including the intermediate checkpoints (you can find them in the stepxxx folder in the repo).

Note that the following model list may include repeated entries, as it is organized by the experiments and conclusions presented in the paper.

Open-source Datasets

We provide the main datasets used in our experiments.

✨Note: We also release the raw dataset before filtering and random selection: Huggingface, Modelscope. This dataset contains around 44k queries, each with 32 responses generated by Qwen3-32B. For each textual response, we also release the token-level log probability and entropy from the teacher model. We believe this might be useful for someone who is interested in response diversity, token probability distribution, or other related topics.

Evaluation

Please refer to the evaluation guide in evaluation/README.md for details.

Citation

If you use our code, model, or dataset in your project, please consider citing us.

@inproceedings{ren2026rethinking,
  title={Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability},
  author={Qihan Ren and Peng Wang and Ruikun Cai and Shuai Shao and Dadi Guo and Yuejin Xie and Yafu Li and Quanshi Zhang and Xia Hu and Jing Shao and Dongrui Liu},
  booktitle={Third Conference on Language Modeling},
  year={2026},
  url={https://openreview.net/forum?id=EvSOFcwnEb}
}

Acknowledgement

We thank the open-source communities and contributors behind verl, TinyZero, lm-evaluation-harness, evalchemy, Transferability-of-LLM-Reasoning, LLM-Extrapolation, and HarmBench.