Parallel-R1

The official repository for "Parallel-R1: Towards Parallel Thinking via Reinforcement Learning".

Updates

• 2026-1-26: 🎉Parallel-R1 was accepted at ICLR2026.
• 2025-10-15: 🎉Parallel-R1 was accepted at Neurips 2025 Efficient Reasoning workshop (Spotlight).
• 2025-10-05: Released Training/Evaluation code; Released Qwen3-4B-Base with adding special tokens with special tokens, along with the Parallel-Unseen (S2) 200-step checkpoint to support reproduction of mid-training experiments.
• 2025-09-11: The cold-start dataset, Parallel-GSM8K, is now available on Hugging Face. (Stay tuned for training/evaluation codes)
• 2025-09-10: We have released our paper, "Parallel-R1: Towards Parallel Thinking via Reinforcement Learning".

About

This project introduces Parallel-R1, a new reinforcement learning (RL) framework designed to teach large language models (LLMs) parallel thinking for complex, real-world reasoning tasks. Unlike existing methods that rely on supervised fine-tuning (SFT) over costly, synthetic data, our approach enables models to learn parallel behaviors through RL.

Key Contributions

• A Novel RL Framework: We present the first RL framework to learn parallel thinking on general mathematical reasoning tasks.
• Progressive Curriculum: The framework uses a progressive curriculum to overcome the "cold-start" problem. It begins with SFT on simpler tasks to teach the model the basic format of parallel thinking before transitioning to RL on harder problems.
• Simple and Scalable Data Pipeline: We developed a simple data pipeline that uses prompting on easy math problems (like GSM8K) to generate high-quality "cold-start" data, which is then used to teach the model the parallel thinking format.
• Strategic Evolution Analysis: We provide an in-depth analysis showing that the model's strategy evolves over the course of training. It shifts from using parallel paths for early-stage computational exploration to using them for late-stage multi-perspective verification.
• Mid-Training Exploration Scaffold: We empirically validate the concept of using parallel thinking as a temporary "mid-training exploration scaffold" to unlock a higher performance ceiling after RL training.

Main Results

🧱 Release Overview

Training Logs

🤗 Training Logs

🚀 Usage

1️⃣ Environment Setup

We recommend using Python 3.10+ and creating a fresh conda environment:

cd verl
conda create -n parallel-r1 python=3.10 -y
conda activate parallel-r1
USE_MEGATRON=0 bash scripts/install_vllm_sglang_mcore.sh
pip install --no-deps -e .

2️⃣ Perform SFT

cd verl
sh training_scripts/sft_exp.sh

2️⃣ Perform RL

To train Parallel-R1-Unseen (S1) from scratch

cd verl
sh training_scripts/rl_exp_s1.sh

To train Parallel-R1-Unseen (S2) from scratch

cd verl
sh training_scripts/rl_exp_s2.sh

To reproduce results of mid-training experiments

cd verl
sh training_scripts/mid_training_exp.sh

Citation

If you think this work is useful, please cite our paper.

@article{zheng2025parallel,
  title={Parallel-r1: Towards parallel thinking via reinforcement learning},
  author={Zheng, Tong and Zhang, Hongming and Yu, Wenhao and Wang, Xiaoyang and Yang, Xinyu and Dai, Runpeng and Liu, Rui and Bao, Huiwen and Huang, Chengsong and Huang, Heng and others},
  journal={arXiv preprint arXiv:2509.07980},
  year={2025}
}