Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search

        

This repository contains the official implementation of Satori-Qwen-7B in the paper Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search (ICML 2025).

News 🎉

• [2025/06/02] We have released our RL training code to help the community reproduce our work!
• [2025/05/01] Our paper has been accepted to present at ICML 2025!
• [2025/02/04] We have released our model and data through HuggingFace.

Introduction

Satori is a 7B parameter LLM that can autoregressive search for reasoning, allowing it to self-reflect and self-explore alternative strategies without external guidance. Built on Qwen-2.5-Math-7B, Satori achieves state-of-the-art reasoning performance through small-scale Format Tuning (FT) and large-scale self-improvement via reinforcement learning (RL).

Key Features

• Autoregressive Search Capabilities: Self-reflection and self-exploration without external feedback.

• Chain-of-Action-Thought (COAT) Reasoning: Leverage several meta-action tokens to guide reasoning,

  • Continue Reasoning (<|continue|>): encourages the LLM to build upon its current reasoning trajectory by generating the next intermediate step.
  • Reflect (<|reflect|>): prompts the model to pause and verify the correctness of prior reasoning steps.
  • Explore Alternative Solution (<|explore|>): signals the model to identify critical flaws in its reasoning and explore a new solution.

• Transferability: Train on math domain, but generalize to unseen domains beyond math.

Training Framework

1. Format Tuning (FT) via Imitation Learning

• Use a multi-agent data synthesis framework (Generator, Critic, Reward Model) to create COAT-style demonstration trajectories.
• Train the base model to imitate the COAT reasoning format.

2. Self-Improvement via Reinforcement Learning (RL)

• Restart and Explore (RAE): Train the model to reason from intermediate states to encourage deeper reflection.
• Iterative Self-Improvement: Alternate between RL training and policy distillation for iterative improvements.

Quick Start 🔧

Installation

To train our model with OpenRLHF, first install the environment with:

cd Satori
pip install -e .

Model and Data

• Download model checkpoints: SFT Model Ckpt, Outcome Reward Model.
• Download RL training data: RL Data with RAE.

Training Script

bash examples/satori/train.sh

Evaluation

Math Reasoning Evaluation

Satori-Qwen-7B achieves SOTA performance and outperforms Qwen-2.5-Math-7B-Instruct which uses the same base model (Qwen-2.5-Math-7B). After round 2 training, Satori-Qwen-7B (Round 2) demonstrates even stronger performance on hard tasks.

General Domain Reasoning Benchmarks

Trained only on math datasets, Satori-Qwen-7B exhibits strong transferability across diverse out-of-domain reasoning benchmarks and outperforms Qwen-2.5-Math-7B-Instruct by a large margin. Moreover, despite not being trained in other domains, Satori-Qwen-7B achieves performance comparable to or exceeding other small-scale general instruct models.

Satori Team Members

Core Contributors

• Maohao Shen, MIT
• Guangtao Zeng, SUTD
• Zhenting Qi, Harvard

Contributors

*: Project lead

• Zhang-Wei Hong, MIT
• Zhenfang Chen, MIT-IBM Watson AI Lab
• Wei Lu, SUTD
• Gregory W. Wornell, MIT
• Subhro Das, MIT-IBM Watson AI Lab
• David Cox, MIT-IBM Watson AI Lab
• Chuang Gan*, UMass, MIT-IBM Watson AI Lab

Contact Information

For questions, please:

• Raise an issue in our GitHub repository
• Contact us at:
  • satori2025@outlook.com

Citation

@misc{shen2025satorireinforcementlearningchainofactionthought,
      title={Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search}, 
      author={Maohao Shen and Guangtao Zeng and Zhenting Qi and Zhang-Wei Hong and Zhenfang Chen and Wei Lu and Gregory Wornell and Subhro Das and David Cox and Chuang Gan},
      year={2025},
      eprint={2502.02508},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.02508}, 
}