R-PRM: Reasoning-Driven Process Reward Modeling

📃 Paper | 📝 Blog | ⚙️ Code | 🤖 Model | 🤗 Dataset | 📭 Contact

Overview

Welcome to the repository of R-PRM, our cutting-edge framework designed to revolutionize process-level evaluation in mathematical reasoning for large language models (LLMs).

• 🚀 We introduce Reasoning-Driven Process Reward Modeling (R-PRM), a novel approach that enhances LLMs' ability to evaluate mathematical reasoning step-by-step. By leveraging stronger LLMs to generate seed data, optimizing preferences without additional annotations, and scaling inference-time computation, R-PRM delivers comprehensive, transparent, and robust assessments of reasoning processes.
• 📈 Our framework significantly boosts evaluation accuracy and generalization, outperforming strong baselines by wide margins on ProcessBench and PRMBench. When guiding policy models, R-PRM consistently improves reasoning performance across diverse datasets, achieving state-of-the-art (SOTA) results.
• 🌐 Overall, R-PRM offers a scalable and data-efficient solution to the challenge of scarce process-level annotations, enabling a more generalizable enhancement of reasoning evaluation capabilities without extensive human labeling.

🏆 Experiment Results

🧪 Data Efficiency

R-PRM demonstrates exceptional data efficiency under varying training scales:

• With just 12.8k training samples, R-PRM reaches F1 = 52.6, already surpassing most open-source PRMs.
• R-PRM achieves +3.6 F1 over Qwen2.5-Math-7B-PRM800K when trained on just 64k samples (vs. Qwen's 265k), and extends this lead to +8.7 F1 when both are trained on comparable data volumes.
• Notably, despite using only ~15% of the data, R-PRM’s performance is already comparable to Qwen2.5-Math-PRM, which was trained on a much larger 1.8M LLM-filtered dataset.

📊 ProcessBench

Our reasoning-driven framework improves over Qwen2.5-Math-7B-PRM800K by +8.7 F1 (SFT) and +13.9 F1 (DPO), demonstrating its powerful evaluation capability.

🧠 PRMBench

R-PRM achieves +8.5 F1 (DPO) over Qwen2.5-Math-7B-PRM800K 📌 Excels in soundness, sensitivity, and multi-dimensional error analysis.

🧪 Best-of-N Strategy

When selecting the best among N reasoning paths, R-PRM improves accuracy by +8.6 points over the Pass@1 baseline, achieving the best results among all PRMs across six math datasets.

🔁 Guide Search Strategy

By guiding reasoning step-by-step, R-PRM surpasses Pass@1 by +8.4 points, outperforming both majority voting and previous PRM-guided methods.

🚀 Inference-Time Scaling

Evaluation performance improves consistently as more reasoning trajectories are sampled at inference. → From 62.8 F1 (2 samples) to 67.6 F1 (4 samples) on ProcessBench. This showcases R-PRM’s ability to deliver robust, ensemble-style judgment through multi-path reasoning.

Citation

If you find this repository helpful, feel free to cite our paper:

@misc{she2025rprmreasoningdrivenprocessreward,
      title={R-PRM: Reasoning-Driven Process Reward Modeling}, 
      author={Shuaijie She and Junxiao Liu and Yifeng Liu and Jiajun Chen and Xin Huang and Shujian Huang},
      year={2025},
      eprint={2503.21295},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2503.21295}, 
}