📚 AwesomeRM: A Survey of Reward Models for LLM

"Good Teachers, Better Students" - A Comprehensive Survey of Reward Models

中文版 | English

---

🌟 Highlights

• 📖 285+ Papers systematically organized (2017-2025)
• 🎯 29 Evaluation Benchmarks covering output quality, human preferences, and process quality
• 🗓️ Development Timeline tracking key milestones in the RM field
• 🔗 Companion Website https://linjh1118.github.io/AwesomeRM

---

📑 Table of Contents

• Core Concepts
• RM Taxonomy
  • Discriminative RM
  • Generative RM
  • Implicit RM
• Application Scenarios
  • Data Processing
  • RL Training
  • Test-Time Scaling
• Evaluation Benchmarks
• Milestone Models
• Challenges and Future
• Complete Paper List
• Citation
• Contributing

---

Core Concepts

Reward Model (RM) is a function that maps input-output pairs to scalar reward signals:

r = R_ψ(x, y)

Where:

• x: Input prompt
• y: Model-generated output
• r: Reward score (scalar)
• ψ: Model parameters

Reward models act as "teachers" throughout the LLM lifecycle, guiding continuous model improvement through feedback signals.

Three Paradigms

---

RM Taxonomy

Discriminative RM

Discriminative reward models evaluate output quality through explicit scoring functions.

🔹 Classification Methods

Transform preference judgments into classification problems (good/bad, win/lose).

Representative Papers:

• Hpsv3: Towards wide-spectrum human preference score - Ma, Yuhang et al., 2025. Proceedings of the IEEE/CVF International Conference on Computer Vision
• HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages - Wang, Zhilin et al., 2025. arXiv:2505.11475
• Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation - Xu, Jiazheng et al., 2024. arXiv:2412.21059
• Helpsteer 2: Open-source dataset for training top-performing reward models - Wang, Zhilin et al., 2024. Advances in Neural Information Processing Systems
• Imagereward: Learning and evaluating human preferences for text-to-image generation - Xu, Jiazheng et al., 2023. Advances in Neural Information Processing Systems
• Helpsteer: Multi-attribute helpfulness dataset for steerlm - Wang, Zhilin et al., 2023. arXiv:2311.09528

See all discriminative RM papers

🔹 Ranking Methods

Learn comparison functions to rank multiple candidate outputs.

🔹 Regression Methods

Directly predict continuous reward scores.

🔹 Model Ensembling

Combine outputs from multiple reward models.

---

Generative RM

Generative reward models implicitly model reward signals through the generation process.

Representative Papers:

• Process-based self-rewarding language models - Zhang, Shimao et al., 2025. arXiv:2503.03746
• MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation - Yutong Wang, et~al., 2025. arXiv:2502.12468
• Self-rewarding correction for mathematical reasoning - Wei Xiong et al., 2025. arXiv:2502.19613
• Self-rewarding language models - Yuan, Weizhe et al., 2024. Forty-first International Conference on Machine Learning
• Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge - Wu, Tianhao et al., 2024. arXiv:2407.19594
• Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark - Chen, Dongping et al., 2024. Forty-first International Conference on Machine Learning
• Judging llm-as-a-judge with mt-bench and chatbot arena - Zheng, Lianmin et al., 2023. Advances in neural information processing systems

See all generative RM papers

---

Implicit RM

Implicit reward models don't require explicit reward function training.

---

Application Scenarios

Data Processing

Reward models are used in data preprocessing for selecting and annotating high-quality training data.

RL Training

Reward models provide core feedback signals in RLHF/GRPO training.

Key Applications:

• Preference Alignment: InstructGPT, GPT-4, Claude series
• Reasoning Verification: Math-Shepherd, DeepSeekMath series

Test-Time Scaling

Reward models guide generation and selection during inference.

---

Evaluation Benchmarks

📊 Output Quality Evaluation (16 benchmarks)

📊 Human Preference Alignment (8 benchmarks)

📊 Process Quality Evaluation (5 benchmarks)

---

Milestone Models

🗓️ Development Timeline

---

Challenges and Future

🚧 Current Challenges

1. Reward Hacking - Models learn to "game" reward functions
2. Long-context Forgetting - Difficulty in preference modeling for long contexts
3. Scalability of Generative RMs - High inference costs
4. Information-sparse Scalar Rewards - Single scores can't capture multi-dimensional preferences
5. Process Supervision Complexity - High cost of step-level annotations
6. Multimodal Alignment Complexity - Difficulty in defining cross-modal preferences

🔮 Future Directions

1. Adaptive Reward Modeling - Dynamically adjust reward functions based on tasks
2. Explainable Reward Signals - Provide human-understandable feedback
3. Multi-objective Reward Optimization - Balance quality, safety, and diversity
4. Active Learning and Online Adaptation - Continuously learn preferences from interactions
5. Cross-domain Transfer - Generalized reward representations

---

Complete Paper List

Statistics

• Total Papers: 285
• By Category:
  • Discriminative RM: 6
  • Generative RM: 7
  • General RM: 112
  • Other: 160
• By Domain:
  • Math Reasoning: 46
  • Code Generation: 11
  • Multimodal: 69
  • Agent: 10

Discriminative Reward Models

• Hpsv3: Towards wide-spectrum human preference score - Ma, Yuhang et al., 2025. Proceedings of the IEEE/CVF International Conference on Computer Vision
• HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages - Wang, Zhilin et al., 2025. arXiv:2505.11475
• Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation - Xu, Jiazheng et al., 2024. arXiv:2412.21059
• Helpsteer 2: Open-source dataset for training top-performing reward models - Wang, Zhilin et al., 2024. Advances in Neural Information Processing Systems
• Imagereward: Learning and evaluating human preferences for text-to-image generation - Xu, Jiazheng et al., 2023. Advances in Neural Information Processing Systems
• Helpsteer: Multi-attribute helpfulness dataset for steerlm - Wang, Zhilin et al., 2023. arXiv:2311.09528

Generative Reward Models

• Process-based self-rewarding language models - Zhang, Shimao et al., 2025. arXiv:2503.03746
• MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation - Yutong Wang, et~al., 2025. arXiv:2502.12468
• Self-rewarding correction for mathematical reasoning - Wei Xiong et al., 2025. arXiv:2502.19613
• Self-rewarding language models - Yuan, Weizhe et al., 2024. Forty-first International Conference on Machine Learning
• Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge - Wu, Tianhao et al., 2024. arXiv:2407.19594
• Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark - Chen, Dongping et al., 2024. Forty-first International Conference on Machine Learning
• Judging llm-as-a-judge with mt-bench and chatbot arena - Zheng, Lianmin et al., 2023. Advances in neural information processing systems

General Reward Models

• A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models - Zheng, Congming et al., 2025. arXiv:2510.08049

• Reward Models in Deep Reinforcement Learning: A Survey - Yu, Rui et al., 2025. arXiv:2506.15421

• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey - Liu, Qiyuan et al., 2025. arXiv:2510.01925

• A comprehensive survey of reward models: Taxonomy, applications, challenges, and future - Zhong, Jialun et al., 2025. arXiv:2504.12328

• Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards - Wu, Xiaobao, 2025. arXiv:2505.02686

• Diffusion model as a noise-aware latent reward model for step-level preference optimization - Zhang, Tao et al., 2025. arXiv:2502.01051

• PRMBench: A fine-grained and challenging benchmark for process-level reward models - Song, Mingyang et al., 2025. arXiv:2501.03124

• Bradley-terry and multi-objective reward modeling are complementary - Zhang, Zhiwei et al., 2025. arXiv:2507.07375

• Unified reward model for multimodal understanding and generation - Wang, Yibin et al., 2025. arXiv:2503.05236

• Unified multimodal chain-of-thought reward model through reinforcement fine-tuning - Wang, Yibin et al., 2025. arXiv:2505.03318

• Generalist Reward Models: Found Inside Large Language Models - Li, Yi-Chen et al., 2025. arXiv:2506.23235

• Reward Inside the Model: A Lightweight Hidden-State Reward Model for LLM's Best-of-N sampling - Guo, Jizhou, Wu, Zhaomin, Yu, Philip S, 2025. arXiv:2505.12225

• Vilbench: A suite for vision-language process reward modeling - Tu, Haoqin et al., 2025. arXiv:2503.20271

• Vrprm: Process reward modeling via visual reasoning - Chen, Xinquan et al., 2025. arXiv:2508.03556

• Benchmarking multimodal cot reward model stepwise by visual program - Gao, Minghe et al., 2025. arXiv:2504.06606

• Visualprm: An effective process reward model for multimodal reasoning - Wang, Weiyun et al., 2025. arXiv:2503.10291

• Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Models - Wang, Shuai et al., 2025. arXiv:2506.09532

• DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoning - Cao, Qi et al., 2025. arXiv:2505.20241

• DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Training - Cao, Qi, Xie, Pengtao, 2025. arXiv:2509.05542

• Vlrmbench: A comprehensive and challenging benchmark for vision-language reward models - Ruan, Jiacheng et al., 2025. arXiv:2503.07478

• Multimodal rewardbench: Holistic evaluation of reward models for vision language models - Yasunaga, Michihiro, Zettlemoyer, Luke, Ghazvininejad, Marjan, 2025. arXiv:2502.14191

• VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models - Li, Lei et al., 2025. Proceedings of the Computer Vision and Pattern Recognition Conference

• Libra: Assessing and improving reward model by learning to think - Zhou, Meng et al., 2025. arXiv:2507.21645

• Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models - Ding, Meidan et al., 2025. arXiv:2508.21430

• RewardBench 2: Advancing Reward Model Evaluation - Malik, Saumya et al., 2025. arXiv:2506.01937

• Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents - Men, Tianyi et al., 2025. arXiv:2506.21252

• Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision - Pala, Tej Deep et al., 2025. arXiv:2505.19706

• Uncertainty and influence aware reward model refinement for reinforcement learning from human feedback - Sun, Zexu et al., 2025. The Thirteenth International Conference on Learning Representations

• Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned - Ong, Brandon et al., 2025. arXiv:2509.23250

• Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models - Hu, Jian et al., 2025. arXiv:2501.03262

• Discriminative Policy Optimization for Token-Level Reward Models - Chen, Hongzhan et al., 2025. arXiv:2505.23363

• Sequence to sequence reward modeling: Improving rlhf by language feedback - Zhou, Jiayi et al., 2025. Proceedings of the AAAI Conference on Artificial Intelligence

• Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs - Sun, Hao et al., 2025. arXiv:2502.04357

• Reinforcement learning from human feedback - Lambert, Nathan, 2025. arXiv:2504.12501

• Reward shaping to mitigate reward hacking in rlhf - Fu, Jiayi et al., 2025. arXiv:2502.18770

• The energy loss phenomenon in rlhf: A new perspective on mitigating reward hacking - Miao, Yuchun et al., 2025. arXiv:2501.19358

• StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Models - Zhang, Xiangxiang et al., 2025. arXiv:2508.05383

• From captions to rewards (carevl): Leveraging large language model experts for enhanced reward modeling in large vision-language models - Dai, Muzhi et al., 2025. arXiv:2503.06260

• EQA-RM: A Generative Embodied Reward Model with Test-time Scaling - Chen, Yuhang, Tan, Zhen, Chen, Tianlong, 2025. arXiv:2506.10389

• Fluent and Accurate Image Captioning with a Self-Trained Reward Model - Moratelli, Nicholas et al., 2025. International Conference on Pattern Recognition

• Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback - Ji, Jiaming et al., 2025. arXiv:2503.17682

• Rovrm: A robust visual reward model optimized via auxiliary textual preference data - Wang, Chenglong et al., 2025. Proceedings of the AAAI Conference on Artificial Intelligence

• Internlm-xcomposer2. 5-reward: A simple yet effective multi-modal reward model - Zang, Yuhang et al., 2025. arXiv:2501.12368

• Generative RLHF-V: Learning Principles from Multi-modal Human Preference - Zhou, Jiayi et al., 2025. arXiv:2505.18531

• BaseReward: A Strong Baseline for Multimodal Reward Model - Zhang, Yi-Fan et al., 2025. arXiv:2509.16127

• Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferences - Bahng, Hyojin et al., 2025. arXiv:2506.02095

• Scalable ranked preference optimization for text-to-image generation - Karthik, Shyamgopal et al., 2025. Proceedings of the IEEE/CVF International Conference on Computer Vision

• Calibrated multi-preference optimization for aligning diffusion models - Lee, Kyungmin et al., 2025. Proceedings of the Computer Vision and Pattern Recognition Conference

• Onereward: Unified mask-guided image generation via multi-task human preference learning - Gong, Yuan et al., 2025. arXiv:2508.21066

• Pref-grpo: Pairwise preference reward-based grpo for stable text-to-image reinforcement learning - Wang, Yibin et al., 2025. arXiv:2508.20751

• Boosting chart-to-code generation in mllm via dual preference-guided refinement - Zhang, Zhihan, Cao, Yixin, Liao, Lizi, 2025. arXiv:2504.02906

• Multimodal LLMs as Customized Reward Models for Text-to-Image Generation - Zhou, Shijie et al., 2025. Proceedings of the IEEE/CVF International Conference on Computer Vision

• Robust reinforcement learning from human feedback for large language models fine-tuning - Ye, Kai et al., 2025. arXiv:2504.03784

• Efficient process reward model training via active learning - Duan, Keyu et al., 2025. arXiv:2504.10559

• PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling - Jian, Ai et al., 2025. arXiv:2510.24235

• FreePRM: Training Process Reward Models Without Ground Truth Process Labels - Sun, Lin et al., 2025. arXiv:2506.03570

• Self-Correcting Reasoning with Reward Model: Enhancing Test-Time Scaling Through Iterative Feedback - Author1, FirstName, Author2, LastName, 2025. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

• LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling - Zecheng Tang et al., 2025. ArXiv

• Lost in the Context: Insufficient and Distracted Attention to Contexts in Preference Modeling - Dou, Shihan and Chen, Jiayi and Huang, Chenhao and Chen, Feng and Chengzhi, Wei and Zheng, Huiyuan and Liu, Shichun and Liu, Yan and Liu, Chenxiao and Xin, Chao and Yan, Lin and Zhang, Zongzhang and Gui, Tao and Zhang, Qi and Huang, Xuanjing, 2025. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

• LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling - Zecheng Tang et al., 2025. arXiv:2510.06915

• Warm: On the benefits of weight averaged reward models - Ram{'e, 2024. arXiv:2401.12187

• Advancing llm reasoning generalists with preference trees - Yuan, Lifan et al., 2024. arXiv:2404.02078

• How to evaluate reward models for rlhf - Frick, Evan et al., 2024. arXiv:2410.14872

• Interpretable preferences via multi-objective reward modeling and mixture-of-experts - Wang, Haoxiang et al., 2024. arXiv:2406.12845

• Metametrics: Calibrating metrics for generation tasks using human preferences - Winata, Genta Indra et al., 2024. arXiv:2410.02381

• Learning multi-dimensional human preference for text-to-image generation - Zhang, Sixian et al., 2024. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

• Generative verifiers: Reward modeling as next-token prediction - Zhang, Lunjun et al., 2024. arXiv:2408.15240

• Generative reward models - Mahan, Dakota et al., 2024. arXiv:2410.12832

• MJ-bench: Is your multimodal reward model really a good judge for text-to-image generation? - Chen, Zhaorun et al., 2024. arXiv:2407.04842

• Acemath: Advancing frontier math reasoning with post-training and reward modeling - Liu, Zihan et al., 2024. arXiv:2412.15084

• Evaluating robustness of reward models for mathematical reasoning - Kim, Sunghwan et al., 2024. arXiv:2410.01729

• Rewardbench: Evaluating reward models for language modeling - Lambert, Nathan et al., 2024. arXiv:2403.13787

• Rag-rewardbench: Benchmarking reward models in retrieval augmented generation for preference alignment - Jin, Zhuoran et al., 2024. arXiv:2412.13746

• RMB: Comprehensively benchmarking reward models in LLM alignment - Zhou, Enyu et al., 2024. arXiv:2410.09893

• M-RewardBench: Evaluating reward models in multilingual settings - Gureja, Srishti et al., 2024. arXiv:2410.15522

• Rm-bench: Benchmarking reward models of language models with subtlety and style - Liu, Yantao et al., 2024. arXiv:2410.16184

• Skywork-reward: Bag of tricks for reward modeling in llms - Liu, Chris Yuhao et al., 2024. arXiv:2410.18451

• Self-generated critiques boost reward modeling for language models - Yu, Yue et al., 2024. arXiv:2411.16646

• Rrm: Robust reward model training mitigates reward hacking - Liu, Tianqi et al., 2024. arXiv:2409.13156

• Improving machine translation with human feedback: An exploration of quality estimation as a reward model - He, Zhiwei et al., 2024. arXiv:2401.12873

• Batch active learning of reward functions from human preferences - Biyik, Erdem, Anari, Nima, Sadigh, Dorsa, 2024. ACM Transactions on Human-Robot Interaction

• Simpo: Simple preference optimization with a reference-free reward - Meng, Yu, Xia, Mengzhou, Chen, Danqi, 2024. Advances in Neural Information Processing Systems

• Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback - Yoon, Eunseop et al., 2024. arXiv:2407.16574

• T-reg: Preference optimization with token-level reward regularization - Zhou, Wenxuan et al., 2024. arXiv:2412.02685

• Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment - Yang, Rui et al., 2024. arXiv:2402.10207

• Advancing Process Verification for Large Language Models via Tree-Based Preference Learning - He, Mingqian et al., 2024. EMNLP

• Vlrm: Vision-language models act as reward models for image captioning - Dzabraev, Maksim, Kunitsyn, Alexander, Ivaniuta, Andrei, 2024. arXiv:2404.01911

• Tldr: Token-level detective reward model for large vision language models - Fu, Deqing et al., 2024. arXiv:2410.04734

• Diffusion model alignment using direct preference optimization - Wallace, Bram et al., 2024. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

• A dense reward view on aligning text-to-image diffusion with preference - Yang, Shentao, Chen, Tianqi, Zhou, Mingyuan, 2024. arXiv:2402.08265

• Musicrl: Aligning music generation to human preferences - Cideron, Geoffrey et al., 2024. arXiv:2402.04229

• Baton: Aligning text-to-audio model with human preference feedback - Liao, Huan et al., 2024. arXiv:2402.00744

• Self-Generated Critiques Boost Reward Modeling for Language Models - Yue Yu et al., 2024. arXiv:2411.16646

• RM-R1: Reward Modeling as Reasoning - Author1, FirstName, Author2, LastName, 2024. Journal of Artificial Intelligence

• Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning - Author1, FirstName, Author2, LastName, 2024. Proceedings of the Neural Information Processing Systems (NeurIPS)

• DPRM: A Dual Implicit Process Reward Model in Multi-Hop Question Answering - Author1, FirstName, Author2, LastName, 2024. Journal of Machine Learning Research

• Reward Modeling and Chain-of-Thought in Large Language Models - Author1, FirstName, Author2, LastName, 2024. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS)

• Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis - Wu, Xiaoshi et al., 2023. arXiv:2306.09341

• Pick-a-pic: An open dataset of user preferences for text-to-image generation - Kirstain, Yuval et al., 2023. Advances in neural information processing systems

• Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model - Deng, Haikang, Raffel, Colin, 2023. arXiv:2310.09520

• Let's reward step by step: Step-Level reward model as the Navigators for Reasoning - Ma, Qianli et al., 2023. arXiv:2310.10080

• Rlaif: Scaling reinforcement learning from human feedback with ai feedback - Lee, Harrison et al., 2023.

• Direct preference optimization: Your language model is secretly a reward model - Rafailov, Rafael et al., 2023. Advances in neural information processing systems

• Vision-language models are zero-shot reward models for reinforcement learning - Rocamonde, Juan et al., 2023. arXiv:2310.12921

• Training a helpful and harmless assistant with reinforcement learning from human feedback - Bai, Yuntao et al., 2022. arXiv:2204.05862

• Uncertainty estimation for language reward models - Gleave, Adam, Irving, Geoffrey, 2022. arXiv:2203.07472

• Active preference-based gaussian process regression for reward learning - B{\i, 2020. arXiv:2005.02575

• Fine-tuning language models from human preferences - Ziegler, Daniel M et al., 2019. arXiv:1909.08593

• Deep reinforcement learning from human preferences - Christiano, Paul F et al., 2017. Advances in neural information processing systems

• Active preference-based learning of reward functions - Sadigh, Dorsa et al., 2017.

• Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment - Shen, Yunyi, Sun, Hao, Ton, Jean-Francois, . Forty-second International Conference on Machine Learning

• R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning - Learning, Through Stable Reinforcement, . Policy

• Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarios - Zang, Jianxiang et al., 2025. arXiv:2512.00920

Papers by Domain

🔢 Math Reasoning

• DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning - Zhihong Shao et al., 2025. arXiv:2511.22570
• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey - Liu, Qiyuan et al., 2025. arXiv:2510.01925
• VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rank - Wu, Tianhe et al., 2025. arXiv:2505.14460
• Co-reward: Self-supervised reinforcement learning for large language model reasoning via contrastive agreement - Zhang, Zizhuo et al., 2025. arXiv:2508.00410
• Vrprm: Process reward modeling via visual reasoning - Chen, Xinquan et al., 2025. arXiv:2508.03556
• Visualprm: An effective process reward model for multimodal reasoning - Wang, Weiyun et al., 2025. arXiv:2503.10291
• Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Models - Wang, Shuai et al., 2025. arXiv:2506.09532
• Ursa: Understanding and verifying chain-of-thought reasoning in multimodal mathematics - Luo, Ruilin et al., 2025. arXiv:2501.04686
• PRM-BAS: Enhancing Multimodal Reasoning through PRM-guided Beam Annealing Search - Hu, Pengfei et al., 2025. arXiv:2504.10222
• MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision - Du, Lingxiao et al., 2025. arXiv:2505.13427
• DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoning - Cao, Qi et al., 2025. arXiv:2505.20241
• Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards - Chen, Honghao et al., 2025. arXiv:2509.19003
• JudgeSQL: Reasoning over SQL Candidates with Weighted Consensus Tournament - Bai, Jiayuan et al., 2025. arXiv:2510.15560
• Big Reasoning with Small Models: Instruction Retrieval at Inference Time - Alkiek, Kenan, Jurgens, David, Vydiswaran, Vinod, 2025. arXiv:2510.13935
• Towards Inference-time Scaling for Continuous Space Reasoning - Wang, Minghan et al., 2025. arXiv:2510.12167
• TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoning - Zou, Jiaru et al., 2025. arXiv:2510.06217
• Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned - Ong, Brandon et al., 2025. arXiv:2509.23250
• EasyARC: Evaluating Vision Language Models on True Visual Reasoning - Unsal, Mert, Akkus, Aylin, 2025. arXiv:2506.11595
• Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning - Guo, Daya et al., 2025. arXiv:2501.12948
• StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Models - Zhang, Xiangxiang et al., 2025. arXiv:2508.05383
• CarBoN: Calibrated Best-of-N Sampling Improves Test-time Reasoning - Yung-Chen Tang, Pin-Yu Chen, Andrea Cavallaro, 2025. arXiv:2510.15674
• Adaptive Test-Time Reasoning via Reward-Guided Dual-Phase Search (DREAM) - Y. Cui, et~al., 2025. arXiv:2509.25420
• Self-rewarding correction for mathematical reasoning - Wei Xiong et al., 2025. arXiv:2502.19613
• Reward Reasoning Model (RRM) - Author1, FirstName, Author2, LastName, 2025. Proceedings of the International Conference on Learning Representations (ICLR)
• Self-Correcting Reasoning with Reward Model: Enhancing Test-Time Scaling Through Iterative Feedback - Author1, FirstName, Author2, LastName, 2025. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)
• Improve mathematical reasoning in language models by automated process supervision - Luo, Liangchen et al., 2024. arXiv:2406.06592
• Advancing llm reasoning generalists with preference trees - Yuan, Lifan et al., 2024. arXiv:2404.02078
• Llm critics help catch bugs in mathematics: Towards a better mathematical verifier with natural language feedback - Gao, Bofei et al., 2024. arXiv:2406.14024
• Acemath: Advancing frontier math reasoning with post-training and reward modeling - Liu, Zihan et al., 2024. arXiv:2412.15084
• Evaluating robustness of reward models for mathematical reasoning - Kim, Sunghwan et al., 2024. arXiv:2410.01729
• Processbench: Identifying process errors in mathematical reasoning - Zheng, Chujie et al., 2024. arXiv:2412.06559
• Scalable and accurate graph reasoning with llm-based multi-agents - Hu, Yuwei et al., 2024. arXiv:2410.05130
• Deepseekmath: Pushing the limits of mathematical reasoning in open language models - Shao, Zhihong et al., 2024. arXiv:2402.03300
• Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement - Yang, An et al., 2024. arXiv:2409.12122
• On designing effective rl reward at training time for llm reasoning - Gao, Jiaxuan et al., 2024. arXiv:2410.15115
• Math-shepherd: Verify and reinforce llms step-by-step without human annotations - Wang, Peiyi et al., 2024. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models - Zhihong Shao et al., 2024. arXiv:2402.03300
• Interpretable Contrastive Monte Carlo Tree Search Reasoning - Zitian Gao et al., 2024. arXiv:2410.01707
• Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning - Zhenni Bi et al., 2024. arXiv:2412.09078
• LLMs cannot find reasoning errors, but can correct them given the error location - Gladys Tyen, Hassan Mansoor, Victor C\u{a, 2024. Findings of the Association for Computational Linguistics: ACL 2024
• RM-R1: Reward Modeling as Reasoning - Author1, FirstName, Author2, LastName, 2024. Journal of Artificial Intelligence
• Evaluating the logical reasoning ability of chatgpt and gpt-4 - Liu, Hanmeng et al., 2023. arXiv:2304.03439
• Math-shepherd: Verify and reinforce llms step-by-step without human annotations - Wang, Peiyi et al., 2023. arXiv:2312.08935
• Let's reward step by step: Step-Level reward model as the Navigators for Reasoning - Ma, Qianli et al., 2023. arXiv:2310.10080
• Progressive-Hint Prompting Improves Reasoning in Large Language Models - Chuanyang Zheng et al., 2023. arXiv:2304.09797
• Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change) - Valmeekam, Karthik et al., 2022. NeurIPS 2022 Foundation Models for Decision Making Workshop

💻 Code Generation

• Codedpo: Aligning code models with self generated and verified source code - Zhang, Kechi et al., 2025. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
• Boosting chart-to-code generation in mllm via dual preference-guided refinement - Zhang, Zhihan, Cao, Yixin, Liao, Lizi, 2025. arXiv:2504.02906
• MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation - Yutong Wang, et~al., 2025. arXiv:2502.12468
• Evocodebench: An evolving code generation benchmark with domain-specific evaluations - Li, Jia et al., 2024. Advances in Neural Information Processing Systems
• Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories - Li, Jia et al., 2024. arXiv:2405.19856
• Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions - Zhuo, Terry Yue et al., 2024. arXiv:2406.15877
• Livecodebench: Holistic and contamination free evaluation of large language models for code - Jain, Naman et al., 2024. arXiv:2403.07974
• StepCoder: Improving Code Generation with Reinforcement Learning from Compiler Feedback - Dou, Shihan et al., 2024. 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand, Aug 11 2024-Aug 16 2024
• Execution-based code generation using deep reinforcement learning - Shojaee, Parshin et al., 2023. arXiv:2301.13816
• Coderl: Mastering code generation through pretrained models and deep reinforcement learning - Le, Hung et al., 2022. Advances in Neural Information Processing Systems
• Evaluating large language models trained on code - Chen, Mark et al., 2021. arXiv:2107.03374

🎨 Multimodal

• A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models - Zheng, Congming et al., 2025. arXiv:2510.08049
• Unified reward model for multimodal understanding and generation - Wang, Yibin et al., 2025. arXiv:2503.05236
• Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step - Guo, Ziyu et al., 2025. arXiv:2501.13926
• Unified multimodal chain-of-thought reward model through reinforcement fine-tuning - Wang, Yibin et al., 2025. arXiv:2505.03318
• VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rank - Wu, Tianhe et al., 2025. arXiv:2505.14460
• Vilbench: A suite for vision-language process reward modeling - Tu, Haoqin et al., 2025. arXiv:2503.20271
• Vrprm: Process reward modeling via visual reasoning - Chen, Xinquan et al., 2025. arXiv:2508.03556
• Benchmarking multimodal cot reward model stepwise by visual program - Gao, Minghe et al., 2025. arXiv:2504.06606
• Visualprm: An effective process reward model for multimodal reasoning - Wang, Weiyun et al., 2025. arXiv:2503.10291
• Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Models - Wang, Shuai et al., 2025. arXiv:2506.09532
• Ursa: Understanding and verifying chain-of-thought reasoning in multimodal mathematics - Luo, Ruilin et al., 2025. arXiv:2501.04686
• PRM-BAS: Enhancing Multimodal Reasoning through PRM-guided Beam Annealing Search - Hu, Pengfei et al., 2025. arXiv:2504.10222
• MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervision - Du, Lingxiao et al., 2025. arXiv:2505.13427
• DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoning - Cao, Qi et al., 2025. arXiv:2505.20241
• Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards - Chen, Honghao et al., 2025. arXiv:2509.19003
• DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Training - Cao, Qi, Xie, Pengtao, 2025. arXiv:2509.05542
• Vlrmbench: A comprehensive and challenging benchmark for vision-language reward models - Ruan, Jiacheng et al., 2025. arXiv:2503.07478
• Multimodal rewardbench: Holistic evaluation of reward models for vision language models - Yasunaga, Michihiro, Zettlemoyer, Luke, Ghazvininejad, Marjan, 2025. arXiv:2502.14191
• VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models - Li, Lei et al., 2025. Proceedings of the Computer Vision and Pattern Recognition Conference
• Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models - Ding, Meidan et al., 2025. arXiv:2508.21430
• Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents - Men, Tianyi et al., 2025. arXiv:2506.21252
• Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision - Pala, Tej Deep et al., 2025. arXiv:2505.19706
• Seeing and Knowing in the Wild: Open-domain Visual Entity Recognition with Large-scale Knowledge Graphs via Contrastive Learning - Zhou, Hongkuan et al., 2025. arXiv:2510.13675
• GenPilot: A Multi-Agent System for Test-Time Prompt Optimization in Image Generation - Ye, Wen et al., 2025. arXiv:2510.07217
• Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned - Ong, Brandon et al., 2025. arXiv:2509.23250
• BALM-TSF: Balanced Multimodal Alignment for LLM-Based Time Series Forecasting - Zhou, Shiqiao, Sch{, 2025. arXiv:2509.00622
• EasyARC: Evaluating Vision Language Models on True Visual Reasoning - Unsal, Mert, Akkus, Aylin, 2025. arXiv:2506.11595
• Video-t1: Test-time scaling for video generation - Liu, Fangfu et al., 2025. arXiv:2503.18942
• StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Models - Zhang, Xiangxiang et al., 2025. arXiv:2508.05383
• From captions to rewards (carevl): Leveraging large language model experts for enhanced reward modeling in large vision-language models - Dai, Muzhi et al., 2025. arXiv:2503.06260
• Controlling Multimodal LLMs via Reward-guided Decoding - Ma{~n, 2025. Proceedings of the IEEE/CVF International Conference on Computer Vision
• Fluent and Accurate Image Captioning with a Self-Trained Reward Model - Moratelli, Nicholas et al., 2025. International Conference on Pattern Recognition
• CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning - Xing, Long et al., 2025. arXiv:2509.22647
• Rovrm: A robust visual reward model optimized via auxiliary textual preference data - Wang, Chenglong et al., 2025. Proceedings of the AAAI Conference on Artificial Intelligence
• BaseReward: A Strong Baseline for Multimodal Reward Model - Zhang, Yi-Fan et al., 2025. arXiv:2509.16127
• ImageReFL: Balancing Quality and Diversity in Human-Aligned Diffusion Models - Sorokin, Dmitrii et al., 2025. arXiv:2505.22569
• Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferences - Bahng, Hyojin et al., 2025. arXiv:2506.02095
• Scalable ranked preference optimization for text-to-image generation - Karthik, Shyamgopal et al., 2025. Proceedings of the IEEE/CVF International Conference on Computer Vision
• Onereward: Unified mask-guided image generation via multi-task human preference learning - Gong, Yuan et al., 2025. arXiv:2508.21066
• Pref-grpo: Pairwise preference reward-based grpo for stable text-to-image reinforcement learning - Wang, Yibin et al., 2025. arXiv:2508.20751
• Rewarddance: Reward scaling in visual generation - Wu, Jie et al., 2025. arXiv:2509.08826
• DanceGRPO: Unleashing GRPO on Visual Generation - Xue, Zeyue et al., 2025. arXiv:2505.07818
• Multimodal LLMs as Customized Reward Models for Text-to-Image Generation - Zhou, Shijie et al., 2025. Proceedings of the IEEE/CVF International Conference on Computer Vision
• Llms meet multimodal generation and editing: A survey - He, Yingqing et al., 2024. arXiv:2405.19334
• Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation - Xu, Jiazheng et al., 2024. arXiv:2412.21059
• Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision - Wang, Zihan et al., 2024. arXiv:2402.02658
• Improve mathematical reasoning in language models by automated process supervision - Luo, Liangchen et al., 2024. arXiv:2406.06592
• Learning multi-dimensional human preference for text-to-image generation - Zhang, Sixian et al., 2024. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
• MJ-bench: Is your multimodal reward model really a good judge for text-to-image generation? - Chen, Zhaorun et al., 2024. arXiv:2407.04842
• Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark - Chen, Dongping et al., 2024. Forty-first International Conference on Machine Learning
• Vlrm: Vision-language models act as reward models for image captioning - Dzabraev, Maksim, Kunitsyn, Alexander, Ivaniuta, Andrei, 2024. arXiv:2404.01911
• Tldr: Token-level detective reward model for large vision language models - Fu, Deqing et al., 2024. arXiv:2410.04734
• Multimodal large language models make text-to-image generative models align better - Wu, Xun et al., 2024. Advances in Neural Information Processing Systems
• A dense reward view on aligning text-to-image diffusion with preference - Yang, Shentao, Chen, Tianqi, Zhou, Mingyuan, 2024. arXiv:2402.08265
• Itercomp: Iterative composition-aware feedback learning from model gallery for text-to-image generation - Zhang, Xinchen et al., 2024. arXiv:2410.07171
• Reno: Enhancing one-step text-to-image models through reward-based noise optimization - Eyring, Luca et al., 2024. Advances in Neural Information Processing Systems
• Parrot: Pareto-optimal multi-reward reinforcement learning framework for text-to-image generation - Lee, Seung Hyun et al., 2024. European Conference on Computer Vision
• Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning - Author1, FirstName, Author2, LastName, 2024. Proceedings of the Neural Information Processing Systems (NeurIPS)
• Imagereward: Learning and evaluating human preferences for text-to-image generation - Xu, Jiazheng et al., 2023. Advances in Neural Information Processing Systems
• Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis - Wu, Xiaoshi et al., 2023. arXiv:2306.09341
• Pick-a-pic: An open dataset of user preferences for text-to-image generation - Kirstain, Yuval et al., 2023. Advances in neural information processing systems
• Vision-language models are zero-shot reward models for reinforcement learning - Rocamonde, Juan et al., 2023. arXiv:2310.12921
• Test-time adaptation with clip reward for zero-shot generalization in vision-language models - Zhao, Shuai et al., 2023. arXiv:2305.18010
• Cgt-gan: Clip-guided text gan for image captioning - Yu, Jiarui et al., 2023. Proceedings of the 31st ACM international conference on multimedia
• Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models - Fan, Ying et al., 2023. Advances in Neural Information Processing Systems
• Fine-grained image captioning with clip reward - Cho, Jaemin et al., 2022. arXiv:2205.13115
• Learning transferable visual models from natural language supervision - Radford, Alec et al., 2021. International conference on machine learning
• 2017 IEEE International Conference on Computer Vision (ICCV) - Lin, Tsung-Yi et al., 2017. 2017 IEEE International Conference on Computer Vision (ICCV)
• R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning - Learning, Through Stable Reinforcement, . Policy

🤖 Agent

• A survey of large language model agents for question answering - Yue, Murong, 2025. arXiv:2503.19213
• Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents - Men, Tianyi et al., 2025. arXiv:2506.21252
• Rethinking Agentic Workflows: Evaluating Inference-Based Test-Time Scaling Strategies in Text2SQL Tasks - Guo, Jiajing et al., 2025. arXiv:2510.10885
• GenPilot: A Multi-Agent System for Test-Time Prompt Optimization in Image Generation - Ye, Wen et al., 2025. arXiv:2510.07217
• TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoning - Zou, Jiaru et al., 2025. arXiv:2510.06217
• Agentar-Scale-SQL: Advancing Text-to-SQL through Orchestrated Test-Time Scaling - Wang, Pengfei et al., 2025. arXiv:2509.24403
• Metal: A multi-agent framework for chart generation with test-time scaling - Li, Bingxuan et al., 2025. arXiv:2502.17651
• Scalable and accurate graph reasoning with llm-based multi-agents - Hu, Yuwei et al., 2024. arXiv:2410.05130
• Reflexion: Language agents with verbal reinforcement learning - Shinn, Noah et al., 2023. Advances in Neural Information Processing Systems
• Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change) - Valmeekam, Karthik et al., 2022. NeurIPS 2022 Foundation Models for Decision Making Workshop

---

Citation

If this project helps your research, please cite our paper:

@article{awesomerm2025,
  title={Good Teachers, Better Students: A Survey of Reward Models for LLM},
  author={[Author List]},
  journal={arXiv preprint arXiv:XXXX.XXXXX},
  year={2025}
}

---

Contributing

We welcome community contributions! To add papers, fix errors, or improve content:

1. Fork this repository
2. Create a new branch (git checkout -b feature/your-contribution)
3. Commit your changes (git commit -m 'Add some papers')
4. Push to the branch (git push origin feature/your-contribution)
5. Create a Pull Request

Format for Adding Papers

Please follow this format when adding papers:

- **[Paper Title](arXiv link)** - Brief description (1-2 sentences)
  - Authors, Conference/Journal, Year
  - Key contributions: [List 1-3 points]
  - Code: [GitHub link if available]

---

📧 Contact Us

• Issue Reports: GitHub Issues
• Email: [Contact Email]
• Project Website: https://linjh1118.github.io/AwesomeRM

---

📜 License

This project is licensed under the MIT License.

---

🙏 Acknowledgments

Thanks to all researchers and developers contributing to the reward models field. Special thanks to:

• OpenAI, Anthropic, Google DeepMind for pioneering work in RLHF
• All contributors of open-source benchmarks and datasets
• Community feedback and suggestions

---

⭐ If this project helps you, please give us a Star! ⭐

Made with ❤️ by the AwesomeRM Team