😄 Welcome to recommend missing papers through Issues and Pull Requests.

Paper List

🔍 Preference Collection

Human Preference

• Deep Reinforcement Learning from Human Preferences 2017 [NeurIPS]
• Batch Active Preference-Based Learning of Reward Functions 2018 [CoRL]
• Reward learning from human preferences and demonstrations in Atari 2018 [NeurIPS]
• Active Preference-Based Gaussian Process Regression for Reward Learning 2020 [RSS]
• Information Directed Reward Learning for Reinforcement Learning 2021 [NeurIPS]
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training 2021 [ICML]
• Improving alignment of dialogue agents via targeted human judgements 2022 [arxiv]
• Training language models to follow instructions with human feedback 2022 [NeurIPS]
• SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning 2022 [ICLR]
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback 2022 [arxiv]
• Active Reward Learning from Multiple Teachers 2023 [AAAI Workshop]
• RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback 2023 [ICML Workshop]
• Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feedback 2023 [NeurIPS]
• Fine-Grained Human Feedback Gives Better Rewards for Language Model Training 2023 [NeurIPS]
• Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback 2024 [ICLR]
• HelpSteer2: Open-source dataset for training top-performing reward models 2024 [arxiv]
• Batch Active Learning of Reward Functions from Human Preferences 2024 [arxiv]
• Towards Comprehensive Preference Data Collection for Reward Modeling 2024 [arxiv]
• RLHF Workflow: From Reward Modeling to Online RLHF 2024 [TMLR]
• Towards Comprehensive Preference Data Collection for Reward Modeling 2024 [arxiv]
• Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data 2024 [arxiv]
• Less is More: Improving LLM Alignment via Preference Data Selection 2025 [arxiv]
• RLTHF: Targeted Human Feedback for LLM Alignment 2025 [arxiv]

AI Preference

• Constitutional AI: Harmlessness from AI Feedback 2022 [arxiv]
• AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback 2023 [NeurIPS]
• Aligning Large Language Models through Synthetic Feedback 2023 [EMNLP]
• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback 2024 [ICML]
• UltraFeedback: Boosting Language Models with Scaled AI Feedback 2024 [ICML]
• SALMON: Self-Alignment with Instructable Reward Models 2024 [ICLR]
• Improving Reward Models with Synthetic Critiques 2024 [arxiv]
• Self-Generated Critiques Boost Reward Modeling for Language Models 2024 [arxiv]
• Safer-Instruct: Aligning Language Models with Automated Preference Data 2024 [NAACL]
• Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization 2024 [EMNLP Findings]
• RLCD: Reinforcement Learning from Contrastive Distillation for LM Alignment 2024 [ICLR]
• West-of-N: Synthetic Preference Generation for Improved Reward Modeling 2024 [ICLR Workshop]
• RMBoost: Reward Model Training With Preference-Conditional Multi-Aspect Synthetic Data Generation 2025 [ICLR Workshop]
• Interpreting Language Model Preferences Through the Lens of Decision Trees 2025 [Online]

🖥️ Reward Modeling

Type-Level

Discriminative Reward

• LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion 2023 [ACL]
• InternLM2 Technical Report 2024 [arxiv]
• Advancing LLM Reasoning Generalists with Preference Trees 2024 [arxiv]
• Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs 2024 [NeurIPS]
• MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferences 2024 [ICLR]
• Nemotron-4 340B Technical Report 2024 [arxiv]
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts 2024 [arxiv]

Generative Reward

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena 2023 [NeurIPS]
• Generative Judge for Evaluating Alignment 2024 [ICLR]
• Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models 2024 [EMNLP]
• CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution 2024 [arxiv]
• LLM Critics Help Catch LLM Bugs 2024 [arxiv]
• LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedback 2024 [arxiv]
• Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge 2024 [arxiv]
• Self-Taught Evaluators 2024 [arxiv]
• Self-Rewarding Language Models 2024 [ICML]
• Direct Judgement Preference Optimization 2024 [arxiv]
• Generative Reward Models 2024 [arxiv]
• Generative Verifiers: Reward Modeling as Next-Token Prediction 2024 [arxiv]
• Beyond Scalar Reward Model: Learning Generative Judge from Preference Data 2024 [arxiv]
• Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint 2024 [ACL Findings]

Implicit Reward

• Direct Preference Optimization: Your Language Model is Secretly a Reward Model 2023 [NeurIPS]
• SLiC-HF: Sequence Likelihood Calibration with Human Feedback 2023 [arxiv]
• A General Theoretical Paradigm to Understand Learning from Human Preferences 2023 [arxiv]
• A Minimaximalist Approach to Reinforcement Learning from Human Feedback 2024 [ICML]
• Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive 2024 [arxiv]
• From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function 2024 [COLM]
• Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs 2024 [arxiv]
• Token-level Direct Preference Optimization 2024 [ICML]
• $β$-DPO: Direct Preference Optimization with Dynamic $β$ 2024 [NeurIPS]
• Generalized Preference Optimization: A Unified Approach to Offline Alignment 2024 [ICML]
• Contrastive Preference Optimization: Pushing the Boundaries of LLM 2024 [ICML]
• Offline Regularised Reinforcement Learning for Large Language Models Alignment 2024 [arxiv]
• Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences 2024 [arxiv]
• ORPO: Monolithic Preference Optimization without Reference Model 2024 [EMNLP]
• Mixed Preference Optimization: A Two-stage Reinforcement Learning with Human Feedbacks 2024 [arxiv]
• LiPO: Listwise Preference Optimization through Learning-to-Rank 2024 [arxiv]
• Noise Contrastive Alignment of Language Models with Explicit Rewards 2024 [NeurIPS]
• SimPO: Simple Preference Optimization with a Reference-Free Reward 2024 [NeurIPS]
• Direct Preference Optimization with an Offset 2024 [ACL Findings]
• Statistical Rejection Sampling Improves Preference Optimization 2024 [ICLR]
• sDPO: Don’t Use Your Data All at Once 2025 [COLING Industry]
• Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization 2025 [ICLR]
• Self-Play Preference Optimization for Language Model Alignment 2025 [ICLR]
• TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights 2025 [ICLR]

Granularity-Level

Outcome Reward

TBD

Process Reward

• Solving math word problems with process- and outcome-based feedback 2022 [arxiv]
• GRACE: Discriminator-Guided Chain-of-Thought Reasoning 2023 [EMNLP Findings]
• Making Language Models Better Reasoners with Step-Aware Verifier 2023 [ACL]
• Let's reward step by step: Step-Level reward model as the Navigators for Reasoning 2023 [arxiv]
• Let’s Reinforce Step by Step 2023 [NeurIPS Workshop]
• Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations 2024 [ACL]
• Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision 2024 [EMNLP Findings]
• Improve Mathematical Reasoning in Language Models by Automated Process Supervision 2024 [arxiv]
• Let's Verify Step by Step 2024 [ICLR]
• AutoPSV: Automated Process-Supervised Verifier 2024 [NeurIPS]
• Process Reward Model with Q-value Rankings 2025 [ICLR]
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning 2025 [ICLR]
• Process Reinforcement through Implicit Rewards 2025 [arxiv]
• The Lessons of Developing Process Reward Models in Mathematical Reasoning 2025 [arxiv]
• AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence 2025 [arxiv]
• Better Process Supervision with Bi-directional Rewarding Signals 2025 [arxiv]
• An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning 2025 [arxiv]
• VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data 2025 [arxiv]

🦾 Usages

Data Selection

• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment 2023 [TMLR]
• RRHF: Rank Responses to Align Language Models with Human Feedback without tears 2023 [NeurIPS]
• Reinforced Self-Training (ReST) for Language Modeling 2023 [arxiv]
• Iterative Reasoning Preference Optimization 2024 [NeruIPS]
• Filtered Direct Preference Optimization 2024 [EMNLP]

Policy Training

• Fine-Grained Human Feedback Gives Better Rewards for Language Model Training 2023 [NeurIPS]
• Aligning Crowd Feedback via Distributional Preference Reward modeling 2024 [arxiv]
• Reward-Robust RLHF in LLMs 2024 [arxiv]
• Bayesian Reward Models for LLM Alignment 2024 [arxiv]
• Prior Constraints-based Reward Model Training for Aligning Large Language Models 2024 [CCL]
• ODIN: Disentangled Reward Mitigates Hacking in RLHF 2024 [ICML]
• Disentangling Length from Quality in Direct Preference Optimization 2024 [ACL Findings]
• WARM: On the Benefits of Weight Averaged Reward Models 2024 [ICML]
• Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble 2024 [arxiv]
• RRM: Robust Reward Model Training Mitigates Reward Hacking 2025 [ICLR]
• Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment 2025 [arxiv]

Inference

• Let's reward step by step: Step-Level reward model as the Navigators for Reasoning 2023 [arxiv]
• ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search 2024 [arxiv]
• Advancing Process Verification for Large Language Models via Tree-Based Preference Learning 2024 [arxiv]
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning 2025 [ICLR]
• Process Reward Models for LLM Agents: Practical Framework and Directions 2025 [arxiv]
• Reward-Guided Speculative Decoding for Efficient LLM Reasoning 2025 [arxiv]

🛠️ Applications

Harmless Dialogue

Dialogue

• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback 2022 [arxiv]
• Constitutional AI: Harmlessness from AI Feedback 2022 [arxiv]
• Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialogue 2023 [arxiv]
• HuatuoGPT, Towards Taming Language Models To Be a Doctor 2023 [EMNLP Findings]
• Empathy Level Alignment via Reinforcement Learning for Empathetic Response Generation 2024 [arxiv]
• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback 2024 [ICML]
• Safe RLHF: Safe Reinforcement Learning from Human Feedback 2024 [ICLR]
• Deliberative Alignment: Reasoning Enables Safer Language Models 2024 [arxiv]
• Training Dialogue Systems by AI Feedback for Improving Overall Dialogue Impression 2025 [arxiv]

Logical Reasoning

Math

• Training Verifiers to Solve Math Word Problems 2022 [arxiv]
• Solving math word problems with process- and outcome-based feedback 2022 [arxiv]
• WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct 2023 [arxiv]
• Let's Verify Step by Step 2024 [ICLR]
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models 2024 [arxiv]
• Improve Mathematical Reasoning in Language Models by Automated Process Supervision 2024 [arxiv]
• Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations 2024 [ACL]
• The Lessons of Developing Process Reward Models in Mathematical Reasoning 2025 [arxiv]
• Retrieval-Augmented Process Reward Model for Generalizable Mathematical Reasoning 2025 [arxiv]

Code

• Let's reward step by step: Step-Level reward model as the Navigators for Reasoning 2023 [arxiv]
• Applying RLAIF for Code Generation with API-usage in Lightweight LLMs 2024 [arxiv]
• Process Supervision-Guided Policy Optimization for Code Generation 2024 [arxiv]
• Performance-Aligned LLMs for Generating Fast Code 2024 [arxiv]
• Policy Filtration in RLHF to Fine-Tune LLM for Code Generation 2024 [arxiv]
• LLM Critics Help Catch LLM Bugs 2024 [arxiv]

Retrieve & Recommendation

Retrieve

• Enhancing Generative Retrieval with Reinforcement Learning from Relevance Feedback 2023 [EMNLP]
• When Search Engine Services meet Large Language Models: Visions and Challenges 2024 [arxiv]
• Syntriever: How to Train Your Retriever with Synthetic Data from LLMs 2025 [arxiv]
• RAG-Gym: Optimizing Reasoning and Search Agents with Process Supervision 2025 [arxiv]
• DeepRAG: Thinking to Retrieval Step by Step for Large Language Models 2025 [arxiv]

Recommendation

• Reinforcement Learning-based Recommender Systems with Large Language Models for State Reward and Action Modeling 2024 [SIGIR]
• RLRF4Rec: Reinforcement Learning from Recsys Feedback for Enhanced Recommendation Reranking 2024 [arxiv]
• Fine-Tuning Large Language Model Based Explainable Recommendation with Explainable Quality Reward 2025 [AAAI]

Other Applications

Text to Audio

• MusicRL: Aligning Music Generation to Human Preferences 2024 [ICML]
• BATON: Aligning Text-to-Audio Model Using Human Preference Feedback 2024 [IJCAI]
• Reinforcement Learning for Fine-tuning Text-to-speech Diffusion Models 2024 [arxiv]

Text to Image

• Aligning Text-to-Image Models using Human Feedback 2023 [arxiv]
• ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation 2023 [NeurIPS]
• DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models] 2023 [arxiv]

Text to Video

• InstructVideo: Instructing Video Diffusion Models with Human Feedback 2024 [CVPR]
• Boosting Text-to-Video Generative Model with MLLMs Feedback 2024 [NeurIPS]
• Harness Local Rewards for Global Benefits: Effective Text-to-Video Generation Alignment with Patch-level Reward Models 2025 [arxiv]

Robotic

• Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models 2023 [ICRA]
• Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Models 2023 [arxiv]
• Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning 2024 [ICLR]

Game

• DIP-RL: Demonstration-Inferred Preference Learning in Minecraft 2025 [arxiv]
• Process Reward Models for LLM Agents: Practical Framework and Directions 2025 [arxiv]

💯 Evaluation

Benchmarks

• RewardBench: Evaluating Reward Models for Language Modeling 2024 [arxiv] [Leaderboard]
• RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style 2024 [arxiv]
• RMB: comprehensively benchmarking reward models in LLM alignment 2024 [arxiv]
• VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models 2024 [arxiv] [Leaderboard]
• How to Evaluate Reward Models for RLHF 2024 [arxiv] [Leaderboard]
• ProcessBench: Identifying Process Errors in Mathematical Reasoning 2024 [arxiv]
• RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment 2024 [arxiv]
• M-RewardBench: Evaluating Reward Models in Multilingual Settings 2024 [arxiv]
• MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? 2024 [arxiv] [Leaderboard]
• PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models 2025 [arxiv]
• Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models 2025 [arxiv]
• VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models 2025 [arxiv]

🤺 Challenges

Data

• Fine-Tuning Language Models from Human Preferences 2019 [arxiv]
• The Expertise Problem: Learning from Specialized Feedback 2022 [arxiv]
• Active Reward Learning from Multiple Teachers 2023 [AAAI Workshop]
• Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models 2024 [ICLR]

Training

• Defining and Characterizing Reward Hacking 2022 [NeurIPS]
• A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift 2023 [arxiv]
• Scaling Laws for Reward Model Overoptimization 2023 [ICML]
• Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback 2023 [EMNLP Findings]
• Honesty to Subterfuge: In-Context Reinforcement Learning Can Make Honest Models Reward Hack 2024 [arxiv]
• Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models 2024 [arxiv]
• Language Models Learn to Mislead Humans via RLHF 2024 [arxiv]
• Towards Understanding Sycophancy in Language Models 2024 [ICLR]
• Reward Model Ensembles Help Mitigate Overoptimization 2024 [ICLR]
• Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification 2024 [NeurIPS]
• Spontaneous Reward Hacking in Iterative Self-Refinement 2024 [arxiv]
• Confronting Reward Model Overoptimization with Constrained RLHF 2024 [ICLR]
• Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree? 2025 [ICLR]
• Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking 2025 [ICLR]
• RRM: Robust Reward Model Training Mitigates Reward Hacking 2025 [ICLR]
• The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking 2025 [arxiv]

Evaluation

• An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Models are Task-specific Classifiers 2024 [arxiv]
• OffsetBias: Leveraging Debiased Data for Tuning Evaluators 2024 [EMNLP Findings]
• Preference Leakage: A Contamination Problem in LLM-as-a-judge 2025 [arxiv]

📊 Analysis

• The Accuracy Paradox in RLHF: When Better Reward Models Don’t Yield Better Language Models 2024 [EMNLP]
• Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking 2024 [COLM]
• Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives 2024 [arxiv]
• RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs 2024 [arxiv]
• Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective 2024 [arxiv]
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study 2024 [ICML]
• Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree? 2025 [ICLR]
• All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning 2025 [arxiv]
• Reward Models Identify Consistency, Not Causality 2025 [arxiv]
• What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-boosted Mathematical Reasoning 2025 [AAAI]

Resources

🌏 Blogs

• Illustrating Reinforcement Learning from Human Feedback (RLHF) [Link]
• Why reward models are key for alignment [Link]
• Reward Hacking in Reinforcement Learning [Link]

📚 Prior Survey

• A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges 2021 [arxiv]
• Reinforcement Learning With Human Advice: A Survey 2021 [Frontiers Robotics AI]
• AI Alignment: A Comprehensive Survey 2023 [arxiv]
• A Survey of Reinforcement Learning from Human Feedback 2023 [arxiv]
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback 2023 [TMLR]
• Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities 2024 [JAIR]
• Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods 2024 [arxiv]
• A Survey on Human Preference Learning for Large Language Models 2024 [arxiv]
• A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More 2024 [arxiv]
• Reinforcement Learning Enhanced LLMs: A Survey 2024 [arxiv]
• Towards a Unified View of Preference Learning for Large Language Models: A Survey 2024 [arxiv]
• A Survey on Post-training of Large Language Models 2025 [arxiv]