Training Recipes for Agentic Reinforcement Learning in LLMs: A Survey
51
6 commits
updated Jan 30, 2026
README
Agentic-RL-Training-Recipes
Training Recipes for Agentic Reinforcement Learning in LLMs: A Survey
✨ If you find our survey useful, a star ⭐ on GitHub helps others discover it and keeps you updated on future releases.
Table of Contents
Training Schemes
| Paper | TLDR | Component | ↳ Focus | Year | Venue |
|---|---|---|---|---|---|
| Reinforced Self-Training (ReST) for Language Modeling | Iterative self-improvement via generating and filtering high-quality trajectories. | Training Schemes | Rollout & Data Strategy | 2023 | arXiv |
| Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models | Self-play fine-tuning against previous iterations. | Training Schemes | Rollout & Data Strategy | 2024 | ICML |
| Scaling Relationship on Learning Mathematical Reasoning with Large Language Models | Simple rejection sampling strategy for data collection. | Training Schemes | Rollout & Data Strategy | 2023 | arXiv |
| DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models | Introduces Group Relative Policy Optimization (GRPO) for group-based sampling. | Training Schemes | Rollout & Data Strategy | 2024 | arXiv |
| Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning | Down-sampling strategy to filter rollouts and reduce compute. | Training Schemes | Rollout & Data Strategy | 2025 | arXiv |
| Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards | Tree-based exploration with branching at high-uncertainty steps. | Training Schemes | Rollout & Data Strategy | 2025 | arXiv |
| Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents | Learn from failure trajectories via contrastive preference pairs. | Training Schemes | Rollout & Data Strategy | 2024 | ACL |
| SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling | Adapts soft actor-critic and hindsight replay for open-ended exploration. | Training Schemes | Rollout & Data Strategy | 2024 | arXiv |
| Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning | Reverse curriculum learning starting from goal proximity. | Training Schemes | Rollout & Data Strategy | 2024 | ICML |
| Voyager: An Open-Ended Embodied Agent with Large Language Models | Uses predefined progressions in open-ended worlds (Minecraft). | Training Schemes | Rollout & Data Strategy | 2023 | arXiv |
| RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments | Dynamically generates tasks based on current agent performance. | Training Schemes | Rollout & Data Strategy | 2025 | arXiv |
| WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning | Self-correcting curriculum for web agents. | Training Schemes | Rollout & Data Strategy | 2025 | ICLR |
| VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models | Uses variance of group rewards to prioritize medium-difficulty tasks. | Training Schemes | Rollout & Data Strategy | 2025 | arXiv |
| Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning | Uses exact matching outcome rewards for definitive tasks. | Training Schemes | Feedback & Credit | 2025 | COLM |
| ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning | Exact matching outcome rewards for search tasks. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesis | Functional verification for open-ended intent satisfaction. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning | Optimizes efficiency by penalizing retrieval costs. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning | Reward modeling for open-ended search tasks. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning | Intent satisfaction rewards for search agents. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agent | Optimizes efficiency and functional verification. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| DeepRAG: Thinking to Retrieve Step by Step for Large Language Models | Optimizes efficiency by explicitly penalizing unnecessary retrieval actions. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| UR^2: Unify RAG and Reasoning through Reinforcement Learning | Efficiency-aware reward modeling for retrieval. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| ReZero: Enhancing LLM search ability by trying one-more-time | Rewards focused on relevance and formatting in IR. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| s3: You Don't Need That Much Data to Train a Search Agent via RL | IR reward modeling for query diversity and relevance. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| AI-SearchPlanner: Modular Agentic Search via Pareto-Optimal Multi-Objective Reinforcement Learning | Planning-centric rewards for search agents. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| DeepDiver: Adaptive Search Intensity Scaling via Open-Web Reinforcement Learning | IR rewards for deep exploration of topics. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| O^2-Searcher: A Searching-based Agent Model for Open-Domain Open-Ended Question Answering | Outcome-oriented rewards for search quality. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning | Tool-augmented reward modeling for long-form agentic tasks. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning | Uses Shortest Path Reward Estimation for trajectory quality. | Training Schemes | Feedback & Credit | 2025 | NeurIPS |
| Synthetic Data Generation and Multi-Step Reinforcement Learning for Reasoning and Tool Use | Holistic history analysis for process rewards. | Training Schemes | Feedback & Credit | 2025 | COLM |
| Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers | Process rewards for search exploration steps. | Training Schemes | Feedback & Credit | 2025 | NeurIPS |
| ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs | Dense supervision via process reward models (PRM). | Training Schemes | Feedback & Credit | 2025 | NeurIPS |
| Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning | Granular evaluation of specific reasoning steps. | Training Schemes | Feedback & Credit | 2025 | NeurIPS |
| Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation | Verification-based process rewards for reasoning. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents | Process supervision for verifiable meta-reasoning steps. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization | Intermediate search quality evaluation. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Coordinating Search-Informed Reasoning and Reasoning-Guided Search in Claim Verification | Hierarchical rewards for information seeking. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learning | Rewards parallel decomposition efficiency. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment | Uses LLMs to grade/critique intermediate steps. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalization | Collaborative grading for UI actions. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic | LLM-based critique for search trajectories. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models | Verbal feedback for self-correction via retrospective analysis. | Training Schemes | Feedback & Credit | 2025 | NeurIPS |
| Reflexion: language agents with verbal reinforcement learning | Verbal reinforcement for iterative refinement. | Training Schemes | Feedback & Credit | 2023 | NeurIPS |
| VinePPO: Refining Credit Assignment in RL Training of LLMs | Statistical credit assignment using rollout branching. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Exploiting Tree Structure for Credit Assignment in RL Training of LLMs | Tree-structured estimation for step-wise credit. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution | Trains dedicated value networks for step influence. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Agentic Reinforcement Learning with Implicit Step Rewards | Implicit value prediction for self-taught reasoners. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| GRPO-$\lambda$: Credit Assignment improves LLM Reasoning | Reformulated objective for granular updates without value heads. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Group-in-Group Policy Optimization for LLM Agent Training | Group-in-Group policy optimization for credit assignment. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design | Multi-turn adaptation of GRPO credit assignment. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Proximal Policy Optimization Algorithms | Standard clipped gradient optimization (Trust Region). | Training Schemes | Policy Optimization | 2017 | arXiv |
| VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks | Variant of PPO adapted for agentic stability. | Training Schemes | Policy Optimization | 2025 | arXiv |
| DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning | Scaled GRPO for reasoning tasks. | Training Schemes | Policy Optimization | 2025 | arXiv |
| DAPO: An Open-Source LLM Reinforcement Learning System at Scale | Systematized GRPO for scale and distributed training. | Training Schemes | Policy Optimization | 2025 | arXiv |
| URPO: A Unified Reward & Policy Optimization Framework for Large Language Models | Unifies policy optimization with reward modeling. | Training Schemes | Policy Optimization | 2025 | arXiv |
| TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling | Geometry-aware objectives for tree-structured policies. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning | Lightweight PPO with entropy bonuses. | Training Schemes | Policy Optimization | 2025 | arXiv |
| A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems | Survey on reasoning degeneracy and regularization. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learning | PRM-free step-level estimation for efficiency. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards | Asymmetric REINFORCE for off-policy data reuse. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts | Selective rollout strategy to optimize compute budget. | Training Schemes | Policy Optimization | 2025 | arXiv |
| A Survey on the Optimization of Large Language Model-based Agents | Survey on efficient optimization strategies. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning | Uses early stopping triggers (KL spikes) for stability. | Training Schemes | Policy Optimization | 2025 | arXiv |
| LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios | Overview of reasoning stability and hacking. | Training Schemes | Policy Optimization | 2025 | arXiv |
| AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement | Multi-objective scaffolds for safety and performance. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Safe RLHF: Safe Reinforcement Learning from Human Feedback | Constrained MDP formulation for safety. | Training Schemes | Policy Optimization | 2023 | ICLR 2024 |
| SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning | Safety constraints for vision-language agents. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Agentic Reinforcement Learning for Search is Unsafe | Penalizes harmful queries in search agents. | Training Schemes | Policy Optimization | 2025 | arXiv |
| MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory | Fine-tunes models to execute explicit read/write API calls. | Training Schemes | Training-Time Memory | 2024 | arXiv |
| MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversation | Trains multi-stage summarization pipelines. | Training Schemes | Training-Time Memory | 2023 | arXiv |
| Augmenting Language Models with Long-Term Memory | Introduces decoupled memory encoders. | Training Schemes | Training-Time Memory | 2023 | NeurIPS |
| Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection | Trains reflection tokens to trigger on-demand retrieval. | Training Schemes | Training-Time Memory | 2024 | ICLR |
| MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents | Uses RLVR to compress context into a constant footprint. | Training Schemes | Training-Time Memory | 2025 | arXiv |
| MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent | Adapts DAPO for streaming document processing. | Training Schemes | Training-Time Memory | 2025 | arXiv |
| Memory-R1: Enhancing large language model agents to manage and utilize memories via reinforcement learning | Uses PPO/GRPO to train a dedicated memory-manager agent. | Training Schemes | Training-Time Memory | 2025 | arXiv |
| LongMemEval: Benchmarking chat assistants on long-term interactive memory | Benchmark for knowledge updates and abstention. | Training Schemes | Training-Time Memory | 2024 | arXiv |
| LongBench-v2: Towards deeper understanding and reasoning on realistic long-context multitasks | Benchmark for extreme-context (2M tokens) tasks. | Training Schemes | Training-Time Memory | 2025 | ACL |
Training Infrastructure
| Paper | TLDR | Component | ↳ Focus | Year | Venue |
|---|---|---|---|---|---|
| Proximal Policy Optimization Algorithms | Standard synchronous execution model for on-policy consistency. | Training Infrastructure | Actor-Learner Architectures | 2017 | ICLR |
| DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models | Utilizes synchronous execution to simplify credit assignment. | Training Infrastructure | Actor-Learner Architectures | 2024 | arXiv |
| AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning | Decouples collection and training with staleness control. | Training Infrastructure | Actor-Learner Architectures | 2025 | NeurIPS |
| Asynchronous Methods for Deep Reinforcement Learning | Introduces asynchronous actor-learner decoupling. | Training Infrastructure | Actor-Learner Architectures | 2016 | ICML |
| IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures | V-trace correction for off-policy lag in asynchronous setups. | Training Infrastructure | Actor-Learner Architectures | 2018 | ICML |
| IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks | Uses clipped target networks to stabilize stale critics. | Training Infrastructure | Actor-Learner Architectures | 2020 | ICLR |
| LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training | Orchestrates disjoint GPU groups with AIPO for lag correction. | Training Infrastructure | Actor-Learner Architectures | 2025 | arXiv |
| Defeating the Training-Inference Mismatch via FP16 | Switches to FP16 to reduce training-inference numerical divergence. | Training Infrastructure | Precision & Acceleration | 2025 | arXiv |
| FP8-LM: Training FP8 Large Language Models | Demonstrates doubled throughput with FP8 training. | Training Infrastructure | Precision & Acceleration | 2023 | arXiv |
| COAT: Compressing Optimizer states and Activations for Memory-Efficient FP8 Training | Compresses optimizer states and activations for efficiency. | Training Infrastructure | Precision & Acceleration | 2025 | ICLR |
| Scaling FP8 training to trillion-token LLMs | Smoothing operation to handle activation outliers in FP8. | Training Infrastructure | Precision & Acceleration | 2025 | ICLR |
| Optimizing Large Language Model Training Using FP4 Quantization | Enables FP4 training via block-wise quantization. | Training Infrastructure | Precision & Acceleration | 2025 | ICML |
| HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs | Hadamard rotations to spread outliers for low-bit precision. | Training Infrastructure | Precision & Acceleration | 2025 | NeurIPS |
| QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs | Uses Adaptive Quantization Noise (AQN) for exploration. | Training Infrastructure | Precision & Acceleration | 2025 | arXiv |
| Efficient Memory Management for Large Language Model Serving with PagedAttention | Optimized inference kernel that contributes to mismatch. | Training Infrastructure | Training-Inference Mismatch | 2023 | SOSP |
| SGLang: Efficient Execution of Structured Language Model Programs | High-throughput inference engine often differing from learner. | Training Infrastructure | Training-Inference Mismatch | 2024 | NeurIPS |
| PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel | High-precision training framework (standard baseline). | Training Infrastructure | Training-Inference Mismatch | 2023 | arXiv |
| Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | High-precision training framework (standard baseline). | Training Infrastructure | Training-Inference Mismatch | 2019 | arXiv |
| MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention | Upcasts output head to FP32 to fix entropy collapse. | Training Infrastructure | Training-Inference Mismatch | 2025 | arXiv |
| Your Efficient RL Framework Secretly Brings You Off-Policy RL Training | Token-level importance sampling to correct distribution shift. | Training Infrastructure | Training-Inference Mismatch | 2025 | Blog |
| When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch | Sequence masking to robustly handle off-policy shift. | Training Infrastructure | Training-Inference Mismatch | 2025 | Blog |
| DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models | Adopts off-policy sequence masking for training stability. | Training Infrastructure | Training-Inference Mismatch | 2025 | arXiv |
Training Environments
| Paper | TLDR | Component | ↳ Focus | Year | Venue |
|---|---|---|---|---|---|
| Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning | Uses outcome-based rewards to train tool invocation without step-level supervision. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use | Modular framework for unified performance across SQL, code, and visual tasks. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| Tool-Augmented Policy Optimization: Synergizing Reasoning and Adaptive Tool Use with Reinforcement Learning | Uses Dynamic-PPO to optimize tool usage for external retrieval. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| Agent-R1: Training Powerful LLM Agents with End-to-End Reinforcement Learning | Formalizes tool use as an MDP to improve multi-step reasoning. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| Simulating Environments with Reasoning Models for Agent Training | Leverages simulated feedback to overcome data scarcity. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| Introducing the Model Context Protocol | Standard for reproducible agent-tool interactions. | Training Environments | Single-Domain Environments | 2024 | Blog |
| Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning | Trains hierarchical T5 planner with V-MPO for web navigation. | Training Environments | Single-Domain Environments | 2024 | SAC |
| AutoWebGLM: A Large Language Model-based Web Navigating Agent | Staged pipeline (SFT $\to$ DPO $\to$ RFT) for stability in web agents. | Training Environments | Single-Domain Environments | 2024 | KDD |
| WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning | Applies on-policy M-GRPO using sparse binary signals in WebArena-Lite. | Training Environments | Single-Domain Environments | 2025 | EMNLP |
| WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning | Uses self-evolving curriculum and outcome reward models for web agents. | Training Environments | Single-Domain Environments | 2025 | ICLR |
| ReTool: Reinforcement Learning for Strategic Tool Use in LLMs | Shows reward-driven learning yields emergent tool behaviors. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| ToRL: Scaling Tool-Integrated RL | Demonstrates emergent tool use from reward-driven learning. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| Reinforcement Learning for Reasoning in Large Language Models with One Training Example | Shows single verifier-rewarded problem can double performance. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback | Decomposes code tasks into curriculum-aligned sub-problems with compiler feedback. | Training Environments | Single-Domain Environments | 2024 | ACL |
| The BrowserGym Ecosystem for Web Agent Research | Aggregates web benchmarks into a fixed schema. | Training Environments | Multi-Domain Environments | 2025 | TMLR |
| AgentGym: Evolving Large Language Model-based Agents across Diverse Environments | Unifies diverse domains via a consistent HTTP interface. | Training Environments | Multi-Domain Environments | 2025 | ACL |
| Mind2Web: Towards a Generalist Agent for the Web | Uses strict splits to penalize layout memorization. | Training Environments | Multi-Domain Environments | 2023 | NeurIPS |
| WebCanvas: Benchmarking Web Agents in Online Environments | Evaluates agents against live, drifting UIs. | Training Environments | Multi-Domain Environments | 2024 | arXiv |
| VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks | Unifies multimodal tasks through consistent GUI interfaces. | Training Environments | Multi-Domain Environments | 2024 | ACL |
| WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models | Unifies multimodal tasks through consistent GUI interfaces. | Training Environments | Multi-Domain Environments | 2024 | ACL |
| AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation | Applies bidirectional evolution to synthesize environment code. | Training Environments | Multi-Domain Environments | 2025 | KDD |
| Eurekaverse: Environment Curriculum Generation via Large Language Models | LLMs write simulation code for embodied control tasks. | Training Environments | Multi-Domain Environments | 2024 | CoRL |
| InSTA: Towards Internet-Scale Training For Agents | Generates verifiable tasks on unlabeled websites at internet scale. | Training Environments | Multi-Domain Environments | 2025 | arXiv |
| Self-Challenging Language Model Agents | Generates verifiable Code-as-Task instances to bootstrap training data. | Training Environments | Multi-Domain Environments | 2025 | arXiv |
| AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning | Dynamically expands horizon from greedy to exploratory settings. | Training Environments | Multi-Domain Environments | 2025 | arXiv |
| Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction | Agents learn to adaptively allocate patience for test-time budgets. | Training Environments | Multi-Domain Environments | 2025 | arXiv |
Benchmarks for Training Environments
| Paper | TLDR | Component | ↳ Focus | Year | Venue |
|---|---|---|---|---|---|
| AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments | Unifies diverse tasks (web, games, databases) under a standard interface. | Benchmarks for Training Environments | Training Gyms | 2025 | ACL |
| The BrowserGym Ecosystem for Web Agent Research | Unifies diverse web tasks under a standard interface for generalization. | Benchmarks for Training Environments | Training Gyms | 2025 | TMLR |
| WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning | Provides functional browser environments with self-evolving curricula. | Benchmarks for Training Environments | Training Gyms | 2025 | ICLR |
| WebArena: A Realistic Web Environment for Building Autonomous Agents | Functional browser environment for learning navigation policies. | Benchmarks for Training Environments | Training Gyms | 2024 | ICLR |
| OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments | High-fidelity environment for computer control (OS). | Benchmarks for Training Environments | Training Gyms | 2024 | NeurIPS |
| AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents | High-fidelity environment for mobile control. | Benchmarks for Training Environments | Training Gyms | 2025 | ICLR |
| ALFWorld: Aligning Text and Embodied Environments for Interactive Learning | Bridges high-level reasoning with low-level embodied physics. | Benchmarks for Training Environments | Training Gyms | 2021 | ICLR |
| Measuring Massive Multitask Language Understanding | Standardized exam metrics for general cognitive reasoning. | Benchmarks for Training Environments | Certification Benchmarks | 2021 | ICLR |
| Training Verifiers to Solve Math Word Problems | Standardized benchmark for grade-school math reasoning. | Benchmarks for Training Environments | Certification Benchmarks | 2021 | arXiv |
| AgentBench: Evaluating LLMs as Agents | Quantifies the gap between commercial and open-source models across 8 modalities. | Benchmarks for Training Environments | Certification Benchmarks | 2024 | ICLR |
| PaperBench: Evaluating AI's Ability to Replicate AI Research | Certification test for research reproduction capabilities. | Benchmarks for Training Environments | Certification Benchmarks | 2025 | ICML |
| SWE-bench: Can Language Models Resolve Real-World GitHub Issues? | Certification test for software engineering and error recovery. | Benchmarks for Training Environments | Certification Benchmarks | 2024 | ICLR |
| CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities | Adversarial scenarios to certify robustness against exploits. | Benchmarks for Training Environments | Certification Benchmarks | 2025 | ICML |
| Do the Rewards Justify the Means? Measuring Trade-offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark | Interactive scenarios to certify robustness against ethical hazards. | Benchmarks for Training Environments | Certification Benchmarks | 2023 | ICML |
| R-Judge: Benchmarking Safety Risk Awareness for LLM Agents | Static trajectories to certify ability to refuse unsafe instructions. | Benchmarks for Training Environments | Certification Benchmarks | 2024 | EMNLP (Findings) |
How to Contribute
We welcome contributions! Feel free to open a PR with improvements, fixes, or additional resources.
Citation
@article{chang2026training,
title={Training Recipes for Agentic Reinforcement Learning in LLMs: A Survey},
author={Chang, Ching and Xiao, Yijia and Xu, Jade and Xu, Fred and Ye, Chenchen and Li, Ruoyan and Zhang, Junkai and Deng, Yihe and Zheng, Kyle and Ji, Ethan and others},
journal={TechRxiv},
year={2026},
publisher={TechRxiv},
doi={10.36227/techrxiv.176972131.13438500/v1},
url={https://www.techrxiv.org/users/1020340/articles/1380448-training-recipes-for-agentic-reinforcement-learning-in-llms-a-survey}
}
Contributors
blacksnail789521
6 commits
Similar repos
Training Recipes for Agentic Reinforcement Learning in LLMs: A Survey
51
6 commits
updated Jan 30, 2026
README
Agentic-RL-Training-Recipes
Training Recipes for Agentic Reinforcement Learning in LLMs: A Survey
✨ If you find our survey useful, a star ⭐ on GitHub helps others discover it and keeps you updated on future releases.
Table of Contents
Training Schemes
| Paper | TLDR | Component | ↳ Focus | Year | Venue |
|---|---|---|---|---|---|
| Reinforced Self-Training (ReST) for Language Modeling | Iterative self-improvement via generating and filtering high-quality trajectories. | Training Schemes | Rollout & Data Strategy | 2023 | arXiv |
| Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models | Self-play fine-tuning against previous iterations. | Training Schemes | Rollout & Data Strategy | 2024 | ICML |
| Scaling Relationship on Learning Mathematical Reasoning with Large Language Models | Simple rejection sampling strategy for data collection. | Training Schemes | Rollout & Data Strategy | 2023 | arXiv |
| DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models | Introduces Group Relative Policy Optimization (GRPO) for group-based sampling. | Training Schemes | Rollout & Data Strategy | 2024 | arXiv |
| Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning | Down-sampling strategy to filter rollouts and reduce compute. | Training Schemes | Rollout & Data Strategy | 2025 | arXiv |
| Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards | Tree-based exploration with branching at high-uncertainty steps. | Training Schemes | Rollout & Data Strategy | 2025 | arXiv |
| Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents | Learn from failure trajectories via contrastive preference pairs. | Training Schemes | Rollout & Data Strategy | 2024 | ACL |
| SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling | Adapts soft actor-critic and hindsight replay for open-ended exploration. | Training Schemes | Rollout & Data Strategy | 2024 | arXiv |
| Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning | Reverse curriculum learning starting from goal proximity. | Training Schemes | Rollout & Data Strategy | 2024 | ICML |
| Voyager: An Open-Ended Embodied Agent with Large Language Models | Uses predefined progressions in open-ended worlds (Minecraft). | Training Schemes | Rollout & Data Strategy | 2023 | arXiv |
| RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments | Dynamically generates tasks based on current agent performance. | Training Schemes | Rollout & Data Strategy | 2025 | arXiv |
| WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning | Self-correcting curriculum for web agents. | Training Schemes | Rollout & Data Strategy | 2025 | ICLR |
| VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models | Uses variance of group rewards to prioritize medium-difficulty tasks. | Training Schemes | Rollout & Data Strategy | 2025 | arXiv |
| Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning | Uses exact matching outcome rewards for definitive tasks. | Training Schemes | Feedback & Credit | 2025 | COLM |
| ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning | Exact matching outcome rewards for search tasks. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesis | Functional verification for open-ended intent satisfaction. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning | Optimizes efficiency by penalizing retrieval costs. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning | Reward modeling for open-ended search tasks. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning | Intent satisfaction rewards for search agents. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agent | Optimizes efficiency and functional verification. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| DeepRAG: Thinking to Retrieve Step by Step for Large Language Models | Optimizes efficiency by explicitly penalizing unnecessary retrieval actions. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| UR^2: Unify RAG and Reasoning through Reinforcement Learning | Efficiency-aware reward modeling for retrieval. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| ReZero: Enhancing LLM search ability by trying one-more-time | Rewards focused on relevance and formatting in IR. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| s3: You Don't Need That Much Data to Train a Search Agent via RL | IR reward modeling for query diversity and relevance. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| AI-SearchPlanner: Modular Agentic Search via Pareto-Optimal Multi-Objective Reinforcement Learning | Planning-centric rewards for search agents. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| DeepDiver: Adaptive Search Intensity Scaling via Open-Web Reinforcement Learning | IR rewards for deep exploration of topics. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| O^2-Searcher: A Searching-based Agent Model for Open-Domain Open-Ended Question Answering | Outcome-oriented rewards for search quality. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning | Tool-augmented reward modeling for long-form agentic tasks. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning | Uses Shortest Path Reward Estimation for trajectory quality. | Training Schemes | Feedback & Credit | 2025 | NeurIPS |
| Synthetic Data Generation and Multi-Step Reinforcement Learning for Reasoning and Tool Use | Holistic history analysis for process rewards. | Training Schemes | Feedback & Credit | 2025 | COLM |
| Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers | Process rewards for search exploration steps. | Training Schemes | Feedback & Credit | 2025 | NeurIPS |
| ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs | Dense supervision via process reward models (PRM). | Training Schemes | Feedback & Credit | 2025 | NeurIPS |
| Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning | Granular evaluation of specific reasoning steps. | Training Schemes | Feedback & Credit | 2025 | NeurIPS |
| Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation | Verification-based process rewards for reasoning. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents | Process supervision for verifiable meta-reasoning steps. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization | Intermediate search quality evaluation. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Coordinating Search-Informed Reasoning and Reasoning-Guided Search in Claim Verification | Hierarchical rewards for information seeking. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learning | Rewards parallel decomposition efficiency. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment | Uses LLMs to grade/critique intermediate steps. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalization | Collaborative grading for UI actions. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic | LLM-based critique for search trajectories. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models | Verbal feedback for self-correction via retrospective analysis. | Training Schemes | Feedback & Credit | 2025 | NeurIPS |
| Reflexion: language agents with verbal reinforcement learning | Verbal reinforcement for iterative refinement. | Training Schemes | Feedback & Credit | 2023 | NeurIPS |
| VinePPO: Refining Credit Assignment in RL Training of LLMs | Statistical credit assignment using rollout branching. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Exploiting Tree Structure for Credit Assignment in RL Training of LLMs | Tree-structured estimation for step-wise credit. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution | Trains dedicated value networks for step influence. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Agentic Reinforcement Learning with Implicit Step Rewards | Implicit value prediction for self-taught reasoners. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| GRPO-$\lambda$: Credit Assignment improves LLM Reasoning | Reformulated objective for granular updates without value heads. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Group-in-Group Policy Optimization for LLM Agent Training | Group-in-Group policy optimization for credit assignment. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Reward Design | Multi-turn adaptation of GRPO credit assignment. | Training Schemes | Feedback & Credit | 2025 | arXiv |
| Proximal Policy Optimization Algorithms | Standard clipped gradient optimization (Trust Region). | Training Schemes | Policy Optimization | 2017 | arXiv |
| VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks | Variant of PPO adapted for agentic stability. | Training Schemes | Policy Optimization | 2025 | arXiv |
| DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning | Scaled GRPO for reasoning tasks. | Training Schemes | Policy Optimization | 2025 | arXiv |
| DAPO: An Open-Source LLM Reinforcement Learning System at Scale | Systematized GRPO for scale and distributed training. | Training Schemes | Policy Optimization | 2025 | arXiv |
| URPO: A Unified Reward & Policy Optimization Framework for Large Language Models | Unifies policy optimization with reward modeling. | Training Schemes | Policy Optimization | 2025 | arXiv |
| TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling | Geometry-aware objectives for tree-structured policies. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning | Lightweight PPO with entropy bonuses. | Training Schemes | Policy Optimization | 2025 | arXiv |
| A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems | Survey on reasoning degeneracy and regularization. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learning | PRM-free step-level estimation for efficiency. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards | Asymmetric REINFORCE for off-policy data reuse. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts | Selective rollout strategy to optimize compute budget. | Training Schemes | Policy Optimization | 2025 | arXiv |
| A Survey on the Optimization of Large Language Model-based Agents | Survey on efficient optimization strategies. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning | Uses early stopping triggers (KL spikes) for stability. | Training Schemes | Policy Optimization | 2025 | arXiv |
| LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios | Overview of reasoning stability and hacking. | Training Schemes | Policy Optimization | 2025 | arXiv |
| AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement | Multi-objective scaffolds for safety and performance. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Safe RLHF: Safe Reinforcement Learning from Human Feedback | Constrained MDP formulation for safety. | Training Schemes | Policy Optimization | 2023 | ICLR 2024 |
| SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning | Safety constraints for vision-language agents. | Training Schemes | Policy Optimization | 2025 | arXiv |
| Agentic Reinforcement Learning for Search is Unsafe | Penalizes harmful queries in search agents. | Training Schemes | Policy Optimization | 2025 | arXiv |
| MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory | Fine-tunes models to execute explicit read/write API calls. | Training Schemes | Training-Time Memory | 2024 | arXiv |
| MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversation | Trains multi-stage summarization pipelines. | Training Schemes | Training-Time Memory | 2023 | arXiv |
| Augmenting Language Models with Long-Term Memory | Introduces decoupled memory encoders. | Training Schemes | Training-Time Memory | 2023 | NeurIPS |
| Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection | Trains reflection tokens to trigger on-demand retrieval. | Training Schemes | Training-Time Memory | 2024 | ICLR |
| MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents | Uses RLVR to compress context into a constant footprint. | Training Schemes | Training-Time Memory | 2025 | arXiv |
| MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent | Adapts DAPO for streaming document processing. | Training Schemes | Training-Time Memory | 2025 | arXiv |
| Memory-R1: Enhancing large language model agents to manage and utilize memories via reinforcement learning | Uses PPO/GRPO to train a dedicated memory-manager agent. | Training Schemes | Training-Time Memory | 2025 | arXiv |
| LongMemEval: Benchmarking chat assistants on long-term interactive memory | Benchmark for knowledge updates and abstention. | Training Schemes | Training-Time Memory | 2024 | arXiv |
| LongBench-v2: Towards deeper understanding and reasoning on realistic long-context multitasks | Benchmark for extreme-context (2M tokens) tasks. | Training Schemes | Training-Time Memory | 2025 | ACL |
Training Infrastructure
| Paper | TLDR | Component | ↳ Focus | Year | Venue |
|---|---|---|---|---|---|
| Proximal Policy Optimization Algorithms | Standard synchronous execution model for on-policy consistency. | Training Infrastructure | Actor-Learner Architectures | 2017 | ICLR |
| DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models | Utilizes synchronous execution to simplify credit assignment. | Training Infrastructure | Actor-Learner Architectures | 2024 | arXiv |
| AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning | Decouples collection and training with staleness control. | Training Infrastructure | Actor-Learner Architectures | 2025 | NeurIPS |
| Asynchronous Methods for Deep Reinforcement Learning | Introduces asynchronous actor-learner decoupling. | Training Infrastructure | Actor-Learner Architectures | 2016 | ICML |
| IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures | V-trace correction for off-policy lag in asynchronous setups. | Training Infrastructure | Actor-Learner Architectures | 2018 | ICML |
| IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks | Uses clipped target networks to stabilize stale critics. | Training Infrastructure | Actor-Learner Architectures | 2020 | ICLR |
| LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training | Orchestrates disjoint GPU groups with AIPO for lag correction. | Training Infrastructure | Actor-Learner Architectures | 2025 | arXiv |
| Defeating the Training-Inference Mismatch via FP16 | Switches to FP16 to reduce training-inference numerical divergence. | Training Infrastructure | Precision & Acceleration | 2025 | arXiv |
| FP8-LM: Training FP8 Large Language Models | Demonstrates doubled throughput with FP8 training. | Training Infrastructure | Precision & Acceleration | 2023 | arXiv |
| COAT: Compressing Optimizer states and Activations for Memory-Efficient FP8 Training | Compresses optimizer states and activations for efficiency. | Training Infrastructure | Precision & Acceleration | 2025 | ICLR |
| Scaling FP8 training to trillion-token LLMs | Smoothing operation to handle activation outliers in FP8. | Training Infrastructure | Precision & Acceleration | 2025 | ICLR |
| Optimizing Large Language Model Training Using FP4 Quantization | Enables FP4 training via block-wise quantization. | Training Infrastructure | Precision & Acceleration | 2025 | ICML |
| HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs | Hadamard rotations to spread outliers for low-bit precision. | Training Infrastructure | Precision & Acceleration | 2025 | NeurIPS |
| QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs | Uses Adaptive Quantization Noise (AQN) for exploration. | Training Infrastructure | Precision & Acceleration | 2025 | arXiv |
| Efficient Memory Management for Large Language Model Serving with PagedAttention | Optimized inference kernel that contributes to mismatch. | Training Infrastructure | Training-Inference Mismatch | 2023 | SOSP |
| SGLang: Efficient Execution of Structured Language Model Programs | High-throughput inference engine often differing from learner. | Training Infrastructure | Training-Inference Mismatch | 2024 | NeurIPS |
| PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel | High-precision training framework (standard baseline). | Training Infrastructure | Training-Inference Mismatch | 2023 | arXiv |
| Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | High-precision training framework (standard baseline). | Training Infrastructure | Training-Inference Mismatch | 2019 | arXiv |
| MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention | Upcasts output head to FP32 to fix entropy collapse. | Training Infrastructure | Training-Inference Mismatch | 2025 | arXiv |
| Your Efficient RL Framework Secretly Brings You Off-Policy RL Training | Token-level importance sampling to correct distribution shift. | Training Infrastructure | Training-Inference Mismatch | 2025 | Blog |
| When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch | Sequence masking to robustly handle off-policy shift. | Training Infrastructure | Training-Inference Mismatch | 2025 | Blog |
| DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models | Adopts off-policy sequence masking for training stability. | Training Infrastructure | Training-Inference Mismatch | 2025 | arXiv |
Training Environments
| Paper | TLDR | Component | ↳ Focus | Year | Venue |
|---|---|---|---|---|---|
| Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning | Uses outcome-based rewards to train tool invocation without step-level supervision. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use | Modular framework for unified performance across SQL, code, and visual tasks. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| Tool-Augmented Policy Optimization: Synergizing Reasoning and Adaptive Tool Use with Reinforcement Learning | Uses Dynamic-PPO to optimize tool usage for external retrieval. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| Agent-R1: Training Powerful LLM Agents with End-to-End Reinforcement Learning | Formalizes tool use as an MDP to improve multi-step reasoning. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| Simulating Environments with Reasoning Models for Agent Training | Leverages simulated feedback to overcome data scarcity. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| Introducing the Model Context Protocol | Standard for reproducible agent-tool interactions. | Training Environments | Single-Domain Environments | 2024 | Blog |
| Navigating WebAI: Training Agents to Complete Web Tasks with Large Language Models and Reinforcement Learning | Trains hierarchical T5 planner with V-MPO for web navigation. | Training Environments | Single-Domain Environments | 2024 | SAC |
| AutoWebGLM: A Large Language Model-based Web Navigating Agent | Staged pipeline (SFT $\to$ DPO $\to$ RFT) for stability in web agents. | Training Environments | Single-Domain Environments | 2024 | KDD |
| WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning | Applies on-policy M-GRPO using sparse binary signals in WebArena-Lite. | Training Environments | Single-Domain Environments | 2025 | EMNLP |
| WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning | Uses self-evolving curriculum and outcome reward models for web agents. | Training Environments | Single-Domain Environments | 2025 | ICLR |
| ReTool: Reinforcement Learning for Strategic Tool Use in LLMs | Shows reward-driven learning yields emergent tool behaviors. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| ToRL: Scaling Tool-Integrated RL | Demonstrates emergent tool use from reward-driven learning. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| Reinforcement Learning for Reasoning in Large Language Models with One Training Example | Shows single verifier-rewarded problem can double performance. | Training Environments | Single-Domain Environments | 2025 | arXiv |
| StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback | Decomposes code tasks into curriculum-aligned sub-problems with compiler feedback. | Training Environments | Single-Domain Environments | 2024 | ACL |
| The BrowserGym Ecosystem for Web Agent Research | Aggregates web benchmarks into a fixed schema. | Training Environments | Multi-Domain Environments | 2025 | TMLR |
| AgentGym: Evolving Large Language Model-based Agents across Diverse Environments | Unifies diverse domains via a consistent HTTP interface. | Training Environments | Multi-Domain Environments | 2025 | ACL |
| Mind2Web: Towards a Generalist Agent for the Web | Uses strict splits to penalize layout memorization. | Training Environments | Multi-Domain Environments | 2023 | NeurIPS |
| WebCanvas: Benchmarking Web Agents in Online Environments | Evaluates agents against live, drifting UIs. | Training Environments | Multi-Domain Environments | 2024 | arXiv |
| VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks | Unifies multimodal tasks through consistent GUI interfaces. | Training Environments | Multi-Domain Environments | 2024 | ACL |
| WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models | Unifies multimodal tasks through consistent GUI interfaces. | Training Environments | Multi-Domain Environments | 2024 | ACL |
| AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation | Applies bidirectional evolution to synthesize environment code. | Training Environments | Multi-Domain Environments | 2025 | KDD |
| Eurekaverse: Environment Curriculum Generation via Large Language Models | LLMs write simulation code for embodied control tasks. | Training Environments | Multi-Domain Environments | 2024 | CoRL |
| InSTA: Towards Internet-Scale Training For Agents | Generates verifiable tasks on unlabeled websites at internet scale. | Training Environments | Multi-Domain Environments | 2025 | arXiv |
| Self-Challenging Language Model Agents | Generates verifiable Code-as-Task instances to bootstrap training data. | Training Environments | Multi-Domain Environments | 2025 | arXiv |
| AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning | Dynamically expands horizon from greedy to exploratory settings. | Training Environments | Multi-Domain Environments | 2025 | arXiv |
| Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction | Agents learn to adaptively allocate patience for test-time budgets. | Training Environments | Multi-Domain Environments | 2025 | arXiv |
Benchmarks for Training Environments
| Paper | TLDR | Component | ↳ Focus | Year | Venue |
|---|---|---|---|---|---|
| AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments | Unifies diverse tasks (web, games, databases) under a standard interface. | Benchmarks for Training Environments | Training Gyms | 2025 | ACL |
| The BrowserGym Ecosystem for Web Agent Research | Unifies diverse web tasks under a standard interface for generalization. | Benchmarks for Training Environments | Training Gyms | 2025 | TMLR |
| WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning | Provides functional browser environments with self-evolving curricula. | Benchmarks for Training Environments | Training Gyms | 2025 | ICLR |
| WebArena: A Realistic Web Environment for Building Autonomous Agents | Functional browser environment for learning navigation policies. | Benchmarks for Training Environments | Training Gyms | 2024 | ICLR |
| OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments | High-fidelity environment for computer control (OS). | Benchmarks for Training Environments | Training Gyms | 2024 | NeurIPS |
| AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents | High-fidelity environment for mobile control. | Benchmarks for Training Environments | Training Gyms | 2025 | ICLR |
| ALFWorld: Aligning Text and Embodied Environments for Interactive Learning | Bridges high-level reasoning with low-level embodied physics. | Benchmarks for Training Environments | Training Gyms | 2021 | ICLR |
| Measuring Massive Multitask Language Understanding | Standardized exam metrics for general cognitive reasoning. | Benchmarks for Training Environments | Certification Benchmarks | 2021 | ICLR |
| Training Verifiers to Solve Math Word Problems | Standardized benchmark for grade-school math reasoning. | Benchmarks for Training Environments | Certification Benchmarks | 2021 | arXiv |
| AgentBench: Evaluating LLMs as Agents | Quantifies the gap between commercial and open-source models across 8 modalities. | Benchmarks for Training Environments | Certification Benchmarks | 2024 | ICLR |
| PaperBench: Evaluating AI's Ability to Replicate AI Research | Certification test for research reproduction capabilities. | Benchmarks for Training Environments | Certification Benchmarks | 2025 | ICML |
| SWE-bench: Can Language Models Resolve Real-World GitHub Issues? | Certification test for software engineering and error recovery. | Benchmarks for Training Environments | Certification Benchmarks | 2024 | ICLR |
| CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities | Adversarial scenarios to certify robustness against exploits. | Benchmarks for Training Environments | Certification Benchmarks | 2025 | ICML |
| Do the Rewards Justify the Means? Measuring Trade-offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark | Interactive scenarios to certify robustness against ethical hazards. | Benchmarks for Training Environments | Certification Benchmarks | 2023 | ICML |
| R-Judge: Benchmarking Safety Risk Awareness for LLM Agents | Static trajectories to certify ability to refuse unsafe instructions. | Benchmarks for Training Environments | Certification Benchmarks | 2024 | EMNLP (Findings) |
How to Contribute
We welcome contributions! Feel free to open a PR with improvements, fixes, or additional resources.
Citation
@article{chang2026training,
title={Training Recipes for Agentic Reinforcement Learning in LLMs: A Survey},
author={Chang, Ching and Xiao, Yijia and Xu, Jade and Xu, Fred and Ye, Chenchen and Li, Ruoyan and Zhang, Junkai and Deng, Yihe and Zheng, Kyle and Ji, Ethan and others},
journal={TechRxiv},
year={2026},
publisher={TechRxiv},
doi={10.36227/techrxiv.176972131.13438500/v1},
url={https://www.techrxiv.org/users/1020340/articles/1380448-training-recipes-for-agentic-reinforcement-learning-in-llms-a-survey}
}
Similar repos
Contributors
blacksnail789521
6 commits