🔮 Awesome-Environment-Scaling

👋 Introduction

Welcome to Awesome-Environment-Scaling!

As agent capabilities continue to evolve, it is infeasible to attain intelligence beyond the human-level merely by supervised fine-tuning (SFT) pretrained models on static datasets. Such datasets are typically manually annotated or curated under human oversight, which makes them costly and labor-intensive to produce at scale, intrinsically bounded by human-level knowledge, and often lacking realism and adaptability. By contrast, reinforcement learning provides a more aligned training paradigm: agents explore and interact with the environment, accumulate experience, and acquire new knowledge and skills. We formalize this interactive process as the Generation–Execution–Feedback (GEF) loop. In this setting, the environment is no longer a mere container for agents’ activities; it has become an active producer of experiential data, underscoring the growing need for scaling environments to create a more complex, realistic, and richly interactive world.

We are the first to comprehensively investigate current environment scaling methods from an environment-centric perspective and curate representative methods along the three stages of the GEF loop:

• Stage 1: Task Generation - Environments generate complex, dynamic, and diverse tasks to continuously challenge agents.

• Stage 2: Task Execution - Environments support real-time interaction (Interactivity) and provide observation that consitent with real world (Realism).

• Stage 3: Feedback - Environments provide evaluative feedback on rollouts with higher density and finer granularity, increase automation and objectivity, and improve robustness, forming useful experience for subsequent learning.

This collection is for researchers, developers, and enthusiasts eager to explore and build the next generation of LLM agents.

🎉 News

• 2025.12 A new version of our survey paper "Environment Scaling for Interactive Agentic Experience Collection: A Survey" is updated on arXiv.
• 2025.11 We have released the survey paper "Scaling Environments for LLM Agents in the Era of Learning from Interaction: A Survey". The arXiv page is now available.
• 2025.10 The survey paper has been accepted by SEA @ NeurIPS 2025 and will be presented on December 7th in San Diego, USA.

📜 Table of Contents

• 🔮 Awesome-Environment-Scaling
  • 👋 Introduction
  • 🎉 News
  • 📜 Table of Contents
  • 🎯 Stage 1: Task Generation
    • ➤ Complexity Scaling
    • ➤ Dynamic Scaling
    • ➤ Diversity Scaling
  • 🧗 Stage 2: Task Execution
    • ➤ Interactivity Scaling
    • ➤ Realism Scaling
  • ⚖️ Stage 3: Feedback
    • ➤ Density Scaling
    • ➤ Granularity Scaling
    • ➤ Automation Scaling
    • ➤ Objectivity Scaling
    • ➤ Robustness Scaling
  • 🙏 Contributing & Citation
  • Star History

🎯 Stage 1: Task Generation

In the task generation stage, the environment is required to propose challenging tasks to push the agent’s capabilities forward. Scaling at this stage therefore targets three facets of task supply: increasing difficulty, introducing dynamics, and expanding diversity. Accordingly, we organize representative approaches into three directions: complexity scaling, dynamic scaling, and diversity scaling.

➤ Complexity Scaling

Static complexity increases a task’s inherent structural intricacy, moving beyond single-step commands to challenges defined by dependencies, logical flows, and hierarchical relationships.

Sequential

• ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use
• $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
• The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

Compositional

• Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL
• Taskcraft: Automated generation of agentic tasks.
• $C^3$-Bench: The Things Real Disturbing LLM based Agent in Multi-Tasking
• Compositional Task Representations for Large Language Models
• WebDancer: Towards Autonomous Information Seeking Agency
• WebWalker: Benchmarking LLMs in Web Traversal

Graph-based

• WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking
• WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learning
• WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents
• WebSailor: Navigating Super-human Reasoning for Web Agent
• WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization

➤ Dynamic Scaling

Scaling dynamics focuses on creating an environment that changes task complexity based on agent performance, helping agents learn and adapt to new challenges and goals.

• RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
• Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairs
• LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Training
• EvoCurr: Self-evolving Curriculum with Behavior Code Generation for Complex Decision-making
• R-Zero: Self-Evolving Reasoning LLM from Zero Data
• All Life is Problem Creation: Learning to Generate Environments that Maximize Performance Gain
• WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
• AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation
• AgentGym: Evolving Large Language Model-based Agents across Diverse Environments
• EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents

➤ Diversity Scaling

Scaling the diversity of environments and tasks is crucial for preventing agents from overfitting to specific action patterns, thereby fostering more robust and generalizable performances.

• RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
• AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learning
• ARE: Scaling Up Agent Environments and Evaluations
• R-Zero: Self-Evolving Reasoning LLM from Zero Data
• AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents
• Taskcraft: Automated generation of agentic tasks.
• AgentSense: Virtual Sensor Data Generation Using LLM Agents in Simulated Home Environments
• BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery
• AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories
• AgentGym: Evolving Large Language Model-based Agents across Diverse Environments

🧗 Stage 2: Task Execution

In the task execution stage, interaction takes the form of the agent taking an action and receiving an observation from the environment. We organize works towards environment scaling in this stage into two dimensions: interactivity scaling and realism scaling. Interactivity learns how agent performance affected by the frequency of tool interaction, while Realism focuses on making observations more realistic to enhance the interaction quality.

➤ Interactivity Scaling

Interactivity concerns whether the environment is executable and whether the agent is blind to the returns of intermediate steps.

• MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents
• Towards General Agentic Intelligence via Environment Scaling
• BrowseMaster: Towards Scalable Web Browsing via Tool-Augmented Programmatic Agent Pair
• UserBench: An Interactive Gym Environment for User-Centric Agents
• Procedural Environment Generation for Tool-Use Agents
• AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

➤ Realism Scaling

Realism aims to ensure that observations obtained from the environment remain consistent with the real world.

• Towards General Agentic Intelligence via Environment Scaling
• Tongyi DeepResearch: A New Era of Open-Source AI Researchers
• Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments
• MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers
• MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers
• MCPToolBench++: A Large Scale AI Agent Model Context Protocol MCP Tool Use Benchmark
• Making REST APIs Agent-Ready: From OpenAPI to Model Context Protocol Servers for Tool-Augmented LLMs
• $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis
• APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay
• WebWalker: Benchmarking LLMs in Web Traversal
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
• WebArena Verified: Reliable Evaluation for Web Agents
• WebArena: A Realistic Web Environment for Building Autonomous Agents
• RestGPT: Connecting Large Language Models with Real-World RESTful APIs
• WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

Multi-agent Realism

• ARE: Scaling Up Agent Environments and Evaluations
• A Parallelized Framework for Simulating Large-Scale LLM Agents with Realistic Environments and Interactions
• Licence to Scale: A Microservice Simulation Environment for Benchmarking Agentic AI
• Enabling multi-agent collaboration in knowledge graph environments
• AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society
• OASIS: Open Agent Social Interaction Simulations with One Million Agents

⚖️ Stage 3: Feedback

In the feedback stage, the environment assesses the trajectories collected during task execution and generates feedback signals for subsequent RL training. Scaling at this stage focuses on how feedback is provided, including its frequency and richness (density and granularity), its level of automation (automation), as well as how objectively and reliably it is delivered (objectivity and robustness).

➤ Density Scaling

The density of feedback refers to how frequently evaluative signals are provided, typically including trajectory-level outcome-based rewards and step-level process-based rewards.

Outcome-based Rewards

• VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use
• Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards
• Toolrl: Reward is all tool learning needs
• OTC: Optimal Tool Calls via Reinforcement Learning

Process-based Rewards

• Know What You Don't Know: Uncertainty Calibration of Process Reward Models
• Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
• Process Reward Models That Think
• What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning
• On Designing Effective RL Reward at Training Time for LLM Reasoning

➤ Granularity Scaling

Granularity scaling refers to increasing the level of detail in feedback as well as enriching the forms of feedback provided.

• DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
• Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
• Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning
• Reinforcement Learning with Rubric Anchors
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
• Checklists Are Better Than Reward Models For Aligning Language Models
• AdaCtrl: Towards Adaptive and Controllable Reasoning via Difficulty-Aware Budgeting
• Toolrl: Reward is all tool learning needs
• R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning

➤ Automation Scaling

Automation scaling involves the shift of the feedback mechanism from slow, costly, and labor-intensive human feedback to automated evaluations.

• Reinforcement Learning with Rubric Anchors
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
• JudgeLRM: Large Reasoning Models as a Judge
• Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems
• Scaling autonomous agents via automatic reward modeling and planning
• Generative verifiers: Reward modeling as next-token prediction

➤ Objectivity Scaling

Objectivity scaling aims to reduce these biases and make automated verifiers fairer.

• ARE: Scaling Up Agent Environments and Evaluations
• Reinforcement Learning with Rubric Anchors
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
• Omni-think: Scaling cross-domain generalization in llms via multi-task rl with hybrid rewards
• Rlpr: Extrapolating rlvr to general domains without verifiers
• Writing-zero: Bridge the gap between non-verifiable tasks and verifiable rewards
• General-reasoner: Advancing llm reasoning across all domains
• Nover: Incentive training for language models via verifier-free reinforcement learning
• Nemotron-crossthink: Scaling self-learning beyond math reasoning
• Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains

➤ Robustness Scaling

Feedback robustness requires the environment to provide stable and reliable reward signals.

Reward-level

• Reward Hacking Mitigation using Verifiable Composite Rewards
• Reinforcement Learning with Rubric Anchors
• Reward Shaping to Mitigate Reward Hacking in RLHF
• Mona: Myopic optimization with non-myopic approval can mitigate multi-step reward hacking
• Navigating noisy feedback: Enhancing reinforcement learning with error-prone language models
• RRM: Robust Reward Model Training Mitigates Reward Hacking
• InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling

Environment-level

• Trinity-RFT: A general-purpose and unified framework for reinforcement fine-tuning of large language models
• Tongyi DeepResearch: A New Era of Open-Source AI Researchers

🙏 Contributing & Citation

We welcome contributions! If you have a paper that fits into this framework, please open a pull request. Let's build this resource together.

If you find our survey and this repository useful for your research, please consider citing our work:

@misc{huang2025scalingenvironmentsllmagents,
      title={Scaling Environments for LLM Agents in the Era of Learning from Interaction: A Survey}, 
      author={Yuchen Huang and Sijia Li and Minghao Liu and Wei Liu and Shijue Huang and Zhiyuan Fan and Hou Pong Chan and Yi R. Fung},
      year={2025},
      eprint={2511.09586},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2511.09586}, 
}

Star History