AgentEvolver: Towards Efficient Self-Evolving Agent System

AgentEvolver is an end-to-end, self-evolving training framework that unifies self-questioning, self-navigating, and self-attributing into a cohesive system. It empowers agents to autonomously improve their capabilities, aiming for efficient, cost-effective, and continuous capability evolution.

📰 News

• [2026-03] ⚡ SeeUPO released — Sequence-Level Agentic RL with convergence guarantees; multi-turn training stack built on AgentEvolver
• [2025-12] 🎮 AgentEvolver Game Arena released — a multi-agent social reasoning arena for interaction, evaluation, and training in games like Avalon and Diplomacy.
• [2025-12] 📢 New preprint CuES on an extended self-questioning method released with code.
• [2025-11] 📄 The AgentEvolver Technical Report is now available, detailing the framework's architecture, methodology, and key findings.
• [2025-11] 🧩 AgentEvolver v1 has been released now!

✨ Why AgentEvolver

🧠 AgentEvolver provides three Self-Evolving Mechanisms from Environment to Policy:

• Automatic Task Generation (Self-Questioning) – Explore the environment and autonomously create diverse tasks, eliminating costly manual dataset construction.
• Experience-guided Exploration (Self-Navigating) – Summarize and reuse cross-task experience, guiding higher-quality rollouts and improving exploration efficiency.
• Attribution-based Credit Assignment (Self-Attributing) – Process long trajectories to uncover the causal contribution of intermediate steps, enabling fine-grained and efficient policy optimization.

🔧 Architecture Design

AgentEvolver adopts a service-oriented dataflow architecture, seamlessly integrating environment sandboxes, LLMs, and experience management into modular services.

• Environment Compatibility – Standardized interfaces for seamless integration with a wide range of external environments and tool APIs.
• Flexible Context Manager – Built-in utilities for managing multi-turn contexts and complex interaction logic, supporting diverse deployment scenarios.
• Modular & Extensible Architecture – Decoupled components allow easy customization, secondary development, and future algorithm upgrades.

🌟 Benchmark Performance

Performance comparison on the AppWorld and BFCL-v3 benchmarks. AgentEvolver achieves superior results while using substantially fewer parameters than larger baseline models.

Performance on two benchmarks. Columns show avg@8 and best@8 for each benchmark, plus their averages (Avg.). All values are in percent (%). Bolded numbers highlight the best results.

🚀 Quick Start

Step 1. Basic Dependency Installation

Make sure you have conda and cuda toolkit installed.

Then, set up the training environment by running the script

bash install.sh

Step 2. Setup Env-Service (Appworld as example)

The script below sets up an environment for appworld.

cd env_service/environments/appworld && bash setup.sh

Step 3. Setup ReMe (Optional)

Set up the ReMe for experience management by running the script:

bash external/reme/install_reme.sh

For more detailed installation, please refer to ReMe.

Step 4. Begin Training! 🚀 🚀

Copy the example.env file to .env and modify the parameters, including your API key, conda path.

Using AgentEvolver launcher to start environment, log dashboard and training process altogether.

conda activate agentevolver

# option 1: minimal example without ReMe (using built-in datasets within environments)
python launcher.py --conf examples/basic.yaml --with-appworld

# option 2: full example with ReMe (questioning + navigating + attributing)
python launcher.py --conf examples/overall.yaml --with-appworld --with-reme

Alternatively, you can use bash scripts for manual execution: bash examples/run_basic.sh or bash examples/run_overall.sh. See Advanced Usage for more details.

🎮 AgentEvolver Game Arena

AgentEvolver Game Arena extends AgentEvolver into multi-agent social game environments, providing a unified arena for interaction, evaluation, and training of AI agents in long-horizon social reasoning tasks.

Key capabilities include:

• Web-based interaction – Observe AI agents' reasoning and communication in real time, or participate as a human player.
• Scalable evaluation – Run large-scale self-play or mixed-model tournaments with configurable settings and leaderboards.
• End-to-end training support – Train LLM agents directly within social game environments using reinforcement learning–based methods (e.g., GRPO).

Web Interface Demo:

Training Example: Training curve for the assassin role in Avalon

For detailed documentation, quick start guides, and configuration examples, see the Game Arena README.

🧩 Advanced Usage

🔧 Manual Execution

For users requiring fine-grained control over the training pipeline, we provide standalone execution scripts:

• bash examples/run_basic.sh - Execute basic RL pipeline with GRPO using built-in datasets within environments.
• bash examples/run_overall.sh - Run the complete self-evolving AgentEvolver pipeline with fully customizable configurations.

Refer to the QuickStart for detailed usage instructions and configuration parameters.

📄 Documentation

For detailed usage and customization, please refer to the following guidelines:

• Environment Service - Set up and manage environment instances, integrate custom environments
• Task Manager - Explore environments, generate synthetic tasks, and curate training data for agent evolution
• Experience Manager - Configure experience pool management and self-navigating mechanisms
• Advantage Processor - Implement self-attributing mechanisms with ADCA-GRPO for fine-grained credit assignment

For API documentation and more details, visit our documentation site.

🔮 Upcoming

• Evolution in multi-agent scenarios – Investigate autonomous co-evolution strategies for agents operating within shared, interactive environments.
• Cross-stage collaborative self-evolution – Explore methods that couple questioning, navigating, and attributing into coordinated loops for mutual enhancement.

🙏 Acknowledgements

This project builds upon the excellent work of several open-source projects:

• ReMe - for experience summarization and management;
• veRL - for distributed RL training;
• mkdocs - for documentation.

📚 Citation

If you find this work useful, please consider citing:

@misc{AgentEvolver2025,
  title         = {AgentEvolver: Towards Efficient Self-Evolving Agent System},
  author        = {Yunpeng Zhai and Shuchang Tao and Cheng Chen and Anni Zou and Ziqian Chen and Qingxu Fu and Shinji Mai and Li Yu and Jiaji Deng and Zouying Cao and Zhaoyang Liu and Bolin Ding and Jingren Zhou},
  year          = {2025},
  eprint        = {2511.10395},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2511.10395}
}

✨ Star History