AEGIS: Automated Error Generation and Identification for Multi-Agent Systems

Paper | Project Page | Code

Overview

AEGIS (Automated Error Generation and Identification for Multi-Agent Systems) is a large-scale dataset and benchmark for detecting errors in Multi-Agent Systems (MAS). It addresses the critical lack of large-scale, diverse datasets with precise, ground-truth error labels for MAS, which has hampered research in understanding MAS error modes.

By systematically injecting controllable and traceable errors into initially successful trajectories, AEGIS creates a rich dataset of realistic failures. This is achieved using a context-aware, LLM-based adaptive manipulator that performs sophisticated attacks like prompt injection and response corruption to induce specific, predefined error modes. The dataset enables the development and evaluation of robust error detection methods across multiple MAS frameworks and supports the error identification task.

Sample Usage

To generate an error dataset using AEGIS, you can use the following Python snippet, as provided in the GitHub repository:

from aegis_core.malicious_factory import MaliciousFactory
from aegis_core.agent_systems import BaseMASWrapper

# Initialize AEGIS
factory = MaliciousFactory()

# Load a MAS framework
mas_wrapper = BaseMASWrapper("dylan", config_path="configs/config_main.yaml")

# Inject errors and generate dataset
results = factory.generate_error_dataset(
    mas_wrapper=mas_wrapper,
    tasks=["task1", "task2"],
    error_modes=["FM-1.1", "FM-2.3"],
    num_samples=100
)

Core Components

Error Injection Factory

The malicious factory provides sophisticated error injection capabilities:

• Prompt Injection: Modifies agent inputs to trigger specific errors
• Response Corruption: Directly manipulates agent outputs
• Context-Aware Attacks: Adapts injection strategies based on task domain and agent role

Multi-Agent System Wrappers

Standardized interfaces for various MAS frameworks are provided, including:

• DyLAN: Dynamic graph-based agent interactions
• AgentVerse: Hierarchical role-based collaboration
• LLM Debate: Multi-agent debate mechanisms
• MacNet: Configurable network topologies
• Magnetic-One: Orchestrator-executor patterns

Error Taxonomy

AEGIS supports 14 failure modes based on the MAST taxonomy, categorized as:

1. Specification Issues (FM-1.x)

   • Task specification deviation
   • Role specification deviation
   • Add redundant steps
   • Remove conversation history
   • Remove termination conditions

2. Inter-Agent Misalignment (FM-2.x)

   • Repeat handled tasks
   • Make request ambiguous
   • Deviate from main goal
   • Hide important information
   • Ignore other agents
   • Inconsistent reasoning

3. Task Verification Failures (FM-3.x)

   • Premature termination
   • Remove verification steps
   • Incorrect verification

Citation

If you use AEGIS in your research, please cite:

@article{kong2025aegis,
  title={AEGIS: Automated Error Generation and Identification for Multi-Agent Systems},
  author={Kong, Fanqi and Zhang, Ruijie and Yin, Huaxiao and Zhang, Guibin and Zhang, Xiaofei and Chen, Ziang and Zhang, Zhaowei and Zhang, Xiaoyuan and Zhu, Song-Chun and Feng, Xue},
  journal={arXiv preprint arXiv:2509.14295},
  year={2025}
}