Shadowgraph
TrendingAbout
Sign in

MurrayTom/Awesome-Agent-Security-Papers

Beyond Environment and User Inputs: A Comprehensive Survey of Skill-Oriented Security in Agentic AI

10

18 commits

updated Aug 28, 2026

See the code

README

Pandora's Toolbox

A Survey of Generalist Agent Security from the Lifecycle Perspective

A curated literature map of risks, evaluations, and defenses across the full agent lifecycle

Lifecycle Topic Status

🌐 Language / 语言: English · 简体中文

Paper · Taxonomy · Contributing

Overview

Generalist agents are no longer bounded chat interfaces. They acquire reusable skills, retrieve long-term memory, select tools, interact with external environments, and preserve artifacts across sessions. Their security therefore cannot be understood as a collection of isolated model or component failures. A vulnerability may enter early, remain latent, be amplified during decision-making, materialize as a consequential action, and then contaminate persistent state.

This repository accompanies Pandora's Toolbox: A Survey of Generalist Agent Security from the Lifecycle Perspective. The survey organizes generalist-agent security through two orthogonal structures:

  • Lifecycle stages: provenance → orchestration → execution, followed by feedback through persistent state.
  • Security perspectives: risks and attacks, evaluation benchmarks, and defense methods at each stage.

The stage assignments below indicate where a risk is introduced, where safety evidence is collected, or where a defense primarily intervenes. They are not mutually exclusive: one attack or method may span multiple stages, and one paper may appear in more than one category.

The Lifecycle Perspective

Lifecycle perspective on generalist agent security: provenance, orchestration, execution, and backward contamination through persistent state.

Figure 1. Risks enter during provenance, propagate and amplify through orchestration, manifest during execution, and persist through contaminated states that affect future workflows.

Lifecycle stageSecurity questionPrimary risk pathTypical evidenceMain intervention point
ProvenanceWhat capabilities and artifacts are admitted?Poisoned metadata, skill files, dependencies, and forged trust signalsSkill packages, manifests, provenance recordsPre-installation inspection and admission control
OrchestrationHow are context, memory, skills, and tools composed into decisions?Malicious requests, retrieved memories, skill instructions, unsafe compositionsSelected skills, plans, candidate tool callsContext filtering, decision integrity, plan auditing
ExecutionWhat actions are committed, and how does the environment respond?Web, GUI, tool-output, API, file, and terminal injectionActions, trajectories, tool evidence, final environment stateStep-level enforcement and trajectory review
Persistent feedbackWhat survives and re-enters future workflows?Contaminated memory, rewritten skills, and compromised artifactsCross-session state and recurrenceProvenance tracking, recovery, and continual safety evolution
View the full taxonomy from the paper

Full taxonomy of security risks, evaluations, and defenses across the generalist agent lifecycle.

Figure 2. Taxonomy of security risks, evaluations, and defenses across the full lifecycle of generalist agents. Click the figure to open the high-resolution version.

Literature Map

1. Provenance

The provenance stage determines which capabilities enter an agent's usable capability space. Skill metadata may be loaded before a task begins, while specifications, scripts, resources, dependencies, and trust signals can remain dormant until a later request activates them.

1.1 Risks and Attacks

Declarative skill file poisoning

Attackers poison skill metadata, instructions, examples, or trigger rules so that malicious natural-language instructions enter the agent's planning context.

Multi-component skill ecosystem poisoning

The attack surface extends beyond the primary skill file to auxiliary scripts, tool interfaces, dependencies, external assets, configuration, retrieval resources, and local caches.

Cross-session skill rewriting and backdoor triggering

An attack modifies reusable skills, caches, project artifacts, or dependencies in one session, while malicious behavior activates only in a later session or under a specific trigger.

1.2 Evaluation Benchmarks

Malicious skill detection

These benchmarks evaluate whether a rule system, model, static analyzer, or verifier can identify semantic, code-level, and mixed attacks in a skill package before execution.

1.3 Defense Methods

Skill risk scanning

Before a skill is installed, integrated, or invoked, scanners inspect natural-language declarations, requested permissions, dependencies, executable resources, and cross-skill data flows.

2. Orchestration

During orchestration, the agent interprets the user request, retrieves memory, selects and composes skills, constructs a plan, and prepares tool calls. This stage is where latent risks become activated and amplified through context and control flow.

2.1 Risks and Attacks

Benign skill misuse and compositional risk

An individually benign skill may be exploitable through adversarial input. Multiple benign skills may also produce unauthorized, privacy-leaking, or destructive behavior when composed along a shared execution path.

User-side attacks

Malicious users submit harmful tasks, jailbreak prompts, obfuscated contexts, or multi-turn inducements through the direct interaction interface.

Manual jailbreak attacks
Heuristic optimization-based jailbreak attacks
Agentic red-teaming for jailbreak

Memory and long-term state risks

Long-term memory is retrieved and reused as trusted context in later tasks. A single poisoning event may therefore create stealthy, persistent, and transmissible manipulation.

Direct memory poisoning
Interaction-based memory injection and backdoors

2.2 Evaluation Benchmarks

Skill-mediated agent safety

Given a user task and one or more skills, the agent executes in an environment. Safety is judged from its response, execution evidence, or the resulting state.

2.3 Defense Methods

Prompt injection and jailbreak guardrails

Boundary detectors and safety classifiers inspect user inputs, retrieved content, web data, tool outputs, and model responses before these signals influence later decisions.

Memory guardrails

Memory defenses govern writing, retrieval, reasoning, and recovery through provenance verification, consistency checking, anomaly detection, causal auditing, and robustness certification.

Decision and control integrity

Harness-externalized defenses restructure or isolate untrusted context and constrain planning through task alignment, tool dependency graphs, and information-flow control.

Alignment-internalized defenses use instruction-hierarchy training, supervised fine-tuning, preference optimization, or reinforcement learning to internalize trust priorities and safe tool-use policies.

Planning-stage auditing

Complete plans are checked before external actions execute, with the goal of detecting dangerous objectives, privilege violations, and hazardous action compositions.

3. Execution

During execution, the agent commits tool calls and environment actions, observes their effects, and updates its plan or state. Security failures become externally consequential here, and can create artifacts that survive the current task.

3.1 Risks and Attacks

Web injection attacks

Malicious instructions are embedded in HTML, CSS, hidden text, images, PDFs, forms, or other page elements to hijack browsing agents.

Malicious injection in computer-use environments

Files, terminal output, notifications, pop-ups, screenshots, buttons, and mobile overlays manipulate perception, grounding, and operating-system actions.

Tool-output injection

Malicious instructions arrive through search results, API responses, database records, emails, command output, or MCP server responses and influence subsequent actions.

3.2 Evaluation Benchmarks

Execution-centered evaluation asks whether an agent performs an unsafe action, follows an attacker-controlled trajectory, or leaves the environment in a harmful final state. It also includes offline auditing of recorded trajectories and candidate actions.

Tool-invocation safety
Web-browsing behavior safety
Computer- and mobile-use behavior safety
Trajectory-level safety detection

Complete interaction records are provided to safety judges, which must detect gradually emerging risks, localize risky steps, and explain causal chains.

Step-level safety detection

Given the current context and a candidate action, the evaluator decides whether the action should be allowed, blocked, or verified further before execution.

3.3 Defense Methods

Step-level monitoring and enforcement

Before each tool call or environment action is committed, a monitor inspects the state, action semantics, and arguments, then blocks, corrects, confirms, or formally verifies the action.

Trajectory-level review

Partial or complete histories are analyzed to detect cross-step privilege escalation, cross-tool information leakage, and cumulative policy violations, supporting root-cause diagnosis and continual improvement.

4. Persistent Feedback and Research Outlook

Execution is not the end of the lifecycle. Agent actions may update long-term memory, rewrite reusable skills, or leave artifacts that become inputs to later workflows. The survey therefore frames future work as a closed safety loop:

Representative Work on Self-Evolving Agent Safety

The Introduction highlights three studies that directly examine how safety can fail when agents update their own models, memories, tools, workflows, or accumulated experience:

Together, these works show that self-evolution creates more than a larger static attack surface: it can make failures persistent, amplify them over time, and decouple capability improvement from reliable or safe behavior.

  1. Risk generation — automatically construct executable, verifiable, and continuously evolving adversarial environments and safety data.
  2. Risk diagnosis — attribute failures to the responsible stage, step, tool, parameter, or state transition along long-horizon trajectories.
  3. Safety evolution — convert newly discovered attacks and diagnosed failure trajectories into feedback for continual, stable, and generalizable policy improvement.

This loop connects adversarial environment generation, fine-grained trajectory attribution, and adversarial-feedback-driven continual safety evolution toward safer recursive self-improvement.

Repository Notes

  • The list follows Figure 2 and the benchmark/defense tables in the survey.
  • Years refer to the first public release of each work.
  • Links prioritize arXiv, OpenReview, ACL Anthology, official project pages, and official repositories.
  • The same paper may contribute an attack, benchmark, and defense, and may therefore appear more than once.
  • The two extracted figures remain copyright of their respective authors and are reproduced here to describe the accompanying survey.

Citation

If this repository supports your research, please cite the survey and the original papers you use:

Yutao Mou, Dingyao Yu, Xiaotian Luan, Zhe Yin, Zhangchi Xue, Peiyang Liu, Pengfei Yang, Tong Zhang, Shikun Zhang, and Wei Ye. Pandora's Toolbox: A Survey of Generalist Agent Security from the Lifecycle Perspective. 2026.

The formal BibTeX entry will be added when the public preprint record becomes available.

Contributing

Issues and pull requests for new papers, code repositories, datasets, or corrections are welcome. Please identify the primary lifecycle stage and security perspective, then use the following format:

- (Year) [Paper Title](Paper URL)

When a work spans multiple stages, place it under the stage where the relevant risk originates, the evaluation collects its principal evidence, or the defense primarily intervenes. Cross-listing is encouraged when it materially improves discoverability.

Provenance → Orchestration → Execution → Persistent Feedback

Contributors

MurrayTom

16 commits

LuanXiaotian598

2 commits

Similar repos

LLMSecurity/awesome-agent-skills-security

🛡️ A curated list of resources on agent skills security: attacks, defenses, frameworks, and…

102

ucsb-mlsec/Awesome-Agent-Security

67

OSU-NLP-Group/AgentSafety

198

8kSec/awesome-ai-security

A practitioner-focused reference for AI/ML security — attacks, tools, research, and defenses.…

4

Joe-B-Security/awesome-prompt-injection

Learn about a type of vulnerability that specifically targets machine learning models

639

YoujunZhao/Awesome-Openclaw-Papers

4

REAL-Lab-NU/Awesome-OpenClaw-Papers

Official companion repository for our survey "A Survey of the OpenClaw Ecosystem: From Platform…

20

h5i-dev/awesome-ai-agent-incidents

A curated corpus of incidents, attack vectors, failure modes, and defensive tools for autonomous AI…

52

MurrayTom/Awesome-Agent-Security-Papers

Beyond Environment and User Inputs: A Comprehensive Survey of Skill-Oriented Security in Agentic AI

10

18 commits

updated Aug 28, 2026

See the code

README

Pandora's Toolbox

A Survey of Generalist Agent Security from the Lifecycle Perspective

A curated literature map of risks, evaluations, and defenses across the full agent lifecycle

Lifecycle Topic Status

🌐 Language / 语言: English · 简体中文

Paper · Taxonomy · Contributing

Overview

Generalist agents are no longer bounded chat interfaces. They acquire reusable skills, retrieve long-term memory, select tools, interact with external environments, and preserve artifacts across sessions. Their security therefore cannot be understood as a collection of isolated model or component failures. A vulnerability may enter early, remain latent, be amplified during decision-making, materialize as a consequential action, and then contaminate persistent state.

This repository accompanies Pandora's Toolbox: A Survey of Generalist Agent Security from the Lifecycle Perspective. The survey organizes generalist-agent security through two orthogonal structures:

  • Lifecycle stages: provenance → orchestration → execution, followed by feedback through persistent state.
  • Security perspectives: risks and attacks, evaluation benchmarks, and defense methods at each stage.

The stage assignments below indicate where a risk is introduced, where safety evidence is collected, or where a defense primarily intervenes. They are not mutually exclusive: one attack or method may span multiple stages, and one paper may appear in more than one category.

The Lifecycle Perspective

Lifecycle perspective on generalist agent security: provenance, orchestration, execution, and backward contamination through persistent state.

Figure 1. Risks enter during provenance, propagate and amplify through orchestration, manifest during execution, and persist through contaminated states that affect future workflows.

Lifecycle stageSecurity questionPrimary risk pathTypical evidenceMain intervention point
ProvenanceWhat capabilities and artifacts are admitted?Poisoned metadata, skill files, dependencies, and forged trust signalsSkill packages, manifests, provenance recordsPre-installation inspection and admission control
OrchestrationHow are context, memory, skills, and tools composed into decisions?Malicious requests, retrieved memories, skill instructions, unsafe compositionsSelected skills, plans, candidate tool callsContext filtering, decision integrity, plan auditing
ExecutionWhat actions are committed, and how does the environment respond?Web, GUI, tool-output, API, file, and terminal injectionActions, trajectories, tool evidence, final environment stateStep-level enforcement and trajectory review
Persistent feedbackWhat survives and re-enters future workflows?Contaminated memory, rewritten skills, and compromised artifactsCross-session state and recurrenceProvenance tracking, recovery, and continual safety evolution
View the full taxonomy from the paper

Full taxonomy of security risks, evaluations, and defenses across the generalist agent lifecycle.

Figure 2. Taxonomy of security risks, evaluations, and defenses across the full lifecycle of generalist agents. Click the figure to open the high-resolution version.

Literature Map

1. Provenance

The provenance stage determines which capabilities enter an agent's usable capability space. Skill metadata may be loaded before a task begins, while specifications, scripts, resources, dependencies, and trust signals can remain dormant until a later request activates them.

1.1 Risks and Attacks

Declarative skill file poisoning

Attackers poison skill metadata, instructions, examples, or trigger rules so that malicious natural-language instructions enter the agent's planning context.

Multi-component skill ecosystem poisoning

The attack surface extends beyond the primary skill file to auxiliary scripts, tool interfaces, dependencies, external assets, configuration, retrieval resources, and local caches.

Cross-session skill rewriting and backdoor triggering

An attack modifies reusable skills, caches, project artifacts, or dependencies in one session, while malicious behavior activates only in a later session or under a specific trigger.

1.2 Evaluation Benchmarks

Malicious skill detection

These benchmarks evaluate whether a rule system, model, static analyzer, or verifier can identify semantic, code-level, and mixed attacks in a skill package before execution.

1.3 Defense Methods

Skill risk scanning

Before a skill is installed, integrated, or invoked, scanners inspect natural-language declarations, requested permissions, dependencies, executable resources, and cross-skill data flows.

2. Orchestration

During orchestration, the agent interprets the user request, retrieves memory, selects and composes skills, constructs a plan, and prepares tool calls. This stage is where latent risks become activated and amplified through context and control flow.

2.1 Risks and Attacks

Benign skill misuse and compositional risk

An individually benign skill may be exploitable through adversarial input. Multiple benign skills may also produce unauthorized, privacy-leaking, or destructive behavior when composed along a shared execution path.

User-side attacks

Malicious users submit harmful tasks, jailbreak prompts, obfuscated contexts, or multi-turn inducements through the direct interaction interface.

Manual jailbreak attacks
Heuristic optimization-based jailbreak attacks
Agentic red-teaming for jailbreak

Memory and long-term state risks

Long-term memory is retrieved and reused as trusted context in later tasks. A single poisoning event may therefore create stealthy, persistent, and transmissible manipulation.

Direct memory poisoning
Interaction-based memory injection and backdoors

2.2 Evaluation Benchmarks

Skill-mediated agent safety

Given a user task and one or more skills, the agent executes in an environment. Safety is judged from its response, execution evidence, or the resulting state.

2.3 Defense Methods

Prompt injection and jailbreak guardrails

Boundary detectors and safety classifiers inspect user inputs, retrieved content, web data, tool outputs, and model responses before these signals influence later decisions.

Memory guardrails

Memory defenses govern writing, retrieval, reasoning, and recovery through provenance verification, consistency checking, anomaly detection, causal auditing, and robustness certification.

Decision and control integrity

Harness-externalized defenses restructure or isolate untrusted context and constrain planning through task alignment, tool dependency graphs, and information-flow control.

Alignment-internalized defenses use instruction-hierarchy training, supervised fine-tuning, preference optimization, or reinforcement learning to internalize trust priorities and safe tool-use policies.

Planning-stage auditing

Complete plans are checked before external actions execute, with the goal of detecting dangerous objectives, privilege violations, and hazardous action compositions.

3. Execution

During execution, the agent commits tool calls and environment actions, observes their effects, and updates its plan or state. Security failures become externally consequential here, and can create artifacts that survive the current task.

3.1 Risks and Attacks

Web injection attacks

Malicious instructions are embedded in HTML, CSS, hidden text, images, PDFs, forms, or other page elements to hijack browsing agents.

Malicious injection in computer-use environments

Files, terminal output, notifications, pop-ups, screenshots, buttons, and mobile overlays manipulate perception, grounding, and operating-system actions.

Tool-output injection

Malicious instructions arrive through search results, API responses, database records, emails, command output, or MCP server responses and influence subsequent actions.

3.2 Evaluation Benchmarks

Execution-centered evaluation asks whether an agent performs an unsafe action, follows an attacker-controlled trajectory, or leaves the environment in a harmful final state. It also includes offline auditing of recorded trajectories and candidate actions.

Tool-invocation safety
Web-browsing behavior safety
Computer- and mobile-use behavior safety
Trajectory-level safety detection

Complete interaction records are provided to safety judges, which must detect gradually emerging risks, localize risky steps, and explain causal chains.

Step-level safety detection

Given the current context and a candidate action, the evaluator decides whether the action should be allowed, blocked, or verified further before execution.

3.3 Defense Methods

Step-level monitoring and enforcement

Before each tool call or environment action is committed, a monitor inspects the state, action semantics, and arguments, then blocks, corrects, confirms, or formally verifies the action.

Trajectory-level review

Partial or complete histories are analyzed to detect cross-step privilege escalation, cross-tool information leakage, and cumulative policy violations, supporting root-cause diagnosis and continual improvement.

4. Persistent Feedback and Research Outlook

Execution is not the end of the lifecycle. Agent actions may update long-term memory, rewrite reusable skills, or leave artifacts that become inputs to later workflows. The survey therefore frames future work as a closed safety loop:

Representative Work on Self-Evolving Agent Safety

The Introduction highlights three studies that directly examine how safety can fail when agents update their own models, memories, tools, workflows, or accumulated experience:

Together, these works show that self-evolution creates more than a larger static attack surface: it can make failures persistent, amplify them over time, and decouple capability improvement from reliable or safe behavior.

  1. Risk generation — automatically construct executable, verifiable, and continuously evolving adversarial environments and safety data.
  2. Risk diagnosis — attribute failures to the responsible stage, step, tool, parameter, or state transition along long-horizon trajectories.
  3. Safety evolution — convert newly discovered attacks and diagnosed failure trajectories into feedback for continual, stable, and generalizable policy improvement.

This loop connects adversarial environment generation, fine-grained trajectory attribution, and adversarial-feedback-driven continual safety evolution toward safer recursive self-improvement.

Repository Notes

  • The list follows Figure 2 and the benchmark/defense tables in the survey.
  • Years refer to the first public release of each work.
  • Links prioritize arXiv, OpenReview, ACL Anthology, official project pages, and official repositories.
  • The same paper may contribute an attack, benchmark, and defense, and may therefore appear more than once.
  • The two extracted figures remain copyright of their respective authors and are reproduced here to describe the accompanying survey.

Citation

If this repository supports your research, please cite the survey and the original papers you use:

Yutao Mou, Dingyao Yu, Xiaotian Luan, Zhe Yin, Zhangchi Xue, Peiyang Liu, Pengfei Yang, Tong Zhang, Shikun Zhang, and Wei Ye. Pandora's Toolbox: A Survey of Generalist Agent Security from the Lifecycle Perspective. 2026.

The formal BibTeX entry will be added when the public preprint record becomes available.

Contributing

Issues and pull requests for new papers, code repositories, datasets, or corrections are welcome. Please identify the primary lifecycle stage and security perspective, then use the following format:

- (Year) [Paper Title](Paper URL)

When a work spans multiple stages, place it under the stage where the relevant risk originates, the evaluation collects its principal evidence, or the defense primarily intervenes. Cross-listing is encouraged when it materially improves discoverability.

Provenance → Orchestration → Execution → Persistent Feedback

Similar repos

LLMSecurity/awesome-agent-skills-security

🛡️ A curated list of resources on agent skills security: attacks, defenses, frameworks, and…

102

ucsb-mlsec/Awesome-Agent-Security

67

OSU-NLP-Group/AgentSafety

198

8kSec/awesome-ai-security

A practitioner-focused reference for AI/ML security — attacks, tools, research, and defenses.…

4

Joe-B-Security/awesome-prompt-injection

Learn about a type of vulnerability that specifically targets machine learning models

639

YoujunZhao/Awesome-Openclaw-Papers

4

REAL-Lab-NU/Awesome-OpenClaw-Papers

Official companion repository for our survey "A Survey of the OpenClaw Ecosystem: From Platform…

20

h5i-dev/awesome-ai-agent-incidents

A curated corpus of incidents, attack vectors, failure modes, and defensive tools for autonomous AI…

52

Contributors

MurrayTom

16 commits

LuanXiaotian598

2 commits