This repository provides a categorized benchmark for backdoor attacks targeting LLM-based agents. With the rapid adoption of LLM agents in applications like finance, healthcare, and web automation, understanding their unique security vulnerabilities is crucial for safe deployment.
We classify current backdoor attacks into four major categories, based on the attack surface, trigger mechanism, and agent behavior manipulation:
| 🧩 Category | 🔍 Attack Surface | 🎯 Trigger Type | 📚 Representative Works | 🧠 Key Characteristics | 🛡 Defense Challenges |
|---|---|---|---|---|---|
| 1. Knowledge Injection | External KB / Retriever | Passive (Query-based) | AgentPoison (NeurIPS 2024), PoisonedRAG (USS 2025), TrojanRAG (2024) | Stealthy, model-agnostic | Hard to detect, needs content/IR validation |
| 2. Model-level Implantation | LLM Parameters | Passive (Hidden triggers) | BadAgent (ACL 2024), WatchOut (NeurIPS 2024) | Robust and precisely triggered | Watermarking / fine-tuning defenses often fail |
| 3. Reasoning Manipulation | Prompt / CoT Context | In-context trigger | BadChain (ICLR 2024), DemonAgent (2025) | Lightweight, stealthy | Requires trace-level consistency checking |
| 4. Active Adversarial Agents | Full agent behavior / tools | Active (Query / Env / Obs) | PI AdvAgent (ICML 2025) | Multi-modal tool-level attacks | Costly to simulate and hard to generalize defenses |
This benchmark aims to:
✅ Systematically categorize agent backdoor threats
✅ Collect representative attacks and implementations
✅ Define standard evaluation metrics (ASR, stealth, transferability)
✅ Facilitate development of effective, agent-specific defenses
| Paper | Code / Link |
|---|---|
| AgentPoison (NeurIPS 2024) | GitHub |
| PoisonedRAG | GitHub |
| TrojanRAG | GitHub |
| BadAgent (ACL 2024) | GitHub |
| BadChain (ICLR 2024) | arXiv |
| AdvAgent (ICML 2025) | Website |
| DemonAgent | GitHub |
| Defense | Description | Highlights |
|---|---|---|
| ShieldAgent (2025) | Uses verifiable reasoning-based safety policies to monitor and block harmful agent actions. | - Model-agnostic logic-based verification - Captures policy violations via trace audit |
| Prompt Filtering | Heuristic-based rejection of suspicious inputs or tools. | Weak against stealthy / CoT-based attacks |
| Safety-Aware Memory | Blacklisting memory traces post detection. | Not effective against dynamically encrypted payloads (e.g., DemonAgent) |
⚠️ Observation: Most existing defenses are insufficient against dynamic, multi-step, or reasoning-injected backdoors.
🧩 These two architectures define key attack surfaces for backdoor triggering and planning logic manipulation.
| Metric | Meaning |
|---|---|
| ASR (Attack Success Rate) | Percentage of queries where the backdoor is triggered and achieves malicious goal |
| Stealth | Attack visibility to human or detector |
| Robustness | Success rate under input variations or defense presence |
| Transferability | Effectiveness across models / agent tasks |
| Trigger Compactness | How small and hidden the trigger can be |
Please cite individual works as needed. A collective survey paper and BibTeX export will be released soon.
We welcome contributions on:
Please open an issue or PR to get involved.
This repository provides a categorized benchmark for backdoor attacks targeting LLM-based agents. With the rapid adoption of LLM agents in applications like finance, healthcare, and web automation, understanding their unique security vulnerabilities is crucial for safe deployment.
We classify current backdoor attacks into four major categories, based on the attack surface, trigger mechanism, and agent behavior manipulation:
| 🧩 Category | 🔍 Attack Surface | 🎯 Trigger Type | 📚 Representative Works | 🧠 Key Characteristics | 🛡 Defense Challenges |
|---|---|---|---|---|---|
| 1. Knowledge Injection | External KB / Retriever | Passive (Query-based) | AgentPoison (NeurIPS 2024), PoisonedRAG (USS 2025), TrojanRAG (2024) | Stealthy, model-agnostic | Hard to detect, needs content/IR validation |
| 2. Model-level Implantation | LLM Parameters | Passive (Hidden triggers) | BadAgent (ACL 2024), WatchOut (NeurIPS 2024) | Robust and precisely triggered | Watermarking / fine-tuning defenses often fail |
| 3. Reasoning Manipulation | Prompt / CoT Context | In-context trigger | BadChain (ICLR 2024), DemonAgent (2025) | Lightweight, stealthy | Requires trace-level consistency checking |
| 4. Active Adversarial Agents | Full agent behavior / tools | Active (Query / Env / Obs) | PI AdvAgent (ICML 2025) | Multi-modal tool-level attacks | Costly to simulate and hard to generalize defenses |
This benchmark aims to:
✅ Systematically categorize agent backdoor threats
✅ Collect representative attacks and implementations
✅ Define standard evaluation metrics (ASR, stealth, transferability)
✅ Facilitate development of effective, agent-specific defenses
| Paper | Code / Link |
|---|---|
| AgentPoison (NeurIPS 2024) | GitHub |
| PoisonedRAG | GitHub |
| TrojanRAG | GitHub |
| BadAgent (ACL 2024) | GitHub |
| BadChain (ICLR 2024) | arXiv |
| AdvAgent (ICML 2025) | Website |
| DemonAgent | GitHub |
| Defense | Description | Highlights |
|---|---|---|
| ShieldAgent (2025) | Uses verifiable reasoning-based safety policies to monitor and block harmful agent actions. | - Model-agnostic logic-based verification - Captures policy violations via trace audit |
| Prompt Filtering | Heuristic-based rejection of suspicious inputs or tools. | Weak against stealthy / CoT-based attacks |
| Safety-Aware Memory | Blacklisting memory traces post detection. | Not effective against dynamically encrypted payloads (e.g., DemonAgent) |
⚠️ Observation: Most existing defenses are insufficient against dynamic, multi-step, or reasoning-injected backdoors.
🧩 These two architectures define key attack surfaces for backdoor triggering and planning logic manipulation.
| Metric | Meaning |
|---|---|
| ASR (Attack Success Rate) | Percentage of queries where the backdoor is triggered and achieves malicious goal |
| Stealth | Attack visibility to human or detector |
| Robustness | Success rate under input variations or defense presence |
| Transferability | Effectiveness across models / agent tasks |
| Trigger Compactness | How small and hidden the trigger can be |
Please cite individual works as needed. A collective survey paper and BibTeX export will be released soon.
We welcome contributions on:
Please open an issue or PR to get involved.
