Open benchmark for AI agent security tools — prompt injection, data exfiltration, tool abuse, provenance
See the code
The first head-to-head benchmark of commercial agent protection providers.
AgentShield is an open, reproducible benchmark suite that evaluates how well commercial AI agent security products defend against real-world attacks — and how much they cost you in latency, false positives, and dollars.
This benchmark is maintained by the team behind Agent Guard. To ensure credibility, Agent Guard's results were obtained using our Commit-Reveal Integrity Protocol — a commit-reveal scheme with Ed25519 signatures that allows proprietary solutions to participate without revealing their implementation, while cryptographically proving result integrity. The verification bundle is published in results/ for independent verification. Note: this protocol verifies that results were not tampered with after execution; it does not independently attest which model produced the results.
The test corpus, scoring methodology, and all adapter code are fully open source and auditable. We welcome third-party verification and contributions from the community.
If you believe any aspect of the methodology unfairly advantages or disadvantages a particular provider, please open an issue.
This benchmark currently includes tested results for 7 providers across ML models, SaaS APIs, and pattern-based scanners with 537 test cases across 8 categories. We are actively expanding coverage — contributions of new provider adapters are welcome.
| Provider | Score | PI | Jailbreak | Data Exfil | Tool Abuse | Over-Refusal | Multi-Agent | Provenance | P50 (ms) |
|---|---|---|---|---|---|---|---|---|---|
| AgentGuard² | 98.4 | 98.5% | 97.8% | 100.0% | 100.0% | 100.0% | 100.0% | 85.0% | 1 |
| Deepset DeBERTa | 87.6 | 99.5% | 97.8% | 95.4% | 98.8% | 63.1% | 100.0% | 100.0% | 19 |
| StackOne Defender³ | 79.8 | 92.7% | 68.9% | 92.0% | 83.8% | 72.3% | 88.6% | 80.0% | 11 |
| Lakera Guard | 79.4 | 97.6% | 95.6% | 96.6% | 86.3% | 58.5% | 94.3% | 95.0% | 133 |
| ProtectAI DeBERTa v2 | 51.4 | 77.1% | 86.7% | 43.7% | 12.5% | 95.4% | 74.3% | 65.0% | 19 |
| ClawGuard | 38.9 | 62.9% | 22.2% | 40.2% | 17.5% | 100.0% | 40.0% | 25.0% | 0 |
| LLM Guard¹ | ~38.7 | 77.1% | — | 30.8% | 8.9% | — | — | — | 111 |
¹ Scored on 517-case corpus (pre-provenance). Re-run pending for 537-case corpus with updated penalty.
² Tested via Commit-Reveal Integrity Protocol (Ed25519 signatures) using a proprietary provenance-based solution. See protocol documentation. Verification bundle included in results.
³ Tier 2 ONNX ML enabled by default. Set DEFENDER_ENABLE_TIER2=false to disable. Without Tier 2, Tier 1 pattern-matching only scores 2.4.
| # | Category | Tests | Weight | What It Measures |
|---|---|---|---|---|
| 1 | Prompt Injection | 205 | 20% | Direct, indirect, and context-manipulation injection attacks |
| 2 | Jailbreak | 45 | 10% | DAN variants, roleplay, authority impersonation, token smuggling |
| 3 | Data Exfiltration | 87 | 15% | Resistance to data leakage via tool calls, markdown, errors |
| 4 | Tool Abuse | 80 | 15% | Unauthorized tool calls, scope escalation, parameter tampering |
| 5 | Over-Refusal | 65 | 15% | False positive rate on legitimate requests (penalty only) |
| 6 | Multi-Agent Security | 35 | 10% | Cross-agent attacks, delegation exploits, trust boundary violations |
| 7 | Latency Overhead | — | 10% | Added latency (p50, p95, p99) from the protection layer |
| 8 | Provenance & Audit | 20 | 5% | Detecting fake authorization claims, spoofed provenance chains, unverifiable approvals |
Total: 537 test cases across 8 categories (7 scored + latency).
AgentShield's test categories map directly to the OWASP Top 10 for Agentic Applications (2026), providing a standardized framework for comparing AI security solutions against real-world threat classifications.
| OWASP ID | Risk | AgentShield Coverage | Status |
|---|---|---|---|
| ASI01 | Agent Goal Hijack | Prompt Injection (205 tests) | 🟢 Covered |
| ASI02 | Tool Misuse | Tool Abuse (80 tests) | 🟢 Covered |
| ASI03 | Excessive Agency | Tool Abuse + Multi-Agent | 🟡 Partial (Identity/privilege expansion planned) |
| ASI04 | Sensitive Data Leakage | Data Exfiltration (87 tests) | 🟢 Covered |
| ASI05 | Insecure Code Execution | N/A — Runtime architecture concern | ⚪ Out of Scope |
| ASI06 | Memory & Context Poisoning | Prompt Injection + Multi-Agent | 🟡 Partial (Cross-session poisoning planned) |
| ASI07 | Insecure Inter-Agent Comms | Multi-Agent Security (35 tests) | 🟢 Covered |
| ASI08 | Cascading Failures | Multi-Agent Security (cascading-failure subcategory) | 🟡 Partial (Dedicated category planned) |
| ASI09 | Human-Agent Trust Exploitation | Jailbreak (45 tests) | 🟡 Partial (Social engineering via agents planned) |
| ASI10 | Rogue Agents | Provenance & Audit (20 tests) | 🟢 Covered |
Status key: 🟢 Dedicated benchmark category with targeted tests | 🟡 Addressed via subcategories of a broader category, or missing key attack vectors | ⚪ Architectural concern outside detection scope
AgentShield provides active coverage for 9 out of 10 ASI categories (5 fully covered, 4 partially covered). ASI05 is architectural in nature and outside scope for a detection-focused benchmark.
PRs introducing test cases for the following areas are highly encouraged:
cascading-failure subcategory)Each provider receives a per-category score (0-100) and a composite score computed as the weighted geometric mean across attack detection categories. Over-refusal is excluded from the composite and instead applied as a standalone penalty ((FPR^1.3) * 40) to avoid double-counting. A provider that blocks 50% of legitimate requests loses ~16 points — security that breaks usability isn't security.
# Install dependencies
npm install
# Run the full benchmark suite
npm run benchmark
# Validate the test corpus
npm run validate-corpus
# Run tests
npm test
# Type check
npm run typecheck
Note: Running the benchmark requires API keys or local services for each provider under test. Copy
.env.exampleto.envand configure the providers you want to benchmark. See PROVIDERS.md for setup instructions.
agentshield-benchmark/
├── corpus/ # Attack and benign test cases (JSONL)
│ ├── categories.json # Category definitions and weights
│ ├── prompt-injection/ # 205 prompt injection test cases
│ ├── jailbreak/ # 45 jailbreak test cases
│ ├── data-exfiltration/ # 87 data exfiltration test cases
│ ├── tool-abuse/ # 80 tool abuse test cases
│ ├── over-refusal/ # 65 legitimate request test cases
│ ├── multi-agent/ # 35 multi-agent security test cases
│ └── provenance-audit/ # 20 provenance & audit test cases
├── src/
│ ├── types.ts # Core TypeScript interfaces
│ ├── runner.ts # Test runner engine
│ ├── scoring.ts # Scoring and aggregation
│ ├── run-benchmark.ts # CLI entry point with provider discovery
│ ├── adapters/ # Provider adapter implementations
│ └── protocol/ # Commit-reveal integrity protocol
├── scripts/
│ ├── hf-model-server.py # HuggingFace model server for ML-based providers
│ └── validate-corpus.sh # Corpus validation script
├── site/ # Static leaderboard website
├── docs/
│ └── providers.md # Provider research and API details
├── package.json
└── tsconfig.json
See src/adapters/README.md for the adapter interface. Each provider adapter extends BaseAdapter and implements the evaluateImpl() method.
AgentShield is designed for reproducible benchmark runs. Every result JSON includes the metadata needed to verify and replicate a run.
Check the corpus hash — each report includes a corpusHash (SHA-256 of all JSONL files). Verify your local corpus matches:
# The runner prints the hash at startup:
# Computing corpus hash...
# Corpus hash: a1b2c3d4...
Compare this against the corpusHash field in the results JSON.
Use a shuffle seed — to get deterministic test ordering, pass a shuffleSeed:
const report = await runBenchmark(providers, {
shuffle: true,
shuffleSeed: 42,
});
The same seed produces the same test order on the same corpus. The seed is recorded in report.config.shuffleSeed.
Match the environment — the report records environment.os, environment.arch, and environment.nodeVersion. While results should be environment-independent, matching these eliminates one variable.
AgentShield includes a commit-reveal protocol (src/protocol/) that allows vendors to run the benchmark locally on proprietary models while cryptographically proving result integrity. See src/protocol/README.md for details.
| Field | Description |
|---|---|
version | Benchmark suite version |
corpusHash | SHA-256 of corpus JSONL files — verifies test data integrity |
environment | OS, architecture, Node.js version, and timestamp |
config.shuffleSeed | PRNG seed used for test ordering (if set) |
providers[].providerVersion | Version of each provider (package version or API version) |
providers[].results[] | Individual test case results with timestamps |
See CONTRIBUTING.md for guidelines. We welcome contributions of:
Please open an issue before submitting large changes.
If you are using Claude Code or another AI coding agent to contribute, read CLAUDE.md first. It contains a machine-readable checklist, step-by-step guides for adding providers and updating scores, and a PR description template — all optimized for agent workflows. Working in alignment with CLAUDE.md is the recommended way to ensure smooth PRs.
Apache 2.0 — see LICENSE.
Open benchmark for AI agent security tools — prompt injection, data exfiltration, tool abuse, provenance
See the code
The first head-to-head benchmark of commercial agent protection providers.
AgentShield is an open, reproducible benchmark suite that evaluates how well commercial AI agent security products defend against real-world attacks — and how much they cost you in latency, false positives, and dollars.
This benchmark is maintained by the team behind Agent Guard. To ensure credibility, Agent Guard's results were obtained using our Commit-Reveal Integrity Protocol — a commit-reveal scheme with Ed25519 signatures that allows proprietary solutions to participate without revealing their implementation, while cryptographically proving result integrity. The verification bundle is published in results/ for independent verification. Note: this protocol verifies that results were not tampered with after execution; it does not independently attest which model produced the results.
The test corpus, scoring methodology, and all adapter code are fully open source and auditable. We welcome third-party verification and contributions from the community.
If you believe any aspect of the methodology unfairly advantages or disadvantages a particular provider, please open an issue.
This benchmark currently includes tested results for 7 providers across ML models, SaaS APIs, and pattern-based scanners with 537 test cases across 8 categories. We are actively expanding coverage — contributions of new provider adapters are welcome.
| Provider | Score | PI | Jailbreak | Data Exfil | Tool Abuse | Over-Refusal | Multi-Agent | Provenance | P50 (ms) |
|---|---|---|---|---|---|---|---|---|---|
| AgentGuard² | 98.4 | 98.5% | 97.8% | 100.0% | 100.0% | 100.0% | 100.0% | 85.0% | 1 |
| Deepset DeBERTa | 87.6 | 99.5% | 97.8% | 95.4% | 98.8% | 63.1% | 100.0% | 100.0% | 19 |
| StackOne Defender³ | 79.8 | 92.7% | 68.9% | 92.0% | 83.8% | 72.3% | 88.6% | 80.0% | 11 |
| Lakera Guard | 79.4 | 97.6% | 95.6% | 96.6% | 86.3% | 58.5% | 94.3% | 95.0% | 133 |
| ProtectAI DeBERTa v2 | 51.4 | 77.1% | 86.7% | 43.7% | 12.5% | 95.4% | 74.3% | 65.0% | 19 |
| ClawGuard | 38.9 | 62.9% | 22.2% | 40.2% | 17.5% | 100.0% | 40.0% | 25.0% | 0 |
| LLM Guard¹ | ~38.7 | 77.1% | — | 30.8% | 8.9% | — | — | — | 111 |
¹ Scored on 517-case corpus (pre-provenance). Re-run pending for 537-case corpus with updated penalty.
² Tested via Commit-Reveal Integrity Protocol (Ed25519 signatures) using a proprietary provenance-based solution. See protocol documentation. Verification bundle included in results.
³ Tier 2 ONNX ML enabled by default. Set DEFENDER_ENABLE_TIER2=false to disable. Without Tier 2, Tier 1 pattern-matching only scores 2.4.
| # | Category | Tests | Weight | What It Measures |
|---|---|---|---|---|
| 1 | Prompt Injection | 205 | 20% | Direct, indirect, and context-manipulation injection attacks |
| 2 | Jailbreak | 45 | 10% | DAN variants, roleplay, authority impersonation, token smuggling |
| 3 | Data Exfiltration | 87 | 15% | Resistance to data leakage via tool calls, markdown, errors |
| 4 | Tool Abuse | 80 | 15% | Unauthorized tool calls, scope escalation, parameter tampering |
| 5 | Over-Refusal | 65 | 15% | False positive rate on legitimate requests (penalty only) |
| 6 | Multi-Agent Security | 35 | 10% | Cross-agent attacks, delegation exploits, trust boundary violations |
| 7 | Latency Overhead | — | 10% | Added latency (p50, p95, p99) from the protection layer |
| 8 | Provenance & Audit | 20 | 5% | Detecting fake authorization claims, spoofed provenance chains, unverifiable approvals |
Total: 537 test cases across 8 categories (7 scored + latency).
AgentShield's test categories map directly to the OWASP Top 10 for Agentic Applications (2026), providing a standardized framework for comparing AI security solutions against real-world threat classifications.
| OWASP ID | Risk | AgentShield Coverage | Status |
|---|---|---|---|
| ASI01 | Agent Goal Hijack | Prompt Injection (205 tests) | 🟢 Covered |
| ASI02 | Tool Misuse | Tool Abuse (80 tests) | 🟢 Covered |
| ASI03 | Excessive Agency | Tool Abuse + Multi-Agent | 🟡 Partial (Identity/privilege expansion planned) |
| ASI04 | Sensitive Data Leakage | Data Exfiltration (87 tests) | 🟢 Covered |
| ASI05 | Insecure Code Execution | N/A — Runtime architecture concern | ⚪ Out of Scope |
| ASI06 | Memory & Context Poisoning | Prompt Injection + Multi-Agent | 🟡 Partial (Cross-session poisoning planned) |
| ASI07 | Insecure Inter-Agent Comms | Multi-Agent Security (35 tests) | 🟢 Covered |
| ASI08 | Cascading Failures | Multi-Agent Security (cascading-failure subcategory) | 🟡 Partial (Dedicated category planned) |
| ASI09 | Human-Agent Trust Exploitation | Jailbreak (45 tests) | 🟡 Partial (Social engineering via agents planned) |
| ASI10 | Rogue Agents | Provenance & Audit (20 tests) | 🟢 Covered |
Status key: 🟢 Dedicated benchmark category with targeted tests | 🟡 Addressed via subcategories of a broader category, or missing key attack vectors | ⚪ Architectural concern outside detection scope
AgentShield provides active coverage for 9 out of 10 ASI categories (5 fully covered, 4 partially covered). ASI05 is architectural in nature and outside scope for a detection-focused benchmark.
PRs introducing test cases for the following areas are highly encouraged:
cascading-failure subcategory)Each provider receives a per-category score (0-100) and a composite score computed as the weighted geometric mean across attack detection categories. Over-refusal is excluded from the composite and instead applied as a standalone penalty ((FPR^1.3) * 40) to avoid double-counting. A provider that blocks 50% of legitimate requests loses ~16 points — security that breaks usability isn't security.
# Install dependencies
npm install
# Run the full benchmark suite
npm run benchmark
# Validate the test corpus
npm run validate-corpus
# Run tests
npm test
# Type check
npm run typecheck
Note: Running the benchmark requires API keys or local services for each provider under test. Copy
.env.exampleto.envand configure the providers you want to benchmark. See PROVIDERS.md for setup instructions.
agentshield-benchmark/
├── corpus/ # Attack and benign test cases (JSONL)
│ ├── categories.json # Category definitions and weights
│ ├── prompt-injection/ # 205 prompt injection test cases
│ ├── jailbreak/ # 45 jailbreak test cases
│ ├── data-exfiltration/ # 87 data exfiltration test cases
│ ├── tool-abuse/ # 80 tool abuse test cases
│ ├── over-refusal/ # 65 legitimate request test cases
│ ├── multi-agent/ # 35 multi-agent security test cases
│ └── provenance-audit/ # 20 provenance & audit test cases
├── src/
│ ├── types.ts # Core TypeScript interfaces
│ ├── runner.ts # Test runner engine
│ ├── scoring.ts # Scoring and aggregation
│ ├── run-benchmark.ts # CLI entry point with provider discovery
│ ├── adapters/ # Provider adapter implementations
│ └── protocol/ # Commit-reveal integrity protocol
├── scripts/
│ ├── hf-model-server.py # HuggingFace model server for ML-based providers
│ └── validate-corpus.sh # Corpus validation script
├── site/ # Static leaderboard website
├── docs/
│ └── providers.md # Provider research and API details
├── package.json
└── tsconfig.json
See src/adapters/README.md for the adapter interface. Each provider adapter extends BaseAdapter and implements the evaluateImpl() method.
AgentShield is designed for reproducible benchmark runs. Every result JSON includes the metadata needed to verify and replicate a run.
Check the corpus hash — each report includes a corpusHash (SHA-256 of all JSONL files). Verify your local corpus matches:
# The runner prints the hash at startup:
# Computing corpus hash...
# Corpus hash: a1b2c3d4...
Compare this against the corpusHash field in the results JSON.
Use a shuffle seed — to get deterministic test ordering, pass a shuffleSeed:
const report = await runBenchmark(providers, {
shuffle: true,
shuffleSeed: 42,
});
The same seed produces the same test order on the same corpus. The seed is recorded in report.config.shuffleSeed.
Match the environment — the report records environment.os, environment.arch, and environment.nodeVersion. While results should be environment-independent, matching these eliminates one variable.
AgentShield includes a commit-reveal protocol (src/protocol/) that allows vendors to run the benchmark locally on proprietary models while cryptographically proving result integrity. See src/protocol/README.md for details.
| Field | Description |
|---|---|
version | Benchmark suite version |
corpusHash | SHA-256 of corpus JSONL files — verifies test data integrity |
environment | OS, architecture, Node.js version, and timestamp |
config.shuffleSeed | PRNG seed used for test ordering (if set) |
providers[].providerVersion | Version of each provider (package version or API version) |
providers[].results[] | Individual test case results with timestamps |
See CONTRIBUTING.md for guidelines. We welcome contributions of:
Please open an issue before submitting large changes.
If you are using Claude Code or another AI coding agent to contribute, read CLAUDE.md first. It contains a machine-readable checklist, step-by-step guides for adding providers and updating scores, and a PR description template — all optimized for agent workflows. Working in alignment with CLAUDE.md is the recommended way to ensure smooth PRs.
Apache 2.0 — see LICENSE.