This is the evaluation suite behind the Prompt Injection Guard model card, packaged so anyone can reproduce the comparison or evaluate their own detector. It covers over-defense (false alarms on benign text that looks suspicious), direct injection and jailbreaks, and indirect injection planted in documents, emails and RAG contexts.
Every row has text, label (1 = injection or jailbreak, 0 = benign), source, kind (direct / indirect) and lang.
Under this suite's definition, plain harmful requests with no attempt to override instructions are benign. That is
why XSTest's "unsafe" prompts are labelled 0.
| config | rows | positives | what it tests | source | license |
|---|---|---|---|---|---|
notinject | 339 | 0 | over-defense: benign prompts with trigger words (all label 0) | leolee99/NotInject | MIT |
xstest | 450 | 0 | safe and unsafe-but-not-injection prompts (all label 0 for injection detection) | Paul/XSTest | CC-BY-4.0 |
orbench_hard | 1319 | 0 | seemingly toxic but benign prompts (all label 0) | bench-llm/or-bench (or-bench-hard-1k) | CC-BY-4.0 |
deepset_test | 116 | 60 | direct injections, en/de | deepset/prompt-injections (test) | Apache-2.0 |
jackhhao_test | 262 | 139 | jailbreak vs benign prompts | jackhhao/jailbreak-classification (test) | Apache-2.0 |
simsonsun_jailbreaks | 2426 | 2426 | contamination-filtered jailbreaks (all label 1) | Simsonsun/JailbreakPrompts | MIT |
boundary_pairs_test | 240 | 120 | agentic minimal pairs | 3nesdeniz/agentic-prompt-injection-boundary-pairs (test) | CC-BY-4.0 |
agentic5k_test | 545 | 280 | agentic injections and hard negatives | 3nesdeniz/agentic-prompt-injection-5k (test) | CC-BY-4.0 |
neuralchemy_test | 942 | 552 | direct injections and hard negatives | neuralchemy/Prompt-injection-dataset (core test) | Apache-2.0 |
slabs_test | 2101 | 1051 | direct injections and hard negatives | S-Labs/prompt-injection-dataset (test) | MIT |
bipia | 3178 | 3000 | indirect injection in email/table/code contexts | geodesic-research/bipia (derived: attacked contexts + the same contexts with the attack removed) | CC-BY-SA-4.0 |
piarena | 1500 | 750 | indirect injection in RAG/QA contexts | sleeepeer/PIArena (derived: injected task inserted at a sentence boundary vs clean context) | MIT |
llmail_phase2 | 2000 | 2000 | real adaptive email injections (all label 1) | microsoft/llmail-inject-challenge (Phase 2, deduplicated sample) | MIT |
synthetic_docs_test | 3354 | 1113 | documents in 45 types / 30 languages: clean, injected, hard-benign | Horizon-Labs (generated with Qwen/Qwen3.8-27B) | Apache-2.0 |
synthetic_direct_test | 3162 | 1585 | direct attacks and look-alike benign messages, 30 languages | Horizon-Labs (generated with Qwen/Qwen3.8-27B) | Apache-2.0 |
synthetic_v2_test | 2839 | 1412 | role-play framing pairs and planted-task documents | Horizon-Labs (generated with Qwen/Qwen3.8-27B) | Apache-2.0 |
Notes:
bipia and piarena are derived: we built positive/negative pairs from the upstream contexts. See
code/data/build_v0.py in the model repo.synthetic_* configs were generated by us and are in-distribution for our own models. Treat them as
supplementary.qualifire/prompt-injections-benchmark and Mindgard/evaded-prompt-injection-and-jailbreak-samples.Results for 9 open detectors: Prompt Injection Detector Leaderboard.
from datasets import load_dataset
from transformers import pipeline
clf = pipeline("text-classification", model="Horizon-Labs/prompt-injection-guard-base", truncation=True)
ds = load_dataset("Horizon-Labs/prompt-injection-eval-suite", "notinject", split="test")
preds = [int(p["label"] == "injection") for p in clf(ds["text"], batch_size=32)]
print("accuracy", sum(int(p == y) for p, y in zip(preds, ds["label"])) / len(preds))
This is the evaluation suite behind the Prompt Injection Guard model card, packaged so anyone can reproduce the comparison or evaluate their own detector. It covers over-defense (false alarms on benign text that looks suspicious), direct injection and jailbreaks, and indirect injection planted in documents, emails and RAG contexts.
Every row has text, label (1 = injection or jailbreak, 0 = benign), source, kind (direct / indirect) and lang.
Under this suite's definition, plain harmful requests with no attempt to override instructions are benign. That is
why XSTest's "unsafe" prompts are labelled 0.
| config | rows | positives | what it tests | source | license |
|---|---|---|---|---|---|
notinject | 339 | 0 | over-defense: benign prompts with trigger words (all label 0) | leolee99/NotInject | MIT |
xstest | 450 | 0 | safe and unsafe-but-not-injection prompts (all label 0 for injection detection) | Paul/XSTest | CC-BY-4.0 |
orbench_hard | 1319 | 0 | seemingly toxic but benign prompts (all label 0) | bench-llm/or-bench (or-bench-hard-1k) | CC-BY-4.0 |
deepset_test | 116 | 60 | direct injections, en/de | deepset/prompt-injections (test) | Apache-2.0 |
jackhhao_test | 262 | 139 | jailbreak vs benign prompts | jackhhao/jailbreak-classification (test) | Apache-2.0 |
simsonsun_jailbreaks | 2426 | 2426 | contamination-filtered jailbreaks (all label 1) | Simsonsun/JailbreakPrompts | MIT |
boundary_pairs_test | 240 | 120 | agentic minimal pairs | 3nesdeniz/agentic-prompt-injection-boundary-pairs (test) | CC-BY-4.0 |
agentic5k_test | 545 | 280 | agentic injections and hard negatives | 3nesdeniz/agentic-prompt-injection-5k (test) | CC-BY-4.0 |
neuralchemy_test | 942 | 552 | direct injections and hard negatives | neuralchemy/Prompt-injection-dataset (core test) | Apache-2.0 |
slabs_test | 2101 | 1051 | direct injections and hard negatives | S-Labs/prompt-injection-dataset (test) | MIT |
bipia | 3178 | 3000 | indirect injection in email/table/code contexts | geodesic-research/bipia (derived: attacked contexts + the same contexts with the attack removed) | CC-BY-SA-4.0 |
piarena | 1500 | 750 | indirect injection in RAG/QA contexts | sleeepeer/PIArena (derived: injected task inserted at a sentence boundary vs clean context) | MIT |
llmail_phase2 | 2000 | 2000 | real adaptive email injections (all label 1) | microsoft/llmail-inject-challenge (Phase 2, deduplicated sample) | MIT |
synthetic_docs_test | 3354 | 1113 | documents in 45 types / 30 languages: clean, injected, hard-benign | Horizon-Labs (generated with Qwen/Qwen3.8-27B) | Apache-2.0 |
synthetic_direct_test | 3162 | 1585 | direct attacks and look-alike benign messages, 30 languages | Horizon-Labs (generated with Qwen/Qwen3.8-27B) | Apache-2.0 |
synthetic_v2_test | 2839 | 1412 | role-play framing pairs and planted-task documents | Horizon-Labs (generated with Qwen/Qwen3.8-27B) | Apache-2.0 |
Notes:
bipia and piarena are derived: we built positive/negative pairs from the upstream contexts. See
code/data/build_v0.py in the model repo.synthetic_* configs were generated by us and are in-distribution for our own models. Treat them as
supplementary.qualifire/prompt-injections-benchmark and Mindgard/evaded-prompt-injection-and-jailbreak-samples.Results for 9 open detectors: Prompt Injection Detector Leaderboard.
from datasets import load_dataset
from transformers import pipeline
clf = pipeline("text-classification", model="Horizon-Labs/prompt-injection-guard-base", truncation=True)
ds = load_dataset("Horizon-Labs/prompt-injection-eval-suite", "notinject", split="test")
preds = [int(p["label"] == "injection") for p in clf(ds["text"], batch_size=32)]
print("accuracy", sum(int(p == y) for p, y in zip(preds, ds["label"])) / len(preds))