Passing the Test You Trained On

Re-evaluating prompt-injection detectors where LLM agents actually use them: on the tool outputs an agent reads.

Teams pick injection detectors by their benchmark scores. We check whether those scores predict behavior inside an agent, and find that they mostly measure how close the benchmark is to the detector's training data.

Findings

Fifteen detectors (nine open, six license-gated including Meta's Prompt Guard 2) and two task-aware LLM judges, on two agent benchmarks (AgentDojo, tau-bench) and on BIPIA. Benign tool outputs come from replaying each benchmark's ground-truth tool calls without an LLM. Detection is TPR at the threshold that gives 1% FPR.

(All 15 detectors and both judges: paper/tab_many.tex.)

1. Detection rankings transfer poorly between benchmarks (Kendall τ over 15 detectors): 0.01 for BIPIA vs AgentDojo, 0.28 for AgentDojo vs tau-bench, 0.31 for BIPIA vs tau-bench; none significant. The BIPIA leader (PIGuard) ranks 13th of 15 on AgentDojo; Prismor drops from 72% on AgentDojo to 15% on tau-bench.
2. False-positive rates on benign tool outputs range from 0% to over 90%, are consistent across the two agent benchmarks (τ = 0.67, p = 0.001), and are not predicted by false positives on ordinary text.
3. The form of the training inputs, not overlap in attack strings, explains the results. PIGuard trained on 1,116 full BIPIA inputs and leads BIPIA. It also saw 53 of InjecAgent's 62 attacks, but only as short prompts; inside tau-bench tool outputs it detects seen and unseen InjecAgent attacks alike (52.1% vs 55.8%). Horizon-Labs shares no data with any benchmark, trained on agent-style inputs, and leads both agent benchmarks.
4. Task-aware 7–8B judges (one trained before AgentDojo existed) reach 54–78% at 1% FPR on agent tool outputs, below the best dedicated detectors. Outside its training distribution every approach misses whole classes of injections; apart from PIGuard, no approach catches more than 39% of task-irrelevant injections.
5. Removing serialization markup or declaring the input type lowers default-threshold false positives by up to about twentyfold but does not fix low-FPR detection.

All numbers are regenerated from the score files by analysis/compute_all.py; the paper reads them only through paper/numbers.tex.

Layout

scripts/     build evaluation sets, score detectors and judges
analysis/    compute_all.py, compute_many.py, compute_robust.py, compute_e6.py, compute_cost.py (numbers),
             make_tables.py, make_figures.py (tables and figures), make_macros.py (-> LaTeX macros)
results/     detector and judge scores used in the paper
paper/       LaTeX source, figures, compiled main.pdf

Reproduce

pip install -r requirements.txt
AGENTDOJO_PY=/path/to/agentdojo-env/bin/python QWEN=/models/Qwen2.5-7B-Instruct LLAMA=/models/Llama-3.1-8B-Instruct bash reproduce.sh

To regenerate only the numbers and figures from the released scores, copy results/*.json into the working directory, rebuild the .jsonl sets (steps 1–3 of reproduce.sh, CPU only, with PYTHONHASHSEED=0), then run the analysis scripts of step 8 and python analysis/make_tables.py && python analysis/make_figures.py.

Robustness and cost experiments (journal version):

• analysis/compute_robust.py: bootstrap intervals and permutation tests for rank transfer; thresholds calibrated on held-out AgentDojo suites.
• scripts/score_e6.py, analysis/compute_e6.py: Prompt Guard 2 in the setting of the community benchmark buried-injections, and a check that the rebuilt sets reproduce the stored scores.
• scripts/latency.py, analysis/compute_cost.py: latency and memory per tool output on GPU and CPU.
• scripts/e5_run.py, scripts/e5_compare.py, scripts/e5_score.py: a real agent (Qwen2.5-7B served by vLLM) runs AgentDojo without attacks; its tool calls are replayed without a model and compared output by output.
• scripts/check_revision.py: which Hugging Face revision produced the stored scores (Horizon-Labs changed its weights on 2026-10-02; the stored scores are revision af3c52e6a9d8).

Evaluation data is rebuilt from the public sources, not redistributed: AgentDojo v1.2, tau-bench (MIT), InjecAgent attacks (MIT), BIPIA (MIT), GitHub READMEs (h1alexbel/github-readmes), AESLC, FineWeb-Edu. Some scripts assume BIPIA and PIGuard are cloned under ~/agentsec/.

Building the paper

paper/usenix-2020-09-xetex.sty is a build copy of the USENIX style that compiles with tectonic/XeTeX. For submission, switch main.tex to the official usenix-2020-09.sty and compile with pdflatex.

Status

Preprint on arXiv; journal version in preparation. Not evaluated against adaptive attacks; both agent benchmarks are simulations and tau-bench's injection point is ours; commercial API detectors not included. See the limitations section of the paper.

License

Code, analysis scripts, and score files: MIT (see LICENSE). Third-party files keep their own terms: the USENIX LaTeX style (paper/usenix-2020-09*.sty) and the benchmarks and detectors the scripts download (AgentDojo, tau-bench, InjecAgent, BIPIA, and each detector's model license).