Awesome Prompt Injection

A curated list of resources on prompt injection, the vulnerability that lets attacker-controlled text act as instructions to an LLM.

Contents

• Introduction
• Introduction Resources
• Articles and Blog posts
• Tutorials
• Research Papers
• Tools
• CTF
• Community
• Contributing

Introduction

Prompt injection is a vulnerability in applications built on large language models. The model receives instructions and data in the same channel, as one stream of text, and has no reliable way to tell which is which. Anyone who can get text in front of the model can therefore try to issue it instructions.

The direct form is a user typing something that overrides the developer's intent. The form that matters more in practice is indirect: the attacker never talks to the model at all, and instead plants text where the model will read it. A web page the agent browses, a document it summarises, an email in the inbox it triages, a tool description it loads, an issue on a repository it has been pointed at. The model follows the planted instructions because, from inside the context window, they are indistinguishable from legitimate ones.

Severity tracks capability rather than cleverness. A chatbot that can only produce text can be made to say something unintended. An agent that reads private data, calls tools and acts on the outside world can be made to leak that data or take actions on the attacker's behalf, which is why the problem has grown alongside agent adoption rather than being solved.

It also has no known general fix. Filters and classifiers raise the cost of an attack without closing the gap, and published defences have repeatedly fallen to adaptive attacks. The approaches that hold up best constrain what a compromised model is able to do, rather than trying to detect bad input.

This repository collects resources for understanding, testing and mitigating these attacks.

Introduction Resources

• OWASP GenAI LLM Top 10 (2026) - The canonical community reference for LLM application risk, with prompt injection at number one. Supersedes the 2025 edition; the per-risk 2025 pages remain online as an archive.
• Agents Rule of Two: A Practical Approach to AI Agent Security - Meta's Oct 2025 framework stating that agents must satisfy no more than two of: (A) processing untrustworthy inputs, (B) access to sensitive data, (C) ability to change state externally, a deterministic architectural approach to bounding blast radius.
• OWASP Top 10 for Agentic Applications (2026) - Companion list for autonomous and agentic systems, released Dec 2025 with input from over 100 contributors. Covers the agent-specific failure modes that the LLM list does not reach, including tool misuse and agent hijacking.

Articles and Blog posts

• Design Patterns for Securing LLM Agents against Prompt Injections - Overview of various strategies to mitigate the risk of prompt injection.
• Prompt injection: What's the worst that can happen? - General overview of Prompt Injection attacks, part of a series.
• ChatGPT Plugins: Data Exfiltration via Images & Cross Plugin Request Forgery - This post shows how a malicious website can take control of a ChatGPT chat session and exfiltrate the history of the conversation.
• Prompt Injection Cheat Sheet: How To Manipulate AI Language Models - A prompt injection cheat sheet for AI bot integrations.
• Prompt injection explained - Video, slides, and a transcript of an introduction to prompt injection and why it's important.
• Adversarial Prompting - A guide on the various types of adversarial prompting and ways to mitigate them.
• Don't you (forget NLP): Prompt injection with control characters in ChatGPT - A look into how to achieve prompt injection from control characters from Dropbox.
• Improving LLM Security Against Prompt Injection: AppSec Guidance For Pentesters and Developers - Using Role Based APIs to Minimize the Risk of Prompt Injection and 13 Guidelines for Writing System Prompts that Minimize the Risk of Prompt Injection.
• Improving LLM Security Against Prompt Injection: AppSec Guidance For Pentesters and Developers – Part 2 - Understanding Transformer Models (especially attention), Causes and How Do We Stop Prompt Injection.
• Synthetic Recollections - A Case Study in Prompt Injection for ReAct LLM Agents - A practical scenario showing how prompt injection can be used to hi-jack the ReAct loop used by LLM agents to inject forged thoughts and associated observations into the LLM context, thus altering the intended behavior.
• Continuously Hardening ChatGPT Atlas Against Prompt Injection Attacks - OpenAI's Dec 2025 disclosure of a real attack chain (malicious email → agent sends resignation letter) and the RL-trained automated attacker they built to find new injection classes before external adversaries do. OpenAI explicitly states deterministic guarantees are not achievable.
• How Microsoft Defends Against Indirect Prompt Injection Attacks - Microsoft MSRC's Jul 2025 post on FIDES, an information-flow control system enforcing privilege separation and prompt isolation to deterministically block IPI in Copilot-class agents.
• ToxicSkills: Snyk Finds Malware and Prompt Injection in 36% of AI Agent Skills - Feb 2026 Snyk research across the ClawHub AI agent skills registry: 36% of audited skills contained security flaws, 1,467 malicious payloads found, and 2.9% used curl | bash remote instruction loading to evade static analysis. Covers indirect injection via poisoned web content and persistent memory tampering.
• New Prompt Injection Papers: Agents Rule of Two and The Attacker Moves Second - Simon Willison's Nov 2025 commentary on both landmark papers, including the finding that 12 published defenses were bypassed at >90% success rate using gradient descent and RL-based adaptive attacks.
• Indirect Prompt Injection Through MCP Tools: A Defense Guide - Feb 2026 guide explaining why any MCP tool that reads data written outside your trust boundary (CRM notes, calendar invites, API responses) is an injection vector, with concrete mitigations per tool category.
• Indirect Prompt Injection Attacks: Hidden AI Risks - CrowdStrike's Dec 2025 analysis of IPI TTPs targeting enterprise GenAI, including attacker-controlled document poisoning, RAG context manipulation, and practical detection signals for SOC workflows.
• Why content-based prompt-injection detection has a structural blind spot - Walks through a disguised "already-scanned, classified DATA" annotation that passes both a regex layer and a local LLM classifier (qwen2.5:3b and 14b), with the gap pinned as a failing test. Detection is necessary but not sufficient.
• Prompt-Injection Cascades in a Multi-Agent Enterprise (ATLAS) - Forensic study in which one poisoned support ticket drives four agents across three model families to a fraudulent transfer in 17 of 18 trials, with no model jailbroken. Argues the failure is topological rather than per-model. Synthetic data, CC BY 4.0.
• Authority Framing in Agentic CI/CD (RELAY) - Pre-registered factorial study of a five-agent CI/CD pipeline built from five production LLMs. A fabricated "pre-approved, do not re-review" claim gets secret-exfiltration code shipped in up to 55% of runs, and roughly 80% of laundered pull requests pass a capable LLM code scanner. Notably the entry agent never leaked its system prompt (0/40), so prompt secrecy held and did not help. Synthetic data, CC BY 4.0.
• Agent-targeted bounty honeypots: a reproducible corpus - Field notes on a live GitHub operation advertising $219,000 across 43 issues, each stapling a request for the contributor's "complete pre-session instructions" onto a genuine engineering task. Documents the sandwich injection in CONTRIBUTING.md, where a truthful human-visible warning that the bounties are symbolic is wrapped in HTML comments telling automated readers to ignore it, and a payout gated on a merge the same file rules out. Every figure ships with the query that reproduces it.

Tutorials

• Prompt Injection - Prompt Injection tutorial from Learn Prompting.
• Google AI Red Team: Securing AI - Google's red team walkthrough of how AI systems are attacked in practice. An early foundational report, useful for framing rather than for current technique.
• Prompt Injection in LLM Agents (ReAct, Langchain) - Theory and hands-on lab on prompt injection against Langchain ReAct agents.
• How AI Prompt Injection Works | Hands-on with LLMs - Jan 2026 AppSecEngineer tutorial with a code-level demo of injecting against a real LLM application and live testing of LLM Guard detection. One of the most practical end-to-end tutorials published to date.
• MCP Prompt Injection: How AI Gets Hacked - Nov 2025 hands-on walkthrough showing how prompt injection exploits tool metadata and trust boundaries in Model Context Protocol-integrated agents, which remain a primary attack surface as MCP adoption widens.

Research Papers

• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection - This paper explores the concept of Indirect Prompt Injection attacks on Large Language Models (LLMs) through their integration with various applications. It identifies significant security risks, including remote data theft and ecosystem contamination, present in both real-world and synthetic applications.
• Universal and Transferable Adversarial Attacks on Aligned Language Models - This paper introduces a simple and efficient attack method that enables aligned language models to generate objectionable content with high probability, highlighting the need for improved prevention techniques in large language models. The generated adversarial prompts are found to be transferable across various models and interfaces, raising important concerns about controlling objectionable information in such systems.
• The Landscape of Prompt Injection Threats in LLM Agents (SoK) - Feb 2026 systematization-of-knowledge paper with a unified taxonomy covering attack payload strategies (heuristic vs. optimisation-based) and defense intervention stages (text, model, execution). Introduces the AgentPI benchmark for context-dependent agent tasks that all prior benchmarks ignored.
• The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections - Oct 2025 paper systematically breaking 12 published defenses using gradient descent, RL, random search, and human-guided exploration. Most defenses originally claimed near-zero attack success rates; adaptive attacks exceeded 90% against all of them.
• Prompt Injection 2.0: Hybrid AI Threats - Jul 2025 paper showing how prompt injections now combine with XSS, CSRF, AI worm propagation, and multi-agent infections to evade traditional WAFs entirely. Evaluates Preamble's classifier, data-tagging, and RL-based defenses against these hybrid scenarios.
• Securing AI Agents Against Prompt Injection Attacks - Nov 2025 benchmark of 847 adversarial test cases across 5 attack categories against 7 LLMs. The combined defense framework reduces attack success from 73.2% to 8.7% while retaining 94.3% of baseline task performance.
• ToolHijacker: Prompt Injection Attack to Tool Selection in LLM Agents - Apr 2025 paper introducing a no-box attack that injects a malicious tool document into an agent's tool library to consistently hijack tool selection. Finds that StruQ, SecAlign, DataSentinel, and perplexity detection are all insufficient defenses.
• Attention Tracker: Detecting Prompt Injection Attacks in LLMs - NAACL 2025 Findings paper detecting prompt injection by tracking attention distribution shifts, with no modification to the underlying model required, making it deployable as a wrapper on any LLM.
• Safety in Embodied AI: Risks, Attacks, and Defenses - A comprehensive survey of 500+ papers covering prompt injection and other attack vectors in embodied AI systems across the full pipeline (perception, cognition, planning, action, agentic). Includes a 5-layer threat taxonomy mapping where new capabilities introduce new attack surfaces.
• Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models - Discovers that text embedding models have severely biased output distributions, and exploits this to find universal adversarial suffixes ("magic words") that bypass embedding-based LLM safeguards. Attacks transfer across models and languages; a train-free debiasing defense is also proposed.
• FinAgent Red-Team: A Benchmark for Regulatory-Control Bypass in Financial LLM Agents - Aug 2026 benchmark measuring whether indirect prompt injection can drive financial LLM agents past regulatory controls (sanctions screening, payment-structuring thresholds, dual-approval, data exfiltration, confused-deputy payee redirection). Replays 48 scenarios under three control postures (none/advisory/enforced) with deterministic state-predicate scoring rather than an LLM grader. Finds a frontier model (GPT-4o) bypasses controls 71% of the time with no stated policy, while hard tool-layer enforcement drives attack success to 0% for every model tested.

Tools

• Garak - Automate looking for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks, and many other weaknesses in LLM's.
• OWASP Agent Memory Guard - Open-source scanner for AI agent memory poisoning attacks (OWASP ASI06). Detects prompt injection payloads, memory manipulation patterns, and data exfiltration attempts in agent memory stores. Available as a Python package (pip install agent-memory-guard) and GitHub Action.
• PIC Standard - Protocol to block unauthorized or unproven agent actions via intent + provenance checks. Mitigates prompt injection & side-effect risks. Open-source (Apache 2.0).
• Agent Threat Rules (ATR) - Open detection standard for AI agent threats (prompt injection, tool poisoning, MCP attacks, skill compromise), using Sigma/YARA-style YAML rules. 330 rules across 9 attack categories with full mapping to OWASP Agentic Top 10 (10/10), MITRE ATLAS (100/113), NIST AI RMF (100%), and SAFE-MCP (78/85). 97.1% recall on the garak probe set (193 probes) and 0% false-positive on 53,577 real-world MCP skills. Shipped in production at Cisco AI Defense and Microsoft agent-governance-toolkit. Apache-2.0.
• Augustus - Feb 2026 open-source tool from Praetorian. A single Go binary with 210+ vulnerability probes across 47 attack categories, 28 LLM providers, 90+ detectors, and 7 payload transformation buffs. Built for penetration testing workflows without Python/npm dependencies.
• InjecGuard - Open-source prompt guard with published training data; achieves +30.8% over prior state-of-the-art on the NotInject benchmark, specifically addressing overdefense false positives that break legitimate use cases.
• brood-box - Hardware-isolated microVM sandbox for running coding agents (Claude Code, Codex, OpenCode) with workspace snapshot isolation, DNS-aware egress control, and MCP authorization profiles to contain damage from prompt injection attacks.
• prompt-shield - Self-learning prompt injection detection engine with novel cross-domain techniques: Smith-Waterman sequence alignment (bioinformatics), stylometric discontinuity detection (forensic linguistics), and adversarial fatigue tracking (materials science). 27 detectors, 6 output scanners, 10 languages, benchmarked on 6 public datasets. Research paper: arXiv:2604.18248. Apache-2.0.
• Guard Bands - Cryptographic data boundary for LLM applications: untrusted content is wrapped in HMAC-SHA256 or Ed25519 signed markers that bind provenance, lifetime and application context, and the verifier reconstructs that context from trusted state before a protected path proceeds. Tamper, replay, expiry and context mismatch fail closed, so enforcement sits in application code rather than in a model asked to classify inputs. Published protocol spec with conformance vectors, plus FastAPI and MCP adapters. MIT.

CTF

• PromptTrace - Free AI security training platform with 7 hands-on prompt injection labs and a 15-level CTF (the Gauntlet) with progressively harder defenses, from prompt-level rules to code guards to LLM classifiers. Unique feature: Context Trace shows the full prompt stack (system prompt, RAG documents, tool definitions, user input) in real-time so you can see exactly how attacks work. Uses real LLMs from OpenAI, Anthropic, Google, Groq, and Cerebras.
• Gandalf - Your goal is to make Gandalf reveal the secret password for each level. However, Gandalf will level up each time you guess the password, and will try harder not to give it away. Can you beat level 7? (There is a bonus level 8).
• Damn Vulnerable LLM Agent - A sample chatbot powered by a ReAct agent, implemented with Langchain. It's designed to be an educational tool for security researchers, developers, and enthusiasts to understand and experiment with prompt injection attacks in ReAct agents.
• AI/LLM Exploitation Challenges - AI, ML, and LLMs CTF Challenges.
• CrowdStrike AI Unlocked - Released Feb 2026, designed to train security, developer, and AI teams on prompt injection against increasingly capable agents. Built by CrowdStrike's Counter Adversary Operations team.
• ai-prompt-ctf by c-goosen - One of the few CTFs that tests indirect injection against tool-calling agents, spanning RAG, function calling, and ReAct agent scenarios using LlamaIndex, ChromaDB, GPT-4o, and Llama 3.2.
• LLMVault - Self-hosted CTF-style training range for the OWASP LLM Top 10, with 25 labs across three tiers. Play Mode uses scripted assistants so flags reproduce every time; Live Mode points the same attacks at a real model on your own machine with the secret regenerated per session, so there is no flag to look up. Each lab pairs the attack with the defense that would have stopped it. MIT, Docker.

Community

• Learn Prompting - Discord server from Learn Prompting.
• OWASP Gen AI Security Project - The standards body behind the LLM and Agentic Top 10 lists, with working groups, cheat sheets and guidance updated continuously by practitioners across the industry.
• Simon Willison's Blog - The most consistent independent tracker of real-world prompt injection incidents, new papers, and tooling across the field.
• r/llmsecurity - The most active subreddit dedicated to LLM security research; a good early-warning channel for real-world incidents and new disclosures.
• MITRE ATLAS - MITRE's adversarial ML threat matrix formally cataloging direct and indirect prompt injection as core adversary techniques, enabling integration into enterprise threat modelling and purple team exercises.

Contributing

Contributions are welcome! Please read the contribution guidelines first.