Awesome AI Security

A practitioner-focused reference for AI/ML security — attacks, tools, research, and defenses. Covers the full spectrum: offensive AI, securing AI systems, AI-assisted security operations, and governance. Last updated March 28, 2026.

---

Contents

• 1. Attacks & Exploitation
  • Prompt Injection
  • Jailbreaking LLMs
  • Agentic & Multi-Agent Attacks
  • Training Data & Privacy Attacks
  • Adversarial ML — Classical Models
  • Supply Chain & Model Poisoning
  • AI Infrastructure Attacks
  • Offensive Use of AI
  • Attack Tutorials & Walkthroughs
• 2. Key Research Papers
  • Prompt Injection & Jailbreaking
  • Privacy & Extraction
  • Agent Security
  • Adversarial ML & Robustness
  • Backdoors & Supply Chain
  • Offensive AI
  • Defenses
• 3. Conference Talks
  • Black Hat
  • RSA Conference
  • DEF CON AI Village
  • USENIX Security
  • IEEE SaTML
  • CAMLIS
  • Talk Archives on YouTube
• 4. Tools — Offense & Red Teaming
  • LLM Red Teaming
  • Adversarial ML
  • Agentic & MCP Attack Tools
  • Benchmarks & Evaluation
  • Vulnerable Labs & CTFs
• 5. Tools — Defense & Detection
  • Guardrails & Output Safety
  • Model & Supply Chain Security
  • Production Monitoring
  • Agent Runtime Security & Sandboxing
  • MCP Security
  • AI Code Security
  • Privacy-Preserving Inference
• 6. AI for Security Operations
  • Penetration Testing & Offensive Security
  • Malware Analysis & Reverse Engineering
  • Vulnerability Research
  • Threat Intelligence & SOC
  • Security-Specialized Models
• 7. Notable Incidents & CVEs
• 8. Attack Frameworks & Knowledge Bases
• 9. Defensive Frameworks & Standards
  • Risk Management
  • Verification Standards
  • Threat Modeling
  • Incident Response
• 10. Regulatory & Compliance
• 11. Community & Practice
  • Communities & Organizations
  • Key Practitioners to Follow
  • Conferences & Venues
  • Bug Bounty Programs
  • Research Blogs Worth Following
  • Newsletters & Podcasts
• Datasets
  • Safety & Attack Datasets
  • Cybersecurity Skill Benchmarks
• Agentic AI Security Skills

---

1. Attacks & Exploitation

Prompt Injection

Prompt injection is the primary attack class against LLM-integrated applications. It splits into two types: direct injection (user-controlled input manipulates the model) and indirect injection (malicious instructions arrive via data the model retrieves — web pages, documents, tool outputs, emails).

Key attack techniques:

Real-world vulnerabilities:

• CVE-2025-53773 — GitHub Copilot RCE via prompt injection. Attacker-controlled code comments triggered Copilot to generate and execute malicious code.
• EchoLeak (CVE-2025-32711) — Zero-click prompt injection in Microsoft 365 Copilot. Chains XPIA classifier bypass + Markdown redaction bypass + auto-fetched image abuse to exfiltrate SharePoint/Teams/OneDrive data without user interaction. CVSS 9.3.
• SpAIware — Persistent memory injection attack in ChatGPT's memory feature. Attacker embeds instructions in a webpage; when a user asks ChatGPT to summarize it, the instructions persist in memory and activate in future sessions (Johann Rehberger / embracethered.com). Write-up

2026 developments:

• ToxicSkills (Feb 2026) — First coordinated malware campaign via AI agent skills. Snyk audited 3,984 skills from ClawHub; 36% contained prompt injection techniques, 76 confirmed malicious payloads for credential theft and SSH key exfiltration. Three lines of markdown in SKILL.md were sufficient to exfiltrate SSH keys. snyk.io/blog
• Agentic browser injection (Trail of Bits, Jan 2026) — Agentic browsers that fetch web pages, read files, and interact with the DOM create XSS/CSRF-equivalent attack surfaces. Magic link authentication URL attacks silently log users into attacker-controlled accounts when an agent summarizes a malicious page. blog.trailofbits.com
• AI Recommendation Poisoning (Microsoft, Feb 2026) — Microsoft documented 50+ real-world cases of prompt injection poisoning AI assistant memory (ChatGPT, Copilot, Claude, Perplexity, Grok) for commercial promotion. 31 companies across 14 industries were exploiting this in the wild. microsoft.com/security/blog
• Perplexity Comet injection (Trail of Bits, Feb 2026) — Audit of Perplexity's Comet browser AI assistant found four prompt injection techniques that could exfiltrate private Gmail data. blog.trailofbits.com
• Reprompt (Varonis, Mar 2026) — Single-link attack against Microsoft Copilot that bypasses data-leak protections and enables persistent session exfiltration even after Copilot is closed. varonis.com/blog

Classification:

• ARC Prompt Injection Taxonomy — The most structured open classification of prompt injection attacks. Four dimensions: attacker intent (13 categories), execution technique (18), filter evasion method (20), input surface. Interactive frontend: https://arcanum-sec.github.io/arc_pi_taxonomy/

• Promptware Kill Chain (2025) — Documents 21 real-world prompt injection incidents from 2025, classifying them via a kill-chain model. Finds persistence capabilities in 12 of 21 attacks.

---

Jailbreaking LLMs

Jailbreaking bypasses the safety alignment of a model to elicit policy-violating outputs. Distinct from prompt injection (which hijacks an integrated application); jailbreaking targets the model's trained refusal behavior directly.

Gradient-based attacks (white-box):

• GCG (Greedy Coordinate Gradient) — Optimizes a universal adversarial suffix that reliably bypasses aligned LLMs and transfers across models including GPT-4, Claude, Bard. Foundational paper: arXiv:2307.15043. Production implementation: BrokenHill (Bishop Fox).
• AutoDAN — Automated generation of human-readable adversarial prompts using genetic algorithms. Produces jailbreaks that are fluent and harder to detect than GCG suffixes. github.com/SheltonLiu-N/AutoDAN
• DiffusionAttacker — Uses a seq2seq diffusion model to generate jailbreak prompts. Outperforms prior methods on fluency, diversity, and attack success rate. EMNLP 2025. arXiv:2412.17522

Black-box attacks (query-only):

• PAIR (Prompt Automatic Iterative Refinement) — An LLM-as-attacker that iteratively refines jailbreak prompts until a target model complies. Achieves jailbreaks in ~20 queries. arXiv:2310.08419
• TAP (Tree of Attacks with Pruning) — Extends PAIR with a tree search to prune ineffective attack branches. More efficient than PAIR on complex safety categories. github.com/RICommunity/TAP
• Bad Likert Judge — Instructs the target LLM to evaluate harmfulness on a Likert scale, then requests examples aligned to the highest-rated category. Boosts success rates >60% across tested models. Palo Alto Unit 42, 2024. unit42.paloaltonetworks.com
• Crescendo — Multi-turn gradual escalation. See Prompt Injection section above.
• Many-Shot — Long-context exploitation. See Prompt Injection section above.

Reasoning model attacks:

• H-CoT (Chain-of-Thought Hijacking) — Universal attack on o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking that hijacks the model's visible intermediate reasoning steps. Under H-CoT, refusal rates drop from 98% to below 2%. arXiv:2502.12893
• DeepSeek-R1 Safety Assessment — R1's baseline refusal rate is ~20% on harmful queries. Design flaw: R1 produces harmful content in its reasoning trace before its safety moderator fires. arXiv:2502.12659

Multimodal attacks:

• Adversarial Image Jailbreaks — Adversarial perturbations on images reliably jailbreak vision-language models (LLaVA, MiniGPT-4, InstructBLIP) even when text-based safety training is intact. Transfers across model families. arXiv:2306.13213
• DiffusionAttacker (multimodal) — See above.
• PoisonedEye — Embeds malicious instructions inside images in RAG-indexed documents. Triggered when a vision-capable agent retrieves and processes the image. openreview.net

2026 jailbreak research:

• Mastermind (Jan 2026) — Hierarchical planning framework that decouples high-level attack objectives from tactical execution, guided by a knowledge repository that autonomously refines effective attack patterns. Achieves 94% ASR on DeepSeek V3, 93% on GPT-4o, 90% on o3-mini, 89% on DeepSeek-R1. arXiv:2601.05445
• RACE — Reasoning-Augmented Conversation (Feb 2026) — Reformulates harmful queries into benign reasoning tasks that lead models to produce harmful content. Up to 96% overall ASR, 82% on o1, 92% on DeepSeek-R1. arXiv:2502.11054
• UltraBreak (Feb 2026) — Universal adversarial patterns for vision-language models that transfer across diverse jailbreak objectives and model families. arXiv:2602.01025
• Reasoning Models as Autonomous Jailbreak Agents (Nature Communications 2026) — When DeepSeek-R1, Gemini 2.5 Flash, Grok 3 Mini, and Qwen3 235B are used as autonomous jailbreak agents against nine target models, overall attack success rate reaches 97.14%. Converts jailbreaking from an expert activity into a non-expert-accessible automated process. nature.com

Fine-tuning as jailbreak:

• Standard fine-tuning on completely benign data degrades alignment. Adversarial fine-tuning with 10 examples costs <$0.20 and strips GPT-3.5's safety guardrails. arXiv:2310.03693

LLM-as-a-Judge exploitation:

• Universal adversarial phrases appended to responses manipulate LLM judges into predicting inflated scores. Critical for red team pipelines that use automated evaluation. arXiv:2402.14016
• System-prompt injection into evaluation pipelines achieves higher success rates than content-layer attacks. arXiv:2504.18333

---

Agentic & Multi-Agent Attacks

AI agents that use tools, browse the web, execute code, and persist across sessions dramatically expand the attack surface beyond single-turn LLM interactions.

Memory poisoning:

• AgentPoison — Backdoor attack targeting RAG-based agents. Optimizes triggers in embedding space so poisoned memory entries are retrieved with >80% probability whenever a trigger appears. No model retraining required. NeurIPS 2024. arXiv:2407.12784
• MemoryGraft — Injects malicious "successful task completion" records into an agent's memory. On future semantically similar tasks, the agent adopts the malicious procedure without any explicit trigger. Persistent cross-session compromise. arXiv:2512.16962
• MINJA — Query-only memory injection achieving >95% success rates via bridging steps and progressive shortening. No privileged access required — exploitable via normal user interactions. arXiv:2503.03704

Control flow & privilege escalation:

• Multi-Agent Control-Flow Hijacking — Compromised subagents re-route task execution to parent orchestrators, achieving access equivalent to the compromised agent: credentials, emails, calendars, files. 97% code execution rates observed. arXiv:2510.17276
• ConfusedPilot — Data corruption and leakage by exploiting Microsoft 365 Copilot's RAG context injection. UT Austin, DEF CON 32. arXiv:2408.04870

MCP (Model Context Protocol) attacks:

• Tool Poisoning — Malicious MCP server embeds prompt injection payloads inside tool descriptions or server instructions, poisoning the agent's context before the user's first interaction ("line jumping"). Invariant Labs research
• CVE-2025-6514 — mcp-remote arbitrary command execution via malicious server URL. CVSS 9.6. nvd.nist.gov
• MCPTox — Benchmark for tool poisoning attacks against real MCP servers. arXiv:2508.14925
• MCP Registry Supply Chain — Malicious servers registered in public MCP registries, impersonating legitimate tools. vulnerablemcp.info
• MCP Rug Pull / Tool Shadowing — MCP servers can silently modify tool definitions between sessions post-approval. A tool approved on Day 1 may be replaced by a malicious version by Day 7, exploiting cached user trust. Formally documented by Unit 42. unit42.paloaltonetworks.com

Self-replicating attacks:

• Morris II (AI Worm) — First self-replicating worm targeting GenAI ecosystems. Adversarial self-replicating prompts cascade through RAG-based multi-agent pipelines without user interaction. Demonstrated against ChatGPT-4, Gemini Pro, and LLaVA in an email assistant simulation. arXiv:2403.02817

Computer-use agent attacks:

• Agents that control a desktop or browser (Anthropic Computer Use, OpenAI Operator) introduce a novel attack surface: malicious content on a rendered webpage can inject instructions via the visual/UI channel, bypassing text-based filters. arXiv:2501.04219

2026 agentic attack research:

• MCP-ITP (Jan 2026) — First automated framework for implicit tool poisoning in MCP. Formulates poisoned tool generation as black-box optimization. Achieves 84.2% attack success rate while suppressing detection to 0.3%. Existing safety alignment largely ineffective. arXiv:2601.07395
• Viral Agent Loop (Feb 2026) — Agents acting as vectors for self-propagating generative worms. Systematizes agentic runtime supply chain attacks: data supply chain (context injection + memory poisoning) and tool supply chain (discovery, implementation, invocation). arXiv:2602.19555
• Sleeper Cell backdoor (Mar 2026) — Novel stealthy backdoor for tool-using agents via SFT-then-GRPO fine-tuning. With 1,000 samples, trains models that are operationally deceptive while maintaining near-perfect stealth on utility benchmarks. arXiv:2603.03371
• ToxicSkills agent skills supply chain (Feb 2026) — 36% of ClawHub agent skills contain prompt injection; 76 confirmed malicious payloads. 91% of malicious skills simultaneously use prompt injection alongside malicious code. Three lines of markdown sufficient to exfiltrate SSH keys. snyk.io
• CVE-2026-0628 — Gemini Chrome panel hijacking (Jan 2026) — Chrome WebView insufficient policy enforcement allows a low-privilege extension to inject code into Gemini Live's side panel and inherit file access, screenshot, and camera/microphone capabilities. CVSS 8.8. unit42.paloaltonetworks.com
• Claude Code CVEs (2025–2026) — CVE-2025-59536: RCE via malicious Hook commands in .claude/settings.json, triggering automatically when an untrusted repository is opened. CVE-2026-21852: API key exfiltration by overriding ANTHROPIC_BASE_URL to an attacker endpoint — every Claude API call then sends the authorization header to the attacker. CVE-2026-31862: Critical command injection in Cloud CLI (CVSS 9.1). research.checkpoint.com
• SesameOp — AI API as C2 — First confirmed real-world backdoor using a commercial AI API (OpenAI Assistants) as covert command-and-control. Threat actor was present for months before discovery. Now documented as MITRE ATLAS case study AML.CS0042. microsoft.com/security/blog

---

Training Data & Privacy Attacks

Training data extraction:

• Carlini et al. (2021) — LLMs memorize and reproduce verbatim training data including PII. Baseline methodology for extraction. arXiv:2012.07805
• Divergence Attack (2023) — Causes ChatGPT to emit memorized training data at 150× the normal rate. Demonstrates gigabyte-scale extraction from production LLMs. arXiv:2311.17035
• Copyrighted Book Extraction (2025) — Gemini 2.5 Pro and Grok 3 directly comply with instructions to extract memorized copyrighted book text. Claude 3.7 and GPT-4.1 require jailbreaking. arXiv:2601.02671
• Diffusion model extraction — Over 1,000 training images (including personal photos) recovered from diffusion models. arXiv:2301.13188

Membership inference attacks (MIAs):

• Determine whether a specific data point was in a model's training set. Directly relevant to GDPR, HIPAA, and data deletion compliance. Foundational paper: arXiv:1610.05820
• Critical evaluation (2024): most published LLM MIAs are methodologically flawed — performance near random chance under rigorous conditions. arXiv:2402.07841 / SaTML 2025: arXiv:2406.17975
• Tokenizer MIA — Novel attack surface: tokenizers trained on pretraining-representative data leak membership. arXiv:2510.05699

Model stealing:

• Carlini et al. (2024) — Extracts the embedding projection layer from production LLMs. Cost: <$20 for GPT Ada/Babbage; <$2,000 for GPT-3.5-turbo's full projection matrix. arXiv:2403.06634
• Logit-based extraction — Most LLMs output logits restricted to a low-dimensional subspace, leaking non-public architecture information via API. Under $1,000 in queries. arXiv:2403.09539

Embedding inversion:

• Reconstruct original text inputs from embedding vectors with high fidelity using only a surrogate model (no access to the target model). Realistic threat to vector database deployments. ACL 2024. arXiv:2406.10280

Federated learning gradient attacks:

• Deep Leakage from Gradients — Shared gradients in federated learning can reconstruct original training inputs with high fidelity. NeurIPS 2019. arXiv:1906.08935
• Inverting Gradients — Extends reconstruction to large batch sizes and high-resolution images, making the attack practical at scale. Achieves quality sufficient to read text in images and identify individuals. NeurIPS 2020. arXiv:2003.14053

---

Adversarial ML — Classical Models

Web-scale poisoning — Controlling a small fraction of web content (common crawl, Wikipedia edits) is sufficient to influence model behavior. Carlini et al., 2023: arXiv:2302.10149

LeftoverLocals (GPU Side-Channel, Trail of Bits, 2024) — Cross-process recovery of LLM inference outputs from GPU local memory. An attacker with local GPU access can read partial KV-cache or logits from another process's LLM inference. Demonstrated against Apple, AMD, and Qualcomm GPUs. CVE-2023-4969. Blog post

---

Supply Chain & Model Poisoning

Backdoor persistence through safety training:

• Sleeper Agents (Anthropic, 2024) — Backdoor behaviors survive RLHF, supervised fine-tuning, and adversarial training. A model trained to insert malicious code when the year is 2024 (but write safe code otherwise) cannot be reliably cleaned. Adversarial training may make backdoors better hidden, not smaller. arXiv:2401.05566

Code completion backdoors:

• CodeBreaker (USENIX Security 2024) — LLM-assisted backdoor attack on code completion models that evades static analysis. Poisoned completions insert CWE-level vulnerabilities that are syntactically valid and bypass Semgrep/CodeQL. arXiv:2406.06822

Malicious models on Hugging Face:

• Two PyTorch models discovered (2025) using 7z compression (not ZIP) to evade Picklescan, hiding malicious payloads in ML model files. GGUF format has no production-ready security scanner as of early 2026.
• PickleBall (CCS 2025) — ~44.9% of Hugging Face repos contain pickle-format models. Proposes a secure deserialization sandbox. arXiv:2508.15987

Watermark attacks:

• Watermark-removal via semantic paraphrase and watermark-spoofing (injecting a target watermark into malicious content) work against major LLM watermarking schemes. arXiv:2402.16187
• Adaptive attackers with GPU access achieve >96% watermark evasion in under 7 hours. arXiv:2410.02440

2026 supply chain:

• LiteLLM TeamPCP supply chain attack (Mar 2026) — Threat actor stole PyPI credentials via a compromised Trivy GitHub Action in LiteLLM's CI/CD pipeline. Published backdoored versions 1.82.7 and 1.82.8 with multi-stage credential stealers. With 3.4 million daily downloads, packages were live for ~3 hours. Tracked by Wiz, Sonatype, and Datadog Security Labs. wiz.io/blog
• MCP ecosystem CVEs — 30 CVEs in 60 days (2026) — First 60 days of 2026 saw 30+ CVEs across MCP servers, clients, and infrastructure. Root causes: missing input validation (43% exec/shell injection), absent authentication, blind trust in tool descriptions. vulnerablemcp.info

RAG poisoning:

• PoisonedRAG — Injecting 5 malicious texts into a database of millions induces target answers. Success rates: 97% (NQ), 99% (HotpotQA), 91% (MS-MARCO) against PaLM 2. USENIX Security 2025. arXiv:2402.07867
• Phantom RAG — Dormant malicious document that remains inactive during normal queries, activating only when specific trigger keywords appear. Significantly harder to detect than always-active poisoned documents. arXiv:2405.20485
• Semantic Chameleon — Gradient-guided corpus-dependent RAG poisoning. Achieves 38% co-retrieval on pure vector retrieval; notably, hybrid BM25+vector retrieval reduces attack success from 38% to 0%, making retrieval strategy a key defensive decision. arXiv:2603.18034
• AgentPoison — Embedding-space backdoor targeting RAG agents (see Agentic Attacks).

---

AI Infrastructure Attacks

These target the MLOps stack — training clusters, model serving, notebook environments, and cloud AI platforms — rather than the model itself.

AI coding assistant attacks:

• IDEsaster — Systematic disclosure of 24+ CVEs affecting Cursor, Windsurf, GitHub Copilot, Zed, Kiro.dev, Cline, and others. Attack chain: Prompt Injection → AI tool use → base IDE features (RCE, credential exfiltration). 100% of tested AI IDEs were vulnerable. arXiv:2601.17548
• Rules File Backdoor — Hidden Unicode characters in .cursorrules / Copilot configuration files silently poison AI-generated code with backdoors that survive code review. A supply chain attack requiring no runtime access. pillar.security
• AI-generated code CVEs (2026 trend) — AI vibe-coded code has a ~45% security failure rate despite 95%+ syntax correctness. 35 CVEs from AI-generated code disclosed in March 2026 alone (up from 6 in January). 86% vulnerable to XSS; 88% to log injection. Tracked by Georgia Tech "Vibe Security Radar." infosecurity-magazine.com

Real-world exploits: ProtectAI AI-Exploits — working PoC exploits for disclosed CVEs in MLflow, Ray, Hugging Face, and other MLOps infrastructure.

---

Offensive Use of AI

Autonomous vulnerability exploitation:

• GPT-4 one-day CVE exploitation — GPT-4 agents autonomously exploit 87% of one-day CVEs given CVE descriptions. GPT-3.5, open-source LLMs, and Metasploit scored 0%. arXiv:2404.08144
• Multi-agent zero-day exploitation — Hierarchical LLM teams achieve 42% success on novel, undisclosed vulnerabilities. arXiv:2406.01637
• CVE-Genie — Automates CVE-to-exploit reproduction using multi-agent LLMs. Reproduces ~51% of 2024–2025 CVEs at ~$2.77 per CVE. arXiv:2509.01835

Google Project Zero — Big Sleep:

• Project Naptime → Big Sleep: a Google DeepMind + Project Zero framework providing AI agents with Code Browser, Python execution, and Debugger tools for autonomous vulnerability research. Discovered a real-world exploitable stack buffer underflow in SQLite — the first publicly documented AI-discovered real-world zero-day. Fixed the same day.
• Project Naptime (June 2024)
• From Naptime to Big Sleep (October 2024)

AI in active threat operations:

• OpenAI disrupted 40+ threat actor networks since 2024 using its models for phishing, influence operations, and SIGINT-style monitoring tool development. openai.com
• Documented cases: AI-generated SVG phishing payloads with obfuscated malicious code; AI-generated spear-phishing with 38% click rates; Dark LLM vendors offering uncensored 80B+ models at $30–$200/month. (Group-IB, 2025)
• GTG-2002 threat actor — Claude Code weaponized to conduct automated attacks against 17+ organizations (2025). anthropic.com
• AI accelerating attack lifecycles (Unit 42, Feb 2026) — Based on 750+ high-stakes incidents, AI accelerated attack lifecycles 4× over the prior year. Fastest cases: initial access to data exfiltration in 72 minutes. paloaltonetworks.com
• CrowdStrike 2026 Global Threat Report — Average eCrime breakout time fell to 29 minutes; fastest observed: 27 seconds. AI-enabled attacks up 89% YoY. 24 new adversaries named; 281+ total tracked. Adversaries actively injecting malicious prompts into GenAI tools at 90+ organizations. crowdstrike.com
• IBM X-Force Threat Intelligence Index 2026 — 44% increase in public-facing application exploitation; AI-enabled attacks documented across vulnerability discovery, spear-phishing generation, and data synthesis for targeting. ibm.com/security/blog
• Google GTIG AI Threat Tracker — DPRK, Iran, China, and Russia all operationalized AI in 2025. PROMPTFLUX and PROMPTSTEAL are first documented AI-native malware families using LLMs at execution time. 100,000+ model extraction attempts observed and mitigated. State-backed actors using Gemini for OSINT synthesis and target profiling. cloud.google.com
• Microsoft "AI as Tradecraft" (Mar 2026) — Detailed analysis of threat actor AI use across the full attack lifecycle: reconnaissance, spear-phishing, malware generation, evasion, and post-exploitation iteration. Documents emerging agentic AI tradecraft. microsoft.com/security/blog
• HiddenLayer 2026 AI Threat Landscape Report — 1 in 8 reported AI breaches now linked to agentic systems; 35% of AI-related breaches sourced from malware in public model/code repositories; 31% of orgs don't know if they experienced an AI security breach. hiddenlayer.com
• CrowdStrike 2026 Global Threat Report — FANCY BEAR deployed LLM-enabled malware (LAMEHUG) for automated recon; FAMOUS CHOLLIMA (DPRK) scaled insider operations using AI-generated personas; average eCrime breakout time fell to 29 minutes (fastest: 27 seconds); AI-enabled attacks up 89% YoY. crowdstrike.com
• LLMjacking — Operation Bizarre Bazaar (Jan 2026) — First large-scale LLMjacking campaign with full commercial monetization. 35,000 attack sessions targeting exposed Ollama instances, OpenAI-compatible APIs, and MCP servers. Stolen LLM access resold at 40–60% discount on silver.inc marketplace. pillar.security
• 91,000+ sessions targeting LLM infrastructure (GreyNoise, Feb 2026) — GreyNoise sensors observed 91,403 sessions targeting Ollama inference servers from Oct 2025 to Jan 2026. A single 11-day campaign tested 73+ model endpoints across GPT-4o, Claude, Llama, Gemini, Mistral, DeepSeek-R1. greynoise.io
• CyberExplorer benchmark (Feb 2026) — AI agents evaluated autonomously performing recon, target selection, and exploitation against 40 real-world CTF-derived web services. arXiv:2602.08023
• Wiz AI Cyber Model Arena (Feb 2026) — 257 real-world challenges (zero-day discovery, CVE exploitation, cloud security). AI agents solved 9 of 10 web challenges; no single model dominates all domains. wiz.io/blog

---

Attack Tutorials & Walkthroughs

Hands-on resources with working code and step-by-step attack execution — not just theory.

Prompt injection:

Jailbreaking:

Agentic & MCP attacks:

RAG poisoning:

Adversarial ML:

AI infrastructure exploitation:

---

2. Key Research Papers

All linked to free arXiv versions or official open-access pages. Organized by attack class.

Prompt Injection & Jailbreaking Papers

Privacy & Extraction Papers

Agent Security Papers

Adversarial ML & Robustness Papers

Backdoors & Supply Chain Papers

Offensive AI Papers

Defense Papers

---

3. Conference Talks

Black Hat

Black Hat USA 2026 (August 5–7, Las Vegas)

• Full-day AI Summit on August 5. Sessions span AI-accelerated attacks, agentic AI threat landscapes, AI infrastructure exploitation, and AI for defensive security operations. Schedule TBA. blackhat.com/us-26

Black Hat Asia 2026 (April 21–24, Singapore)

• AI Security Summit alongside the main briefings program. Sessions cover AI-driven attacks, enterprise AI hardening, agentic AI threat landscapes, and practical GenAI threat intelligence using RAG and multi-agent workflows. blackhat.com/asia-26

Black Hat USA 2025

• "AI Enterprise Compromise: 0Click Exploit Methods" — Michael Bargury & Tamir Ishay Sharbat (Zenity Labs). Silent hijacking of enterprise AI agents (Jira, GitHub Copilot Studio, Slack) via zero-click MCP exploit chains. blackhat.com/us-25
• "Breaking Out of the AI Cage: Pwning AI Providers with NVIDIA Vulnerabilities" — Container escapes via CVE-2024-0132, CVE-2025-23266, CVE-2025-23319 affecting GPU cloud infrastructure. YouTube
• "LLMDYARA: LLMs-Driven Automated YARA Rules Generation" — Automated malware YARA rule creation using LLMs. Slides
• "Autonomous Timeline Analysis and Threat Hunting: An AI Agent for Timesketch" — Sec-Gemini digital forensic agent for log analysis. YouTube · Google Cloud blog

Black Hat Europe 2024

• "SpAIware & More: Advanced Prompt Injection Exploits in LLM Applications" — Johann Rehberger. Persistent memory injection and advanced injection chains. blackhat.com/eu-24

Black Hat USA 2024

• "Practical LLM Security: Takeaways From a Year in the Trenches" — Richard Harang (NVIDIA). Slides: PDF
• "From MLOps to MLOops: Exposing the Attack Surface of Machine Learning Platforms" — Shachar Menashe (JFrog). Slides: PDF
• "Isolation or Hallucination? Hacking AI Infrastructure Providers for Fun and Weights" — Hillai Ben-Sasson & Sagi Tzadik (Wiz). Cross-tenant attacks on Hugging Face, Replicate, SAP AI Core.
• "Deep Backdoors in Deep Reinforcement Learning Agents" — Mavroudis et al. (Alan Turing Institute). Slides: PDF
• "What Lies Beneath the Surface? Evaluating LLMs for Offensive Cyber Capabilities" — Kouremetis et al. (MITRE). Slides: PDF
• Full AI talk index across BSidesLV + Black Hat + DEF CON 2024: tldrsec.com

Black Hat Asia 2024

• "LLM4Shell: Discovering and Exploiting RCE in LLM-Integrated Applications" — RCE vulnerabilities in LLM-integrated apps. Slides: PDF

Black Hat USA 2023

• "Compromising LLMs: The Advent of AI Malware" — Kai Greshake & Christoph Endres. Indirect prompt injection weaponized as AI malware. Slides: PDF

RSA Conference

RSA Conference 2026 (March 23–26, San Francisco)

• Agentic security was the dominant theme. Key session: "Securing AI Agent Toolchains: Exploiting and Hardening MCP Servers." Cisco announced DefenseClaw open-source agentic security framework. Multiple vendor announcements on AI-SPM (AI Security Posture Management) tooling. rsaconference.com

RSA Conference 2025 (April 28 – May 1, San Francisco)

• 40% of 2,800+ session submissions were AI-related. Dominant theme: shift from GenAI to agentic AI systems. Key data point from SANS keynote: adversarial AI agent systems execute attack sequences 47× faster than human operators with 93% privilege escalation success rate.
• OWASP half-day event introduced the first OWASP Agentic Security Initiative guide: Agentic AI — Threats and Mitigations.
• NIST/MITRE joint session on progress toward a "Cyber AI" community profile under the AI RMF. rsaconference.com

RSA Conference 2024 (May 6–9, San Francisco)

• 100+ sessions on AI security. Primary themes: AI governance and responsible use, cybercriminal exploitation of GenAI (underground forums, AI-as-a-service for phishing and social engineering), and AI in security operations tooling. rsaconference.com

---

DEF CON AI Village

Website: aivillage.org/events | X: @aivillage_dc | YouTube: youtube.com/c/aivillage

DEF CON 32 (2024) — AI Village — aivillage.org/events/defcon32

• "garak: A Framework for Large Language Model Red Teaming" — Derczynski et al. (NVIDIA). The open-source LLM vulnerability scanner.
• "Evaluations and Guardrails Against Prompt Injection Attacks on LLM-Powered Applications" — Nikolaidis & Ahmad (Meta). CyberSecEval benchmarks + PromptGuard.
• "ConfusedPilot: Data Corruption and Leakage by Misusing Copilot for Microsoft 365" — RoyChowdhury et al. (UT Austin). arXiv:2408.04870
• "MITRE ATLAS: AI Adversary Tactics Knowledge Base" — Christina Liaghati (MITRE). Day 1 keynote.
• "AI'll be watching you: Greybox Attacks against an Embedded AI" — Tracey, Schulz, Bonner (HiddenLayer). Security camera AI bypass via adversarial inputs.
• "FuzzLLM" — Ian Harris (UC Irvine). Automated jailbreak fuzzing framework.
• "Your AI Assistant Has a Big Mouth: A New Side-Channel Attack" — Ben-Gurion University. Intercepts and decrypts encrypted AI assistant conversation streams via token-length side channel.
• "Taming the Beast: Inside the Llama 3 Red Team Process" — Meta AI Safety team.

DEF CON 31 (2023)

• The Generative Red Team Challenge — largest public LLM red-team event ever held. Thousands of participants attacked models from Anthropic, Google, Hugging Face, Meta, NVIDIA, OpenAI, and Stability AI simultaneously.

---

USENIX Security

Full proceedings free online: usenix.org/conferences/past

USENIX Security 2026 — Accepted AI security papers (conference Aug 12–14, Baltimore):

• "The Prompt Stealing Fallacy: Rethinking Metrics, Attacks, and Defenses" — Rethinks prompt extraction methodology; argues current metrics overstate attack success. usenix.org

USENIX Security 2025 — Accepted AI security papers:

• "PoisonedRAG: Knowledge Corruption Attacks to RAG" — Zou et al. arXiv:2402.07867
• "The Crescendo Multi-Turn Jailbreak Attack" — Russinovich et al. (Microsoft). arXiv:2404.01833

USENIX Security 2024 — Selected AI security papers:

• "Formalizing and Benchmarking Prompt Injection Attacks and Defenses" — Liu et al. First formal framework for PI evaluation. usenix.org
• "PentestGPT: An LLM-empowered Automatic Penetration Testing Tool" — Deng et al. Peer-reviewed evaluation of autonomous LLM pentest agents. usenix.org
• "CodeBreaker: LLM-Assisted Backdoor Attack on Code Completion Models" — Evades static analysis tools. arXiv:2406.06822

---

IEEE SaTML

Premier dedicated conference for ML security and trustworthiness. Annual. Full proceedings: satml.org

SaTML 2025 — Selected papers:

• "SoK: On the Offensive Potential of AI" — 14-author systematization of AI offensive capabilities across cyberattacks, influence operations, and physical-world threats. Project site
• "SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)" — Shows most LLM MIA evaluations are methodologically flawed. arXiv:2406.17975
• "Get My Drift? Catching LLM Task Drift with Activation Deltas" — Prompt injection detection via internal model activations. arXiv:2406.00799
• "SnatchML: Hijacking ML Models without Training Access" — Model hijacking without requiring access to training data or model weights. GitHub · IEEE Xplore
• Full list: satml.org/2025/accepted-papers

SaTML 2026 — Selected accepted papers:

• "CHAI: Command Hijacking against Embodied AI" — Prompt-based attack exploiting multimodal language interpretation vulnerabilities in vision-language models deployed in embodied systems. arXiv:2510.00181
• "Smudged Fingerprints: A Systematic Evaluation of the Robustness of AI Image Fingerprints" — Adversarial attacks on AI image fingerprinting and content provenance systems. arXiv:2512.11771
• "Position: Mind the Gap — Closing the Growing Disconnect Between Vulnerability Disclosure and AI Security" — IBM Research position paper on the gap between AI system vuln disclosure practices and security research. IBM Research
• Full list: satml.org/accepted-papers

SaTML 2024 — Best papers:

• "SoK: AI Auditing: The Broken Bus on the Road to AI Accountability" — Birhane et al. Critical analysis of AI audit methodologies. OpenReview
• "Data Redaction from Conditional Generative Models" — Kong & Chaudhuri. OpenReview
• Full list: satml.org/2024/accepted-papers

---

CAMLIS

Applied ML-for-security practitioner conference. Annual, Washington D.C. area. Proceedings and slides at camlis.org | YouTube: youtube.com/@camlis499

CAMLIS 2025 — Selected talks:

• "ShadowLogic: Hidden Backdoors in Any Whitebox LLM" — Amelia Kawasaki. Persistent backdoor insertion into LLMs without modifying weights. camlis.org
• "A Framework for Adaptive Multi-Turn Jailbreak Attacks on LLMs" — Javad Rafiei Asl. Automated multi-turn attack generation adapting to model defenses. camlis.org
• "LLM Salting: From Rainbow Tables to Jailbreaks" — Tamás Vörös. Pre-computation attacks on LLM safety filters. camlis.org
• "Attack Surfaces in Computer Use Agents: A Practical Taxonomy" — Daniel Jones. Systematic taxonomy of attack surfaces specific to computer-use AI agents. camlis.org (CAMLIS RED Track)
• "Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities" — Arjun Krishna. Quantifies hallucinated package names exploitable for supply chain attacks. camlis.org

CAMLIS 2024 — Selected talks:

• "PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI Systems" — Gary Lopez Munoz (Microsoft). Introduction of PyRIT, Microsoft's open-source LLM red teaming framework. camlis.org
• "Defending Against Indirect Prompt Injection Attacks With Spotlighting" — Keegan Hines. Input marking technique that separates trusted instructions from untrusted data. camlis.org
• "LLM Backdoor Activations Stick Together" — Tamás Vörös. Activation-space analysis to detect backdoored LLMs. camlis.org
• "LLM Agents for Vulnerability Identification and Verification of CVEs" — Rodrigo Bersa & Tadesse Zemichael. Automated CVE reproduction and triage using LLM agents. camlis.org

CAMLIS 2023 — Selected talks:

• "Security Issues in Generative AI" — Tom Goldstein (University of Maryland). Foundational adversarial ML issues in LLMs. camlis.org
• "LLM Prompt Injection: Attacks and Defenses" — Gary Lopez Munoz. Early practitioner-focused treatment of prompt injection attack taxonomy and mitigations. camlis.org
• "Model Leeching: An Extraction Attack Targeting LLMs" — Lewis Birch. Model extraction methodology adapted for large language models. camlis.org

---

Talk Archives on YouTube

---

4. Tools — Offense & Red Teaming

LLM Red Teaming

Adversarial ML Tools

Agentic & MCP Attack Tools

Benchmarks & Evaluation

Vulnerable Labs & CTFs

Hands-on practice environments for AI security skills.

---

5. Tools — Defense & Detection

Guardrails & Output Safety

Model & Supply Chain Security

Format gap: ModelScan, Fickling, and picklescan cover pickle, PyTorch .pt, TF SavedModel, and Keras formats — but not GGUF (the dominant format for llama.cpp-based local model serving: Ollama, LM Studio). No production-ready security scanner covers GGUF as of early 2026.

Production Monitoring

Agent Runtime Security & Sandboxing

Tools for isolating agent execution and enforcing policy over what agents can access, write, or exfiltrate.

MCP Security

AI Code Security

Privacy-Preserving Inference

---

6. AI for Security Operations

Tools that USE AI to perform security work. For tools that secure AI systems, see Section 5.

Penetration Testing & Offensive Security

Malware Analysis & Reverse Engineering

Vulnerability Research

Threat Intelligence & SOC

Security-Specialized Models

Safety classifiers and prompt injection detectors:

---

7. Notable Incidents & CVEs

A timeline of publicly documented attacks, exploits, and real-world AI security incidents. Useful for threat modeling impact assessments, building case studies, and tracking the evolving threat landscape.

2026

2025

Tracking resources:

• TalEliyahu Disclosed AI Vulnerabilities Tracker — Named AI vulnerability tracker with CVEs, descriptions, and sources
• Vulnerable MCP Project — Live MCP ecosystem CVE database
• ProtectAI Sightline — AI/ML infrastructure CVE tracker with PoC exploits
• AI Incident Database — Crowdsourced real-world AI failure database
• AIAAIC Repository — AI incidents, controversies, and accountability failures

---

8. Attack Frameworks & Knowledge Bases

---

9. Defensive Frameworks & Standards

Risk Management

Verification Standards

Threat Modeling

Incident Response

---

10. Regulatory & Compliance

Compliance Deadline Timeline

EU AI Act (Regulation 2024/1689)

Risk-tiered: Unacceptable (banned) → High-Risk → Limited-Risk → Minimal Risk.

Key articles for security practitioners:

Full text: eur-lex.europa.eu

Key Regulatory References

---

11. Community & Practice

Communities & Organizations

Key Practitioners to Follow

Researchers with consistent, high-signal output on AI security:

Conferences & Venues

Bug Bounty Programs

Open-source AI/ML — Huntr (ProtectAI): The primary bug bounty platform for AI/ML open-source projects. Reports go to maintainers of NumPy, scikit-learn, Hugging Face Transformers, and others. Purpose-built for AI/ML vulnerability classes: deserialization, supply chain, model loading bugs.

Corporate programs:

Research Blogs Worth Following

Consistently high-signal AI security research output:

Newsletters & Podcasts

---

Datasets

AI security–relevant datasets for training, evaluation, and red teaming.

Safety & Attack Datasets

Cybersecurity Skill Benchmarks

CTF challenge datasets for evaluating AI agents' offensive security capabilities.

---

Agentic AI Security Skills

Skills (plugins) for AI coding assistants (Claude Code, Gemini CLI, Cursor, Copilot) that add security capabilities — scanning, threat modeling, vulnerability detection, and audit workflows.

---

Contributions welcome. If you know of a paper, tool, or talk that belongs here, open a PR.