Awesome List of Cybersecurity and AI

Below a so-called awesome list with background reading accompanying my talk AI in Offensive and Defensive Cyber at the 2024 One Conference in The Hague. The Hype Cycle section aggregates core themes, the other sections provide for further exploration of specific topics.

Hype Cycle

• Security of AI Embeddings
  • Common Pitfalls To Avoid When Using Vector Databases
  • A look at the Pinecone vector database from a privacy, security, and risk management perspective
• Model-stealing Attacks
  • Stealing Part of a Production Language Model
• AI Sleeper Agents
  • Two-faced AI language models learn to hide deception
  • Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
• Greedy Coordinate Gradient
  • Accelerating Greedy Coordinate Gradient via Probe Sampling
  • Boosting Jailbreak Attack with Momentum
• AI Bill of Materials (AIBOMs)
  • Manifest AIBOM (MLBOM) Wiki
  • Trail of Bits' response to the US Army’s RFI on developing AIBOM tools
• LLM Jacking
  • LLMjacking: Stolen Cloud Credentials Used in New AI Attack
• Open Source Pair Programmers
  • Devon: An open-source pair programmer
• LLM 0/1-Day Exploitation
  • No, LLM Agents can not Autonomously Exploit One-day Vulnerabilities
  • Project Naptime: Evaluating Offensive Security Capabilities of Large Language Models
• LLM Proxies and Load Balancers
  • LiteLLM
  • Networking The AI Data Center
• LLM-Based (Malware) Pseudocode Analysis
  • MALSIGHT: Exploring Malicious Source Code and Benign Pseudocode for Iterative Binary Malware Summarization
  • Large Language Models for Malware Analysis
• Dark LLMs and Blackhat GPTs
  • Malla: Demystifying Real-world Large Language Model Integrated Malicious Services
• Backdoored Model Files
  • Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor
  • Model Files are Invisible Viruses
• Mechanistic Interpretability
  • Mapping the Mind of a Large Language Model
  • Mechanistic Interpretability for AI Safety - A Review
  • Logit Prisms: Decomposing Transformer Outputs for Mechanistic Interpretability
• AI-Based Intrusion Detection
  • Outside The Closed World: On Using Machine Learning for Network Intrusion Detection
  • IDS-ML: Intrusion Detection System Development Using Machine Learning
• LLM-Based Honeypots
  • LLM in the Shell: Generative Honeypots
  • Galah: LLM-powered web honeypot designed to mimic various applications
• LLM-Based Social Engineering
  • Disrupting deceptive uses of AI by covert influence operations
  • AI Will Increase the Quantity—and Quality—of Phishing Scams
  • Digital Deception: Generative Artificial Intelligence in Social Engineering and Phishing
• Machine Unlearning
  • Verifying in the Dark: Verifiable Machine Unlearning by Using Invisible Backdoor Triggers
  • Backdoors Against Natural Language Processing: A Review
  • Machine Unlearning in 2024
• Prompt Injection
  • LLMs’ Data-Control Path Insecurity
  • Hacking Google Bard - From Prompt Injection to Data Exfiltration
  • Prompt Injection Defenses

---

Policies

• What policy makers need to know about AI (and what goes wrong if they don’t)
• NCSC-NL: The future of cyber attacks with Large Language Models
• The EU Artificial Intelligence Act
• The EU AI Act: National Security Implications
• NCSC-UK: Cyber security risks to artificial intelligence: findings
• NCSC-UK: Guidelines for secure AI system development
• National Security Agency (NSA): Deploying AI Systems Securely: Best Practices for Deploying Secure and Resilient AI Systems
• Australian Government: Policy for the responsible use of AI in government
• AI Safety Ethics & Society: Safe Design Principles
• Verzamelbrief AI en Algoritmes
• RDI (NL): Toezicht op kunstmatige intelligentie
• Monetary Authority of Singapore: Cyber Risks Associated with Generative Artificial Intelligence

---

Frameworks

• MITRE: ATLAS, a globally accessible, living knowledge base of adversary tactics and techniques against Al-enabled systems
• CyberSecEval 2 - A Comprehensive Evaluation Framework for Cybersecurity Risks and Capabilities of Large Language Models
• AI Vulnerability Database (AVID)
• AI Risk Repository: database of over 700 AI risks categorized by their cause and risk domain
• RAID (Real World AI Definitions)

---

Defensive Use of AI (Blue Teams)

• Leveraging AI for efficient incident response
• Large Language Models and Intelligence Analysis
• Cybersecurity & ChatGPT - Part 2 - Generative AI for Blue Teams
• The Power of Artificial Intelligence - From Search to Detection Rule at Light Speed
• Analyzing Malware in Binaries and Executables with AI
• Defending AI Model Files from Unauthorized Access with Canaries
• Prompt Injection Defenses
• Prepare for AI Hackers
• Automated LLM Bugfinders
• Applying LLMs to Threat Intelligence
• STRIDE GPT: AI-powered threat modelling leveraging LLMs to generate threat models and attack trees
• Sinon: creates Windows-based deception hosts and uses GPT-4 to generate content (files, emails, etc.)
• AI-Goat: a deliberately vulnerable AI infrastructure simulating the OWASP ML Top 10
• Where the Wild Things Are: Second Order Risks of AI

---

Offensive Use of AI (Red Teams)

• Cybersecurity & ChatGPT - Part 3 - Generative AI for Red Teams
• Generative AI in Cybersecurity: Assessing impact on current and future malicious software
• Supply chain attacks and the many different ways I've backdoored your dependencies
• Machine Learning Attack Series: Backdooring Keras Models and How to Detect It
• AI-Powered Bug Hunting - Evolution and Benchmarking
• Automatic Tool Invocation when Browsing with ChatGPT - Threats and Mitigations
• Adversary use of Artifical Intelligence and LLMs and Classification of TTPs
• Red Teaming LLM Applications

---

Academic Research

• Zhang, Andy K., et al. "Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models.", 2024
• Miles, Farmer, et al. "Reinforcement Learning for Autonomous Resilient Cyber Defence." Black Hat USA 2024, Black Hat, 2024.]
• Mirsky, Yisroel, et al. "The threat of offensive ai to organizations." Computers & Security 124 (2023): 103006.
• Park, Peter S., et al. "AI deception: A survey of examples, risks, and potential solutions." Patterns 5.5 (2024).
• Marchal, Nahema, et al. "Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data." arXiv preprint arXiv:2406.13843 (2024).
• Bezzi, Michele. "Large Language Models and Security." IEEE Security & Privacy (2024).
• Bhatt, Manish, et al. "Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models." arXiv preprint arXiv:2404.13161 (2024).
• Derczynski, Leon, et al. "garak: A Framework for Security Probing Large Language Models." arXiv preprint arXiv:2406.11036 (2024).
• Anwar, Usman, et al. "Foundational challenges in assuring alignment and safety of large language models." arXiv preprint arXiv:2404.09932 (2024).
• Gibney, Elizabeth. "Not all 'open source' AI models are actually open: here's a ranking." Nature. 2024
• Acemoglu, Daron. "The Simple Macroeconomics of AI." Massachusetts Institute of Technology, 2024.

---

Whitepapers

• Goldman Sachs Research. Gen AI: too much spend, too little benefit? 2024.
• Oprea, Alina, and Apostol Vassilev. Adversarial machine learning: A taxonomy and terminology of attacks and mitigations. No. NIST Artificial Intelligence (AI) 100-2 E2023. National Institute of Standards and Technology, 2023.
• Stanford University. 2024 AI Index Report: Measuring Trends in AI

---

Miscellaneous

• CIA director Bill Burns and MI6 chief Richard Moore talk to FT editor Roula Khalaf
• AI Model Review: Compare AI Models with Prompts Side by Side
• Cognitive Attack Taxonomy (CAT) of over 350 cognitive vulnerabilities, exploits, and TTPs
• Blog series by Harriet Farlow on the intersection of AI and national security
• I Will Fucking Piledrive You If You Mention AI Again
• How LLMs Work, Explained Without Math
• Lessons after a half-billion GPT tokens
• The “it” in AI models is the dataset
• Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy