README
⭐ If you find Awesome GenAI CyberHub useful, please consider giving it a ⭐ to help it grow and reach more people! ⭐
🚀 Awesome GenAI CyberHub ✨
A curated collection of the best LLM-based resources for cybersecurity practitioners, researchers, and enthusiasts.
Welcome to Awesome GenAI CyberHub, your focused repository for Agentic AI Large Language Model (LLM) applications in cybersecurity. Below you’ll find links to articles, papers, PoCs, frameworks, datasets, and community projects organized by topic, making it easy to navigate the ever-expanding landscape of LLM‑powered security.
📋 Table of Contents
- About
- Topic-Based Directory Structure
- Resource Listings
- How to Navigate & Contribute
- License
- Contact
About
This repo aggregates the most impactful LLM‑centric work in cybersecurity, grouped by real‑world problem domains. Whether you’re building an autonomous SOC analyst, crafting threat profiles, or automating detection rule writing or exploring offensive workflows like red teaming, jump straight to the topic area that matters most.
Topic-Based Directory Structure
awesome-genai-cyberhub/
├── Resources/
├── ai-soc/ # Agentic AI for SOC workflows
├── cti/ # Cyber Threat Intelligence via LLMs
├── malware-analysis/ # LLM driven malware Analysis
├── vulnerability-analysis/ # LLMs in vuln discovery
├── detection-engineering/ # Generative AI rule writing & alerts
├── threat-hunting/ # AI-driven threat hunting & hypothesis gen
├── phishing-analysis/ # Phishing Analysis & detection
├── honeypots/ # LLM-driven honeypot frameworks
├── offensive-security/ # Offensive security workflows
└── digital-forensics/ # LLM applications in Digital Forensics
Resource Listings
🏅 General
-
📄 Generative AI in Cybersecurity: A Comprehensive Review of LLM Applications and Vulnerabilities [Paper/Review] - Explores LLM applications in hardware security, intrusion detection, CTI, malware/phishing detection, LLM vulnerabilities (prompt injection, data poisoning), mitigation strategies, and evaluates LLMs in cybersecurity knowledge.
-
📄 A Comprehensive Review of Large Language Models in Cyber Security [Paper/Review] - Reviews LLM applications in cybersecurity, focusing on their potential in threat detection domains like CTI, phishing detection, and log analysis, while also discussing challenges and future directions.
-
📄 Large Language Models in Cybersecurity: State-of-the-Art [Paper/Review] - Provides a characterization of both defensive (mapped to NIST framework) and adversarial (mapped to MITRE ATT&CK) applications of LLMs in cybersecurity, identifying research gaps.
-
📄 Large Language Models for Cyber Security: A Systematic Literature Review [Paper/Review] - A systematic literature review analyzing over 185 papers to map the current landscape of LLM applications across various cybersecurity tasks, including vulnerability detection, malware analysis, and phishing detection.
-
🌐 An Introduction to AI in Cybersecurity [Guide/Resource] - An introductory resource discussing GenAI for cybersecurity, covering various topics including CVE enrichment and CWE assignment.
-
🤖 Cyber Security llm Agents [Tool/GitHub] - A collection of agents using LLMs (built on AutoGen) to perform common cybersecurity tasks, showcased at RSAC 2024.
-
📚 awesome-foundation-agents [Awesome List/GitHub] - Curated list of foundation-model agent resources (general agent ecosystem context).
-
🛠️ AttackGen [Tool/Website] - Interactive helper for generating/structuring attack content and scenarios.
-
🤖 CAI (aliasrobotics/cai) [Tool/GitHub] - Cybersecurity-focused AI tooling from Alias Robotics (automation/agent experiments).
-
📄 A Comprehensive Overview of Large Language Models (LLMs) for Cyber Defences: Opportunities and Directions [Paper/Review] - Surveys LLM applications in cyber defense, categorizing them into threat intelligence, vulnerability assessment, network security, privacy, training, automation, and ethical guidelines. Also covers LLM concepts from Transformers to GPT.
-
📄 From Texts to Shields: Convergence of Large Language Models and Cybersecurity [Paper/Report] - Explores the convergence of LLMs and cybersecurity, synthesizing insights from network security, AI, formal methods, and human-centered design. It examines LLM applications in software/network security, 5G vulnerability analysis, generative security engineering, and the role of agentic LLMs.
-
📄 When LLMs meet cybersecurity: a systematic literature review [Paper/Review] - A systematic literature review of over 300 works, covering 25+ LLMs and 10+ downstream scenarios, addressing the construction of cybersecurity-oriented LLMs, their applications, challenges, and future research.
-
📄 Application of Large Language Models in Cybersecurity: a Systematic Literature Review [Paper/Review] - A systematic literature review of 177 articles (2018-2024) on LLM applications in offensive/defensive cybersecurity, cyberethics, legal frameworks, and cybersecurity governance.
-
📄 Cyber Shadows: Neutralizing Security Threats with AI and Targeted Policy Measures (28 Jan 2025) [Paper]
-
📄 Comparative Analysis of AI-Driven Security Approaches in DevSecOps (arXiv:2504.19154) [Paper]
-
📄 The Dual Role of Artificial Intelligence and LLM in Cybersecurity (IEEE) [Paper] - Examines the dual offensive and defensive role of AI and LLMs in cybersecurity, covering both how AI strengthens defenses and how it can be weaponized by adversaries.
-
📄 Large Language Models for Cybersecurity Intelligence: A Systematic Review of Emerging Threats, Defensive Capabilities, and Security Evaluation Frameworks (Computers, Materials & Continua, 2026) [Paper/Review] - A PRISMA 2020 systematic review spanning eight scholarly databases (Jan 2022 – Dec 2025), mapping LLM-enabled threats, defensive capabilities, and the evaluation frameworks used to measure them.
-
📰 Threat modeling with LLMs: two years in (hype, hope, and a look at Gemini 2.5 Pro) [Blog]
🥇 MCP Servers (Model Context Protocol)
This section covers resources related to Model Context Protocol (MCP) servers for security. These systems facilitate more complex and coordinated AI-driven security operations.
- 🖥️ ORKL MCP Server [Github] - A Model Context Protocol (MCP) server for querying the ORKL API.
- 🖥️ VirusTotal MCP Server [Smithery] - A Model Context Protocol (MCP) server for querying the VirusTotal API.
- 🖥️ OTX Alien Vault MCP Server [Smithery] - A Model Context Protocol (MCP) server for querying the Alien Vault API.
- 🖥️ RSTCloud MCP Server [Smithery] - A Model Context Protocol (MCP) server for querying the RSTCloud API.
- 🖥️ OpenCTI MCP Server [Github] - A Model Context Protocol (MCP) server for orchestrating and querying OpenCTI via LLM agents.
- 🖥️ External Reconnaissance MCP Server [Github] - A Model Context Protocol (MCP) server for performing active external reconnaissance activities against a domain.
🥈 Models
-
🧠 SecureBERT [Model/HuggingFace] - A BERT model pre-trained on a vast corpus of cybersecurity texts.
-
🧠 Lily Cybersecurity 7B v0.2 [Model/HuggingFace] - A 7B parameter Mistral fine-tune, acting as a cybersecurity assistant trained on approximately 22,000 hand-crafted cybersecurity and hacking-related data pairs.
-
🧠 Foundation Sec 8B [Model/HuggingFace] - An 8-billion parameter base language model (extending Llama-3.1-8B) specialized for cybersecurity applications through continued pretraining on a curated corpus of cybersecurity-specific text.
-
🧠 Antares Model Collection (fdtn-ai) [Collection/HuggingFace] - A collection of specialized open cybersecurity models developed by Foundation AI (fdtn-ai).
-
🧠 Google launches Sec-Gemini 1.0, a new AI model to advance cybersecurity (Google Security Blog) [Blog/Model Announcement] - Announces Sec-Gemini v1.0, an experimental AI model focused on cybersecurity, combining Gemini's advanced capabilities with near real-time cybersecurity knowledge.
-
🧠 PRIMUS: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training [Paper/Dataset/Model]
🥉 Benchmarks & Datasets
-
📊 SECURE: Benchmarking LLMs for Cybersecurity [Paper/Benchmark]
-
📊 CTIBench: Benchmark for Evaluating LLMs in CTI [Paper/Benchmark]
-
📊 CyberSecEval 2: Wide-Ranging Cybersecurity Evaluation Suite [Paper/Benchmark]
-
📊 CS-Eval: Comprehensive LLM Benchmark for Cybersecurity [Paper/Benchmark]
-
📊 CyberBench: A Multi-Task Benchmark for Evaluating Large Language Models in Cybersecurity [Paper/Benchmark/GitHub] GitHub: jpmorganchase/CyberBench
-
📊 Crashbench: A LLM benchmark to measure bug-finding and reporting capabilities of LLMs [Benchmark/GitHub]
-
📊 Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models [Benchmark/Framework/Paper] See also: [Project Page] (https://cybench.github.io/)
-
📊 PhishLLM [Benchmark/Tool/GitHub] See also: Project Page
-
📊 SEvenLLM: Benchmarking, Eliciting, and Enhancing Abilities of Large Language Models in Cyber Threat Intelligence [Benchmark/Dataset/GitHub]
-
📊 NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security [Benchmark/Dataset/GitHub]
-
📊 Benchmarking Foundation Models in Cybersecurity: A Study on OpenAI's o1 [Paper/Benchmark]
-
📊 CyberLLMInstruct: A New Dataset for Analysing Safety of Fine-Tuned LLMs Using Cyber Security Data [Dataset/Paper]
-
📊 TM-Bench: A Benchmark for LLM-Based Threat Modeling [Benchmark/Website]
-
📊 SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity [Benchmark/Dataset/Paper]
-
📊 Introducing AI Cyber Model Arena: a real-world benchmark for AI agents in cybersecurity [Blog]
-
📊 ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense [Paper/Benchmark]
-
📊 CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale [Paper/Benchmark]
-
📊 CyberGym-E2E (sunblaze-ucb) [Benchmark/GitHub] - A large-scale benchmark built from real-world vulnerabilities in widely used open-source projects to evaluate AI agents' end-to-end cybersecurity capabilities (from discovering vulnerabilities to generating PoCs to writing patches).
-
📊 SANDBOXESCAPEBENCH: Quantifying Frontier LLM Capabilities for Container Sandbox Escape [Paper/Benchmark] - Introduces an open benchmark that safely measures an LLM agent's capacity to break out of sandboxes, implemented as a CTF evaluation utilizing a nested sandbox architecture.
-
📊 Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agents [Paper/Benchmark]
-
📊 Benchmark Best Practices for Evaluating AI Agents in Cybersecurity [Paper]
-
📊 Evaluating Large Language Models in Cybersecurity: A Systematic Taxonomy and Empirical Analysis (MDPI Electronics, May 2026) [Paper/SoK] - A systematization of knowledge proposing a three-dimensional taxonomy matrix over offensive domains, defensive applications, and inherent architectural flaws, showing how static metric success rarely translates into end-to-end adversarial efficacy.
-
📊 FAITH (Foundation AI Testing Hub) [Benchmark/GitHub] - Benchmarking tool for assessing LLM competency on cybersecurity knowledge and tasks.
-
📊 BotsBench [Benchmark/Leaderboard] - Live leaderboard benchmarking AI agents (Claude Code, OpenAI Codex, etc.) on SOC investigation tasks using the Splunk BOTSv3 corpus. Covers 100+ log/alert providers, ATT&CK-mapped questions, efficiency tradeoffs, and contamination hygiene. By Graphistry. GitHub: graphistry/bots-bench.
-
📊 DefenseBench [Benchmark/Website] - Evaluates how well AI agents perform real-world defensive cybersecurity tasks — from triaging alerts to investigating incidents. Currently benchmarks agents on the Splunk Boss of the SOC v3 (BOTSv3) dataset using Splunk search under time pressure.
-
📊 EVMBench (paradigmxyz) [Benchmark/Tool/GitHub] - A benchmark and harness for finding and exploiting smart contract bugs.
-
📊 StealthBench [Benchmark/GitHub] - A benchmark for evaluating whether autonomous agents operate with operational security (OPSEC) discipline during security engagements.
-
📊 CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research? [Paper/Benchmark] - An expert-annotated benchmark collected from the daily CTI workflow of a world-leading company to assess LLMs on practical threat intelligence tasks (triage, deep search, and TI drafting).
-
📊 AgentCyberRange [Benchmark/Website] - An open, multi-range infrastructure for measuring autonomous cyber attack capability across web exploitation and post-exploitation stages. See also the Research Paper (arXiv:2606.14295) and the WebExploitBench GitHub Repo.
How to Navigate & Contribute
- Browse the topic folder that matches your use case.
- Dive into its README for detailed descriptions and annotations, including resource type tags.
- To add a resource, open a PR in the appropriate folder’s
README.md. - Follow existing formatting and include the identifier tag (e.g., [Paper], [Blog], [GitHub], [Tool], [Video], [Model], [Dataset]).
📘 See CONTRIBUTING.md for full guidelines, style tips, and how to submit your PR.
License
Distributed under the MIT License. See MIT License for details.
Contact
For suggestions or PRs, open an issue or reach out to me at ashfaaqf@proton.me .
Similar repos
README
⭐ If you find Awesome GenAI CyberHub useful, please consider giving it a ⭐ to help it grow and reach more people! ⭐
🚀 Awesome GenAI CyberHub ✨
A curated collection of the best LLM-based resources for cybersecurity practitioners, researchers, and enthusiasts.
Welcome to Awesome GenAI CyberHub, your focused repository for Agentic AI Large Language Model (LLM) applications in cybersecurity. Below you’ll find links to articles, papers, PoCs, frameworks, datasets, and community projects organized by topic, making it easy to navigate the ever-expanding landscape of LLM‑powered security.
📋 Table of Contents
- About
- Topic-Based Directory Structure
- Resource Listings
- How to Navigate & Contribute
- License
- Contact
About
This repo aggregates the most impactful LLM‑centric work in cybersecurity, grouped by real‑world problem domains. Whether you’re building an autonomous SOC analyst, crafting threat profiles, or automating detection rule writing or exploring offensive workflows like red teaming, jump straight to the topic area that matters most.
Topic-Based Directory Structure
awesome-genai-cyberhub/
├── Resources/
├── ai-soc/ # Agentic AI for SOC workflows
├── cti/ # Cyber Threat Intelligence via LLMs
├── malware-analysis/ # LLM driven malware Analysis
├── vulnerability-analysis/ # LLMs in vuln discovery
├── detection-engineering/ # Generative AI rule writing & alerts
├── threat-hunting/ # AI-driven threat hunting & hypothesis gen
├── phishing-analysis/ # Phishing Analysis & detection
├── honeypots/ # LLM-driven honeypot frameworks
├── offensive-security/ # Offensive security workflows
└── digital-forensics/ # LLM applications in Digital Forensics
Resource Listings
🏅 General
-
📄 Generative AI in Cybersecurity: A Comprehensive Review of LLM Applications and Vulnerabilities [Paper/Review] - Explores LLM applications in hardware security, intrusion detection, CTI, malware/phishing detection, LLM vulnerabilities (prompt injection, data poisoning), mitigation strategies, and evaluates LLMs in cybersecurity knowledge.
-
📄 A Comprehensive Review of Large Language Models in Cyber Security [Paper/Review] - Reviews LLM applications in cybersecurity, focusing on their potential in threat detection domains like CTI, phishing detection, and log analysis, while also discussing challenges and future directions.
-
📄 Large Language Models in Cybersecurity: State-of-the-Art [Paper/Review] - Provides a characterization of both defensive (mapped to NIST framework) and adversarial (mapped to MITRE ATT&CK) applications of LLMs in cybersecurity, identifying research gaps.
-
📄 Large Language Models for Cyber Security: A Systematic Literature Review [Paper/Review] - A systematic literature review analyzing over 185 papers to map the current landscape of LLM applications across various cybersecurity tasks, including vulnerability detection, malware analysis, and phishing detection.
-
🌐 An Introduction to AI in Cybersecurity [Guide/Resource] - An introductory resource discussing GenAI for cybersecurity, covering various topics including CVE enrichment and CWE assignment.
-
🤖 Cyber Security llm Agents [Tool/GitHub] - A collection of agents using LLMs (built on AutoGen) to perform common cybersecurity tasks, showcased at RSAC 2024.
-
📚 awesome-foundation-agents [Awesome List/GitHub] - Curated list of foundation-model agent resources (general agent ecosystem context).
-
🛠️ AttackGen [Tool/Website] - Interactive helper for generating/structuring attack content and scenarios.
-
🤖 CAI (aliasrobotics/cai) [Tool/GitHub] - Cybersecurity-focused AI tooling from Alias Robotics (automation/agent experiments).
-
📄 A Comprehensive Overview of Large Language Models (LLMs) for Cyber Defences: Opportunities and Directions [Paper/Review] - Surveys LLM applications in cyber defense, categorizing them into threat intelligence, vulnerability assessment, network security, privacy, training, automation, and ethical guidelines. Also covers LLM concepts from Transformers to GPT.
-
📄 From Texts to Shields: Convergence of Large Language Models and Cybersecurity [Paper/Report] - Explores the convergence of LLMs and cybersecurity, synthesizing insights from network security, AI, formal methods, and human-centered design. It examines LLM applications in software/network security, 5G vulnerability analysis, generative security engineering, and the role of agentic LLMs.
-
📄 When LLMs meet cybersecurity: a systematic literature review [Paper/Review] - A systematic literature review of over 300 works, covering 25+ LLMs and 10+ downstream scenarios, addressing the construction of cybersecurity-oriented LLMs, their applications, challenges, and future research.
-
📄 Application of Large Language Models in Cybersecurity: a Systematic Literature Review [Paper/Review] - A systematic literature review of 177 articles (2018-2024) on LLM applications in offensive/defensive cybersecurity, cyberethics, legal frameworks, and cybersecurity governance.
-
📄 Cyber Shadows: Neutralizing Security Threats with AI and Targeted Policy Measures (28 Jan 2025) [Paper]
-
📄 Comparative Analysis of AI-Driven Security Approaches in DevSecOps (arXiv:2504.19154) [Paper]
-
📄 The Dual Role of Artificial Intelligence and LLM in Cybersecurity (IEEE) [Paper] - Examines the dual offensive and defensive role of AI and LLMs in cybersecurity, covering both how AI strengthens defenses and how it can be weaponized by adversaries.
-
📄 Large Language Models for Cybersecurity Intelligence: A Systematic Review of Emerging Threats, Defensive Capabilities, and Security Evaluation Frameworks (Computers, Materials & Continua, 2026) [Paper/Review] - A PRISMA 2020 systematic review spanning eight scholarly databases (Jan 2022 – Dec 2025), mapping LLM-enabled threats, defensive capabilities, and the evaluation frameworks used to measure them.
-
📰 Threat modeling with LLMs: two years in (hype, hope, and a look at Gemini 2.5 Pro) [Blog]
🥇 MCP Servers (Model Context Protocol)
This section covers resources related to Model Context Protocol (MCP) servers for security. These systems facilitate more complex and coordinated AI-driven security operations.
- 🖥️ ORKL MCP Server [Github] - A Model Context Protocol (MCP) server for querying the ORKL API.
- 🖥️ VirusTotal MCP Server [Smithery] - A Model Context Protocol (MCP) server for querying the VirusTotal API.
- 🖥️ OTX Alien Vault MCP Server [Smithery] - A Model Context Protocol (MCP) server for querying the Alien Vault API.
- 🖥️ RSTCloud MCP Server [Smithery] - A Model Context Protocol (MCP) server for querying the RSTCloud API.
- 🖥️ OpenCTI MCP Server [Github] - A Model Context Protocol (MCP) server for orchestrating and querying OpenCTI via LLM agents.
- 🖥️ External Reconnaissance MCP Server [Github] - A Model Context Protocol (MCP) server for performing active external reconnaissance activities against a domain.
🥈 Models
-
🧠 SecureBERT [Model/HuggingFace] - A BERT model pre-trained on a vast corpus of cybersecurity texts.
-
🧠 Lily Cybersecurity 7B v0.2 [Model/HuggingFace] - A 7B parameter Mistral fine-tune, acting as a cybersecurity assistant trained on approximately 22,000 hand-crafted cybersecurity and hacking-related data pairs.
-
🧠 Foundation Sec 8B [Model/HuggingFace] - An 8-billion parameter base language model (extending Llama-3.1-8B) specialized for cybersecurity applications through continued pretraining on a curated corpus of cybersecurity-specific text.
-
🧠 Antares Model Collection (fdtn-ai) [Collection/HuggingFace] - A collection of specialized open cybersecurity models developed by Foundation AI (fdtn-ai).
-
🧠 Google launches Sec-Gemini 1.0, a new AI model to advance cybersecurity (Google Security Blog) [Blog/Model Announcement] - Announces Sec-Gemini v1.0, an experimental AI model focused on cybersecurity, combining Gemini's advanced capabilities with near real-time cybersecurity knowledge.
-
🧠 PRIMUS: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training [Paper/Dataset/Model]
🥉 Benchmarks & Datasets
-
📊 SECURE: Benchmarking LLMs for Cybersecurity [Paper/Benchmark]
-
📊 CTIBench: Benchmark for Evaluating LLMs in CTI [Paper/Benchmark]
-
📊 CyberSecEval 2: Wide-Ranging Cybersecurity Evaluation Suite [Paper/Benchmark]
-
📊 CS-Eval: Comprehensive LLM Benchmark for Cybersecurity [Paper/Benchmark]
-
📊 CyberBench: A Multi-Task Benchmark for Evaluating Large Language Models in Cybersecurity [Paper/Benchmark/GitHub] GitHub: jpmorganchase/CyberBench
-
📊 Crashbench: A LLM benchmark to measure bug-finding and reporting capabilities of LLMs [Benchmark/GitHub]
-
📊 Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models [Benchmark/Framework/Paper] See also: [Project Page] (https://cybench.github.io/)
-
📊 PhishLLM [Benchmark/Tool/GitHub] See also: Project Page
-
📊 SEvenLLM: Benchmarking, Eliciting, and Enhancing Abilities of Large Language Models in Cyber Threat Intelligence [Benchmark/Dataset/GitHub]
-
📊 NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security [Benchmark/Dataset/GitHub]
-
📊 Benchmarking Foundation Models in Cybersecurity: A Study on OpenAI's o1 [Paper/Benchmark]
-
📊 CyberLLMInstruct: A New Dataset for Analysing Safety of Fine-Tuned LLMs Using Cyber Security Data [Dataset/Paper]
-
📊 TM-Bench: A Benchmark for LLM-Based Threat Modeling [Benchmark/Website]
-
📊 SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity [Benchmark/Dataset/Paper]
-
📊 Introducing AI Cyber Model Arena: a real-world benchmark for AI agents in cybersecurity [Blog]
-
📊 ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense [Paper/Benchmark]
-
📊 CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale [Paper/Benchmark]
-
📊 CyberGym-E2E (sunblaze-ucb) [Benchmark/GitHub] - A large-scale benchmark built from real-world vulnerabilities in widely used open-source projects to evaluate AI agents' end-to-end cybersecurity capabilities (from discovering vulnerabilities to generating PoCs to writing patches).
-
📊 SANDBOXESCAPEBENCH: Quantifying Frontier LLM Capabilities for Container Sandbox Escape [Paper/Benchmark] - Introduces an open benchmark that safely measures an LLM agent's capacity to break out of sandboxes, implemented as a CTF evaluation utilizing a nested sandbox architecture.
-
📊 Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agents [Paper/Benchmark]
-
📊 Benchmark Best Practices for Evaluating AI Agents in Cybersecurity [Paper]
-
📊 Evaluating Large Language Models in Cybersecurity: A Systematic Taxonomy and Empirical Analysis (MDPI Electronics, May 2026) [Paper/SoK] - A systematization of knowledge proposing a three-dimensional taxonomy matrix over offensive domains, defensive applications, and inherent architectural flaws, showing how static metric success rarely translates into end-to-end adversarial efficacy.
-
📊 FAITH (Foundation AI Testing Hub) [Benchmark/GitHub] - Benchmarking tool for assessing LLM competency on cybersecurity knowledge and tasks.
-
📊 BotsBench [Benchmark/Leaderboard] - Live leaderboard benchmarking AI agents (Claude Code, OpenAI Codex, etc.) on SOC investigation tasks using the Splunk BOTSv3 corpus. Covers 100+ log/alert providers, ATT&CK-mapped questions, efficiency tradeoffs, and contamination hygiene. By Graphistry. GitHub: graphistry/bots-bench.
-
📊 DefenseBench [Benchmark/Website] - Evaluates how well AI agents perform real-world defensive cybersecurity tasks — from triaging alerts to investigating incidents. Currently benchmarks agents on the Splunk Boss of the SOC v3 (BOTSv3) dataset using Splunk search under time pressure.
-
📊 EVMBench (paradigmxyz) [Benchmark/Tool/GitHub] - A benchmark and harness for finding and exploiting smart contract bugs.
-
📊 StealthBench [Benchmark/GitHub] - A benchmark for evaluating whether autonomous agents operate with operational security (OPSEC) discipline during security engagements.
-
📊 CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research? [Paper/Benchmark] - An expert-annotated benchmark collected from the daily CTI workflow of a world-leading company to assess LLMs on practical threat intelligence tasks (triage, deep search, and TI drafting).
-
📊 AgentCyberRange [Benchmark/Website] - An open, multi-range infrastructure for measuring autonomous cyber attack capability across web exploitation and post-exploitation stages. See also the Research Paper (arXiv:2606.14295) and the WebExploitBench GitHub Repo.
How to Navigate & Contribute
- Browse the topic folder that matches your use case.
- Dive into its README for detailed descriptions and annotations, including resource type tags.
- To add a resource, open a PR in the appropriate folder’s
README.md. - Follow existing formatting and include the identifier tag (e.g., [Paper], [Blog], [GitHub], [Tool], [Video], [Model], [Dataset]).
📘 See CONTRIBUTING.md for full guidelines, style tips, and how to submit your PR.
License
Distributed under the MIT License. See MIT License for details.
Contact
For suggestions or PRs, open an issue or reach out to me at ashfaaqf@proton.me .