CTIBench

CTIBench is a benchmark for evaluating large language models on practical Cyber Threat Intelligence (CTI) tasks. It covers CTI knowledge, vulnerability root-cause mapping, vulnerability severity prediction, ATT&CK technique extraction, and threat actor attribution.

Highlights

• First broad CTI benchmark for evaluating LLMs across practical intelligence tasks
• 5 task families covering CTI knowledge, CVE/CWE reasoning, CVSS scoring, ATT&CK technique extraction, and threat attribution
• 4,610 released benchmark examples plus a 2021 root-cause mapping comparison split
• Evaluations for ChatGPT-3.5, ChatGPT-4, Gemini-1.5, LLAMA3-70B, and LLAMA3-8B
• Released datasets, formatted model responses, raw logs, evaluation notebooks, and project page

Resources

Repository Layout

Dataset Overview

For CTI-TAA, data/cti-taa.tsv contains the URL, anonymized report text, and prompt. The formatted response file evaluation/responses/cti-taa-responses.tsv includes the ground-truth threat actor labels used by the evaluation notebook.

Dataset details are also available on Hugging Face: https://huggingface.co/datasets/AI4Sec/cti-bench

Evaluation

The evaluation/ directory contains notebooks for generating predictions and evaluating formatted responses. The response TSVs include predictions for:

• ChatGPT-3.5
• ChatGPT-4
• Gemini-1.5
• LLAMA3-70B
• LLAMA3-8B

The main evaluation metrics are accuracy for CTI-MCQ and CTI-RCM, mean absolute deviation for CTI-VSP, F1 for CTI-ATE, and correct/plausible accuracy for CTI-TAA.

Star History

Citation

If you use CTIBench, please cite:

@article{alam2024ctibench,
  title={Ctibench: A benchmark for evaluating llms in cyber threat intelligence},
  author={Alam, Md Tanvirul and Bhusal, Dipkamal and Nguyen, Le and Rastogi, Nidhi},
  journal={Advances in Neural Information Processing Systems},
  volume={37},
  pages={50805--50825},
  year={2024}
}