vulnerability-attack-techniques-llm-scaling

⚠️ The labels in this dataset are machine-generated by an LLM, not analyst-curated — and the paper that produced them found they do not improve a classifier trained on the expert gold set. It is published for reproducibility and for research on LLM-assisted labeling. For training, use the curated gold set CIRCL/vulnerability-attack-techniques.

This dataset contains 984 CVEs labeled with MITRE ATT&CK (Enterprise) techniques by ollama/qwen3.5:122b, following the same MITRE CTID "Mapping ATT&CK to CVE for Impact" methodology as the gold set (exploitation technique / primary impact / secondary impact, one assertive prompt per CVE, a single call each). It was built to answer one question in the paper Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion (arXiv:2607.25572): can LLM-assisted labeling extend a small expert gold set?

DOI: 10.57967/hf/9622

What the paper found

The labeler agrees with the expert gold labels at micro-F1 ≈ 0.39 (measured on the 121-CVE gold test split). At that agreement level, folding these rows into the gold training set (five seeds, validation-split checkpoint selection):

No expansion size from 100 to 984 added CVEs produced a reliable improvement, and at ~1,000 added rows rare-technique coverage (macro-F1) measurably degrades: the LLM's head-technique bias drowns out the tail that the expert labels cover. The checkpoint trained on gold + these 984 rows is published as CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded for side-by-side comparison.

Intended use

• Reproducing the paper's expansion and scaling experiments (the trainer-logs/ in the paper repository cover every run).
• Research on LLM-assisted labeling: this is a documented baseline at ≈0.39 agreement — filtering, adjudication, ensembling, or stronger labelers can be evaluated against it.
• Not recommended as training augmentation as-is. If you mix it with the gold set anyway, keep the label_sources column so the gold-only subset can always be recovered.

Fields

The schema is concatenation-compatible with the gold set, plus two provenance columns:

Usage

from datasets import load_dataset

dataset = load_dataset("CIRCL/vulnerability-attack-techniques-llm-scaling")

for entry in dataset["train"].select(range(3)):
    print(entry["id"], entry["techniques"], "-", entry["llm_comment"][:80])

Licensing of upstream sources

Titles and descriptions come from CIRCL/vulnerability-scores (CC BY 4.0). MITRE ATT&CK® is a registered trademark of The MITRE Corporation; ATT&CK content is used in accordance with the MITRE ATT&CK terms of use.

Related artifacts

Citation

@misc{bonhomme2026mappingcvesmitreattck,
      title={Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion},
      author={Cédric Bonhomme and Alexandre Dulaunoy},
      year={2026},
      eprint={2607.25572},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2607.25572},
}

Acknowledgements

Developed at CIRCL in the context of the AIPITCH project, co-funded by the European Union.