DeltaSecommits: Real-World Vulnerability-Fixing Commits

from Do Language Models Prefer Vulnerable Code? A Probabilistic Study of Insecure Code Preference @ICST 2026

A curated dataset of 2,493 paired vulnerable/secure C code samples from real-world vulnerability-fixing commits, covering 25 CWE (Common Weakness Enumeration) categories and 2,422 unique vulnerability identifiers.

Overview

DeltaSecommits provides paired code snapshots—the vulnerable version (pre-patch) and the secure version (post-patch) of the same file from a single commit. This structure eliminates confounds from stylistic variation between authors and projects, making it ideal for mechanistic analysis of how models encode security properties.

Each pair includes:

• Vulnerable code: Pre-patch version with the security flaw
• Secure code: Post-patch version with the fix applied
• CWE identifier: The specific vulnerability type (CWE-119, CWE-89, etc.)
• Severity rating: CVE severity (if available)

Data Source & Curation

The dataset is drawn from authoritative sources:

• Open Source Vulnerability (OSV) Database: https://osv.dev
• National Vulnerability Database (NVD): https://nvd.nist.gov

Filtering pipeline (for quality assurance):

1. Single-file commits only — excludes multi-file changes that cloud the signal
2. Single-commit fixes — removes vulnerabilities requiring multiple commits to resolve
3. Patch retrieval — uses GitHub API to fetch commit metadata and validate patches
4. Label deduplication — removes duplicate vulnerability IDs

This resulted in the final dataset of 2,493 high-quality pairs from 278 publicly disclosed vulnerabilities.

CWE Coverage

The dataset spans 25 CWE categories across four major families:

Memory Safety (3 CWEs, 610 pairs)

• CWE-119: Buffer Overflow
• CWE-125: Out-of-bounds Read
• CWE-787: Out-of-bounds Write

Injection (2 CWEs, 471 pairs)

• CWE-89: SQL Injection
• CWE-79: Cross-Site Scripting (XSS)

Resource Management (3 CWEs, 248 pairs)

• CWE-401: Memory Leak
• CWE-400: Resource Consumption
• CWE-416: Use-After-Free

Information Disclosure (1 CWE, 143 pairs)

• CWE-200: Sensitive Information Exposure

And 16 additional CWEs with varying sample counts (minimum 20 pairs per type).

Dataset Statistics

Usage

Loading the Dataset

from datasets import load_dataset

# Load full dataset
ds = load_dataset("deltasecommits")

# Access a single example
example = ds["train"][0]
print(example["vulnerable_code"])
print(example["secure_code"])
print(example["cwe"])

Data Format

Each example contains:

{
    "vulnerable_code": str,      # Pre-patch code with vulnerability
    "secure_code": str,          # Post-patch code with fix
    "cwe": str,                  # CWE identifier (e.g., "CWE-119")
    "severity": str,             # CVE severity (if available)
    "vulnerability_id": str,     # Unique identifier
    "source": str                # Source (osv, nvd)
}

Use Cases

Security-focused ML research:

• Training vulnerability detection models
• Analyzing how LLMs encode security information
• Benchmarking code security analyzers

Mechanistic interpretability:

• Understanding model representations of code safety
• Probing vulnerability direction geometry
• Analyzing activation patterns across CWE types

Code generation safety:

• Evaluating whether language models prefer secure or vulnerable patterns
• Testing security alignment in code assistants
• Debugging security misalignment in LLM-generated code

Limitations & Known Issues

1. Language scope: C code only. Multi-language analysis requires separate datasets (SVEN, PreciseBugs).
2. Real-world bias: Reflects OSV/NVD coverage—some vulnerability types may be overrepresented.
3. Patch simplicity: Many fixes are removals/simplifications rather than structural additions. Mean vulnerable code is 20% longer.
4. CWE distribution: Some CWE categories have <30 pairs; use caution with small-sample types.

Replication & Validation

The core findings have been validated on two additional C vulnerability datasets:

• SVEN (Semantic Vulnerability Extraction Network): 423 paired samples across 6 CWEs, synthetically curated
• PreciseBugs: 4,101 paired samples from open-source repositories across 9 CWEs

Results consistently replicate across all three datasets.

Related Work

• CVSS: Common Vulnerability Scoring System (severity ratings)
• CWSS: Common Weakness Scoring System (weakness prioritization)
• SonarQube: Static analysis for vulnerability detection
• Semgrep: Lightweight vulnerability scanning

Contact & Issues

For dataset issues, questions, or contributions:

• GitHub: https://github.com/rufimelo/DeltaSecommits/edit/main/README.md)
• Email: rmelo@cs.cmu.edu

License

DeltaSecommits is provided under the MIT License. Code samples are derived from publicly disclosed vulnerabilities in the OSV and NVD databases and are used in accordance with those databases' terms.

📚 Citation

If you use this work in your research, please cite:

@inproceedings{melo2026vulnerable,
  title     = {Do Language Models Prefer Vulnerable Code? A Probabilistic Study of Insecure Code Preference},
  author    = {Melo, Rui and Reis, Sofia and Catarino, Andre and Abreu, Rui},
  booktitle = {Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST)},
  year      = {2026},
  publisher = {IEEE}
}