EU Cyber Threat Landscape LLM Benchmark — Responses

15,988 LLM-generated cyber threat landscape assessments from 7 models across 3 continents, designed to measure geopolitical bias in attribution framing.

What this is

The complete response corpus from running the EU Cyber LLM Benchmark prompts against 7 locally deployed models via Ollama. Each record contains the full model output, pre-extracted analytical sections, CVE mentions, refusal flags, and latency measurements.

All experiments ran fully offline on Apple Silicon. No cloud APIs were used.

Corpus summary

Model panel

Schema

How to use

from datasets import load_dataset

ds = load_dataset("eromang/eu-cyber-llm-benchmark-responses")

# Filter by model
llama = ds["train"].filter(lambda x: x["model"] == "llama3.1:8b-instruct-q4_K_M")
print(f"llama3.1 records: {len(llama)}")

# Compare hedging across attribution conditions for a model
from collections import Counter
conditions = Counter(r["sensitivity_level"] for r in llama)
print(conditions)

Reproduce the analysis

# CVE fixation by model
from collections import Counter
ds = load_dataset("eromang/eu-cyber-llm-benchmark-responses", split="train")

for model in set(ds["model"]):
    subset = ds.filter(lambda x: x["model"] == model)
    cve_records = subset.filter(lambda x: x["cve_count"] > 0)
    all_cves = [c for r in cve_records for c in r["cves"]]
    top = Counter(all_cves).most_common(3)
    print(f"{model}: {len(cve_records)}/{len(subset)} with CVEs, top: {top}")

Key findings

• Certainty calibration is universal: all 7 models reduce hedging from Suspected to Confirmed attribution
• No systematic geopolitical bias: Phase 1 China-sensitivity disappears at scale; 0/5 China-vs-rest tests significant for any model
• CVE fixation is model-specific: deepseek-r1 fixates on PwnKit (73%), llama3.1 on Log4Shell (49%), phi4 on Log4Shell (60%)
• Chain-of-thought amplifies everything: qwen3 thinking vs nothink pair shows CoT increases calibration, CVE rate, and actor uniformity
• Refusal patterns are model-specific: llama3.1 has 17.7% US_Confirmed refusal at T=0.7; gemma3n and mistral have near-zero refusals

Citation

@misc{romang2026eucyberbenchmark,
  author       = {Eric Romang},
  title        = {EU Cyber Threat Landscape LLM Benchmark: Geopolitical Bias in Local Language Models},
  year         = {2026},
  url          = {https://github.com/eromang/researches/tree/main/LLM-Benchmark},
  note         = {Research benchmark for evaluating actor-asymmetric framing in local LLMs}
}

Related

• Prompt dataset — Reusable evaluation prompts
• GitHub repository — Full analysis scripts, per-model reports, methodology

License

MIT