MalSkillBench

A runtime-verified benchmark of malicious agent skills, with the pipeline that builds it.

AI coding agents (Claude Code, OpenCode, Gemini CLI, ...) extend themselves with third-party skills: SKILL.md packages that bundle natural-language instructions, executable scripts, and tool permissions. A skill is at once code and agent-facing instruction, so it opens an attack surface that is neither pure software nor pure prompt. MalSkillBench is the first large-scale, taxonomy-driven, runtime-verified benchmark for malicious agent skills that jointly covers code injection, prompt injection, and mixed instruction-code attacks.

Cite Me

@misc{guo2026malskillbenchruntimeverifiedbenchmarkmalicious,
      title={MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills}, 
      author={Wenbo Guo and Wei Zeng and Chengwei Liu and Xiaojun Jia and Yijia Xu and Lei Tang and Yong Fang and Yang Liu},
      year={2026},
      eprint={2606.07131},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2606.07131}, 
}

What's in the benchmark

Generated samples are produced with overall 85.6% verification yield (CI 94.5%, MIXED 91.9%, PI 75.8%) and cover all 108 taxonomy cells.

Attack taxonomy

Every malicious skill is labeled along three dimensions; the valid combinations form 108 cells (9x4 CI + 15x3 PI + 9x3 MIXED).

1. Attack vector

2. Malicious behavior (B1-B9 work via CI/PI/MIXED; B10-B15 target the agent and are PI-only)

3. Insertion strategy

Repository structure

MalSkillBench/
├── Core/
│   ├── generation/      # LLM-based malicious skill generation
│   ├── verification/    # Docker sandbox + two-layer verification
│   ├── taxonomy/        # Behavior definitions, material labeling
│   ├── collection/      # Benign/wild skill scraping
│   └── analyzers/
├── Dataset/
│   ├── Codes/           # CI knowledge base (malicious package corpus)
│   ├── Prompts/         # PI knowledge base (jailbreak/injection corpora)
│   └── Skills/
│       ├── malware/     # all malicious skills (generated + wild + tool-test), one dir each
│       └── benign/      # benign reference set
├── Baselines/           # Detector baselines (3 groups, see below)
├── Experiment/          # RQ3/RQ4 runner code + Results/ (RQ1/RQ2 are placeholders)
├── Configs/             # LLM configuration (template only is tracked)
└── Utils/               # LLM client, logging, token accounting

Each released skill is a complete, self-contained skill package — SKILL.md plus any scripts/, references/, or assets/ it ships. Generation- and verification-time artifacts (_expected.json, _evidence.json, _runtime/, etc.) are kept local and not part of the release.

Setup

cp Configs/llm_config.template.json Configs/llm_config.json
# edit Configs/llm_config.json and add your own API keys / endpoints

All llm_config*.json files except the template are gitignored. The generator and verifier read their own config, so you can point each at whichever provider (openai / azure / ollama / custom) you prefer.

Generation

# one cell
python Core/generation/agent.py --vector CI --behavior B1 --strategy "New Script File"

# the whole taxonomy, N samples per cell
python Core/generation/agent.py --generate_all --per_cell 3 --workers 6

The pipeline retrieves real attack patterns from the knowledge base, wraps them in a benign skill template as camouflage, synthesizes the skill, and writes _expected.json declaring the behavior to verify.

Verification

A candidate enters the benchmark only if its declared behavior is actually observed at runtime.

• CI / MIXED: the skill runs in a Docker sandbox (OpenCode agent + strace + inotifywait); Layer 1 matches declared IOCs in the syscall trace, Layer 2 is an LLM judge over the trace.
• PI: no syscall trace, so the Layer-2 judge classifies the agent's response as triggered, detected, or ineffective (the first two count as positive).

docker build -t malskillbench/sandbox Core/verification/docker/

# one skill
python Core/verification/agent.py --skill_dir Dataset/Skills/malware/<sample>

# everything, with closed-loop feedback
python Core/verification/agent.py --verify_all --workers 6 --with_feedback

Rejected candidates loop back to the generator with structured feedback (up to 3 retries).

Baselines

Evaluated under Experiment/RQ3 and Experiment/RQ4, grouped under Baselines/:

The central question is not whether a detector catches malicious code or malicious text alone, but whether it can reason over the relationship between a skill's code and its instructions.

Cite Me

@misc{guo2026malskillbenchruntimeverifiedbenchmarkmalicious,
      title={MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills}, 
      author={Wenbo Guo and Wei Zeng and Chengwei Liu and Xiaojun Jia and Yijia Xu and Lei Tang and Yong Fang and Yang Liu},
      year={2026},
      eprint={2606.07131},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2606.07131}, 
}

License

For academic research use only.