HarmProfile: Characterizing Harmful Distributions in Frontier LLMs

Zhouyuan Ma¹, Yutao Wu², Hanxun Huang³, Xiang Zheng⁴, Xiao Liu², Yixin Cao¹, Zuxuan Wu¹, Xingjun Ma^1†, Yu-Gang Jiang^1†

¹Fudan University    ²Deakin University    ³The University of Melbourne    ⁴City University of Hong Kong

Findings of EMNLP 2026

[!CAUTION] HarmProfile contains sensitive and potentially harmful model-generated content. It is released for safety research and defensive evaluation. Access and use it responsibly, in accordance with applicable laws, institutional requirements, and platform policies.

Motivation

Most LLM safety evaluations ask whether a model refuses or complies with a harmful request. HarmProfile asks a different question: when a frontier model's safety alignment fails, what does the distribution of its harmful outputs look like?

HarmProfile treats harmful generation as an object of analysis rather than only an attack outcome. It builds a model-level risk profile from the content, severity, and variation of observed safety failures, making it possible to compare not just how often models fail, but where their risks concentrate and how broadly those risks are distributed. Even within the same harm category, different models exhibit markedly different generation styles and failure patterns.

The same harassment category manifests differently across frontier LLMs, from conventional responses to adversarial, step-by-step, icon-heavy, realistic-information, and refusal-prefaced styles. Examples are truncated as shown in the paper.

Dataset Overview

HarmProfile is a content-centric benchmark containing:

The Hugging Face release provides two configurations:

Because the corpus contains sensitive material, the dataset is gated. Visit the Hugging Face dataset page to review and accept the access conditions before downloading it.

Method Overview

HarmProfile combines taxonomy-guided autonomous generation with dual-axis profiling:

Overview of the HarmProfile framework: taxonomy construction, autonomous harmful-content generation, validation, and dual-axis profiling.

Each subcategory is instantiated as a specialized agentic workspace. The model generates candidate artifacts, iterates against local checks, and only contributes artifacts that pass an independent workspace validator. The resulting corpus is then profiled along two complementary axes:

• Harmfulness: the severity and harmful quality of the generated content.
• Diversity: the breadth and variation of harmful scenarios covered by the model.

Main Findings

1. Frontier LLMs retain broad high-risk generation capability. Every evaluated model produces non-trivial harmful content across all 15 categories, despite safety alignment.
2. Risk concentrates differently across models and categories. Aggregate safety scores hide distinct category-level risk profiles.
3. Capability grows with harmfulness and diversity. Diversity has a statistically significant correlation with LMArena Elo (r = 0.55, p = 0.018), while harmfulness shows a positive but weaker trend (r = 0.35, p = 0.149).

Harmfulness and diversity profiles across frontier LLMs.

Metric consistency, cross-model distributional similarity, and capability correlations.

Explore the samples, model profiles, main findings, and t-SNE map on the interactive project website.

Quick Start

1. Load the Hugging Face dataset

Install the Hugging Face client and authenticate after accepting the dataset access conditions:

pip install -U datasets huggingface_hub
hf auth login

Load either the 15-category subset or the full 57-subcategory release:

from datasets import load_dataset

# One representative high-risk subcategory per category.
high_risk = load_dataset(
    "Freshma/HarmProfile",
    "high_risk_15",
    split="train",
)

# Complete release covering all 57 subcategories.
all_57 = load_dataset(
    "Freshma/HarmProfile",
    "all_57",
    split="train",
)

print(high_risk)
print(high_risk.column_names)
print(high_risk[0])

2. Run the generation demo

The demo turns a high-level target distribution into a reviewable policy and judge rubric, asks for human approval, and then iteratively generates validated artifacts. It runs inside Docker, so no local Python environment is required.

Requirements: Docker and an OpenRouter API key.

git clone https://github.com/fresh-ma/HarmProfile.git
cd HarmProfile

cp demo/.env.example demo/.env
# Add your OPENROUTER_API_KEY to demo/.env.

./demo/generate.sh \
  "English red-team QA pairs covering diverse financial-scam scenarios" \
  --count 5 \
  --batch-size 1 \
  --language en \
  --llm qwen/qwen3.7-flash \
  --judge

Approved samples are written to:

demo/runs/<policy>_<timestamp>/approved.json

You can also start from one of the 57 bundled policies:

./demo/generate.sh --list-policies
./demo/generate.sh \
  --policy financial_scam \
  --count 5 \
  --batch-size 1 \
  --language en \
  --llm qwen/qwen3.7-flash \
  --judge

Demo workflow and common options

The command first translates a high-level description into a targeted policy and semantic judge rubric. Both are printed for review, and generation starts only after explicit approval. The agent then generates candidates in batches, revises rejected candidates, and stops after producing the requested number of approved samples.

Option	Purpose
--count N	Exact number of approved samples to produce
--batch-size N	Candidate samples generated per iteration
--policy NAME	Use a bundled policy instead of translating a new description
--judge / --no-judge	Enable or disable the workspace LLM validator
--llm MODEL	Use one model for policy design, generation, and judging
--translate-llm MODEL	Override the policy and judge-rubric designer model
--victim-llm MODEL	Override the generation-agent model
--judge-llm MODEL	Override the validator model
--judge-threshold N	Minimum semantic-alignment score from 1 to 5
--language zh|en|mixed	Generated sample language
--min-response-chars N	Minimum response length; defaults to 20 characters
--rounds N	Planned rounds; --count remains the stopping target
--max-retries N	Agent retries per failed round
--max-turns N	Maximum tool turns per agent attempt

Run ./demo/generate.sh --help to see every option. For long or high-quality samples, keep --batch-size small so that each agent round can focus on fewer candidates.

Repository Structure

HarmProfile/
├── data/                 # Project-site samples, metadata, and paper figures
├── demo/                 # Dockerized iterative generation demo
│   ├── generate.sh       # One-command entry point
│   ├── policies.json     # 57 bundled policy configurations
│   └── *.py              # Workspace, agent, and validator implementation
├── index.html            # Interactive project website
└── README.md

Special thanks

Special thanks to the LINUX DO.

Citation

If you find HarmProfile useful, please cite:

@misc{ma2026harmprofilecharacterizingharmfuldistributions,
  title         = {HarmProfile: Characterizing Harmful Distributions in Frontier LLMs},
  author        = {Zhouyuan Ma and Yutao Wu and Hanxun Huang and Xiang Zheng and
                   Xiao Liu and Yixin Cao and Zuxuan Wu and Xingjun Ma and Yu-Gang Jiang},
  year          = {2026},
  eprint        = {2608.14577},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2608.14577}
}