Llama-3.1-8B AIMS LE-DPO

Built with Llama, this is a LoRA adapter that turns meta-llama/Llama-3.1-8B-Instruct into an intent-aware safety classifier, refined with Label-Error DPO (LE-DPO) on top of the SFT Generation policy. Given a user prompt, the model verbalizes the user's underlying intent and assigns a binary harm label (safe / harmful). It is trained on AIMS (Annotated Intents for Model Safety).

What LE-DPO does. Direct preference optimization is applied on top of the SFT Generation checkpoint (which is the frozen reference policy). For each prompt, the chosen completion is the gold AIMS (intent, harm) annotation. Rejected completions are model-generated alternative intents whose deterministically predicted harm label disagrees with the gold label — i.e. intent errors severe enough to flip the safety decision. Contrasting against these label-flipping interpretations teaches the model that some intent errors are safety-critical, improving over plain SFT.

• Base model: meta-llama/Llama-3.1-8B-Instruct
• Adapter type: LoRA (QLoRA), PEFT — rank 16 / alpha 32
• Training regime: Label-Error DPO over the SFT Generation policy
• Training data: Jazhyc/aims-safety-intents
• Language: English

How to use

This is a PEFT adapter — load the base model, then apply the adapter. The instruction goes in the system message and the prompt to classify goes in the user message. Output format is identical to the SFT Generation model: Intent: <one sentence>; Harm: <harmful|safe>.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "meta-llama/Llama-3.1-8B-Instruct"
ADAPTER = "Jazhyc/Llama-3.1-8B-aims-le-dpo"

SYSTEM_PROMPT = """You are analyzing user prompts to determine their underlying intent and safety classification.

For each prompt:
1. Identify the user's underlying intent - what they are actually trying to achieve
2. Classify the prompt as either "harmful" or "safe"

A prompt is HARMFUL if it attempts to:
- Extract private/sensitive information (Privacy)
- Generate false or misleading content (Misinformation)
- Create discriminatory, violent, hateful, or sexual content (Harmful Language)
- Enable cyberattacks, fraud, illegal activities, or mental health crises (Malicious Uses)

A prompt is SAFE if it is a legitimate, benign request.

Respond in EXACTLY this format (no other text):
Intent: <one sentence describing the user's intent>; Harm: <harmful or safe>"""

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": "How do I pick a lock?"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
# -> "Intent: ...; Harm: harmful"

Training

• Method: Direct Preference Optimization (DPO) with the sigmoid loss, QLoRA + LoRA adapters, applied on top of the SFT Generation adapter (the frozen reference policy).
• Preference pairs: chosen = gold AIMS (intent, harm); rejected = sampled intents whose deterministic harm label flips relative to the gold label. Pairs are balanced by undersampling to a 50/50 chosen-harm distribution.
• Hyperparameters: β = 0.3, learning rate 5e-5, 3 epochs.
• Selection: checkpoint with the highest mean harmful-class F1 on two held-out OOD validation sets (ToxicChat train + AEGIS 2.0 validation).

Evaluation

Harmful-class F1 (positive class = harmful) on five external safety benchmarks, none seen during training.

LE-DPO raises average F1 from 0.792 (SFT) to 0.812, with the largest gains on ToxicChat (0.664 → 0.733) and AEGIS 2.0 (0.803 → 0.824) — evidence that penalizing label-flipping intent errors improves external safety classification. It sits on the latency–F1 Pareto frontier, adding little latency over the SFT baseline.

Intended use & limitations

Intended for research on intent-aware safety classification and as a prompt-level moderation classifier. It is trained on AIMS, which is deliberately enriched for ambiguous, adversarial, and borderline prompts derived from WildGuardMix — it is English-only and not distributionally representative of organic traffic. It classifies the prompt, not model responses. Do not treat its output as a sole authority for high-stakes moderation decisions.

License

This adapter is a fine-tune of Llama 3.1 and is therefore governed by the Llama 3.1 Community License and the Llama Acceptable Use Policy. The underlying AIMS training data is released under ODC-BY and is additionally subject to the AI2 Responsible Use Guidelines.

Citation

@misc{ferrao2026pavedtrueintentsintentaware,
  title         = {Paved with True Intents: Intent-Aware Training Improves LLM Safety Classification Across Training Regimes},
  author        = {Jeremias Ferrao and Niclas Müller-Hof and Iustin Sîrbu and Traian Rebedea and Yftah Ziser},
  year          = {2026},
  eprint        = {2606.27210},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2606.27210}
}