Overview

📊 Dataset  •  🤖 Trained Models  •  📝 Paper  •  💻 Repo

This dataset provides benchmark–target evaluation splits for studying Rubric-Induced Preference Drift (RIPD) in LLM-based evaluation pipelines. The full implementation, rubric search pipeline, and downstream alignment experiments are available in the official GitHub repository: https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface.

We construct four benchmark–target settings from five widely used human-preference datasets. All data is converted to a uniform pairwise preference format.

• Bench domains enforce rubric preservation during development;
• Target domains simulate deployment and are used to measure preference drift and generalization.

Downstream alignment experiments use Ultra-Real and Anthropic–SafeRLHF.

Settings

We provide additional details for the datasets used in our experimental settings. Helpfulness: We build domain-specific subsets from ChatbotArena (Arena Human Preference) by grouping on category labels (real world and creative writing). These subsets serve as target domains; we use UltraFeedback as the benchmark domain, giving two benchmark–target pairs: Ultra-Real and Ultra-Creative. Harmlessness: We form pairs by (i) combining PKU-SafeRLHF and RMB (SafeRLHF–RMB), and (ii) pairing PKU-SafeRLHF with Anthropic hh-rlhf (Anthropic–SafeRLHF).

Helpfulness

Harmlessness

Labeled Preference Sets

For downstream alignment experiments, we release DPO training data labeled under two rubric conditions:

• seed: Labels derived from the standard human-crafted rubric, used as the unmodified baseline for evalua- tion and comparison.
• biased: Labels derived from a benchmark-preserving but target-drifting rubric.

These datasets enable controlled studies of how rubric-induced preference drift propagates into downstream policy training.

Data Structure

data/
├── helpfulness/
│   ├── Ultra-Real/
│   │   ├── Ultra-Real-Bench/
│   │   │   ├── train.jsonl
│   │   │   ├── val.jsonl
│   │   │   └── test.jsonl
│   │   ├── Ultra-Real-Target/
│   │   │   ├── train.jsonl
│   │   │   ├── val.jsonl
│   │   │   └── test.jsonl
│   │   ├── dpo_labeled_bench/
│   │   │   ├── train.jsonl
│   │   │   ├── test.jsonl
│   │   │   ├── seed/
│   │   │   │   └── labeling.jsonl
│   │   │   └── biased/
│   │   │       └── labeling.jsonl
│   │   └── dpo_labeled_target/
│   │   │   ├── train.jsonl
│   │   │   ├── test.jsonl
│   │   │   ├── seed/
│   │   │   └── biased/
│   └── Ultra-Creative/...
├── harmlessness/...

Each JSONL line is a pairwise preference record:

{
  "prompt": "...",
  "response_a": "...",
  "response_b": "...",
  "ground_truth": "A"
}

Benchmark–Target Protocol

• Bench: Domain used for rubric development and validation.
• Target: Held-out deployment domain used only to measure generalization and preference drift.

Rubric edits are validated only on Bench data.

Data Splits

Each domain has three splits:

For tasks involving policy model training and evaluation, we provide two additional sampled subsets in dpo_labeled_* folder:

Intended Use

This dataset is intended for:

• Evaluation-time robustness analysis
• Studying cross-domain preference drift
• Benchmark–deployment generalization research
• Alignment pipeline robustness studies

It is not intended for training general-purpose language models.

Cite

If you use this dataset, please cite our paper:

@misc{ding2026rubricsattacksurfacestealthy,
      title={Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges},
      author={Ruomeng Ding and Yifei Pang and He Sun and Yizhong Wang and Zhiwei Steven Wu and Zhun Deng},
      year={2026},
      eprint={2602.13576},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2602.13576},
}