Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges

📊 Dataset  •  🤖 Trained Models  •  📝 Paper  •  💻 Repo

This repository contains code for the paper Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges by Ruomeng Ding*, Yifei Pang*, He Sun, Yizhong Wang, Steven Wu, and Zhun Deng.

We study Rubric-Induced Preference Drift (RIPD) in LLM-based evaluation and alignment pipelines, showing that rubric edits which pass benchmark validation can nonetheless induce systematic, directional preference drift on target domains that are difficult to detect with standard metrics. We further demonstrate rubric-based preference attacks and show how the resulting bias propagates through downstream post-training, leading to persistent policy misalignment.

Set Up

1. Clone Rubrics-as-an-Attack-Surface repository.

    git clone https://github.com/ruomengd/Rubrics-as-an-Attack-Surface.git
    cd Rubrics-as-an-Attack-Surface

2. Create the environment.

    conda create -n rubrics python=3.9
    conda activate rubrics
    pip install -r requirements.txt

Dataset

We use five human-preference datasets (UltraFeedback, ChatbotArena, RMB, Anthropic hh-rlhf, PKU-SafeRLHF) to construct four benchmark–target settings: Ultra-Real and Ultra-Creative for helpfulness (UltraFeedback → ChatbotArena), and SafeRLHF–RMB and Anthropic–SafeRLHF for harmlessness. All data is converted to a uniform pairwise preference format; benchmarks enforce rubric preservation, while targets measure deployment-relevant preference drift, with downstream policy experiments on Ultra-Real and Anthropic–SafeRLHF.

Scripts

The full data pipeline (download, preprocessing, filtering, and domain splitting) is run with:

sh ./scripts/dataset.sh

Alternatively, you can download the data directly from Hugging Face.

Directory Structure

After the pipeline completes, the directory layout is:

data/
├── helpfulness/
│   ├── Ultra-Real/
│   │   ├── Ultra-Real-Bench/
│   │   │   ├── train.jsonl
│   │   │   ├── val.jsonl
│   │   │   └── test.jsonl
│   │   └── Ultra-Real-Target/
│   │       ├── train.jsonl
│   │       ├── val.jsonl
│   │       └── test.jsonl
│   └── ...
├── harmlessness/
│   ├── Anthropic-SafeRLHF/
│   │   ├── Anthropic-SafeRLHF-Bench/
│   │   │   ├── train.jsonl
│   │   │   ├── val.jsonl
│   │   │   └── test.jsonl
│   │   └── Anthropic-SafeRLHF-Target/
│   │       ├── train.jsonl
│   │       ├── val.jsonl
│   │       └── test.jsonl
│   └── ...

Bench vs. Target.
For each dataset configuration, Bench denotes the benchmark domain used during rubric development, while Target denotes a held-out deployment domain used to evaluate generalization and preference drift. Rubric edits are validated exclusively on the Bench domain and never optimized using Target data.

Data splits and usage.

• train.jsonl: Used for rubric search and refinement.
• val.jsonl: Used for rubric selection, ensuring benchmark compliance.
• test.jsonl: Used exclusively for evaluation of Rubric-Induced Preference Drift (RIPD) and is never accessed during rubric editing.

Biased Rubric Search

The rubric search code lives under rubrics_search/search/ and implements a population-based evolutionary procedure to find benchmark-preserving but target-biased rubric variants.

1. Run evolutionary search to generate candidate rubrics with main.py.
2. Select top-k per generation with select_rubrics.py.
3. Evaluate selected rubrics on target-val (measure induced drift) for later selection.
4. Evaluate selected rubrics from target-val with select_final.py for later selection.

To run the full rubric-search pipeline:

sh ./scripts/rubrics_search_helpfulness.sh
sh ./scripts/rubrics_search_harmlessness.sh

Rubric Selection

Rubrics are selected under a benchmark-preserving constraint: candidates must match or exceed the seed rubric’s agreement on a held-out benchmark validation split. Among feasible candidates, we choose the rubric that maximally degrades agreement on the target validation split.

sh ./scripts/rubrics_selection.sh

To assess the transferability of optimized rubrics, we provide scripts for cross-model evaluation. To take rubrics optimized on a source model (Model A) and test their performance on a target model (Model B):

sh ./scripts/rubrics_cross_model_eval.sh

Downstream Policy Misalignment Evaluation

For downstream policy misalignment experiments, we focus on Ultra-Real (helpfulness) and Anthropic–SafeRLHF (harmlessness), training policy models directly on preference labels generated by the selected rubrics.

DPO Training

1. Generate preference labels using the selected rubrics:

sh scripts/dpo_labelling.sh

2. Train the policy with the labeled training data:

sh scripts/dpo_train.sh

Policy Evaluation

Run the evaluation pipeline (you can execute any subset of steps) via:

sh scripts/dpo_eval.sh

The evaluation script supports the following stages:

• Generate model responses
• Score responses (using evaluators/reward models)
• Analyze win-rates
• Select Best-of-N (BoN) responses
• Evaluate final outputs with a third-party judge

Cite Our Work

@misc{ding2026rubricsattacksurfacestealthy,
      title={Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges}, 
      author={Ruomeng Ding and Yifei Pang and He Sun and Yizhong Wang and Zhiwei Steven Wu and Zhun Deng},
      year={2026},
      eprint={2602.13576},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2602.13576}, 
}