Fairness and Robustness in Risk Detection Models

Author: Leah Weldon Supervisor: Eoin Delaney

Overview

This repository contains the code, experiment configurations, and evaluation artifacts for the MSc dissertation:

“Fairness and Robustness in Risk Detection Models.”

The project evaluates fairness and robustness in automated toxicity and risk-detection systems across multiple datasets and adversarial evaluation settings. The analysis covers counterfactual fairness testing, adversarial robustness under meaning-preserving reformulation, calibration, distribution shift, and mitigation-oriented threshold and decision-policy analyses. The central aim is to characterise the conditions under which current classifiers fail systematically, by demographic group, attack type, or deployment context, and to assess the scope for targeted mitigation.

Research questions

The dissertation addresses two formal research questions:

RQ1: Group Fairness in Risk Detection To what extent do current toxicity and risk detection classifiers exhibit systematic group-level disparities in prediction behaviour, and how do these disparities vary across target groups, models, and evaluation paradigms?

Sub-questions:

• Do models assign different risk scores to otherwise-identical content depending on which demographic group is named?
• What is the direction and magnitude of these disparities, and for which groups are they most severe?
• Can threshold-based mitigation reduce FPR disparities without unacceptable F1 loss?

RQ2: Robustness to Adversarial and Meaning-Preserving Reformulation How robust are current toxicity and risk detection classifiers to meaning-preserving reformulation of harmful content, and does adversarial vulnerability vary systematically across target groups and attack strategies?

Sub-questions:

• How do paraphrase flip rates vary across backend sophistication (rules → HF-T5 → LLM API)?
• Which attack families (dog whistle, euphemism, policy framing, etc.) are most effective at bypassing detection?
• Does adversarial vulnerability differ across target groups, and does this constitute a secondary fairness problem?

Full research question documentation: docs/project/research_questions.md

Repository structure

configs/experiments/    YAML experiment configurations
src/                    Core pipeline implementation
src/data/               Dataset loaders, generators, tagging, and transforms
src/models/             Hugging Face model wrappers and prediction utilities
src/eval/               Evaluation and post-processing scripts
src/viz/                Plotting and visualisation scripts
docs/                   Dissertation-facing documentation
docs/datasets/          Dataset cards
docs/models/            Model cards
docs/experiments/       Experiment cards
docs/evaluation/        Evaluation methodology documentation
docs/analysis/          Generated analysis outputs and summaries
docs/project/           Research questions, scope, and contributions
docs/paper_notes/       Per-paper reading notes (one file per bib entry)
results/                Timestamped experiment run directories
data/                   Local/generated datasets (not fully versioned)
bib/                    Bibliography files
jobs/                   HPC / job submission scripts
scripts/                Utility scripts (setup, documentation, diagnostics)

Key documentation entry points:

• docs/datasets/README.md
• docs/models/README.md
• docs/experiments/README.md
• docs/evaluation/README.md
• docs/experiment_overview.md
• docs/reproducibility.md

Models evaluated

The main comparative evaluation uses the following core baselines:

Additional dataset-specific evaluation includes:

Model documentation:

• docs/models/README.md

Datasets

The dissertation evaluates models across several benchmark datasets:

Dataset documentation:

• docs/datasets/README.md

Synthetic evaluation benchmarks

In addition to external datasets, the repository includes internally generated evaluation benchmarks designed to isolate specific failure modes.

These include:

Synthetic benchmark documentation:

• docs/datasets/dataset_cards/synthetic_benchmarks.md

Evaluation dimensions

The dissertation evaluates models across several linked dimensions:

Running experiments

Experiments are defined through YAML configs under configs/experiments.

Example config:

• configs/experiments/exp10_civilcomments_multi_models.yaml

Run an experiment with:

python3 src/run_experiment.py \
  --config configs/experiments/exp10_civilcomments_multi_models.yaml

Results are written to:

results/runs/<timestamp>_<experiment_name>/

Each run stores the exact config used, metadata, predictions, and metrics. Multi-model runs also write comparison artifacts, and robustness runs additionally write paraphrase robustness outputs.

Experiment documentation:

• docs/experiments/README.md

Quick start

Set up a virtual environment and install dependencies:

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

Run a benchmark experiment:

python3 src/run_experiment.py \
  --config configs/experiments/exp10_civilcomments_multi_models.yaml

This will create a timestamped run directory under results/runs/.

Evaluation pipeline

The repository follows a structured evaluation workflow:

1. Load a dataset through the appropriate loader.
2. Run model inference through the Hugging Face classifier wrapper.
3. Apply repository-level binary projection when required.
4. Save predictions, metrics, and run metadata.
5. Run downstream fairness, robustness, calibration, and comparison analyses.

The core implementation is concentrated in:

• src/run_experiment.py
• src/models/
• src/eval/

Main entry points

The most important code entry points for navigating the repository are:

Reproducing results

Detailed reproducibility guidance is documented in:

• docs/reproducibility.md

The repository uses:

• config-driven experiments
• explicit experiment names and run logging
• deterministic seeds where possible
• experiment cards generated from configs
• cached paraphrase generation for some robustness workflows

These mechanisms make it possible to trace reported results back to exact run artifacts and configuration files.

Key analysis outputs

Generated analysis outputs and summaries are stored under:

• docs/analysis/

These include:

• fairness slice reports
• calibration analyses
• robustness summaries
• adversarial benchmark summaries
• threshold and mitigation analyses
• final comparison tables and figures
• SORRY-Bench detection rates and category-level heatmaps
• failure mode taxonomy for adversarial evasion
• deployment scenario analysis and decision-policy comparisons
• effect sizes (Cohen's d) and statistical tests (permutation p-values, bootstrap CIs)
• human paraphrase annotation study results (quality ratings across meaning, fluency, identity preservation, and toxicity preservation)

Experiment design overview

The complete experiment grid and evaluation strategy are summarised in:

• docs/experiment_overview.md

Individual experiment configurations are documented in:

• docs/experiments/

Citation

This repository accompanies the following dissertation:

Leah Weldon. Fairness and Robustness in Risk Detection Models. MSc dissertation, Trinity College Dublin.

Any work that draws on the experimental outputs, benchmarks, or evaluation methodology should also cite the original datasets and models as documented in:

• docs/datasets/README.md
• docs/models/README.md

License

This repository is released under the MIT License.