RobustStress: Stress-Testing AI-Generated Text Detectors under Gradual Perturbations

Introduction

Detecting text generated by large language models (LLMs) is a critical line of defense for ensuring information security. While recent studies have increasingly focused on evaluating the robustness of detectors, existing benchmarks suffer from two limitations: (1) a lack of ability to grade and control perturbation intensity, making it difficult to systematically assess how performance changes with varying attack intensity; and (2) a failure to observe the combined effects of perturbation intensity on detector robustness in complex scenarios, such as cross-domain and cross-model settings. To address these challenges, we propose RobustStress, a novel stress testing framework for intensity-controlled perturbations. We achieve quantitative classification of perturbation intensity by varying the proportion of synonym replacement and employing graded paraphrasig attacks via LLMs.

Project Structure

The project is organized into the following directories:

● Detectors: Houses the core code for all detectors. Instructions for launching each detector are provided in the respective main methods.

● Draw: Contains the code used to generate figures and charts for the associated paper.

● SynonymReplacement: Implements the synonym replacement attack strategy to test model robustness.

● utility: A collection of helper functions and utility classes used across the project.