StealthRL Benchmark

A paired benchmark for evaluating AI-text detectors under adversarial rewriting. Each row contains an AI-generated source text and a StealthRL rewrite.

Links:

• Paper: https://arxiv.org/abs/2602.08934
• Model: https://huggingface.co/suraj-ranganath/StealthRL
• Demo: https://stealthrl.pages.dev/
• Source dataset: https://huggingface.co/datasets/yaful/MAGE

Dataset construction

This benchmark uses the same filtered MAGE test-pool AI examples used in the StealthRL paper. We start from the MAGE test split, apply the paper's evaluation filtering, and keep the 14,656 AI-generated examples used for the full detector evaluation.

For each source example, we include the original AI-generated text and a StealthRL rewrite generated directly with the released StealthRL model.

Columns

• sample_id: stable row identifier from the filtered MAGE test pool.
• source: raw MAGE source field.
• source_domain: parsed source/domain family, e.g. imdb, xsum, squad, yelp, cmv.
• source_generator: parsed original generator when available, e.g. gpt4, text-davinci-003, gpt-3.5-trubo.
• source_task: parsed generation mode when available, e.g. continuation, specified, topical, paraphrase, direct.
• ai_generated_text: original AI-generated text evaluated in the paper.
• stealthrl_text: StealthRL-model rewrite of the AI-generated text.

Usage

from datasets import load_dataset

ds = load_dataset("suraj-ranganath/StealthRL-Benchmark")["train"]

Use ai_generated_text to measure clean AI-text detection and stealthrl_text to measure detector robustness to StealthRL adversarial paraphrases.

Provenance

The source examples come from MAGE (yaful/MAGE, Apache-2.0). The subset corresponds to the filtered full-MAGE evaluation in the StealthRL paper.