BiScope: AI-generated Text Detection by Checking Memorization of Preceding Tokens

---

Table of Contents

• BiScope: AI-generated Text Detection by Checking Memorization of Preceding Tokens
  • Table of Contents
  • Overview
  • Dataset
  • Code Implementation
    • Code Structure
    • Running the Code
  • Cite Our Work

Overview

• This is the official implementation for NeurIPS 2024 paper "BiScope: AI-generated Text Detection by Checking Memorization of Preceding Tokens".
• [video] | [slides] | [poster] | [paper]

Dataset

• We extend existing datasets by crafting more AI-generated data using five latest commercial LLMs, including GPT-3.5-Turbo, GPT-4-Turbo, Claude-3-Sonnet, Claude-3-Opus, and Gemini-1.0-Pro.
• Our Datasets consist of 2 short natural language datasets (Arxiv, Yelp), 2 long natural language datasets (Creative, Essay), and 1 code dataset (Code).
• We craft both the non-paraphrased version (./Dataset) and paraphrased version (./Paraphrased_Dataset) for each AI-generated data.
• We have uploaded our dataset to Hugging Face . Please refer to the link for more details.
• Detailed dataset statistics:

Code Implementation

Code Structure

The project is organized into two main Python files:

• biscope.py:
  This is the main script that:

  • Parses command line arguments.
  • Sets up evaluation strategies:
    • In-domain evaluation: 5-fold cross validation when training and testing datasets are identical.
    • Out-of-domain evaluation:
      • Cross-model OOD (same task): When the task is the same but the generative model or paraphrase status differs, the classifier is trained on human and GPT training data and tested only on GPT test data (using accuracy).
      • Cross-domain OOD (different task): When the task changes, both human and GPT test data are used for evaluation (using F1 score).
  • Saves feature files and evaluation results in a structured output directory.

• biscope_utils.py:
  Contains utility functions for feature extraction using pretrained language models from Hugging Face Transformers. The features are computed by:

  • Generating a summary-based prompt.
  • Tokenizing and computing loss values over segments of the text.
  • Saving the features as pickle files.

---

Running the Code

1. Set up your environment:

conda create -n biscope_env python=3.11
conda activate biscope_env
# install pytorch, please refer to https://pytorch.org/get-started/previous-versions/ to install the newest version that fit your device. We only show the install commands for the latest pytorch version.
pip3 install torch torchvision torchaudio
# install other dependencies
pip install transformers scikit-learn tqdm numpy openai tenacity accelerate

2. Run the code:

First, please set up the GPU devices, e.g.,

export CUDA_VISIBLE_DEVICES=0,1

If you want to use openai models (e.g., gpt-4o-mini) as the summary model, please also set up the openai key:

export OPENAI_API_KEY={Your OpenAI Key}

Then, you can run the code with the following command:

python bisocpe.py --train_dataset={train_dataset} --test_dataset={train_dataset} --detect_model={detect_model}

For more details of the arguments, please refer to the table below:

3. Run the motivation example:

The motivation example demonstrates the core intuition behind BiScope by comparing Next Token and Last Token features for human vs. AI-generated text.

# Run feature collection and generate visualization
python motivation.py \
    --detect_model llama2-7b \
    --dataset nonparaphrased_Yelp_gpt-4-turbo-preview \
    --phase all

# Or run in separate phases
# Phase 1: Collect features only
python motivation.py \
    --detect_model llama2-7b \
    --dataset nonparaphrased_Yelp_gpt-4-turbo-preview \
    --phase collect

# Phase 2: Generate plot from saved features
python motivation.py \
    --detect_model llama2-7b \
    --dataset nonparaphrased_Yelp_gpt-4-turbo-preview \
    --phase plot

The script will generate a 2×3 subplot figure showing:

• (a) Next Token Rank histogram
• (b) Last Token Rank histogram
• (c) Both Rank scatter plot
• (d) Next Token CE histogram
• (e) Last Token CE histogram
• (f) Both CE scatter plot

Results are saved in ./motivation_results/ by default.

Cite Our Work

If you find our work helpful, please consider citing our paper and giving us a star ☆:

@article{guo2024biscope,
  title={BiScope: AI-generated Text Detection by Checking Memorization of Preceding Tokens},
  author={Guo, Hanxi and Cheng, Siyuan and Jin, Xiaolong and Zhang, Zhuo and Zhang, Kaiyuan and Tao, Guanhong and Shen, Guangyu and Zhang, Xiangyu},
  journal={Advances in Neural Information Processing Systems (NeurIPS)},
  volume={37},
  pages={104065--104090},
  year={2024}
}