BiScope: AI-generated Text Detection by Checking Memorization of Preceding Tokens

This dataset is associated with the official implementation for the NeurIPS 2024 paper "BiScope: AI-generated Text Detection by Checking Memorization of Preceding Tokens". It contains human-written and AI-generated text samples from multiple tasks and generative models. Human data is always nonparaphrased, while AI-generated data is provided in both nonparaphrased and paraphrased forms.

Overview

BiScope detects AI-generated text by measuring the extent to which a language model memorizes preceding tokens. The dataset comprises samples from the following tasks:

• Arxiv
• Yelp
• Essay
• Creative
• Code

AI-generated text is produced by the following models:

• gpt-3.5-turbo
• gpt-4-turbo-preview
• claude-3-sonnet
• claude-3-opus
• gemini-1.0-pro

There are 50 configurations in total for AI-generated texts, resulting from the combination of:

1. Paraphrase: paraprhased or nonparaphrased
2. Task: Arxiv, Yelp, Essay, Creative, Code
3. Generative_Model: gpt-3.5-turbo, gpt-4-turbo-preview, claude-3-sonnet, claude-3-opus, gemini-1.0-pro

Note: Human data is always loaded from the nonparaphrased files.

How to Use the Dataset via the Hugging Face API

You can load a specific configuration using the Hugging Face datasets library. For example, to load Arxiv human data and the nonparaphrased Arxiv dataset generated by gpt-3.5-turbo:

from datasets import load_dataset

# 1. Load the entire dataset (assuming it has a single "train" split).
dataset = load_dataset("HanxiGuo/BiScope_Data", split="train")
print("Total records in dataset:", len(dataset))

# 2. Filter for Arxiv human data
arxiv_human = dataset.filter(
    lambda ex: ex["task"] == "Arxiv" and ex["source"] == "human"
)
arxiv_human = [s["text"] for s in arxiv_human]
print("Arxiv human data size:", len(arxiv_human))

# 3. Filter for Arxiv gpt-3.5-turbo nonparaphrased data
arxiv_gpt_nonpara = dataset.filter(
    lambda ex: ex["task"] == "Arxiv"
    and ex["source"] == "gpt-3.5-turbo"
    and ex["paraphrased"] is False
)
arxiv_gpt_nonpara = [s["text"] for s in arxiv_gpt_nonpara]
print("Arxiv gpt-3.5-turbo nonparaphrased size:", len(arxiv_gpt_nonpara))

# 4. Show an example from each subset
if len(arxiv_human) > 0:
    print("\nFirst Arxiv human sample:\n", arxiv_human[0])
if len(arxiv_gpt_nonpara) > 0:
    print("\nFirst Arxiv gpt-3.5-turbo nonparaphrased sample:\n", arxiv_gpt_nonpara[0])

Cite Our Work

If you find our dataset helpful, please consider citing our paper and giving us a like ♥:

@article{guo2024biscope,
  title={BiScope: AI-generated Text Detection by Checking Memorization of Preceding Tokens},
  author={Guo, Hanxi and Cheng, Siyuan and Jin, Xiaolong and Zhang, Zhuo and Zhang, Kaiyuan and Tao, Guanhong and Shen, Guangyu and Zhang, Xiangyu},
  journal={Advances in Neural Information Processing Systems (NeurIPS)},
  volume={37},
  pages={104065--104090},
  year={2024}
}