Training-free LLM-generated Text Detection by Mining Token Probability Sequences

This project provides the core code for the two main methods, Lastde and Lastde++ , as presented in our paper.

We follow the standard testing procedures outlined in Fast-DetectGPT to evaluate each detection method.

>

Environment

• Python3.8
• Pytorch2.0.0
• Other dependencies:

  pip install -r requirements.txt
  

  (Note: Our experiments were conducted on two RTX 3090 GPUs with 24GB of memory each.)

Source Models and Proxy Models

The pretrain_models directory is used to store open-source models, including those used as proxies or for generating text produced by LLMs. Here, we take gpt-j-6b and Llama-3-8B as examples, and the model weights can be downloaded from the following addresses:

• gpt-j-6b
• Llama-3-8B

Datasets

The main dataset is divided into two parts:

• The human_original_data directory contains raw text in json format, with the Xsum dataset (i.e., xsum.json) as an example.
• The human_llm_data_for_experiment directory stores the complete data used for experiments, with xsum_llama3_8b.raw_data.json as an example. This dataset needs to be obtained by running

  python py_scripts/data_generations/data_generation_opensource.py
  

  (Note : We have already provided the data here, so there is no need to run) Each complete data entry contains two parts: 'original' (human-written text) and 'sampled' (LLM-generated text), with the content of the two types of text corresponding to each other. The 'sampled' text is generated by using the first 30 tokens of the corresponding 'original' text as prompt input to the source model (in this case, Llama-3-8B) for continuation, and all entries are truncated to the same length.
• The perturbation_data_detectgpt_npr and regeneration_data_dnagpt directories store the complete data of the DetectGPT/DetectNPR and DNA-GPT detection experiments respectively.
• The remaining directories correspond to the robustness section in our paper.

Detection

Running detection_white_box.sh or detection_black_box.sh in shell_scripts will trigger white-box and black-box detection, respectively.

cd shell_scripts

# white-box setting
./detection_white_box.sh 

# black-box setting
./detection_black_box.sh

The detection methods include:

• Likelihood, LogRank, Entropy, DetectLRR, Lastde(ours). Results will be saved in experiment_results/statistic_detection_results.
• DetectGPT. Results will be saved in experiment_results/detectgpt_detection_results.
• DetectNPR. Results will be saved in experiment_results/npr_detection_results.
• DNA-GPT. Results will be saved in experiment_results/dna_gpt_detection_results.
• Fast-DetectGPT. Results will be saved in experiment_results/fast_detectgpt_detection_results.
• Lastde++(ours). Results will be saved in experiment_results/lastde_doubleplus_detection_results.

The code for the above detection methods is encapsulated in py_scripts/baselines.

Other experimental scripts are located in the shell_scripts directory.

Baselines

We provide the following baseline implementations.

• sample-based methods: Likelihood, LogRank, Entropy, DetectLRR, Lastde(ours), Binoculars
• Distribution-based methods: DetectGPT, DetectNPR, DNA-GPT, Fast-DetectGPT, Lastde++(ours)
• Plug and Play versions: Likelihood_tocsin, LogRank_tocsin, DetectLRR_tocsin, Lastde_tocsin, Fast-DetectGPT_tocsin, Lastde++_tocsin
• supervised-based methods: RoBERTa_Base, RoBERTa_Large, ReMoDetect

We thank the authors of open source projects and models such as Fast-DetectGPT, TOCSIN, ReMoDetect and Binoculars.

Main results (AUROC)

White-box

• Datasets : XSum, SQuAD, WritingPrompts
• Sampling || Scoring model : specific source model || specific source model
• Hyperparameters of Lastde/Lastde++ :
  • Lastde: $s$ = 3, $\varepsilon=10\times n$, $\tau^{\prime}=5$, $Agg=Std$.
  • Lastde++: $s$ = 4, $\varepsilon=8\times n$, $\tau^{\prime}=10$, $Agg=Std$.

Black-box

• Datasets : XSum, WritingPrompts, Reddit
• Sampling || Scoring model :
  • Sample-based methods : None || GPT-j-6B
  • Distribution-based methods : GPT-j-6B || GPT-j-6B
• Hyperparameters of Lastde/Lastde++ :
  • Lastde: $s$ = 3, $\varepsilon=10\times n$, $\tau^{\prime}=5$, $Agg=Std$.
  • Lastde++: $s$ = 4, $\varepsilon=8\times n$, $\tau^{\prime}=10$, $Agg=Std$.

GPT-4-Turbo || GPT-4o || Claude-3-Haiku (Black-box)

• Sampling || Scoring model :
  • Sample-based methods : None || GPT-j-6B
  • Distribution-based methods : GPT-j-6B || GPT-j-6B
• Hyperparameters of Lastde/Lastde++ :
  • Lastde: $s$ = 3, $\varepsilon=1\times n$, $\tau^{\prime}=15$.
  • Lastde++: $s$ = 4, $\varepsilon=8\times n$, $\tau^{\prime}=10$.

Notes

• Currently, we recommend that you set the aggregate function (Agg) to ExpStd.

Citation

If you find this work useful, you can cite it with the following BibTex entry:

@articles{
  xu2025trainingfree,
  title={Training-free {LLM}-generated Text Detection by Mining Token Probability Sequences},
  author={Yihuai Xu and Yongwei Wang and Yifei Bi and Huangsen Cao and Zhouhan Lin and Yu Zhao and Fei Wu},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025},
}