Fast-DetectGPT Worker

A RunPod serverless worker implementation of Fast-DetectGPT, an efficient zero-shot detector for machine-generated text based on the ICLR 2024 research paper. This worker analyzes text samples to determine the probability that they were generated by an AI model versus authored by humans.

Research Background

Fast-DetectGPT is based on the paper "Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature" presented at ICLR 2024. The method achieves:

• 75% improvement in detection performance over DetectGPT
• 340x speedup in computational efficiency
• State-of-the-art results: 0.9887 AUROC on 5-model generations, 0.9338 AUROC on ChatGPT/GPT-4

How It Works

Fast-DetectGPT uses conditional probability curvature to efficiently detect machine-generated text:

1. Conditional Probability Curvature: Measures subtle differences in word choice patterns between AI and human text
2. Efficient Sampling: Replaces DetectGPT's perturbation step with optimized sampling for 340x speedup
3. Zero-shot Detection: No training required, works across different language models
4. Statistical Classification: Uses pre-calibrated normal distributions to compute interpretable probability scores

Supported Model Combinations

The worker supports these pre-calibrated model pairs optimized for Fast-DetectGPT:

• falcon-7b + falcon-7b-instruct: 89.38% accuracy (default, recommended)
• gpt-j-6B + gpt-neo-2.7B: 81.22% accuracy
• gpt-neo-2.7B + gpt-neo-2.7B: 82.22% accuracy

Note: The original Fast-DetectGPT research recommends gpt-j-6B and falcon-7b models for optimal performance.

API Reference

Input Format

{
  "input": {
    "prompt": "Text to analyze for AI detection",
    "sampling_model_name": "falcon-7b",
    "scoring_model_name": "falcon-7b-instruct",
    "device": "cuda",
    "cache_dir": "./cache"
  }
}

Parameters

• prompt (required): Text to analyze for AI generation
• sampling_model_name (optional): Reference model, defaults to "falcon-7b"
• scoring_model_name (optional): Scoring model, defaults to "falcon-7b-instruct"
• device (optional): Computation device, defaults to "cuda"
• cache_dir (optional): Model cache directory, defaults to "./cache"

Available Models

Base Models:

• gpt2, gpt2-xl
• facebook/opt-2.7b, facebook/opt-13b
• EleutherAI/gpt-neo-2.7B, EleutherAI/gpt-j-6B, EleutherAI/gpt-neox-20b
• huggyllama/llama-13b, TheBloke/Llama-2-13B-fp16
• bigscience/bloom-7b1
• tiiuae/falcon-7b, tiiuae/falcon-7b-instruct
• sberbank-ai/mGPT, stanford-crfm/pubmedgpt
• google/mt5-xl

Output

The worker returns the analyzed text along with console output showing:

• Criterion score (sampling discrepancy value)
• Probability percentage that the text is machine-generated

Example output:

Fast-DetectGPT criterion is 2.1234, suggesting that the text has a probability of 87% to be machine-generated.

Local Testing

Setup Environment

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate  # macOS/Linux
# venv\Scripts\activate   # Windows

# Install dependencies
pip install -r requirements.txt

Run Locally

# The script automatically reads test_input.json
python3 rp_handler.py

Update test_input.json with your text sample:

{
  "input": {
    "prompt": "Your text to analyze goes here. This could be a paragraph or longer text sample that you want to check for AI generation.",
    "sampling_model_name": "falcon-7b",
    "scoring_model_name": "falcon-7b-instruct"
  }
}

Docker Deployment

Build and Push Image

# Build for RunPod deployment (linux/amd64 required)
docker build -t alxgirarddavila/worker-fast-detect-gpt:latest --platform linux/amd64 .

# Push to Docker Hub
docker push alxgirarddavila/worker-fast-detect-gpt:latest

Pre-built Image: A ready-to-use Docker image is available at alxgirarddavila/worker-fast-detect-gpt on Docker Hub.

RunPod Configuration

1. Create a new serverless endpoint on RunPod
2. Use the Docker image: alxgirarddavila/worker-fast-detect-gpt:latest
3. Configure GPU requirements (recommended: >= 8GB VRAM for larger models)
4. Set appropriate timeout values for model loading and inference

Performance Notes

• GPU Recommended: Tesla A100 with 80GB memory recommended (as per original research)
• Memory Requirements: Larger models (13B+) require substantial GPU memory
• First Run: Initial execution includes model download time (~340x faster than DetectGPT after setup)
• Model Caching: Models are cached locally to speed up subsequent runs

Technical Details

This implementation is based on the official Fast-DetectGPT repository and uses:

• Conditional Probability Curvature: Core Fast-DetectGPT algorithm for efficient detection
• Transformers: For model loading and tokenization
• PyTorch: For neural network operations (tested with PyTorch 1.10.0)
• SciPy: For statistical probability calculations
• Zero-shot Detection: No fine-tuning required, works across different model architectures

The algorithm analyzes subtle differences in word choice patterns between machine-generated and human text, utilizing pre-calibrated normal distribution parameters for interpretable probability scores.

References

• Paper: Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature
• Conference: ICLR 2024
• Official Repository: https://github.com/baoguangsheng/fast-detect-gpt
• Online Demo: https://fastdetect.net