AI Detection at Scale

Cross-register stylometric detection of AI-generated text using 35 interpretable features (the production API uses the original 11-feature model; 35 features are available for extended experiments). Evaluated on 2.77M texts across four registers and twelve model variants from the RAID benchmark.

This repository has been expanded into a production-hardened hybrid detector incorporating stylometric ensembling, multi-class model source attribution, and local neural observer diagnostics (surprisal analytics).

---

Benchmark Results

1. Stylometric RF Baselines

2. Multi-Benchmark Evaluation Results

We evaluated our models against 4 major AI detection benchmarks, obtaining the following results:

3. Public Detector Results (New)

Per-benchmark evaluation of publicly available HuggingFace detectors, 2000 samples, 512 tokens (1024 for MAGE):

• TuringBench improves from 0.4691 → 0.9146 by switching to roberta-large-openai-detector.
• HC3 reaches near ceiling with the chatgpt-detector-roberta public model.
• MAGE benefits from combining public detector probability with stylometric features.
• Public Detector API: tool/public_api.py exposes /detect/public, /detect/public/batch, and /public/detectors endpoints for these models.
• Beemo Hybrid Ensemble: Fine-tuning a semantic BERT-mini model and ensembling its logits with GPT-2 perplexity and stylometrics raised our Toloka Beemo performance from 0.5256 to 0.7616 AUC. This is an ensemble improvement over our own stylometric baseline, not a claim of state-of-the-art over all published methods.
• Local Neural Observer: tool/neural_detector.py computes perplexity and burstiness from a locally cached GPT-2 model. The API calls it on every request but now catches failures and returns a safe fallback (perplexity: null, burstiness: null) instead of crashing. Production deployments should still pre-cache GPT-2 to avoid degraded responses.

4. SOTA Model Results (New)

The MAGE paper's Longformer detector (nealcly/detection-longformer) evaluated at 512 tokens:

• MAGE jumps from 0.7801 (public detector ensemble) to 0.9796 with the MAGE Longformer at 2000 samples, matching published SOTA.
• TuringBench remains best with roberta-large-openai-detector (0.9146); the MAGE Longformer does not generalize across generators.

5. TuringBench Fine-Tuned RoBERTa-large (New)

A roberta-large model was fine-tuned on the full TuringBench 19-generator training split (331k texts) on Kaggle and evaluated on the validation split:

• Fine-tuning improves TuringBench validation AUC from 0.9146 to 0.9991 (+0.0845) and accuracy from 0.6665 to 0.9948.
• The fine-tuned model weights are saved on Kaggle; the local copy is at models/turingbench_roberta_large/ (1.3 GB, git-ignored). Kaggle notebook: notebooks/kaggle_turingbench_finetune/kaggle_turingbench_finetune.ipynb.
• Results are saved in results/turingbench_finetuned.csv.

6. DeBERTa-v3-large Fine-Tuning & Ensemble (In Progress)

A microsoft/deberta-v3-large model is being fine-tuned on the full TuringBench training split with 512 tokens, 3 epochs, batch 8, and gradient accumulation 8 (effective batch 64) on Kaggle. Earlier attempts on Kaggle P100 failed due to FP16 incompatibility, so the current run uses FP32.

The ensemble script combines multiple fine-tuned models with a logistic regression over their AI-class probabilities.

---

Production Features & API Suite

1. Local Neural Observer Diagnostics

• Implemented in neural_detector.py.
• Evaluates token-level Perplexity (PPL) and Burstiness (surprisal standard deviation) using GPT-2 Small.
• Utilizes lazy loading to prevent overhead on API startup.

2. Stylometric Model-Source Group Attribution

• Trained in train_attribution.py and integrated in attribution.py.
• A multi-class Logistic Regression classifier trained on up to 30,000 samples per source group (~210,000 total) from data/corpus_features.parquet to predict the generative source group (not the individual model):
  • OpenAI (GPT-2 / GPT-3 / GPT-4 / ChatGPT grouped together)
  • Meta Llama
  • Mistral
  • Cohere
  • MPT
  • Human
• scripts/train_attribution.py reports held-out accuracy, a classification report, and a confusion matrix, and saves metrics to models/attribution_metrics.json. Run it to regenerate models/attribution_classifier.joblib.

3. Sentence-level Heatmap Analysis

• Implemented in sentence_analyzer.py.
• Utilizes a 3-sentence sliding window to evaluate localized probability scores, returning character offsets (start, end) to allow visual formatting on frontend clients.

4. API Hardening

• Located in api.py and public_api.py.
• CORS Support: Configurable via the CORS_ORIGINS environment variable; defaults to ["*"] only when unset.
• Authentication: Optional API key enforcement via API_KEY on /detect, /detect/batch, /detect/public, /detect/public/batch, and /public/detectors.
• Rate Limiting: Token-bucket limit of 60 requests per minute per IP applies to all endpoints above; state is bounded by RATE_LIMIT_MAX_IPS and stale entries are evicted after RATE_LIMIT_IP_TTL_SECONDS.
• Response Caching: In-memory MD5-keyed cache with TTL (CACHE_TTL_SECONDS) and a MAX_CACHE_ENTRIES eviction limit.
• Known Production Gaps: In-memory cache and rate-limit state are not shared across workers; Redis is recommended for multi-process deployments.

---

Quickstart

git clone https://github.com/vedangvatsa/ai-detection-at-scale.git
cd ai-detection-at-scale

# Set up environment
python -m venv .venv && source .venv/bin/activate
pip install pandas numpy scikit-learn joblib tqdm transformers torch

# Download pre-built data (3.5 GB) and models (4 GB)
python scripts/download_assets.py

# Start the inference API
uvicorn tool.api:app --host 0.0.0.0 --port 8000

Try the endpoint:

curl -X POST http://localhost:8000/detect \
  -H "Content-Type: application/json" \
  -d '{"text": "Furthermore, the results clearly demonstrate that this approach is effective.", "return_features": true}'

---

Project Layout

├── tool/                    Inference API + Feature Extractors
│   ├── api.py               Production API with Caching, Rate Limiting & CORS
│   ├── feature_extractor.py 35-feature extraction (11 used by production API)
│   ├── neural_detector.py   Perplexity & Burstiness observer (GPT-2)
│   ├── attribution.py       RAID multi-class source classifier
│   ├── sentence_analyzer.py Sliding-window sentence highlights
│   └── multilingual.py      5-language word lists
├── scripts/                 Pipeline & Analysis Scripts (01–14)
│   └── train_attribution.py Attribution model training script
├── results/                 CSV outputs + generated figures
├── models/                  Trained classifier joblibs
└── data/                    Corpus datasets

---

Known Limitations & Audit Notes

The following limitations have been identified and either fixed or explicitly documented. This list is meant to help reviewers and downstream users interpret the results correctly:

• Within-register AUC is inflated by document length. The paper acknowledges this confound (Section 3.2); the headline 0.933–0.978 range mixes length-matched and unmatched evaluations. Use the cross-domain AUC 0.728 as the most trustworthy generalization metric.
• GPT-4 detection AUC 0.983 is from the per-model breakdown in results/per_model_auc.csv (reproduced by scripts/05_mustdo_analyses.py), not a dedicated GPT-4-only benchmark.
• Public detector scores (HC3 0.9997, TuringBench 0.9146, MAGE 0.7801) are evaluations of existing HuggingFace models (Hello-SimpleAI/chatgpt-detector-roberta, roberta-large-openai-detector, etc.), not detectors we trained.
• Model-source attribution predicts model groups (OpenAI, Llama, Mistral, Cohere, MPT, Human), not individual models. scripts/train_attribution.py now reports held-out accuracy, a classification report, and a confusion matrix.
• Calibration now supports a trained Platt/isotonic calibrator (scripts/train_calibration.py); the script was previously broken because it imported non-existent modules. If no trained model is present, tool/calibration.py falls back to the original length-based heuristic.
• Adversarial defense in tool/adversarial_defense.py is a character-level preprocessor (homoglyph normalization, zero-width/control-char stripping, whitespace/punctuation cleanup). It does not defend against paraphrase, prompt injection, synonym substitution, or back-translation.
• API production gaps are partially closed: CORS is configurable via CORS_ORIGINS, cache has a TTL (CACHE_TTL_SECONDS), /detect, /detect/batch, /detect/public, /detect/public/batch, and /public/detectors now check API keys (API_KEY) and IP rate limits, and neural signal failures are caught with a safe fallback. Rate-limit state is bounded by RATE_LIMIT_MAX_IPS with stale-entry eviction; the cache remains in-memory, so a Redis-backed deployment is still recommended for multiple workers.
• Feature quality: 11 of the 35 features are standard stylometric metrics; the remaining 24 are extended counts. extract_features(..., use_pos_tags=True) enables NLTK POS tags for adjective, adverb, and nominalization densities; the default remains the legacy suffix heuristics so existing models are not broken. A duplicate 'flawed' in the NEGATIVE_WORDS list was removed. The capitalized_entity_density regex was tightened to require at least two consecutive capitalized words, reducing sentence-initial false positives.
• Reproducibility: scripts/download_assets.py now validates download sizes, retries failed transfers, and supports optional SHA256 checksum verification via --checksums <file>. A --verify-only mode is also provided. Run scripts/generate_checksums.py to create a checksum manifest from locally verified assets.
• Notebooks: Kaggle notebooks under notebooks/ were not individually tested in CI. Run scripts/audit_notebooks.py to check for hardcoded paths, missing tool.* imports, or missing scripts/ executables.
• Test coverage is still limited but now covers the defense preprocessor, feature extractor, and mocked API endpoints (tests/test_adversarial_defense.py, tests/test_feature_extractor.py, tests/test_api.py). Broader unit tests should be added before deployment.

---

Citation

@misc{ai_detection_at_scale,
  title={AI Detection at Scale: Cross-Register Stylometric Detection of AI-Generated Text},
  author={Vatsa, Vedang},
  year={2026},
  url={https://github.com/vedangvatsa/ai-detection-at-scale}
}