Explainable AI Text Risk Analyzer

An explainable, cautious AI-generated text risk analyzer for coding agents and local workflows.

简体中文 · Install · Evaluation · Use Cases · Detector API

🚀 New: the Lynote AI Detector API is live. Need AI-text detection inside your own app, LMS, or moderation pipeline? Call Lynote's production detector with one REST request — no model to train or host. Get an API key → lynote.ai/ai-detector-api

Other Quality Projects
AI Humanize Text:https://github.com/lynote-ai/humanize-text
AI Image Detector:https://github.com/lynote-ai/ai-image-detector

Two tiers in this repo. This page documents the lightweight local analyzer — an explainable, no-network CLI/skill for fast triage. The production two-level model (trained, multilingual, served) lives in detection-model-pipeline/; see How the production model works.

Why This Exists

Most AI text detectors are either overconfident, opaque, or awkward to embed inside agent workflows.

ai-detector-skill takes the opposite approach:

• explainable weighted signals instead of hidden model claims
• a local CLI and Python API that are easy to script
• a short-text guardrail that refuses to overstate weak evidence
• skill-ready packaging for Codex, Claude Code, and other repo-aware agents
• reproducible benchmark and dataset evaluation scripts

If you want a triage tool that stays cautious and leaves room for human review, this repo is built for that.

How the production model works

The heavyweight detector behind Lynote is a two-level model that, in one forward pass, labels a document (human / ai / mixed) and every sentence (human / ai / paraphrased). It is trained, not rule-based — the full pipeline (data prep → train → calibrate → eval → serve) lives in detection-model-pipeline/.

flowchart LR
    csv["detection-log export (CSV)"] --> prep["prepare · dedup + soft labels"]
    prep --> train["train · joint doc + sentence heads"]
    hub["public backbone · XLM-R / ModernBERT"] --> train
    train --> cal["calibrate · temperature scaling"]
    train --> eval["eval · two-level metrics"]
    cal --> serve["serve · FastAPI, {code,msg,data}"]

• One model, two levels. Document and sentence heads share a backbone; L = L_doc + α·L_sent. Differentiable sentence-probability statistics feed the document head — which is what makes the mixed class trainable.
• Multilingual backbones. XLM-R-large (default) or ModernBERT-base, optimized for en / zh / pt / es.
• Calibrated, not overconfident. Temperature scaling softens probabilities without changing predictions; confidence bands reject < 0.33 ≤ low < 0.6 ≤ medium < 0.8 ≤ high.
• Deterministic & bounded. Hash-pinned splits, synthetic demo data, and make check gates (manifest / demo / docs / tests). No weights, datasets, or service logs are shipped, and no accuracy claims are made.

Full method: MODEL.md · ARCHITECTURE.md · DATA_CONSTRUCTION.md · API.md

Lynote AI Detector API — for developers

The local analyzer here is built for cautious, offline triage, and the production model above ships as a pipeline, not weights. If you need production-grade detection inside your own product, the Lynote AI Detector API serves Lynote's production detector through a single endpoint:

• One synchronous REST call — send text, get structured JSON back with an overall classification, confidence, and sentence-level signals.
• Covers major models — detects patterns associated with ChatGPT, Gemini, Claude, DeepSeek, and more.
• Multilingual — the same endpoint handles multiple languages.
• Server-side keys — Bearer-token auth, with separate credentials per environment.
• Simple billing — 1 credit per input word (Chinese is counted per character: 1 character = 1 word).

curl https://api.lynote.ai/v1/detect \
  -H "Authorization: Bearer $LYNOTE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text": "The rapid adoption of large language models has reshaped how teams write."}'

Plans, rate limits, and full docs: lynote.ai/ai-detector-api. As with any detector, treat results as a signal for human review, not proof of authorship.

Install

pip install -e .
ai-detect examples/sample_ai_like.txt --json

Or bootstrap a local environment:

bash scripts/setup.sh

Example output:

Conclusion: AI-like signals are present, but this medium-confidence score is a risk estimate rather than proof.
Score: 84/100
Confidence: medium
Verdict: high_ai_likelihood
Words analyzed: 256

What You Get

• score: 0-100 AI-like writing risk estimate
• verdict: insufficient_text, low_ai_likelihood, mixed_or_uncertain, or high_ai_likelihood
• confidence: currently low or medium
• signals: strongest weighted evidence signals
• caveats: warnings that stay attached to the result
• next_steps: practical follow-up actions when useful

This is designed to help an agent say:

• "AI-like signals are present."
• "The sample is too short for a meaningful estimate."
• "This should be reviewed against known writing samples."

Not:

• "This was definitely written by AI."
• "The detector proves misconduct."

Usage

CLI

ai-detect examples/sample_ai_like.txt
cat essay.txt | ai-detect --json
python scripts/detect.py examples/sample_human_like.txt --json

Python API

from aidetect import analyze_text

text = open("essay.txt", encoding="utf-8").read()
result = analyze_text(text)

print(result.score, result.confidence, result.verdict)
for signal in result.strongest_signals():
    print(signal.name, signal.note)

Local Skill Install

cp -R ai-detector-skill "$CODEX_HOME/skills/"

Use the root SKILL.md as the portable skill definition, and keep AGENTS.md at the repo root for repo-aware agents.

Evaluation

We ran a reproducible evaluation on the public HC3 dataset, using the English finance, medicine, and open_qa subsets with the first 100 rows from each subset.

Snapshot of the lightweight local analyzer on that slice (not the production two-level model):

• Human mean score: 5.4
• AI mean score: 18.4
• Mean separation: 13.0 points
• Human coverage: 0.427
• AI coverage: 0.920
• Covered accuracy at score >= 45: 0.317

What that means in practice:

• the detector separates human and AI answers on average, but only weakly on HC3
• the short-text guardrail is doing useful work, especially on shorter human answers
• the current thresholds are conservative, which keeps false confidence down but also lowers recall
• this tool works better as triage + explanation than as a stand-alone classifier

See the full report in docs/HC3_EVALUATION.md.

Reproduce it with:

make eval-hc3

Use Cases

Teacher Triage

A teacher receives a polished 400-word reflection and wants a cautious signal before doing manual review.

Suggested workflow:

1. Run ai-detect submission.txt --json.
2. Read the strongest signals and caveats.
3. Compare the passage with known writing samples before making any judgment.

Editorial Review

An editor wants to spot formulaic product reviews or guest posts before spending time on manual edits.

Why it fits:

• medium-length prose works better than short snippets
• explainable signals help justify why a draft feels templated

Trust And Safety Queueing

A moderation team wants to sort suspicious long-form posts into a manual review queue, not auto-remove them.

Why it fits:

• the tool is conservative by design
• it helps with prioritization more than enforcement

Internal Content QA

A team compares human drafts and AI-assisted drafts to see where language starts sounding too generic.

Why it fits:

• the score is useful as a relative signal across versions
• strongest signals can guide rewriting

Not For

• disciplinary decisions about a named student or employee
• treating a single score as proof of cheating or fraud
• very short samples under about 80 words
• high-stakes authorship disputes without known-sample comparison

Project Structure

detection-model-pipeline/   # production two-level model — prepare → train → calibrate → eval → serve
├── code/                    # pipeline stages (i..m) + harness validators
├── model_entity/            # joint model: ModernBERT + XLM-R backbones
├── data/demos/              # synthetic demo data (hash-pinned)
└── docs/                    # ARCHITECTURE / MODEL / DATA_CONSTRUCTION / API

ai-detector-skill/           # lightweight local analyzer (this page)
├── SKILL.md
├── scripts/
│   ├── detect.py
│   ├── setup.sh
│   ├── benchmark.py
│   └── evaluate_hc3.py
├── references/
│   └── api-reference.md
├── assets/
│   ├── hero.svg
│   ├── score-bands.svg
│   ├── workflow.svg
│   └── templates/
│       └── report.md
├── src/aidetect/
├── tests/
├── AGENTS.md
└── README.md

Dev Commands

make test
make demo
make benchmark
make eval-hc3

CI

GitHub Actions automatically runs:

• make test on Python 3.9, 3.11, and 3.13
• make benchmark to regenerate the synthetic benchmark report
• make eval-hc3 to regenerate the HC3 evaluation report
• upload of docs/BENCHMARK.md and docs/HC3_EVALUATION.md as workflow artifacts

Contributing

See CONTRIBUTING.md.

Good contributions usually improve one of these:

• clearer evidence signals
• safer wording and UX
• multilingual handling that stays explainable
• reproducible evaluation coverage
• agent integration ergonomics

Need an API?

🔌 Visit lynote.ai/ai-detector-api to get your AI Detector API key.

License

MIT