NLPCC 2026 Shared Task 6 — LLM-Generated Chinese Text Detection

Ternary classification of Chinese text: HWT (human-written, 0) · LGT (LLM-generated, 1) · HLT (LLM-refined, 2). Metric: macro-averaged F1.

This repository is the code & data package for our method. It contains the full data-processing / augmentation / training / inference source and the training-set statistics scripts. (The training data data/train_data.json is ~200 MB and is not committed to git — add it under data/ before zipping code_and_data.zip.)

---

Method: RoBERTa Creator-Editor with style de-shortcutting & editor-bias calibration

The classifier is a Creator-Editor dual-head over chinese-roberta-wwm-ext-large: each text gets 4 logits [creator_H, creator_L, editor_H, editor_L], decoded by the product rule (HWT=(H,H), LGT=(L,L), HLT=(H,L)). On top of the plain dual head we add:

1. Style de-shortcutting (--normalize_style): training human text carries web-crawl surface artifacts (half-width punctuation, repeated fragments) that a detector can shortcut on. We normalize them (CJK-context half→full-width punctuation + crawl-repeat collapse) uniformly across all classes, as a training-set data augmentation. The flag is persisted in the checkpoint (preprocess_config.json) so inference matches training.
2. Random-window augmentation (--rand_window) + windowed inference (--window): a 512-token encoder otherwise sees only the prefix of long texts; we train on random windows and average the logits over overlapping windows at inference (still one text in → one label out).
3. Content-controlled supervised contrastive (--contrast_weight): using parallel HWT/HLT/LGT triples of the same source, contrast on the Creator and Editor axes so the editor representation encodes the edit operation rather than content/domain. Training-only.
4. Generator-adversarial head (--gen_adv_weight): a gradient-reversal head over the 4 training generators makes features generator-invariant (for unseen generators). Training-only.
5. Editor-bias test-time calibration: the editor head over-predicts HLT out-of-distribution; we add a single fixed bias on the editor logit and recombine by the product rule. The bias is selected on a labeled leave-one-domain hold-out of the training set and then applied to each text independently (see Calibration below).

---

Compliance (NLPCC 2026 Task 6, Phase 2)

• Only the official training set is used. All augmentations (1–4 above, plus length-truncation and LGT-prefix alignment) are training-set transforms. No external data.
• No test data is used to develop or tune the detector. In particular the editor-bias is chosen on a training hold-out, never on the test set.
• Single-text prediction. Inference is per-sample: one text → 4 logits (windowed) → add the fixed editor_bias to the editor logit → product rule → label. No cross-sample / batch statistics are used; predict.py --text "…" runs the method on a single text.
• Training-set statistics scripts (the verifiable "features on the training set"): analysis/style_report.py (the punctuation/crawl artifact that motivates --normalize_style) and analysis/calibrate_editor.py fit (selects the editor-bias on the training hold-out).

---

Environment

pip install -r requirements.txt        # torch, transformers, pandas, scikit-learn, tqdm

Tested with Python 3.12 · PyTorch 2.6 (CUDA 12.4) · Transformers 5.4. The calibration scripts need only numpy.

For the reviewer-requested DeBERTa-v2-710M and MegatronBERT-1.3B experiments, including local model downloads, memory-saving flags, four flat/CEDAR runs and Phase-1/2 evaluation, see baselines/advanced_encoders/README.md.

---

Reproduce

1. Train (full training set)

python -m baselines.creator_editor.train_roberta \
    --model_name hfl/chinese-roberta-wwm-ext-large \
    --data_path  data/train_data.json \
    --output_dir checkpoints/ce-final \
    --normalize_style --rand_window \
    --contrast_weight 0.2 --gen_adv_weight 0.8 \
    --epochs 2 --batch_size 12 --lr 1e-5 --fp16

--batch_size counts triples when contrastive/adversarial are on (×3 samples/step). The (contrast, gen_adv) weights are chosen with a leave-one-generator OOD sweep (--ood_val_model).

2. Select the editor-bias on a training hold-out (compliant calibration)

# 2a) train a model holding out one domain (its slice will be unseen → reproduces OOD saturation)
python -m baselines.creator_editor.train_roberta ... --ood_val_domain Thesis \
    --output_dir checkpoints/ce-ho-thesis   # (same flags as step 1)

# 2b) build the labeled hold-out slice (from TRAIN) and predict it with that model
python analysis/make_oodval_probe.py --hold_out_domain Thesis --max_per_field 1500 \
    --out analysis/probes/holdout_thesis.json
python -m baselines.creator_editor.predict --arch roberta --window \
    --checkpoint checkpoints/ce-ho-thesis/best \
    --input analysis/probes/holdout_thesis.json \
    --output analysis/probes/holdout_thesis_pred.json

# 2c) pick the bias by max macro-F1 on the labeled hold-out (this is the statistics script)
python analysis/calibrate_editor.py fit \
    --ce_debug analysis/probes/holdout_thesis_pred_debug.json \
    --labels   analysis/probes/holdout_thesis.json --out analysis/calib.json   # → editor_bias

3. Inference + apply the fixed bias

# single text (demonstrates single-text usability)
python -m baselines.creator_editor.predict --arch roberta --window \
    --checkpoint checkpoints/ce-final/best --text "某段需要检测的中文文本"

# test set → CE logits, then apply the FIXED bias per-sample → submission
python -m baselines.creator_editor.predict --arch roberta --window \
    --checkpoint checkpoints/ce-final/best --input data/test.json \
    --output checkpoints/ce-final/test_pred.json
python analysis/calibrate_editor.py apply \
    --ce_debug checkpoints/ce-final/test_pred_debug.json \
    --editor_bias 2.5 --creator_bias 0 --out prediction.json

prediction.json is the submission format: [{"id": ..., "label": 0|1|2}, ...].

---

Layout

benchmark/                      data loading / preprocessing (normalize_style) / dataset
                                (rand_window) / leak-free & leave-one-domain splits
baselines/creator_editor/       the Creator-Editor model, training (contrast + gen-adv), inference
baselines/bert_roberta/         encoder 3-class baseline
baselines/advanced_encoders/    DeBERTa-710M / MegatronBERT-1.3B adapters + shell runners
analysis/
├── style_report.py             training-set surface-artifact statistics (de-shortcut evidence)
├── make_oodval_probe.py        build a labeled leave-one-domain hold-out from TRAIN
└── calibrate_editor.py         fit (select editor-bias on the hold-out) / apply (fixed, per-sample)