cot-monitor-robustness

When does activation-level monitoring beat text-level monitoring of chain-of-thought — and do the two channels fail independently under adversarial pressure?

This repo contains the full pipeline and analysis for a study of monitor robustness on a silent hint-use task: a model (Qwen 3.6 27B) is given multiple-choice questions with an injected hint pointing to a specific wrong answer, and we study transcripts where it follows the hint without ever verbalizing it. Four monitors score each transcript, then we attack the chain-of-thought with adversarial rewrites and measure how each monitor degrades — and whether text-level and activation-level failures are correlated per item.

Headline results

• The text judge is robust to rewording and laundering but breaks under social engineering: recruit rewrites (which address the monitor directly with verification claims and authority framing) cut its AUC from 0.917 to 0.635. The probe moves much less (0.736 to 0.630) — injected text is just more tokens to the activations.
• Text and activation failures are roughly independent per item (phi ≈ +0.09 / +0.03 / +0.10 for paraphrase / launder / recruit, all Fisher p > 0.7): OR-ing the two channels recovers 8–17 points of catch rate over the judge alone. Defense-in-depth is supported — with the caveat that at n ≈ 35 per attack only large correlations were detectable.
• Mandatory controls: shuffled-label probe at chance; a presence-vs-use decomposition showing the hint-presence direction alone does not explain the probe's hint-use AUC; a correctness-confound control.

All figures are in figures/ (design schematics, clean leaderboard, score distributions, rewrite anatomy, degradation heatmap and CIs, failure independence, probe learning curve / interrogation / correctness confound); every number in them is read from results/ artifacts by src/10, 11, 15, 17, 24.

Reproduce the results (10 minutes, CPU only)

All numbers and figures regenerate from saved artifacts — no GPU, no API keys. The artifacts live in a gated Hugging Face dataset (gated because the question pool includes GPQA-diamond items, whose authors ask that question text not be posted openly; accepting the terms is one click).

pip install -r requirements.txt
hf auth login                       # needed once, for the gated dataset
jupyter lab notebooks/reproduce_results.ipynb

The notebook downloads the dataset, recomputes the leaderboard AUCs (with bootstrap CIs), the degradation matrix, and the failure-correlation tables from the raw score parquets, asserts they match the published JSON artifacts to numerical precision, refits the activation probe from pooled features, and regenerates the figures.

A second notebook, notebooks/jlens_readout_audit.ipynb, opens up the J-Lens arm (the one that scored at or near chance): it loads the saved per-position top-10 vocabulary readouts for a seeded 20-per-class sample of transcripts (core/jlens_readouts/, written by src/19_jlens_readouts.py), checks they reproduce the published hint-salience scores, and audits what the lens actually surfaces on POS vs NEG transcripts — lens sanity by layer, hint-letter salience at positions the text never mentions it, vocabulary enrichment against a permutation null, concept lexicons, and a transcript viewer. CPU only, no tokenizer or model needed. Its companion notebooks/jlens_readout_audit_late.ipynb runs the same audit on the late-layer rerun of the arm (layers 56/60/62, src/22_jlens_late.py): src/21_jlens_anchor_check.py showed Qwen 3.6 27B is undecodable at 16/32/48 (next-token top-10 hit 4–8%) and only becomes readable from layer ~56 (40%) to the lens anchor 62 (75%). Re-run there, J-Lens is still at chance (clean AUC 0.503, CI 0.38–0.61), so the null is a real negative for the unsupervised readout rather than a wrong-layer artifact.

Dataset: mjkenney/cot-monitor-robustness (questions, hints, labels, canonical + attacked transcripts, judge scores and justifications, pooled activations, all result artifacts; full per-token activations in an optional 17 GB config needed only for the J-Lens arm).

Reproduce the full pipeline (GPU, ~1–2 days, ~$100)

The generation pipeline (src/00–09) needs an 80 GB-class GPU (we used an H200 on RunPod), vLLM for rollouts, HF transformers for activation caching, and an OpenRouter key for the judge (Claude Opus 4.5) and attacker (Kimi K3). Stages, in order:

Every stage writes artifacts to results/ and human-readable random samples to review/; analysis scripts only ever read artifacts. All seeds are in config.yaml. scripts/setup_pod.sh records the exact pod environment.

The study design was written down before the data came in: docs/preregistration.md. Technical reports (every number traced to an artifact): docs/study_report.md (consolidated), docs/jlens_diagnosis.md (why the J-Lens arm was at chance, late-layer rerun, readout audits), docs/probe_c_sensitivity.md, docs/dupfix_diff.md (data-integrity fix, before/after), docs/hint_types_figure.md.

Data-integrity note (2026-09-02)

16 of the 284 items in the analysis set carry two canonical transcripts (a hinted one → POS or NEG-inert, and an unhinted one → NEG-clean). The original pipeline keyed its activation, judge and lens caches by item id alone, so each pair shared one cached transcript. Every cache is now keyed by transcript (tkey, see src/23_dupfix_migrate.py), the 32 affected transcripts were re-cached / re-judged, and all downstream stages were re-run; the numbers above are the corrected ones. The change moved the hint-blind judge from 0.895 to 0.917 and left the probe at 0.735 → 0.736; the before/after diff of every artifact is in docs/dupfix_diff.md.

Limitations (stated up front)

One model, one unfaithfulness type (hint-following), hand-designed rather than optimized attacks, n = 100 silent-positive items (CIs are wide — phi estimates especially are underpowered), forced-forward-pass activations over rewritten CoTs (measures the model processing a laundered CoT, not producing it), and residual label noise from LLM-assisted labeling (hand-checked samples in review/).

Citation

Michael Kenney, Monitor reliability under adversarial pressure: text vs activation channels on silent hint-use, 2026. MATS 12.0 application project (Neel Nanda stream).