SlopShape - release package

Companion package for the paper "SlopShape: Identifying AI-Generated Commercial Web Content" (Jochen Madler, Sitefire; arXiv:2609.15369, https://arxiv.org/abs/2609.15369). It contains everything needed to verify every number in the paper and to rebuild the study end to end, without redistributing any copyrighted post text.

License (LICENSE file): the code (study_b/, code/, setup.sh, requirements.txt, env.example) is released under the PolyForm Noncommercial License 1.0.0; everything else is all rights reserved, public for verification and audit. Researchers who want to reuse the instrument, prompts, or data files beyond verification, need a commercial code license, or need the gated document-level data (per-document feature answers, briefs, mirrors, model refits), can request access under a non-commercial research agreement: jochen@sitefire.ai.

The study

StoryScope (Russell et al. 2026, arXiv 2604.03136) showed AI-generated fiction is detectable from discourse-level structural choices alone (93.2 macro-F1), with a small feature core, per-model fingerprints, and human stories occupying rarer regions of structural space. We replicate the full pipeline on a domain with none of fiction's machinery: 2,250 pre-ChatGPT human B2B blog posts (268 company domains, Wayback Machine snapshots) paired with 11,250 AI mirrors from the same five generator models, measured with a B2B-native 214-feature structural instrument frozen before analysis.

Findings: structural features alone reach 98.0 test macro-F1 (95% domain-cluster CI 96.7-99.2) on domain-disjoint splits; the structural and style instruments fail on nearly disjoint documents (19 vs 101 test errors, 4 shared); every secondary phenomenon replicates, consistent in direction and larger in magnitude - variant ordering (98.1 > 98.0 > 88.1), a 10-feature core at 93.5, six-way source attribution at 79.2 (chance 16.7), and an amplified human rarity gap (Cohen's d 1.83 vs the original's 0.83). The structural signal survives rewording: with every AI test post rewritten by its own model, structural detection is unchanged (98.0 -> 98.1 macro-F1). A human gold session validates the LLM-run instrument (human-human kappa 0.928, human-model 0.946). Claims are scoped to single-pass generation from the five studied AI models, both as generated and after rewording.

What is in this package

The analysis scripts ship in study_b/ at the repository root; the vendored original pipeline is created locally by setup.sh (a pristine clone of the upstream StoryScope repository at its pinned commit, with our declared deviations applied from artifacts/our-fork.patch). Nothing else is required.

What is gated, and why

Per-document feature answers (from 148,500 scoring calls, plus 15,950 rescoring calls for the rewording test), deterministic model refits, briefs (2,250), mirrors (11,250), and the rewritten test mirrors (1,450) are available to researchers on request under a non-commercial research agreement; MANIFEST.md lists them with sizes. Two reasons, stated in the paper's competing-interests section: the author operates Sitefire, a commercial GEO/AEO product, and plans a content-scoring product informed by this work, which makes the ready-to-use scoring assets direct product substrate; and the mirrors embed content derived from copyrighted source posts. Everything gated is rebuildable from this public release. Request process: email jochen@sitefire.ai; access is granted under a non-commercial research agreement.

How to verify the paper

1. Look up any paper number in VERIFICATION.md; open the named artifact file.
2. Check integrity: every data file's sha256 is in MANIFEST.md, and the pre-training freeze manifest's hash chain (taxonomy, exclusions, style boundary, splits) is verifiable in-package - commands in instrument/README.md.
3. To recompute rather than read: the regeneration script named in VERIFICATION.md rebuilds the artifact; the faithful-protocol scripts assert the 0.9803 headline on refit and fail loudly on divergence. Recomputation of classifier numbers requires the gated encoded matrix or a full rebuild (below).

How to rebuild from scratch

0. Environment: run setup.sh (pinned virtualenv from requirements.txt; clones the pinned upstream pipeline into vendor/storyscope and applies artifacts/our-fork.patch; verifies with study_b/verify_reference.py).
1. Corpus: replay fetch/ledger.csv against the Wayback Machine (deterministic; no LLM calls; fetch/README.md). Full funnel re-run from the frames is also scripted, seeds committed in-script.
2. Generation: briefs then mirrors with the five generator models (study_b/extract_briefs.py, study_b/generate_mirrors.py; ~$160 at 2026 prices).
3. Measurement: schema is frozen (artifacts/TEMPLATE_SCHEMA.md); templates, comparison, discovery, screen, dedup, and application per code/README.md (~$1,350; gpt-5.6-terra, gemini-3.6-flash, F2LLM-4B embeddings).

Measured total cost of the original run was ~$1,650. Model availability caveat: the pipeline pins 2026 model versions by exact name; a rebuild after provider deprecation is a new measurement, not a verification.

Future work

Scoring posts produced with Sitefire's own product with this instrument is planned as separate work. It is not part of this study.

License

See the LICENSE file. The code directories - study_b/, code/, plus setup.sh, requirements.txt, and env.example - are licensed under the PolyForm Noncommercial License 1.0.0 (noncommercial research use permitted; commercial use by separate agreement). Everything else in the package - the artifacts, instrument, prompts, fetch data, manifest, documentation, and figures - remains all rights reserved, published for verification and audit, with reuse available to researchers on request (jochen@sitefire.ai). The vendored original pipeline (created locally by setup.sh, not distributed here) retains its upstream license.

Citation

Madler, J. (2026). SlopShape: Identifying AI-Generated Commercial Web Content. arXiv:2609.15369. https://arxiv.org/abs/2609.15369

@misc{madler2026slopshape,
  title         = {SlopShape: Identifying AI-Generated Commercial Web Content},
  author        = {Madler, Jochen},
  year          = {2026},
  eprint        = {2609.15369},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2609.15369}
}

A CITATION.cff file is included for GitHub's "Cite this repository" button.

Please also cite the original study this work replicates: Russell et al. (2026), StoryScope: Investigating idiosyncrasies in AI fiction, arXiv 2604.03136.