writing-eval

Built by David Paluy from Majestic Labs.

writing-eval gives teams a repeatable way to measure a draft against a chosen editorial voice. Build a reusable style profile from reference prose, check a draft against it, and get specific evidence about differences in clarity, readability, sentence rhythm, vocabulary, and detected writing patterns.

The project also includes a corpus evaluation pipeline for comparing generated outputs with a reference corpus. Use it for local diagnostics, regression checks, and repeatable comparisons between writing systems.

Everything runs locally on the CPU and produces deterministic results. The tool does not call hosted models, upload source material, train models, optimize detectors, or reproduce a proprietary evaluation method.

Installation

Python 3.11 or newer and uv are required.

uv sync

The release version is the [project].version value in pyproject.toml. Update this value for every release. To show the installed release version, run:

./writing-eval --version

Quick start

Build a style profile from a directory of an author's posts, then check a draft against it:

./writing-eval profile build acme --from posts/acme
./writing-eval check draft.md --style acme

The first command ingests the .md and .txt files under posts/acme and writes a reusable profile named acme into data/profiles/acme/. The second audits draft.md and produces a scored, profile-relative assessment. The human-readable report shows four section scores, actionable issues with current and target values, editing instructions, success criteria, source locations, and general statistics. The profile name is kept out of the report body; it remains in JSON metadata for reproducibility.

Use with LLM agents

The repository includes an agent skill at skills/writing-eval/SKILL.md. Agent harnesses that support SKILL.md instructions can load it to select the correct command, build or choose a profile, interpret exit codes and JSON, protect private source material, and report results without overstating the heuristic score.

The skill controls the local CLI and does not bundle the executable. Use it from a repository checkout after running uv sync.

Example requests:

Use writing-eval to check docs/draft.md against the acme profile. Summarize the
highest-priority issues with their source locations.

Build a writing-eval profile named product-docs from the authorized prose in
data/product-docs, then check docs/new-guide.md against it.

Compare the writing systems in runs/release-candidate against
data/reference-corpus.jsonl and explain the report verdict.

What it includes

Why we built this

AI-generated writing is easy to demo and difficult to evaluate consistently. A draft can be grammatically correct while still missing an organization's voice, preferred structure, or editorial constraints.

writing-eval turns those expectations into a local, versioned measurement process. Teams can run the same checks after changing a prompt, model, reference corpus, or rule set and see what improved or regressed.

This project is one narrow example of a broader Majestic Labs principle: an AI workflow needs a company-controlled definition of acceptable work. Read Building Private AI Evals for the broader approach.

Documentation

• Use with LLM agents
• Single-document checks
• Style profiles
• Metrics
• Corpus evaluation and benchmark
• Limitations
• License and contributions
• Managed hosting
• About Majestic Labs

Single-document check

./writing-eval check audits one draft the way a linter audits one source file, without any JSONL wrapping. It takes a Markdown or plain-text file, or - to read from standard input.

./writing-eval check draft.md
./writing-eval check draft.md --references data/reference-corpus.jsonl
cat draft.md | ./writing-eval check -

Options:

• --rules selects the rule file (default: the builtin rule set that ships with the package). The repository also ships an optional overlay with extra AI-writing tells; see The anti-ai overlay.
• --references is an optional JSONL reference corpus. When omitted, the token 1-gram L2 metric is skipped, rendered as n/a, and a note is printed to standard error.
• --style compares the draft against a named style profile and renders a scored assessment (see Style profiles). It is mutually exclusive with --references; passing both is a usage error.
• --profiles-root locates profiles for --style (default data/profiles).
• --format text|json selects human-readable text or JSON on standard output (default text).
• --json PATH writes the same JSON result to a file while preserving the selected standard-output format.

Without --style, text output retains the linter format: one finding per line, sorted by position, using 1-indexed line and column offsets computed from the real character positions of each match, followed by a metrics block:

draft.md:1:1 [warn] metadiscourse_openers: Remove the metadiscourse opener and state the point directly. | span: In this article,
draft.md:1:21 [warn] polish_vocab: Replace overused polish vocabulary with specific language. | span: delve
metrics:
  word_count: 15
  tell_rates_by_severity:
    warn: 400.000000
  mean_sentence_length: 7.500000
  sentence_length_variance: 2.250000
  repeated_opening_rate: 0.000000
  token_1gram_l2: n/a
quality_metrics (informational):
  flesch_reading_ease: 52.000000
  flesch_kincaid_grade: 9.000000
  mtld: 14.000000
  paragraph_stats:
    paragraph_count: 1.000000
    mean_paragraph_sentence_count: 2.000000
    single_sentence_paragraph_rate: 0.000000

The example values are illustrative. Scores below 10 tokens render mtld as n/a, and text without a sentence renders the readability scores as n/a. With --style, text output uses the scored assessment described in Check a draft against a profile.

Exit codes distinguish completed checks from input errors:

• 0: the check completed, with or without findings.
• 1: a usage or input error (missing file, unreadable rules, invalid JSONL).

Style profiles

A style profile is a deterministic fingerprint of one author's writing, built from a corpus of their prose. Build a profile once, then check any draft against it to see how far the draft sits from that voice and which vocabulary and structure differ. The author's own voice is just one profile among many.

Build a profile

./writing-eval profile build <name> --from <dir-or-files...> [--profiles-root data/profiles] [--rules PATH]

<name> must be exactly one non-absolute path component. ., .., absolute paths, and nested names are rejected.

--from accepts directories (their .md and .txt files are ingested recursively) or individual files. --rules selects the rule file used to precompute the profile cache (default: the builtin rule set). A leading YAML frontmatter block is stripped from each source, and each document becomes one reference record with a stable ID derived from its filename. The command writes two files into <profiles-root>/<name>/:

• references.jsonl: one {"id", "text", "file"} record per source document, reused as the reference corpus by check --style.
• profile.json: the profile name, its creation date, a numeric metrics_version field (currently 2), a references_sha256 digest of the paired references.jsonl file, per-source word counts, the total word count, and the corpus statistics (mean sentence length and variance, repeated-opening rate, Flesch reading ease and grade, MTLD, paragraph statistics, and the top 20 content tokens after a small stop list).

metrics_version pins the metric semantics the stored statistics were computed with. Version 2 covers curly-apostrophe (U+2019) sentence openers, markdown-aware readability word counts, and MTLD tail, threshold, and sequence-input lowercase behavior. Older profiles are rejected with a rebuild instruction until profile build runs again with the current tool. references_sha256 binds profile.json to the installed references.jsonl. A missing, invalid, or mismatched digest is the same rebuild error, so a mixed pair from an interrupted write is not loaded.

For example, put all articles for one author under a dedicated directory:

posts/acme/
├── choosing-a-market.md
├── distribution-first.md
├── founder-notes.txt
└── archive/
    └── early-lessons.md

Then build the profile from the directory:

./writing-eval profile build acme --from posts/acme

The directory is scanned recursively, so this imports all four .md and .txt articles, including archive/early-lessons.md. There is no need to write a wildcard or list every file. A successful build reports the number of imported sources and words, for example:

built profile 'acme': 30 sources, 15742 words -> data/profiles/acme

To import selected articles instead, list each file after --from:

./writing-eval profile build acme --from \
  posts/acme/choosing-a-market.md \
  posts/acme/distribution-first.md

Recommended corpus size

The command accepts a single non-empty article, but a small profile makes the score depend on which articles you happened to include. For a profile used as a style baseline, use at least 25 articles, and prefer 40 or more.

That number is measured, not assumed. Holding a draft fixed and varying only which articles form the profile, the standard deviation of the resulting score falls with more articles. Two points is the rubric's smallest unit, the deduction for one excess warn occurrence, so below that threshold the sampling noise is smaller than anything the score can express. The mean standard deviation crosses 2 points around N = 20, but individual drafts vary widely, so the number that matters is coverage: the share of drafts that have actually settled below 2 points at a given profile size.

25 articles covers 7 in 10 drafts. 30 articles is not reliably better than 25; the difference sits inside trial noise. 40 articles covers 9 in 10, and 50 covers 24 in 25, which is why 40 is the preferred target. As a rule that holds at every corpus size tested: treat a score difference under 3 points as noise, whether between two drafts or between two runs of the same draft.

Article count drives stability, not word count. At a fixed word budget, a profile built from more, shorter articles is consistently more stable than one built from fewer, longer articles. At 40,000 words, 28 articles gave a standard deviation of 1.3 while 11 articles gave 2.5. Prior versions of this document recommended a 15,000-word minimum; that figure was not supported by measurement and has been removed. At a fixed article count, per-article length still matters up to a point: a corpus averaging about 1,200 words per article was roughly 0.4 points noisier than one averaging about 2,350, with no further gain past about 2,350 words per article. The floor below 517 words per article, the shortest article in the study corpus, is unmeasured.

Method, full results, and the reproduction script are in docs/profile-size-study.md. The study used one 101-article corpus of long-form nonfiction by a single author. The direction of the effect should hold generally; the exact crossover point may move for other genres and article lengths. Run the script on your own corpus to check.

Prefer articles from the same author and the kinds of writing the profile should represent; mixing unrelated authors, genres, or registers creates a blended profile.

Update a profile

There is no incremental append command. Keep the article directory as the authoritative corpus, add the new article to it, and rebuild using the same profile name:

posts/acme/new-article.md

./writing-eval profile build acme --from posts/acme

Rebuilding replaces data/profiles/acme/references.jsonl and data/profiles/acme/profile.json with results computed from every article currently in posts/acme. Always pass the complete corpus when rebuilding. Passing only posts/acme/new-article.md would replace the profile with a one-article profile rather than add that article to the existing profile.

List profiles

./writing-eval profile list [--profiles-root data/profiles]

One line per profile that load_profile accepts: name, source count, and total words. Directories that fail that load (wrong metrics_version, mismatched references_sha256, unreadable metadata) are omitted. The CLI prints a skip note on stderr for each omitted directory.

Profile cache

profile build precomputes reference statistics into <profile>/cache/. After a rule change, refresh that cache with:

./writing-eval profile cache <name> [--rules PATH]

--rules must match the rule file that later check runs will use. Caches also invalidate automatically when detector or tokenizer code changes in a new release, since each entry records a digest of that code. Checks remain correct with a stale or missing cache; they only get slower until the cache is rebuilt.

Check a draft against a profile

./writing-eval check draft.md --style <name> [--profiles-root data/profiles]

--style resolves the profile's references.jsonl as the reference corpus, so it is mutually exclusive with --references. The default text report is Markdown-shaped so both a person and an LLM can use it directly:

# Writing Evaluation

File: `draft.md`

## Article score (heuristic)

**86/100 - Moderate alignment**

This score measures detected style patterns and alignment with the target
profile. It does not measure factual accuracy or overall content quality.

| Section | Score |
|---|---:|
| Clarity and directness | 25/25 |
| Readability | 23/25 |
| Rhythm and structure | 13/25 |
| Vocabulary and style | 25/25 |
| **Total** | **86/100** |

## Issues to improve

### 1. Sentence rhythm differs from the target profile.

- Section: Rhythm and structure
- Priority: High
- Deduction: -8 points

| Measure | Article | Target profile | Direction |
|---|---:|---:|---|
| Average sentence length | 8.1 words | 16.0 words | increase |
| Sentence-length variance | 29.3 | 104.8 | increase |

Editing instruction:

Combine selected explanatory sentences and adjust the mixture of short, medium,
and long sentences toward the target profile. Preserve deliberate emphasis; do
not mechanically force every sentence to the target.

Success criteria:

- Move average sentence length closer to the target.
- Move sentence-length variation closer to the target.
- Preserve short or long sentences that serve a clear rhetorical purpose.

## General statistics

| Statistic | Article | Target profile | Interpretation |
|---|---:|---:|---|
| Word count | 485 | n/a | informational |
| Average sentence length | 8.1 words | 16.0 words | shorter than target |
| Repeated openings | 18.5% | 8.2% | higher than target |
| Reading ease | 59.8 | 60.8 | closely aligned |

The example is abbreviated. Each issue includes stable identifiers in JSON, numeric current and target values, an editing direction, an instruction, success criteria, and any known line and column locations. Consecutive repeated-opening findings are grouped into one run, so deliberate anaphora can be reviewed as a pattern instead of as several disconnected warnings.

Raw rule findings remain in JSON. For a profile check, each rule is also run over the profile references. The profile occurrence rate is scaled to the draft length and rounded up to an allowance. Only occurrences above that allowance appear as rule issues. This lets an author's demonstrated style outrank a generic rule while keeping the evidence visible.

Excess warn findings appear under Issues to improve and can lower the score. Excess info findings appear separately under Review candidates and never lower it. There is no strengths section. If the draft has no scorable prose sentence, the report is explicitly Unscored instead of assigning a misleading number.

JSON for LLMs and automation

Print JSON to standard output:

./writing-eval check draft.md --style acme --format json

Keep the human report on standard output and write the same JSON payload to a file:

./writing-eval check draft.md --style acme --json evaluation.json

Profile checks preserve the raw file, findings, metrics, quality_metrics, and style_gap fields and add an assessment object. This abbreviated example shows the shape:

{
  "assessment": {
    "schema_version": 2,
    "rubric_version": "profile-alignment-v2",
    "basis": "rules_and_target_profile",
    "status": "scored",
    "profile": {
      "id": "acme"
    },
    "score": {
      "total": 86,
      "maximum": 100,
      "label": "Moderate alignment",
      "sections": [
        {
          "id": "clarity_directness",
          "label": "Clarity and directness",
          "score": 25,
          "maximum": 25,
          "deduction": 0
        },
        {
          "id": "readability",
          "label": "Readability",
          "score": 23,
          "maximum": 25,
          "deduction": 2
        },
        {
          "id": "rhythm_structure",
          "label": "Rhythm and structure",
          "score": 13,
          "maximum": 25,
          "deduction": 12
        },
        {
          "id": "vocabulary_style",
          "label": "Vocabulary and style",
          "score": 25,
          "maximum": 25,
          "deduction": 0
        }
      ]
    },
    "issues": [
      {
        "id": "sentence_rhythm",
        "kind": "improvement",
        "section": "rhythm_structure",
        "priority": "high",
        "deduction": 8,
        "summary": "Sentence rhythm differs from the target profile.",
        "comparisons": [
          {
            "metric": "mean_sentence_length",
            "label": "Average sentence length",
            "current": 8.137931,
            "target": 16.031049,
            "delta": -7.893118,
            "direction": "increase",
            "unit": "words_per_sentence"
          }
        ],
        "instruction": "Combine selected explanatory sentences while preserving deliberate emphasis.",
        "success_criteria": [
          "Move average sentence length closer to the target."
        ],
        "locations": []
      }
    ],
    "statistics": [
      {
        "id": "word_count",
        "label": "Word count",
        "value": 485,
        "unit": "words",
        "target": null,
        "interpretation": "informational"
      }
    ]
  }
}

JSON numbers retain full precision. The profile ID stays in JSON for reproducibility but is rendered only as Target profile in the human report. The unscored form uses status: "unscored", a reason, null total, label, and section scores, an empty issue list, and the available statistics. The assessment's rule_baseline object records the profile and draft word counts and one sorted entry per observed rule: profile count, profile rate per 1,000 words, scaled draft allowance, raw draft count, and excess count.

Scoring rubric

The profile-alignment-v2 rubric starts four sections at 25 points. Version 2 adds profile-relative rule allowances; direct library callers that omit rule baseline data retain the version 1 contract.

• Clarity and directness: directness-related rule findings.
• Readability: Flesch reading ease and Flesch-Kincaid grade relative to the profile.
• Rhythm and structure: sentence length, sentence-length variance, and repeated openings relative to the profile.
• Vocabulary and style: all other style rules.

Rule deductions are 2 points per excess warn occurrence and 0 per info occurrence. An excess occurrence is one above the allowance derived from the selected profile's aggregate rate. A rule absent from the profile has an allowance of zero. A rule is assigned to exactly one section, and repeated openings are scored only in rhythm and structure to prevent double-counting. Section deductions are capped at 25.

Profile-relative metric deductions use a tolerance before any points are removed, then increase linearly to a cap:

For sentence length and variance, the relative gap is abs(article - target) / max(abs(target), 1). Other rows use the absolute gap. At or below the tolerance, the deduction is zero. Above it, the unrounded deduction is:

maximum × (min(gap, cap) - tolerance) / (cap - tolerance)

Each issue deduction is rounded half up to an integer before the section cap is applied. The displayed arithmetic is invariant:

sum(issue deductions)
  = sum(section deductions)
  = 100 - total score

Score labels describe alignment, not universal quality:

MTLD, token 1-gram L2, overrepresented terms, and paragraph statistics remain informational. The MTLD comparison is especially sensitive to comparing one draft with an aggregated profile corpus, so it does not affect the score.

Privacy

writing-eval runs locally and does not send drafts, references, or profiles to a hosted service. Profiles are stored under the git-ignored data/ directory by default.

Only use prose that you are permitted to process. Profiles built from third-party published content should remain private and must not be committed or quoted at length in tracked files.

Metrics

• Tell rate by severity: style findings in a severity group per 1,000 output words. Lower values indicate fewer detected tendencies.
• Token 1-gram L2: Euclidean distance between normalized output and reference token frequency vectors. Lower values indicate closer vocabulary distributions. Reported as n/a when either side has zero tokens.
• Overrepresented terms: tokens whose output frequency most exceeds their reference frequency. Counts or rates explain the ranking.
• Shared tokenization: lowercase word tokens keep ASCII and curly-apostrophe contractions together.
• Mean sentence length: average number of words per sentence.
• Sentence length variance: population variance of sentence word counts. It describes how much sentence lengths vary within the corpus.
• Repeated opening rate: share of adjacent sentence pairs that begin with the same normalized opening. The denominator is the number of adjacent sentence pairs, or zero when fewer than two sentences exist. Openers keep curly apostrophes (U+2019), so a contraction such as We'll is one opener unit, matching tokenization.

Quality metrics (informational)

The metrics below broaden the set toward general readability and structure. They appear in reports and check output and never enter the corpus benchmark's decision gate or any pre-registered threshold. In a profile check, reading ease and reading grade contribute only to the versioned heuristic alignment score described above; MTLD and paragraph statistics remain informational.

• Flesch reading ease and Flesch-Kincaid grade: standard readability scores from word, sentence, and syllable counts. Word and sentence counts use the same markdown-aware segmentation as mean sentence length: headings are excluded and list markers are stripped. Syllable counts use a vowel-group heuristic with silent terminal -e, -es, and -ed adjustments, not a dictionary, so individual word estimates can be wrong. Reported as n/a when the text has no sentence.
• MTLD: Measure of Textual Lexical Diversity, the mean length of word runs that keep a type-token ratio above 0.72, averaged over forward and backward passes. Higher means more varied vocabulary. Reported as n/a below 10 tokens, where the measure is unreliable. An unfinished tail that stays above the threshold counts as one factor, so a fully unique 10-token input returns 10.0. The threshold must be a finite value in (0, 1). Token-sequence inputs are lowercased to match text inputs.
• Paragraph statistics: markdown-aware paragraph count, mean sentences per paragraph, and single-sentence paragraph rate. Paragraphs are blank-line separated and headings are excluded, so results depend on the input's markdown formatting.

Reports retain the aggregate tell_rate metric and also emit normalized tell_rates_by_severity values. Repeated-opening corpus rates count only adjacent pairs within each document, so reordering JSONL records does not alter the result.

These metrics describe observable text patterns. They do not establish factual accuracy, originality, reader preference, or overall writing quality on their own.

Corpus evaluation and benchmark

This is the original evaluation pipeline, kept for regression checks and the pre-registered benchmark.

Benchmark corpus generation and revision use the external OpenAI Codex CLI through benchmark/generate_runs.py. The Codex executable and its authentication are optional benchmark dependencies. They are not required for the writing-eval CLI, evaluation of existing output files, or scripts/dry_run.sh.

CLI usage

Run a corpus evaluation with your own output and reference files:

uv run python scripts/run_eval.py --outputs path/to/outputs --references path/to/references.jsonl --report /tmp/writing-eval-report.md

The command reads each output JSONL file, evaluates its text, compares it with the reference corpus, and writes a Markdown report to the requested path.

The optional --json path writes the same report data as JSON. JSON provenance includes the tool name writing-eval, the installed package version, and identity for the reference corpus and style rule set: source paths, record or rule counts, content SHA-256 hashes, and the rule-set version. The Markdown report renders the reference-corpus and rule-set provenance; it does not render tool identity. Markdown reports keep the caller-supplied system order. JSONL writers keep object key order from dict insertion.

For a checked repository run against test fixtures, use:

scripts/dry_run.sh

The corpus evaluation is also available as the eval subcommand (./writing-eval eval --outputs ... --references ... --report ...). The flat invocation shown above is kept for backward compatibility and routes to the same behavior.

Revision runs also record a literal_preservation result per output in their metadata. The decision-gate report summarizes the same comparison as an informational diagnostic. It checks normalized double-quoted spans, URLs, dates, and numeric literals for additions or removals. Quote whitespace is collapsed, URL terminal punctuation is ignored, date case and commas are normalized, and numeric grouping commas are ignored. It does not change any registered criterion or the gate verdict.

JSONL schemas

Each JSONL file contains one JSON object per line. IDs must be nonempty, stable, and unique within a file. A reference file may use a different ID namespace than the outputs. All output files in one run must contain the same ID set.

Prompts

{"id":"prompt-001","use_case":"article_section","prompt":"Explain why clear release notes reduce support work."}

Required fields:

• id: stable prompt identifier
• use_case: article_section, product_writing, or exec_communication
• prompt: writing assignment text

References

{"id":"ref-001","text":"Clear release notes help customers understand what changed and what to do next. That context prevents avoidable support requests."}

Required fields:

• id: stable reference identifier aligned to a prompt by numeric suffix
• text: accepted reference prose

Outputs

{"id":"prompt-001","text":"Good release notes explain the change and its effect. Customers can act without opening a support ticket."}

Required fields:

• id: prompt identifier
• text: generated prose to evaluate

Each file in the directory passed to --outputs represents one system. Output files must contain the same prompt ID set so system comparisons stay consistent.

Rule schema

Style rules live in YAML. Each rule has these fields:

id, severity, detector, and message are required when you define a brand-new rule id. When you override an existing rule id in an overlay, supply only the fields you want to change.

Rules are validated when loaded, after any overlay merge. Missing fields, unknown keys in standalone rule files, empty exception strings, unsupported detectors, and malformed definitions fail with a clear error. enabled is overlay-only.

The builtin rule set

One rule set ships with the project, at a single in-package path. It holds 34 rules: 16 warn and 18 info. The builtin rules are recommendations, and no rule blocks the build. Both check and corpus eval use it by default. The version: key in the YAML header is a schema version, not a content version; the reported SHA-256 and rule fingerprint carry content identity. The regular passive-voice pattern is deliberately precision-first: it matches a be plus regular -ed participle only when the participle ends the sentence or is followed by a likely agent, preposition, or frequency determiner, so unlisted adjectival participles can pass through undetected.

Customizing rules

Point --rules at your own YAML file. A file without an extends key replaces the rule set outright. A file with an extends key overlays the base set:

extends: builtin
rules:
  - id: my_jargon_ban      # new id: appended to the end of the rule set
    severity: warn
    detector: '(?i)\b(?:synergy|ideate)\b'
    message: Use plain language.
  - id: em_dash_ban        # existing id: override only the listed fields
    severity: warn
  - id: passive_voice      # existing id: removed from the effective set
    enabled: false

Merge semantics:

• extends: builtin resolves through the installed package, so it works from any working directory.
• Any other extends value is a path resolved against the overlay file's own directory, never the current working directory. An overlay can extend another overlay, up to a chain depth of 16.
• Ordering is the base order with overrides and removals applied in place, then new rules appended in the order they appear in the overlay.
• An override replaces only the fields present in the overlay entry. Unlisted fields keep the base values.
• enabled: false on an id that is not in the base set is an error, as is a duplicate id inside one overlay file.

The anti-ai overlay

Use the bundled overlay when a draft needs a stricter pass for reader-facing AI-writing tells than the builtin set provides, for example when reviewing machine-assisted copy before publication. Pass it with --rules; it extends the builtin set rather than replacing it:

./writing-eval check draft.md --rules rules/anti-ai.yaml

The command, output format, exit codes, and JSON schema are identical to the default check; only the finding set grows:

draft.md:1:5 [warn] polish_vocab: Replace overused polish vocabulary with specific language. | span: synergy
draft.md:2:1 [info] connector_openers: Replace the formal additive opener with also, and, or a plain sentence start. | span: Furthermore

The overlay lives at rules/anti-ai.yaml in the repository (outside the installed package) and begins with extends: builtin, so the effective rule set is the builtin rules plus four appended rules:

• narrative_cliches: stock narrative phrases such as "couldn't help but", "little did I know", "stumbled upon".
• significance_markers: meta commentary that labels a moment instead of showing it, such as "that's the part that got me", "let that sink in".
• generation_artifacts: bracketed scaffolding tokens (insert, todo, tbd, placeholder, xxx), utm_source=chatgpt.com links, and model self-reference such as "as an AI language model". Ordinary bracketed phrases such as [your account] are not flagged.
• connector_openers: sentence-initial "furthermore", "moreover", "additionally".

It also widens six builtin rules (polish_vocab, recap_endings, throat_clearing, faux_insight, importance_puffery, and collaborative_artifacts) with further phrases, keeping their severities and messages.

The overlay combines with every check form, including --style, --format json, and reading from standard input. The builtin rule set stays the default everywhere. Because rule fingerprints identify comparable runs, keep the default rules for corpus evaluation and benchmark comparisons and use the overlay for single drafts.

Limitations

• The bundled fixtures are synthetic and intentionally exaggerated to exercise detectors.
• Results are diagnostic only and should support, not replace, editorial review.
• The article score is a heuristic combination of configured style rules and target-profile distances. It measures alignment under the documented rubric, not factual accuracy, argument strength, originality, or overall article quality.
• Corpus metrics are sensitive to sample size, genre, and reference selection.
• Simple token and sentence boundaries can differ from linguistic parsers.
• A rule match identifies a review candidate, not a universal writing error.
• The harness does not evaluate factual accuracy, coverage, or human preference.
• Literal preservation is a normalized multiset comparison, not semantic factuality. It can detect a changed number, date, URL, or quoted span but cannot decide whether an unprotected claim is true or whether a paraphrase preserves meaning.
• Distributional scores (token 1-gram L2 and the top overrepresented terms) depend heavily on the composition and size of the chosen reference corpus or profile; small profiles make them noisy.
• L2 distance is sensitive to output length, so systems whose word counts differ a lot are not directly comparable on it.
• Benchmark generation uses a frozen decoding config and measured per-metric noise floors (benchmark/THRESHOLDS.md); deltas below the documented floor are inconclusive.
• Existing profiles with metric semantics other than version 2 must be rebuilt before use.
• Readability scores ignore heading text and list markers, so heading-heavy Markdown can differ from a plain-text reading.
• profile list omits directories that load_profile rejects.

License and contributions

writing-eval is source available under the Elastic License 2.0. You may use, copy, modify, redistribute, and self-host the software subject to that license. You may not provide it to third parties as a hosted or managed service where the service gives users access to any substantial set of the software's features or functionality.

Copyright 2026 Majestic Labs LLC.

Contributions are welcome. Read CONTRIBUTING.md before submitting a pull request. Contributors retain ownership of their work and must accept the Contributor License Agreement, which gives Majestic Labs the rights needed to keep contributions in the public project and use them in an official hosted service.

For OEM, hosted-service, or managed-service licensing, contact Majestic Labs.

Managed hosting

Majestic Labs is considering an official hosted writing-eval service for people and teams that cannot or prefer not to deploy and maintain the tool. The local CLI remains complete and self-hostable.

If managed hosting would help your team, tell us about your workflow.

About Majestic Labs

Majestic Labs is a software foundry for operator-built software.

writing-eval is part of our open-source projects.

If your team needs to evaluate an AI workflow against company-specific standards, start with Majestic AI.

This project is not a DFT clone or training system. It is a small deterministic harness for writing diagnostics.