Parsimony

Among agents that successfully solve the same SWE-bench issue, which leave the smallest code footprint?

Parsimony imports public SWE-bench Verified predictions and published evaluation results, statically measures their Python changes, and compares successful solutions. It makes no LLM API calls, runs no SWE-bench reruns, installs nothing from target repositories and never executes submitted code.

Status: beta / research prototype. No official cross-model ranking or validated full-benchmark score exists. Everything in examples/ is exploratory and must not be read as a model recommendation. Analyzer 0.5.0-beta replaced lexical tokens with coding units (AST elements) as the primary metric (see the changelog). The current two-submission 500-task audit uses it. Older audits are kept for history and are not comparable.

Quick start

Python 3.12+, no runtime dependencies for Python analysis. Optional tracks: JavaScript/TypeScript use Node.js and the official TypeScript parser (npm ci installs the pinned typescript), Go uses pip install '.[languages]' (the current checksum-pinned grammar builds from source, requiring a C compiler and Python development headers); see language tracks for pinned parsers, scope and reproducible commands. Analyzer 0.8 introduced a versioned Go grammar/EOF fix and 0.9 rejects malformed hunk headers and duplicate file sections instead of skipping or double-counting them; published historical cohorts keep their recorded analyzer identities. Run from this checkout (pip install -e . is optional). Measurements depend on the interpreter's tokenizer, so use one pinned Python version for anything you compare.

# Metadata, base commits and human patches for all 500 Verified tasks.
python -m parsimony dataset --output verified.jsonl

# Analyze five successful tasks per submission (alphabetical, NOT a representative sample).
python -m parsimony analyze \
  20250522_sweagent_claude-4-sonnet-20250514 \
  20250225_sweagent_claude-3-7-sonnet \
  --dataset verified.jsonl --limit 5 --output results.jsonl

# Compare only tasks solved AND analyzed by every selected agent.
python -m parsimony leaderboard results.jsonl --shared > leaderboard.json

python -m unittest discover -s tests -v

Useful analyze options:

• --limit N only limits analysis attempts. It never changes the resolve-rate denominator.
• --task TASK_ID (repeatable) analyzes a chosen cohort while keeping all 500 records.
• --include-failed also measures explicitly failed patches (see failed patches).
• --resume appends missing records to an interrupted full single-submission run and retries fetch_error records. It verifies agent, results hash, ref and analyzer/Python versions first.
• --patch-only gives hunk-only estimates without fetching sources. These are a separate, less reliable mode that is never pooled with full-file results.

Output has one record per agent/task, including failures, unavailable analyses and skipped tasks. leaderboard rejects duplicate agent/task records. --agent NAME selects agents.

Methodology

1. Correctness gate. Only tasks listed as resolved in the published results earn efficiency credit. The resolve rate is published resolved / 500. Parsimony trusts published evaluation and does not revalidate correctness.
2. Before vs. after. Each touched implementation file is fetched at the dataset's exact base_commit, and the patch is applied in memory with strict position/context validation. Unapplicable patches are recorded as errors, never scored as zero. Network failures are recorded as retryable fetch_error.
3. Coding units. Each side is parsed with ast (docstrings removed) and walked in source order. A unit is one syntactic element: a statement (if, return, assignment, def), an expression (call, attribute access, arithmetic or boolean operation), each comparison, a name or a literal. Each statement block also ends with one EndBlock unit, so nesting and moving code into or out of a block count. Punctuation, brackets, commas, load/store markers, comments, formatting and name length are not units: if x.count(y) > 0: foo(a, b) is 12 units. Units are unknown (null), never zero, when a file does not parse on both sides (lexical_files).
4. Alignment. Unit counts come from an exact minimum edit script (Myers diff) over the unit sequence. Rewrites needing more than 500 edits use a line-anchored approximation and are flagged (approximate_files).
5. Footprint. net_units = added − deleted, churn = added + deleted. Negative net is genuine shrinking. A renamed name or changed literal is one changed unit; the structural_churn diagnostic ignores identifier/literal values. The pre-0.5 metric, normalized lexical tokens (net_tokens, token_churn), remains a diagnostic. files_changed counts implementation files whose normalized code differs.
6. Scope. Only .py implementation files count. Excluded: test/doc/example/benchmark directories, test filenames, setup.py/conftest.py, generated files (a header comment in the base commit), protobuf output, migration scripts, vendored and build output. Django's db/migrations/ and test/ framework packages count as implementation. Every record lists touched_files and excluded_files. See implementation() for the exact rules.
7. Structure (diagnostic). Full-file AST-node delta and a simple cyclomatic proxy delta (functions/lambdas, branches, loops, handlers, extra boolean operands, comprehension loops/filters, non-default match cases). These are null if either side does not parse.
8. Human comparison. The dataset's human patch (not test_patch) is measured identically. model_human_ratio = model churn / human churn, or null when the human churn is zero.

leaderboard reports medians (net, churn, human ratio, files, AST and complexity deltas), sample counts and resolve rate, sorted by median net units. --shared uses the intersection of resolved and analyzable tasks, which avoids task-mix confounding but can favor easy tasks.

Archived experimental score (80% net / 20% churn; not the website default)

See the scoring specification. Per-task percentiles against a frozen reference panel are blended 80/20 and averaged with equal task weights. Successes score 1 to 100. Explicitly failed attempts get a bounded nonpositive growth/churn penalty down to −25. Missing measurements, unknown outcomes and out-of-scope successes (every touched file excluded) produce bounds, not invented scores.

python -m parsimony.scoring score examples/ten-model-score-panel.json \
  examples/ten-model-results.jsonl --output /tmp/parsimony-scores.json
python -m parsimony.sensitivity examples/ten-model-score-panel.json \
  examples/ten-model-results.jsonl --output /tmp/sensitivity.json

sensitivity runs a paired-task bootstrap and a grid over net weight, failure cap and a net_floor=0 anti-deletion variant. It covers neither panel composition nor task-selection bias.

Beta release tooling: frozen population and coverage

Freeze the task population before looking at candidate scores:

# Commit analyzer changes first; freeze requires a clean checkout unless --analyzer-commit is given.
python -m parsimony.release freeze verified.jsonl --name verified-500-beta-v1 --output population.json
python -m parsimony analyze SUBMISSION --dataset verified.jsonl --ref EXPERIMENTS_COMMIT \
  --include-failed --output all-results.jsonl
python -m parsimony.release audit population.json all-results.jsonl \
  --dataset verified.jsonl --output coverage.json

The manifest pins dataset bytes, task IDs/base commits, Python version and analyzer commit. audit checks records against it: population, duplicates, base commits, resolve totals and the analyzer_commit every record carries since 0.4.0. It reports coverage, statuses, exclusions, zero-footprint, value-only, out-of-scope, lexical and approximate records. This is coverage reporting, not authenticity certification. An official score additionally needs a preregistered, deduplicated successful-reference panel covering every frozen task.

Artifacts, failed patches and caching

Legacy layout. Predictions come from the public swe-bench-submissions S3 bucket (verified/<submission>/all_preds.jsonl), and results/results.json from SWE-bench/experiments on GitHub. Current mini-SWE-agent layout (used on 404): per_instance_details.json supplies per-task booleans, and patches come from logs/<task>/patch.diff via metadata.yaml. Unsupported layouts fail explicitly. For custom or local artifacts, write a JSON manifest:

{"agent": "unique-agent-name", "submission_url": "https://example.org/submission",
 "prediction_url": "all_preds.jsonl", "results_url": "results.json"}

python -m parsimony analyze --manifest submission.json --dataset verified.jsonl --output results.jsonl

Failed patches

Mere absence from resolved is never treated as failure. With --include-failed, failures must come from explicit evidence:

• The current layout uses resolved: false per instance.
• Legacy submissions (whose results.json has no failure list) use per-task logs/<task>/report.json with resolved: false and an applied patch.

Missing logs or reports remain unknown, and a warning is printed when a submission publishes no explicit failures.

Provenance and caching

--ref COMMIT pins GitHub results (S3 is not versioned by it). Records keep SHA256 hashes of predictions, results and each patch, artifact URLs, base commit, reference patch hash, analyzer version/commit/source hash and Python version. Downloads are cached atomically by URL under .parsimony-cache/ (--cache PATH goes before the subcommand). Cached URLs are never refreshed, so use a new cache for mutable upstream changes. Keep the metadata JSONL and cache alongside results for reproducibility. To share exactly the bytes a result set depends on, pack a verified snapshot:

python -m parsimony.snapshot create examples/beta-500-v4/*-results.jsonl --output cache-v4.tar.gz
python -m parsimony.snapshot restore cache-v4.tar.gz --cache .parsimony-cache  # checks every SHA256 first

Website

Current website ranking: lowest mean net coding units added first, by default across measured in-scope solved attempts (Count attempts: Solved only; All measured, successful and failed, is selectable), with equal 1/N weight for each counted attempt, without correctness weighting or passing-reference selection. Missing measurements are excluded, never zero-filled. Footprint values are raw mean coding units, not percentages; Solved % is upstream resolved / full frozen population. Columns are Rank, Model, Net units added (mean), Net units added over solved attempts only (mean), upstream Solved %, and Measured eligible/population; any numeric column can be sorted, and rank stays on the all-attempt mean. Equal means tie and models with no measurements stay unranked. Partial coverage can bias the order; this is smallest measured footprint, not best coding ability. Failed no-ops and large deletions can rank first, and a few very large failed deletions can dominate a mean (on Verified, the #1 model's mean is −244 against +10 over solved attempts). A 10% trimmed mean was used briefly on 2026-10-01 and removed at the owner's request. The historical 80/20 scores, CI and score ranks remain available only as explicitly labelled optional archived diagnostics; they do not determine website rank.

python -m parsimony.site PANEL RECORDS... --output site/index.html renders one self-contained page offline from a frozen panel and measured JSONL. Each page has:

• the chart before the sortable leaderboard table with the columns above;
• a Sources bar chart after the leaderboard, derived from frozen population manifests: DeepSWE Python/JavaScript/TypeScript/Go have 34/5/35/34 tasks and 4 attempts per task; Verified has 500 tasks and Live Python 300, each with 1 attempt per task. Bar widths are task counts relative to the largest population, not ranking weights. The active board is marked, with report links and manifest hashes; boards remain separate, never pooled;
• a task-ID selector that swaps the same table and plot to one task's attempts, including failures (there is no second task table);
• a scatterplot with one borderless circle per model, colored by developer company: OpenAI black, xAI purple, Google green, Anthropic orange and DeepSeek blue. Every generation is labelled by name in plain HTML text, 8px, normal weight (400), black, with no halo, shadow or border. Each model line's newest generation (highest version, e.g. Claude Opus 5 over 4.8) has full-opacity circles and labels; older circles use 70% opacity and older labels 75%. Every label is vertically centered a fixed 10 CSS pixels to the right of its dot, without leader lines or variable-distance placement. Task failures are disclosed in the tooltip and accessible (ARIA) label, never with a dashed point outline. An uncaptioned green area marks the better corner on both axes (top-left for the default view: smaller footprint, more solved) and is omitted when an axis has no better direction. You pick the X/Y metrics from the table; the defaults are mean net units added vs. solved %, except on the DeepSWE boards, where Y defaults to the external Artificial Analysis Intelligence Index from a pinned snapshot (--external, see parsimony/external.py). That index is a model-level score, not a result on the board's tasks; it is matched only for the same model and effort (weaker matches disclosed on hover), unmatched configurations have no point, and it never affects rank. Solved %, the AA Coding Index and the solved-only mean are also selectable. A Chart picker also offers a bar view of mean net units added only (same company colours and fading; ?chart=bar links straight to it) and a Show self-run checkbox that removes self-run configurations from the chart, bars and table (?selfrun=0); axis ticks sit at round values inside the data-fitted range, with any self-run configurations in a separate group below because they cover only a task subset. The DeepSWE Python page also shows self-run configurations (--self-run, see examples/self-run): newer models run by the owner through pi or Claude Code on the first 3–10 tasks, measured with the board's analyzer, not graded. They are faded on the chart, labelled self-run on hover, listed in grey rows and never ranked, and appear only in the all-tasks view. Numeric axes fit the visible points with 5% padding. In All tasks only, Gemini 3.1 Pro (including Preview and effort variants) is excluded from numeric-axis fitting when at least two other points have finite values on that axis and exclusion changes the fit; Gemini 3.1 Professional and Gemini 3.1 Pro Plus are not excluded. Solved normally stays at 0–100%; when this exclusion changes its fit, it uses the other points' 5%-padded range bounded to 0–100%. Coverage stays fixed at 0–100%, and task-view scaling is unchanged. Points outside fitted axes are extrapolated at their true coordinates using the same linear scales, never clamped or replaced with edge markers; labels and hover disclose this, and hover details and the table retain actual values. Data and ranks are unchanged; there is no Pareto overlay.

Each board has a Count attempts: All measured / Solved only selector. Solved only is the default. It ranks by mean net units over measured, in-scope upstream successes and updates footprint charts consistently; failed and unknown-outcome attempts are excluded from task views. Measured then shows measured solved / upstream solved attempts, including repeated attempts. Models without measured successes remain unranked and absent from charts; ungraded self-runs are hidden. The second footprint column shows the alternative mean. Upstream Solved %, external indices and archived score diagnostics remain unchanged. Link to the alternative with ?attempts=all (combine with &chart=bar for bars). Solved-only means still compare different task mixes, not a shared-solved cohort, and benchmark success does not establish maintainability.

Visible copy stays compact, with detailed caveats and archived-score explanations collapsed under How to read this. Pages are always light mode and show an instant tooltip with the model name and both selected axis names and values, plus the task outcome in task view. The archived 80/20 Score, its bootstrap 95% CI and rank ranges (--sensitivity) are available only as labelled optional graph metrics. They never affect the net ranking, and the CI never affects Score.

--nav LABEL=URL links the boards. All pages are served at parsimony.lasu.dev (Cloudflare Pages, output directory site). Edit parsimony/template.html for layout and copy, then rebuild all six pages (HANDOFF.md has the exact commands).

Bring your own benchmark

Every website page includes a local JSON importer near the bottom. Explore any benchmark with known code amounts—lines, tokens, bytes or your own defined units—without running the analyzer. Download the example, edit it and open it in the browser. Imports stay local, are labeled unverified and never mix with published boards. See the format and public contribution guide.

Contributing

See CONTRIBUTING.md. python -m parsimony.contribute export turns existing JSONL into a checksummed bundle under submissions/. PR checks validate it offline. contribute verify (also a manual CI job) re-downloads recorded patches, checks hashes and re-measures them.

Examples (all exploratory)

The smoke and ten-model samples were regenerated with 0.5.0 (python -m examples.run_ten_models for the latter). Earlier versions are in git history.

Limits and interpretation

See the repository audit and release-readiness review for engineering fixes, validation commands and outstanding release gates, and the browser/accessibility audit for tested behavior and remaining gaps. These audits are not independent benchmark certification or an accessibility-conformance claim.

Code footprint is not technical debt. Small code can be cryptic, incorrect beyond the benchmark tests, insecure or hard to maintain. Larger changes may add valuable validation. Human patches are a baseline, not an optimum. Net-weighted scores reward deletion, and SWE-bench tests do not protect untested code, so check the net_floor sensitivity variant.

The existing Python boards measure Python only. Optional JavaScript/TypeScript and Go tracks (introduced in analyzer 0.6.0; JavaScript/TypeScript moved to the official TypeScript parser in 0.7.0, Go to a versioned corrected grammar/EOF policy in 0.8.0) are kept separate from Python and each other; raw counts are not cross-language comparable. Scope is unified text diffs only: binary/rename-only diffs, quoted paths, malformed hunk headers and repeated sections for one implementation file are unsupported (recorded as errors, never scored). There is no call-graph, runtime, maintainability or behavioral-equivalence modeling. Path and generated-file filters are heuristic. Interpreter tokenizer/AST changes (notably f-strings) change results. Coding units measure size and nesting, not readability: a test-specific hardcoded branch can cost fewer units than a general fix.

Roadmap (priority order)

1. Freeze the full task population and independently reproduce public patch/result artifacts. Publish coverage and missingness before rankings. The v5 audit does this for two submissions with a fresh-artifact recheck; more harnesses and independent review remain.
2. Validate footprint against adversarial patches. Done for 0.5.1: formatting, renames, excluded files, generated headers and deletions are tested; unrelated deletion, moved code and hardcoded test inputs remain known limitations.
3. Evaluate score sensitivity to panel composition, weights, failure cap and task mix, with paired uncertainty and per-task outcomes. Stability report for 33 models × 500 tasks (python -m parsimony.stability): weights and cap never change a rank; most neighbouring models are statistically tied, and the agent version alone moves a model by up to 5.6 points.
4. Validate units against blind human preferences on patch pairs, and add readability diagnostics (test-input hardcoding, nesting depth, reuse of existing helpers).
5. Expand to more models and harnesses on the same frozen tasks. Optional JS/TS and Go tracks are published with 7,696 additional attempt records and audited coverage (72 of 74 additional tasks calibratable; JS/TS remeasured in 0.7.0 with the official TypeScript parser). SWE-Atlas investigation: newer-model aggregate scores exist, but public per-attempt model patches/results were not found.