Did it even learn?

A contamination-controlled audit of whether community LLM fine-tunes on HuggingFace actually improve on their base models, or just get better at the benchmark.

Leaderboard →; static source in site/, regenerated by scripts/build_site.py.

The question

Thousands of people fine-tune open-weight models and publish them on HuggingFace claiming an improvement. Nobody has systematically checked whether that's true: the platform's own Open LLM Leaderboard retired in 2025, and nothing replaced it for the long tail of community models.

There's a specific trap: a fine-tune can score higher on a benchmark because the questions leaked into its training data somewhere, not because it actually got better at the underlying skill. This is a recognized problem (known as "benchmaxxing") and it's rarely measured directly.

This project runs 150 matched pairs (a base model and a community fine-tune of it) across 164 unique models, on three task families (grade-school math, general knowledge, instruction following), each in two conditions:

• Public: the standard benchmark item, verbatim. Potentially memorized.
• Fresh: a newly generated equivalent, testing the same skill, never published anywhere before this run.

The gap between a model's public-condition score and its fresh-condition score is the memorization signal. Comparing that gap between a base model and its fine-tune answers the real question: does fine-tuning make a model better, or just better at a test it's already seen?

Why now

This only works with unrestricted access to a large number of open-weight models, which per-token API pricing makes expensive to test at scale. Nobody benchmarks the long tail because it isn't worth paying for. This run uses a flat-rate subscription that makes breadth affordable for a short, fixed window, which is also why the run is time-boxed rather than open-ended.

Methodology

Full detail, including the statistical tests, validity checks, and (written and committed before any data existed) the exact things this study cannot conclude, is in ANALYSIS_PLAN.md. Pre-registering it was deliberate: the analysis code has to match this document, and any deviation has to be disclosed, not quietly applied after seeing a result that looked better a different way.

Short version:

• Pairs are selected by joining HuggingFace's base_model metadata against Featherless's live catalogue, stratified so a handful of popular base models (Llama-3.1-8B-Instruct, Qwen2.5-7B) can't dominate the sample, and sampled across download-count tertiles so genuine long-tail, rarely-touched fine-tunes are represented, not just the already-popular ones.
• Every task is deterministically graded: no LLM-as-judge anywhere, which removes the most common line of attack on this kind of solo benchmark work.
• The primary comparison unit is one matched pair on one task family, not one item: a pair contributes a single data point per metric.
• Statistical tests: paired bootstrap (10,000 resamples) for the headline questions, McNemar's exact test for MMLU's item-level check, Benjamini-Hochberg correction across every reported comparison, and a label-shuffle null check that the measured effect must not survive.
• A model with a failure rate above 20% is excluded from the primary analysis and reported separately (a threshold fixed in the plan before any data existed, specifically to prevent picking exclusions after seeing results).

Results

Fine-tune accuracy on fresh, never-seen items dropped significantly across all three task families (paired bootstrap, BH-corrected, n=143-144 pairs each): GSM8K −6.9pp [−12.1, −1.9], MMLU −6.0pp [−8.6, −3.5], IFEval −6.5pp [−9.1, −4.0]. On average, community fine-tunes did not get better at the underlying skill; they got worse. That's the one clean result from this run.

The other headline question, whether the contamination gap itself grows or shrinks after fine-tuning, does not currently pass the pre-registered label-shuffle validity check for MMLU or IFEval, and isn't being reported as a finding here. Investigating why is open work, not swept under the rug: full detail on both families' failure mode, including what's already been ruled out, is in ANALYSIS_PLAN.md's Results section. Only GSM8K's contamination-gap result currently passes that check, and it doesn't exclude zero (not significant).

Status

Complete. Final results: the leaderboard, published from site/index.html (regenerate locally with scripts/build_site.py). 128,857 responses collected, 5,295 permanently failed, 429 usable matched pairs across all three task families.

This has been a genuinely hard engineering problem, not just a data-collection wait. Ten real, live-caught bugs, in order of discovery: uneven claim ordering causing cold-start latency, an uncaught exception crashing an unattended overnight run, an item-ID collision between two independently-seeded item sets, a scheduling stall from a single unresponsive model monopolizing concurrency, an account-wide API rate limit on switching between models that turned out to be the dominant throughput bottleneck for most of the run, a duplicate process silently doubling the effective load against that same rate limit, a persistently-failing model whose backlog was large enough to pin the scheduler on it indefinitely despite per-attempt retries, the same failure mode compounding across a cluster of several simultaneously-bad models taking turns, and a restarted process silently forgetting which models the database already proved were currently unhealthy. Each is documented in detail in its fixing commit and, where it changed how the harness behaves, in the relevant module's docstring. The design of src/finetune_audit/runner.py in particular is close to a running log of what actually breaks when you point an evaluation harness at 164 real, uncurated models instead of a handful of well-behaved ones.

One disclosed deviation from the mechanical retry policy: in the run's final hours, 3 models (segolilylabs/Lily-Cybersecurity-7B-v0.2, rhaymison/Mistral-portuguese-luana-7b-chat, derikk/sfting) were manually marked failed_permanent before exhausting the normal 6 retry attempts per item. Their errors were deterministic (a chat-template incompatibility rejected by the API on every single call, and one model no longer deployed at all), so retrying could never have changed the outcome; this only spared the scheduler's circuit breaker from wasting cooldown cycles on cases already certain to fail. It does not affect any result: those 1,841 items would have reached failed_permanent on their own regardless.

Repo structure

src/finetune_audit/
  client.py          Minimal async client for Featherless's OpenAI-compatible API
  runner.py           SQLite-backed, resumable work queue and scheduler
  lockfile.py          Exclusive process lock (prevents duplicate runs)
  pairing.py            Base/fine-tune pair discovery via HF metadata
  catalogue.py           Featherless model catalogue lookup
  grading.py               Deterministic graders -- no LLM-as-judge anywhere
  tasks/                     GSM8K, MMLU, and IFEval item generators
  analysis.py            Bootstrap, McNemar, Benjamini-Hochberg, effect sizes
  pair_analysis.py         Raw results -> per-pair metrics, per ANALYSIS_PLAN.md

scripts/
  select_run_scope.py    Stratified pair selection
  run_full_evaluation.py   Enqueue + run from scratch
  resume_evaluation.py      Resume an in-progress run
  export_results.py           Live DB -> results/raw_results.csv
  build_site.py                 Results -> site/index.html
  (plus exploratory probe_*.py, smoke_*.py, and survey_pairs.py from
   early-stage catalogue and latency exploration -- not part of the main
   pipeline, kept for provenance)

ANALYSIS_PLAN.md   Pre-registered analysis plan, written before any data existed
run_scope.json       Exactly which 150 pairs were selected, and why
tests/                186 tests, no network calls, run in under 5 seconds, CI-verified on every push

Running it

uv sync
uv run pytest tests/

A real run needs a Featherless API key in .env (FEATHERLESS_API_KEY=..., gitignored) and will hit the live API. Everything else here is fully offline.

License

MIT: see LICENSE.