Frozen general-purpose features beat a fine-tuned baseline on ChestX-ray14

A linear probe on frozen google/gemma-4-31B-it hidden states, on all 112,120 images of NIH ChestX-ray14, using the official test_list.txt. No fine-tuning, no radiology pretraining, no augmentation. One Linear(d, 14) under BCE, which is fourteen logistic regressions.

Ahead on 12 of 14 findings.

Read the split before you compare anything

This is the part that matters and the part that is usually wrong.

The widely quoted numbers, CheXNet 0.8414 and Yao 2017 0.8027, are on a different split. CheXNet section 5: "We randomly split the dataset into training (70%), validation (10%), and test (20%) sets... We ensure that there is no patient overlap between the splits." That is their own random partition, patient-disjoint but not the official list.

Which split is harder is not established, and we are not going to assert it. Wang et al. scored 0.7381 on a random partition and 0.7451 on the official one, so for their model the official split was very slightly easier. The honest statement is only that the two are not comparable, which is why a single split-matched row is the head-to-head above and the rest is context.

Do not compare this 0.7590 to 0.8414. Different test sets.

Per-finding, split-matched

Hernia has 227 positives in the entire dataset and 86 in the test split, so that column is thin for everyone and should not carry weight in either direction.

Controls, and why each one is there

Shuffled labels (0.5002). Labels permuted within the training split and the probe refit. Anything above 0.5 on held-out data is leakage or a bug.

View position only (0.5883). Portable AP films are taken of sicker, bedbound patients, so view alone is a real route to a high AUROC that involves no pathology. A finding that does not clear this baseline has not been detected. The baseline is folded (max(vw, 1-vw)): Hernia's raw view-only AUROC is 0.3033, which is 0.6967 of shortcut once flipped, and reporting the raw figure would have flattered the probe.

View position is a single binary feature, so every one of its AUROC comparisons is a tie. Scoring it needs rank averaging within tied groups, or the answer becomes an artefact of the sort order and moves between machines. An earlier release of this card said 0.5896 for that reason.

Patient-level cluster bootstrap. Confidence intervals resample patients, not images. The test split is 25,596 films from 2,797 patients, roughly 9 per patient, and those films are anything but independent. Resampling rows treats correlated images as fresh evidence and yields intervals about 1.5x too narrow.

Patient overlap is asserted to be zero and the script refuses to run otherwise.

Scope

Detection, not early detection. These labels describe what is visible in the image in front of you. Nothing here speaks to catching disease before it is apparent; that needs longitudinal data with outcomes.

One backbone, one probe class, and the number depends on the backbone. The probe is deliberately linear because anything stronger measures the probe rather than the representation. The identical probe and split on other frozen backbones gives Qwen3-Omni-30B-A3B 0.7650 and Aria 0.7080, so the 0.7590 here is a gemma-4 result. Averaging three backbones' logits reaches 0.7774 at no added parameters: see RiverRider/srt-cxr14-pooled-probe.

Labels are NLP-mined from radiology reports by the dataset authors, with their own reported precision and recall. Every model on this dataset inherits that ceiling.

Banked negatives

Kept because they bound the claim.

Nine variants (3 depths x 3 poolings) are in results/cxr14_pool_sweep.json.

Contents

All three state files cover the identical 112,120 images in manifest row order, so they can be indexed against each other directly and against the manifest without realignment.

Three backbones, and what falls out of having them

Against a split-matched 0.7451 (Wang et al. 2017, Table 17, ResNet-50 fine-tuned end to end). Averaging logits adds no parameters, and the paired patient-clustered bootstrap gives +0.0124 with 95% CI [+0.0082, +0.0168].

A probe fitted on one backbone and read on another, through a ridge map fitted on training rows only, scores 0.7511 against 0.7440 native. Transport cost is negative, and four of six cross directions beat the target backbone's own probe. Weights for the pooled probe are at RiverRider/srt-cxr14-pooled-probe.

Reproducing

python scripts/get_cxr14.py --shards 12
python scripts/cxr_probe.py --states states/cxr14_gemma4.npz \
    --manifest manifests/cxr14_manifest.json --out probe.json

Align on the manifest key, never on array position: any row the encoder dropped shifts every later row.

Part of the SRT program, https://github.com/space-bacon/SRT.