Jacobian lenses for the OLMo-3-7B post-training ladder

Fitted Jacobian lenses for the full open OLMo-3-7B post-training ladder — the base model and ten post-trained variants. These are the artifacts behind the first quantitative, fully-reproducible test of Anthropic's post-training "point of view" observation — plus a run of the ignition test that Dehaene & Naccache asked for and that had not been run.

Anthropic report (qualitatively, no numbers, on the closed Sonnet 4.5) that the J-space is present already in the pretrained model, but that "during post-training, the J-space develops some signatures of adopting 'Claude's point of view'": "in the base model, the J-space mostly tracks what's needed to predict upcoming text; in the post-trained model, it starts holding Claude's own reactions." In plain terms: post-training shifts the model's internal workspace from what word comes next to also carrying its own stance as the Assistant — its reactions, caveats, and values. These lenses give the first quantitative version of that shift, on fully open weights.

A Jacobian lens is not a sparse autoencoder. It is the averaged forward Jacobian J_l = E[∂h_final/∂h_l] of a frozen model; reading unembed(J_l · h) recovers the tokens a layer's activation is poised to report — the model's "verbalizable" content. Method and reference code: Anthropic's jacobian-lens (Apache-2.0). Fit and analysed with the companion jlens-lab.

Why this exists

The Anthropic result rests on Sonnet 4.5, whose activations no external party can access — the paper's own invited commentators (Dehaene & Naccache) could not check it. AI2's OLMo-3 is the only open family shipping a base model, its post-trained variants, and a clean method-vs-domain factorial. These lenses make the claim checkable by anyone.

Contents

RL-Zero-Mix is deliberately absent: it is olmo2-retrofit, a different base architecture, so its J-space is not comparable to the olmo3 arms.

Each lens covers 11 source layers (0, 3, …, 30 — every third of 32) at d_model=4096, fit on 616 wikitext prompts, stored fp16. For a full 31-layer base lens (used for the ignition readout), use Anthropic's published olmo-3-1025-7b lens on Neuronpedia's neuronpedia/jacobian-lens.

Load and use

pip install git+https://github.com/anthropics/jacobian-lens.git
pip install git+https://github.com/m9h/jlens-lab.git

import jlens
from jlens import JacobianLens
from transformers import AutoModelForCausalLM, AutoTokenizer

lens = JacobianLens.from_pretrained("mhough/olmo3-jacobian-lenses",
                                    filename="lenses/olmo-3-7b-instruct.pt")

tok = AutoTokenizer.from_pretrained("allenai/Olmo-3-7B-Instruct")
hf  = AutoModelForCausalLM.from_pretrained("allenai/Olmo-3-7B-Instruct",
                                           dtype="bfloat16", device_map="cuda").eval()
model = jlens.from_hf(hf, tok)

# read the workspace content at the last token, layer 18:
lens_logits, model_logits, ids = lens.apply(model, "The capital of France is",
                                             layers=[18], positions=[-1])

See reproduce.py for a runnable check.

Result 1 — post-training installs a viewpoint, decoupled from capability

Distance from the base lens, cos(J_base, J_arm), mean over the 11 shared layers, against a ~0.97 same-model refit floor (the floor is layer-dependent — see External validation):

• Method sets the magnitude, not domain: instruction/CoT tuning moves the J-space ~5× more than RLVR; varying the RLVR domain (math/code/IF/general) at matched capability adds only ~1%.
• Decoupled from capability: MMLU is flat-to-slightly-down across post-training while the J-space moves ~31% — a representational shift with no competence gain, i.e. a viewpoint change, not a prediction gain.
• Anchor-gated, and now externally cross-validated: the base fit reproduces the published olmo-3-1025-7b lens (identity_distance 0.2199 vs 0.2209, 0.4% error). Against Neuronpedia's independently fitted lens for the same model, the final layer agrees to cosine 0.9998 — see below.
• ⚠ Corrected 2026-08-02 — read the pooled numbers above against a per-layer floor. The table's cosines are means over 11 layers whose refit floors run 0.884 (layer 0) to 1.000 (layer 30). Recomputed as excess over the per-layer floor: Instruct 29.4%, Think 25.1%, RL-Zero ~2.9%. Two consequences: the method ratio strengthens from ~5× to ~10×, and RLVR's ~6% halves to ~3% — about half of it was two fits disagreeing. The movement is also early-layer concentrated (Instruct 49.9% beyond floor at layers 0–9 vs 10.6% at 21–30), which the pooled statistic hides. Full working: results/posttrain/perlayer_floor_correction.md.
• Capability-controlled: RL-Zero overall-MMLU spread is 1.7pp (in noise), no domain diagonal — so the ~1% geometry differences are not capability artifacts.

Full write-up with the null correction and caveats: results/post-training-point-of-view.md.

Result 2 — Ignition test (Dehaene & Naccache's "decisive experiment")

Their commentary singles out ignition — all-or-none entry into the workspace — as the signature the paper "does not establish," and prescribes grading a concept's evidence and testing whether J-space representations "switch on with a threshold-like nonlinearity, while earlier layers rise monotonically." Nobody had run it (Claude closed, Anthropic didn't). Run here on OLMo-3 via the published 31-layer lens, using an on−off concept contrast:

• No concept without evidence (contrast ≈0 at k=0), then all-or-none entry (a step at the first clue, then saturation — not a graded ramp).
• Sustained in the workspace, transient in early layers: early layers respond to the first clue then decay (keep ~16%), the workspace band holds it (~67%) — exactly Dehaene's ignition ("a sustained, broadly distributed state") vs subliminal ("a delimited wave … which quickly dies away").

This first pass establishes the persistence half of ignition. The graded mid-range threshold and the bifurcation are delivered by the v2 clue set — see the full reviewer battery below. Write-up + confound analysis: results/ignition_result.md.

Result 3 — the full reviewer battery (every test Dehaene & Naccache proposed)

All six tests the reviewers asked for, implemented and run on OLMo-3 (results/reviewer_tests_results.md, raw in modal_reviewer_tests.py). Honest mixed outcome — two clean workspace signatures, one partial, three inconclusive under first-pass adaptations whose flaws are documented:

This is more of the human-consciousness battery than has been run on the record for any model, and every number is reproducible.

Result 4 — metacognition: a covert error signal in the base model, made reportable by post-training

A new direction, connecting to the LLM-metacognition literature (survey: arXiv 2607.11881) and the introspection debate. Reading the base model's workspace recovers an uncertainty signal that predicts whether its own answer is wrong — beyond the model's output confidence (n=200 TriviaQA):

• Structure: per-layer AUROC rises ~0.5 (early) → 0.69 (workspace) → 0.72 (late) — the metacognitive signal emerges in the workspace band.
• Covert (the key control): within matched output-entropy terciles the workspace still separates right from wrong (AUROC 0.64 on confident-looking answers) — it "knows more than it says." Output-confidence baselines for reference: ECE 0.11, meta-sensitivity (type-2 AUROC) 0.80.
• Across the ladder — post-training installs reportability: the base model's verbal self-evaluation (P(True)) is at chance (0.51) while its workspace covertly tracks errors (0.66); post-training raises verbal self-eval to 0.78 (Instruct) / 0.72 (Think) — via SFT+DPO, not RLVR (RL-Zero 0.52–0.55). Post-training does not create the self-monitoring signal; it makes a pre-existing covert one reportable.

This is the "mechanistic evidence of a dissociable second-order process" the introspection Reality Check (Singh, Linzen & Ravfogel) calls for. Full write-up: results/metacognition_result.md; lineage + concept-injection tooling: docs/technique-lineage.md.

External validation & the refit-noise floor

Neuronpedia (with Anthropic) publishes pre-fitted J-lenses at neuronpedia/jacobian-lens, including olmo-3-1025-7b. That gives an independent second-party fit of the same base model — the first external check of this pipeline. Full write-up + reproduce snippet: results/neuronpedia_crossvalidation.md.

These are genuinely independent fits (0 of 11 shared layers bit-identical; both converged at n_prompts=616 under the same stopping rule — a protocol coincidence, not a copy). The estimator is identical: jlens-lab calls jlens.fitting.jacobian_for_prompt from Anthropic's reference package, the same library Neuronpedia vendors, so the documented convention — sum the cotangent over later target positions, then average over source positions — and the attention-sink position masking are shared. J̄ is deliberately not a plain per-position Jacobian average; that is a property of the shared reference implementation, not a difference between the two fits.

Mean cosine over the 11 shared layers is 0.966 (computed in float64).

The refit floor is strongly layer-dependent, and this matters for reading Result 1. Two independent fits of the same model differ by 48% in Frobenius norm at layer 0 and 2% at layer 30. So the pooled ~0.97 floor understates the noise early and overstates it late:

• In the mid-to-late workspace band, where the ignition and metacognition readouts live, the floor is 0.99+ — a post-training cosine of 0.69 is far above noise.
• At layer 0, the floor is only ~0.88, so early-layer J-space comparisons carry much less weight than the pooled number implies.

Result 1 stands — the post-training shift is well above the floor where it is measured — but the honest presentation is excess over a per-layer floor, not a single pooled one. Early-layer statements should be read with that in mind.

Numerical note: compute these cosines in float64. Over 4096×4096 matrices, float32 accumulation returns values above 1.0 (we saw 1.0057).

Scope relative to Neuronpedia's lenses

Neuronpedia's J-lens covers 38 models, and for OLMo-3 it ships the base models (olmo-3-1025-7b, olmo-3-1125-32b) fitted on wikitext. This repo's contribution is the post-training axis — ten post-trained variants spanning SFT / DPO / final for both Instruct and Think, plus four RL-Zero arms — which is what makes the post-training claim testable rather than a single before/after pair.

How they were fit

Matching the published olmo-3-1025-7b config, on B200 / dim_batch=128:

n_prompts=616  dataset=Salesforce/wikitext (wikitext-103-raw-v1, train)
max_seq_len=128  layer_step=3  target_layer=final  dtype=bfloat16

Fit is a plain mean over prompts (sharded exactly across 8 workers). Reproduce end-to-end: m9h/jacobian-lens/modal_olmo_ladder.py (anchor → capability_all → ladder → analysis).

Caveats

• Cosine is a magnitude measure — how much the J-space moved, not in what organized way.
• The same-model refit floor is layer-dependent (~0.88 at layer 0, ~0.9998 at layer 30); early-layer comparisons are weakly determined and should not be read against the pooled floor.
• RL-Zero is a light intervention (~6% from base); the ~1% domain signal is small.
• Capability is measured by multiple-choice log-probability, which under-scores chat-tuned models; it rules out a gross capability confound, not subtle ones.
• One model family (OLMo-3); a second-family replication is designed but not yet run.
• Not peer-reviewed. Derived from open artifacts only.

Provenance & license

Apache-2.0, matching OLMo-3 (weights) and jacobian-lens (method). Derived from allenai/Olmo-3-* via the open jlens / jlens-lab stack. Work by Morgan Hough, Orthogonal Research and Education Lab (OREL); not affiliated with Anthropic or AI2.