srt-nla-av-gemma2-2b-v1 — Activation Verbalizer for Gemma-2-2B (L19)

Read a single hidden activation as a sentence — on a third backbone family. A 5.31M-parameter prefix adapter over a fully frozen google/gemma-2-2b that, given a layer-19 last-token hidden state v ∈ ℝ²³⁰⁴, generates text whose own re-encoded L19 hidden state h maximizes the anisotropy-corrected reconstruction fve_nrm_cen(h, v) = ½(1 + cos(h − μ, v − μ)).

This is the third-backbone replication of RiverRider/srt-nla-av-v1 (Qwen2.5-7B) and RiverRider/srt-nla-av-llama32-3b (Llama-3.2-3B). Same training pipeline, same hyperparameters, third model family (Google's Gemma-2 lineage). Result: every qualitative finding of the original paper reproduces. See paper_nla.md §11.

TL;DR: at best-of-64 sampling the AV exceeds the paraphrase ceiling (fve_nrm_cen = 0.631 > 0.598, ρ_norm = 1.33). Gemma-2-2B has the highest anisotropy of the three backbones (‖μ‖ ≈ 156, vs Qwen 55, Llama 7.2), making this the cleanest case for the centring claim of paper_nla.md §§4–5: raw greedy fve_nrm (0.664) is below the raw random floor (0.675), so the centred metric is non-optional on this backbone.

Card metadata

Files

How to load

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer
from srt.nla import ActivationVerbalizer, NLAConfig

repo = "RiverRider/srt-nla-av-gemma2-2b-v1"
cfg = NLAConfig.from_json(hf_hub_download(repo, "config.json"))

bb = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-2b", torch_dtype=torch.bfloat16
).cuda().eval()
for p in bb.parameters():
    p.requires_grad = False
tok = AutoTokenizer.from_pretrained("google/gemma-2-2b")

av = ActivationVerbalizer(cfg, backbone=bb, tokenizer=tok).cuda().eval()
state = torch.load(hf_hub_download(repo, "best_av.pt"), map_location="cuda",
                   weights_only=False)
av.load_state_dict(state, strict=False)

To verbalize an activation vector v ∈ ℝ²³⁰⁴ extracted from layer 19 of the frozen backbone, draw a best-of-K rollout and score each candidate by fve_nrm_cen (centred cosine vs v); pick argmax. See scripts/centered_eval.py for the canonical eval loop.

Evaluation

fve_nrm_cen = anisotropy-corrected (subtract pool μ before cosine). Pool size 2,000 in all rows.

M=200 oracle ceiling (scripts/oracle_ceiling.py)

Gemma-2-2B's centred ceiling–floor gap is 0.100, smaller than Qwen's and Llama's. This is a substantive cross-backbone fact: Gemma's paraphrase distribution is sharper in the centred geometry, so the normalized scale of ρ_cen is more compressed on this backbone.

M=200 centered eval (K=64; scripts/centered_eval.py)

Note: raw greedy (0.664) sits below the raw random floor (0.675). This is the strongest empirical case across the three backbones for the non-optionality of the centring move — any uncentred reading on Gemma-2-2B L19 reports the verbalizer as worse-than-random.

M=200 K-curve (scripts/rerank_eval.py)

Log-linear: ~+0.021 centred per doubling of K (shallower than Qwen's +0.030 and Llama's +0.034, consistent with the smaller ceiling–floor gap; the shape is preserved across all three backbones).

• logp-rerank gives 0.512 centred (-0.014 vs greedy 0.527, Spearman +0.030 with the oracle) — same death-of-logp-rerank result as Qwen and Llama. Third backbone, same finding: the AV's own confidence is uncorrelated with how well the candidate re-encodes.
• NN-anchor rerank gives 0.600 centred, well above greedy.

Known limitations

• Highest-anisotropy backbone tested. ‖μ‖ ≈ 156 is roughly 3× Qwen-2.5-7B's and 22× Llama-3.2-3B's. Any uncentred metric on this backbone is dominated by the rotation-into-μ component.
• Compressed centred scale. The 0.100 ceiling–floor gap on Gemma vs 0.258 on Llama means ρ_cen is more sensitive to absolute changes; small differences in centred fve correspond to large differences in ρ_cen. Compare in centred fve directly across backbones, not in ρ_cen.
• Greedy gap is the open problem here too. Best-of-64 oracle rerank is required to beat the paraphrase ceiling.
• Same-layer transfer only. The release uses ℓ=19 (73% depth). Other layers were not evaluated.

Recommended deployment

Best-of-K oracle rerank (sample K, score each by fve_nrm_cen, return argmax). At K=64 this delivers fve_nrm_cen ≈ 0.63 (centred) / 0.75 (raw), exceeding the paraphrase ceiling.

Citation

@misc{lancaster2026nlareframe,
  title  = {Natural-Language Activation Verbalization:
            Probing the Decodability of Frozen Hidden States via Prefix-Tuned Generation},
  author = {Lancaster, Burton},
  year   = {2026},
  note   = {Draft; see github.com/space-bacon/SRT/blob/main/paper_nla.md (§11 cross-backbone Gemma)},
}

Related

• Code: https://github.com/space-bacon/SRT (nla branch)
• Targets dataset: RiverRider/srt-nla-targets-gemma2-2b-v1
• Qwen sibling: RiverRider/srt-nla-av-v1
• Llama sibling: RiverRider/srt-nla-av-llama32-3b