Exemplar Partitioning Dictionaries for Gemma-2-2B

Pretrained Exemplar Partitioning (EP) dictionaries for Gemma-2-2B and Gemma-2-2B-it across multiple layers and resolution percentiles. Each dictionary is a Voronoi partition of the centered unit sphere of residual-stream activations, anchored on observed activation directions (exemplars) drawn from the construction stream.

EP dictionaries are built by streaming Pile activations through leader clustering with a single calibrated cosine-distance threshold $\theta_p$. Construction terminates when no new regions are produced for one batch (saturation). The result is a feature dictionary with one hyperparameter ($p$, the percentile setting $\theta_p$) and no learned parameters.

Contents

All builds use ctx=128, batch=128, seed=0, per-position extraction, and saturation criterion sat_window=1.

Format

Each subdirectory contains:

• gemma-2-2b_layer{L}.pkl (or gemma-2-2b-it_layer{L}.pkl): pickled Dictionary object (see cas/discovery/dictionary.py in the source repo).
• gemma-2-2b_layer{L}_metadata.json: build configuration and run metadata.

The Dictionary object exposes:

d.center            # np.ndarray, shape (d_model,) — fixed calibration centre μ
d.threshold         # float — cosine-distance threshold θ_p
d.partitions        # list[Partition] of length K
d._exemplars        # np.ndarray, shape (K, d_model) — exemplar matrix (centered unit directions)

Each Partition exposes:

p.exemplar_direction       # np.ndarray, shape (d_model,) — first-arrival exemplar
p.mean_member_direction    # np.ndarray, shape (d_model,) — spherical mean of members
p.member_count             # int
p.member_coherence         # float in [0, 1]
p.sample_prompts           # list of (distance, prompt_text, position) tuples
p.sample_members           # list of cached member activation vectors
p.constituent_sample_indices  # list of indices into the construction stream

Loading

import pickle

with open("gemma-2-2b_L12_p10/gemma-2-2b_layer12.pkl", "rb") as f:
    d = pickle.load(f)

print(f"K = {len(d.partitions)} regions, threshold = {d.threshold:.3f}")
for p in d.partitions[:3]:
    print(p.member_count, p.sample_prompts[:1])

The companion code repository (with the cas package needed to unpickle) is at the URL referenced in the paper.

Citation

@misc{rumbelow2026exemplar,
  title  = {Exemplar Partitioning: Understanding Activation Geometry through Streaming Voronoi Dictionaries},
  author = {Rumbelow, Jessica},
  year   = {2026},
  note   = {NeurIPS 2026 submission}
}