Shadowgraph
About
Sign in

chanind/gemma-2-2b-batch-topk-matryoshka-saes-w-32k-l0-40

Model

2

stars

54

commits

2

repos using this model

2

linked in READMEs

Apr 3, 2025

updated

saelens

README

SAEs for use with the SAELens library

This repository contains the following batch topk Matryoshka SAEs for Gemma-2-2b. All SAEs have 32k width and are trained with k=40 on 750M tokens from the Pile using SAELens. The SAEs were trained with Matryoshka layers of width 128, 512, 2048, 8192, and 32768. This release contains both standard Matryoshka SAE and snap loss Matryoshka SAEs.

This repository contains the following SAEs:

Snap Loss Matryoshka SAEs

layerSAE IDwidthl0explained variance
0snap/blocks.0.hook_resid_post32768400.919964
1snap/blocks.1.hook_resid_post32768400.863969
2snap/blocks.2.hook_resid_post32768400.858767
3snap/blocks.3.hook_resid_post32768400.815844
4snap/blocks.4.hook_resid_post32768400.821094
5snap/blocks.5.hook_resid_post32768400.797083
6snap/blocks.6.hook_resid_post32768400.79815
7snap/blocks.7.hook_resid_post32768400.78946
8snap/blocks.8.hook_resid_post32768400.779236
9snap/blocks.9.hook_resid_post32768400.759022
10snap/blocks.10.hook_resid_post32768400.743998
11snap/blocks.11.hook_resid_post32768400.731758
12snap/blocks.12.hook_resid_post32768400.725974
13snap/blocks.13.hook_resid_post32768400.727936
14snap/blocks.14.hook_resid_post32768400.727065
15snap/blocks.15.hook_resid_post32768400.757408
16snap/blocks.16.hook_resid_post32768400.751874
17snap/blocks.17.hook_resid_post32768400.763654
18snap/blocks.18.hook_resid_post32768400.77644
19snap/blocks.19.hook_resid_post32768400.768622
20snap/blocks.20.hook_resid_post32768400.761658
21snap/blocks.21.hook_resid_post32768400.765593
22snap/blocks.22.hook_resid_post32768400.741098
23snap/blocks.23.hook_resid_post32768400.729718
24snap/blocks.24.hook_resid_post32768400.754838

Standard Matryoshka SAEs

layerSAE IDwidthl0explained variance
0standard/blocks.0.hook_resid_post32768400.91832
1standard/blocks.1.hook_resid_post32768400.863454
2standard/blocks.2.hook_resid_post32768400.841324
3standard/blocks.3.hook_resid_post32768400.814794
4standard/blocks.4.hook_resid_post32768400.820418
5standard/blocks.5.hook_resid_post32768400.796252
6standard/blocks.6.hook_resid_post32768400.797322
7standard/blocks.7.hook_resid_post32768400.787601
8standard/blocks.8.hook_resid_post32768400.779433
9standard/blocks.9.hook_resid_post32768400.75697
10standard/blocks.10.hook_resid_post32768400.745011
11standard/blocks.11.hook_resid_post32768400.732177
12standard/blocks.12.hook_resid_post32768400.726209
13standard/blocks.13.hook_resid_post32768400.719405
14standard/blocks.14.hook_resid_post32768400.719056
15standard/blocks.15.hook_resid_post32768400.756888
16standard/blocks.16.hook_resid_post32768400.742889
17standard/blocks.17.hook_resid_post32768400.757294
18standard/blocks.18.hook_resid_post32768400.76921
19standard/blocks.19.hook_resid_post32768400.766661
20standard/blocks.20.hook_resid_post32768400.760939
21standard/blocks.21.hook_resid_post32768400.759883
22standard/blocks.22.hook_resid_post32768400.740612
23standard/blocks.23.hook_resid_post32768400.729678
24standard/blocks.24.hook_resid_post32768400.747313

Load these SAEs using SAELens as below:

from sae_lens import SAE

sae, cfg_dict, sparsity = SAE.from_pretrained("chanind/gemma-2-2b-batch-topk-matryoshka-saes-w-32k-l0-40", "<sae_id>")

Contributors

chanind

54 commits

Similar repos

google/gemma-scope

Model

208

google/gemma-scope-2-27b-it

Model

24

google/gemma-scope-9b-pt-res

Model

7

google/gemma-scope-2b-pt-transcoders

Model

14

google/gemma-scope-9b-it-res

Model

12

andreuka18/deepseek-r1-distill-llama-8b-lmsys-openthoughts

Model

1

decoderesearch/SAELens

Training Sparse Autoencoders on Language Models

1,528

jbloomAus/SAELens

Training Sparse Autoencoders on Language Models

1,528

Used in code

rpp5501/sae-hierarchy-diagnostic

0

1 file

alphaXiv/ica-lens-paper-e1fdec2f

0

1 file

Linked in READMEs

jbloomAus/SAELens

Training Sparse Autoencoders on Language Models

1,528

README badge

hijohnnylin/neuronpedia

open source interpretability platform 🧠

1,133

README badge

chanind/gemma-2-2b-batch-topk-matryoshka-saes-w-32k-l0-40

Model

2

stars

54

commits

2

repos using this model

2

linked in READMEs

Apr 3, 2025

updated

saelens

README

SAEs for use with the SAELens library

This repository contains the following batch topk Matryoshka SAEs for Gemma-2-2b. All SAEs have 32k width and are trained with k=40 on 750M tokens from the Pile using SAELens. The SAEs were trained with Matryoshka layers of width 128, 512, 2048, 8192, and 32768. This release contains both standard Matryoshka SAE and snap loss Matryoshka SAEs.

This repository contains the following SAEs:

Snap Loss Matryoshka SAEs

layerSAE IDwidthl0explained variance
0snap/blocks.0.hook_resid_post32768400.919964
1snap/blocks.1.hook_resid_post32768400.863969
2snap/blocks.2.hook_resid_post32768400.858767
3snap/blocks.3.hook_resid_post32768400.815844
4snap/blocks.4.hook_resid_post32768400.821094
5snap/blocks.5.hook_resid_post32768400.797083
6snap/blocks.6.hook_resid_post32768400.79815
7snap/blocks.7.hook_resid_post32768400.78946
8snap/blocks.8.hook_resid_post32768400.779236
9snap/blocks.9.hook_resid_post32768400.759022
10snap/blocks.10.hook_resid_post32768400.743998
11snap/blocks.11.hook_resid_post32768400.731758
12snap/blocks.12.hook_resid_post32768400.725974
13snap/blocks.13.hook_resid_post32768400.727936
14snap/blocks.14.hook_resid_post32768400.727065
15snap/blocks.15.hook_resid_post32768400.757408
16snap/blocks.16.hook_resid_post32768400.751874
17snap/blocks.17.hook_resid_post32768400.763654
18snap/blocks.18.hook_resid_post32768400.77644
19snap/blocks.19.hook_resid_post32768400.768622
20snap/blocks.20.hook_resid_post32768400.761658
21snap/blocks.21.hook_resid_post32768400.765593
22snap/blocks.22.hook_resid_post32768400.741098
23snap/blocks.23.hook_resid_post32768400.729718
24snap/blocks.24.hook_resid_post32768400.754838

Standard Matryoshka SAEs

layerSAE IDwidthl0explained variance
0standard/blocks.0.hook_resid_post32768400.91832
1standard/blocks.1.hook_resid_post32768400.863454
2standard/blocks.2.hook_resid_post32768400.841324
3standard/blocks.3.hook_resid_post32768400.814794
4standard/blocks.4.hook_resid_post32768400.820418
5standard/blocks.5.hook_resid_post32768400.796252
6standard/blocks.6.hook_resid_post32768400.797322
7standard/blocks.7.hook_resid_post32768400.787601
8standard/blocks.8.hook_resid_post32768400.779433
9standard/blocks.9.hook_resid_post32768400.75697
10standard/blocks.10.hook_resid_post32768400.745011
11standard/blocks.11.hook_resid_post32768400.732177
12standard/blocks.12.hook_resid_post32768400.726209
13standard/blocks.13.hook_resid_post32768400.719405
14standard/blocks.14.hook_resid_post32768400.719056
15standard/blocks.15.hook_resid_post32768400.756888
16standard/blocks.16.hook_resid_post32768400.742889
17standard/blocks.17.hook_resid_post32768400.757294
18standard/blocks.18.hook_resid_post32768400.76921
19standard/blocks.19.hook_resid_post32768400.766661
20standard/blocks.20.hook_resid_post32768400.760939
21standard/blocks.21.hook_resid_post32768400.759883
22standard/blocks.22.hook_resid_post32768400.740612
23standard/blocks.23.hook_resid_post32768400.729678
24standard/blocks.24.hook_resid_post32768400.747313

Load these SAEs using SAELens as below:

from sae_lens import SAE

sae, cfg_dict, sparsity = SAE.from_pretrained("chanind/gemma-2-2b-batch-topk-matryoshka-saes-w-32k-l0-40", "<sae_id>")

Similar repos

google/gemma-scope

Model

208

google/gemma-scope-2-27b-it

Model

24

google/gemma-scope-9b-pt-res

Model

7

google/gemma-scope-2b-pt-transcoders

Model

14

google/gemma-scope-9b-it-res

Model

12

andreuka18/deepseek-r1-distill-llama-8b-lmsys-openthoughts

Model

1

decoderesearch/SAELens

Training Sparse Autoencoders on Language Models

1,528

jbloomAus/SAELens

Training Sparse Autoencoders on Language Models

1,528

Used in code

rpp5501/sae-hierarchy-diagnostic

0

1 file

alphaXiv/ica-lens-paper-e1fdec2f

0

1 file

Linked in READMEs

jbloomAus/SAELens

Training Sparse Autoencoders on Language Models

1,528

README badge

hijohnnylin/neuronpedia

open source interpretability platform 🧠

1,133

README badge

Contributors

chanind

54 commits