neuronpedia.org 🧠🔍

Open source interpretability platform
_{api · steering · activations · circuits/graphs · natural language autoencoders · jacobian lens · autointerp · scoring · inference · search · filter · dashboards · benchmarks · cossim · umap · embeds · probes · saes · lists · exports · uploads}

• About Neuronpedia
• Setting Up Your Local Environment
  • "I Want to Use a Local Database / Import More Neuronpedia Data"
  • "I Want to Do Webapp (Frontend + API) Development"
  • "I Want to Run/Develop Inference Locally"
  • 'I Want to Run/Develop the Graph Server Locally'
  • 'I Want to Run/Develop Autointerp Locally'
  • 'I Want to Do High Volume Autointerp Explanations'
  • 'I Want to Generate My Own Dashboards/Data and Add It to Neuronpedia'
• Architecture
  • Requirements
  • Services
    • Services Are Standalone Apps
    • Service-Specific Documentation
  • OpenAPI
  • Monorepo Directory Structure
• Security
• Contact / Support
• Contributing
• Appendix
  • 'Make' Commands Reference
  • Import Data Into Your Local Database
  • Why an OpenAI API Key Is Needed for Search Explanations

About Neuronpedia

Check out our blog post about Neuronpedia, why we're open sourcing it, and other details. There's also a tweet thread with quick demos.

Feature Overview

A diagram showing the main features of Neuronpedia as of March 2025.

Setting Up Your Local Environment

Every Neuronpedia service runs directly on your machine. Start by setting up your local database.

"I Want to Use a Local Database / Import More Neuronpedia Data"

What This Does + What You'll Get

These steps show you how to configure and connect to your own local database. You can then download sources/SAEs of your choosing:

https://github.com/user-attachments/assets/d7fbb46e-8522-4f98-aa08-21c6529424af

⚠️ Warning: Your database will start out empty. You will need to use the admin panel to import sources/data (activations, explanations, etc).

⚠️ Warning: The local database environment does not have any inference servers connected, so you won't be able to do activation testing, steering, etc initially. You will need to configure a local inference instance.

Steps

1. Install Postgres 16+ along with the pgvector extension, which Neuronpedia uses to search explanations by meaning.

   # macos (homebrew)
   brew install postgresql@16 pgvector && brew services start postgresql@16
   

   # debian / ubuntu
   sudo apt install postgresql-16 postgresql-16-pgvector && sudo systemctl start postgresql
   

   For other platforms, see the pgvector installation notes.
2. Check that Neuronpedia can reach your database

   make db-check
   

   ➡️ Connection details live in apps/webapp/.env.localhost and default to user postgres, password postgres, and database postgres on port 5432. If yours differ, edit POSTGRES_PRISMA_URL and POSTGRES_URL_NON_POOLING there.

3. Create the tables and seed the initial rows

   make db-init
   

4. Bring up the webapp by following webapp development below, then go to localhost:3000 to see your local instance connected to your local database
5. See the warnings above for caveats, and next steps to finish setting up

Next Steps

1. Click here for how to import data into your local database (activations, explanations, etc), because your local database will be empty to start
2. Click here for how to bring up a local inference service for the model/source/SAE you're working with

"I Want to Do Webapp (Frontend + API) Development"

What You'll Get

The webapp serves the frontend and the API. Running it in development mode gives you fast reloads on every change and more informative debug output. If you are purely interested in doing frontend/api development for Neuronpedia, you don't need to set up anything else!

Steps

1. Install Node.js via Node Version Manager

   make install-nodejs
   

2. Install the webapp's dependencies

   make webapp-install
   

3. Run the development server

   make webapp-dev
   

4. Go to localhost:3000 to see your local webapp instance

Doing Local Webapp Development

• Auto-reload: When you change any files in the apps/webapp subdirectory, the localhost:3000 will automatically reload
• Install commands: You do not need to run make install-nodejs again, and you only need to run make webapp-install if dependencies change
• Production build: make webapp-build followed by make webapp-run serves an optimized build instead - slower to build, faster to run, and without debug information

"I Want to Run/Develop Inference Locally"

What This Does + What You'll Get

This subsection shows you how to run an inference instance locally so you can do things like steering, activation testing, etc on the sources/SAEs you've downloaded.

⚠️ Warning: For the local environment, we only support running one inference server at a time. This is because you are unlikely to be running multiple models simultaneously on one machine, as they are memory and compute intensive.

Steps

1. Ensure you have installed uv

2. Install the inference server's dependencies

   make inference-install
   

3. Run the inference server, using the MODEL_SOURCESET argument to specify the .env.inference.[model_sourceset] file you're loading from. For this example, we will run gpt2-small, and load the res-jb sourceset/SAE set, which is configured in the .env.inference.gpt2-small.res-jb file. You can see the other pre-loaded inference configs or create your own config as well.

   make inference-dev MODEL_SOURCESET=gpt2-small.res-jb
   

   ➡️ The server picks its own backend and device: vLLM on CUDA where the architecture supports it, otherwise eager PyTorch. Models are read from your normal Hugging Face cache at ~/.cache/huggingface, so weights you've already downloaded are reused.

4. Wait for it to load (first time will take longer). When you see Initialized: True, the local inference server is now ready on localhost:5002

5. Tell the webapp about it. The webapp looks up every GPU server in the ComputeHost table, so a server it has never been told about is invisible to it:

   make host-add SERVICE=INFERENCE MODEL=gpt2-small URL=http://127.0.0.1:5002 SOURCES=6-res-jb
   

   Leave SOURCES off to say "this host can serve anything for the model", which is what jlens and steering with vectors need. make host-list shows what is registered and make host-remove takes one away.

Using the Inference Server

To interact with the inference server, you have a few options - note that this will only work for the model / selected source you have loaded:

1. Load the webapp with the local database setup, then using the model / selected source as you would normally do on Neuronpedia.
2. Use the OpenAPI spec at apps/inference/openapi.json to make calls with any client of your choice, or to generate one. You can get a Swagger interactive spec at /docs after the server starts up. See the apps/inference/README.md for details. (Set environment variable INFERENCE_SERVER_SECRET to localhost-secret, or whatever it's set to in apps/webapp/.env.localhost if you've changed it.)

Pre-Loaded Inference Server Configurations

We've provided some pre-loaded inference configs as examples of how to load a specific model and sourceset for inference. View them by running make inference-list-configs:

$ make inference-list-configs

Available Inference Configurations (.env.inference.*)
================================================

deepseek-r1-distill-llama-8b.llamascope-slimpj-res-32k
    Model: meta-llama/Llama-3.1-8B
    Source/SAE Sets: '["llamascope-slimpj-res-32k"]'
    make inference-dev MODEL_SOURCESET=deepseek-r1-distill-llama-8b.llamascope-slimpj-res-32k

gemma-2-2b-it.gemmascope-res-16k
    Model: gemma-2-2b-it
    Source/SAE Sets: '["gemmascope-res-16k"]'
    make inference-dev MODEL_SOURCESET=gemma-2-2b-it.gemmascope-res-16k

gpt2-small.res-jb
    Model: gpt2-small
    Source/SAE Sets: '["res-jb"]'
    make inference-dev MODEL_SOURCESET=gpt2-small.res-jb

Making Your Own Inference Server Configurations

Look at the .env.inference.* files for examples on how to make these inference server configurations.

The MODEL_ID is the Hugging Face repo id of the model (openai-community/gpt2, google/gemma-2-2b), which is what the weights load from. Each of SAE_SETS is the text after the layer number and hyphen in a Neuronpedia source ID - for example, if you have a Neuronpedia feature at url http://neuronpedia.org/gpt2-small/0-res-jb/123, the 0-res-jb is the source ID, and the item in the SAE_SETS is res-jb. This example matches the .env.inference.gpt2-small.res-jb file exactly.

You can find Neuronpedia source IDs in the SAELens pretrained SAEs YAML file or by clicking into models in the Neuronpedia datasets exports directory.

Using Models Not Officially Supported by TransformerLens Look at the .env.inference.deepseek-r1-distill-llama-8b.llamascope-slimpj-res-32k to see an example of how to load a model not officially supported by TransformerLens. This is mostly for swapping in weights of a distilled/fine-tuned model.

Loading Non-SAELens Sources/SAEs

• TODO #2 Document how to load SAEs/sources that are not in SAELens pretrained YAML

Doing Local Inference Development

• The Pydantic models are the spec: To add or change an endpoint, edit the models under apps/inference/neuronpedia_inference/schemas/, then run make inference-openapi and make webapp-openapi to refresh the committed openapi.json and the webapp's TypeScript types. There is no schema file to edit first. See OpenAPI below and the "Cross-server APIs" section of AGENTS.md.
• No auto-reload: When you change any files in the apps/inference subdirectory, the inference server will NOT automatically reload, because server reloads are slow: they reload the model and all sources/SAEs. If you want to enable autoreload, then append AUTORELOAD=1 to the make inference-dev call, like so:

  make inference-dev \
       MODEL_SOURCESET=gpt2-small.res-jb \
       AUTORELOAD=1
  

'I Want to Run/Develop the Graph Server Locally'

What This Does + What You'll Get

The graph server powers the attribution graph generation functionality, built on top of circuit-tracer by Piotrowski & Hanna. This service handles the backend processing when you create new graphs through the Neuronpedia Circuit Tracer interface.

Steps

1. Ensure you have installed uv

2. Install the graph server's dependencies

   make graph-install
   

3. Within the apps/graph directory, create a .env file with HF_TOKEN (see apps/graph/.env.example)

   • Make sure your HF_TOKEN has access to the Gemma-2-2B model on Hugging Face.
   • The server secret passed in the x-secret-key request header defaults to localhost-secret. Override it with make graph-dev LOCALHOST_SECRET=your-secret, and set GRAPH_SERVER_SECRET in apps/webapp/.env.localhost to match.

4. Run the graph server:

   make graph-dev
   

5. Wait for it to load. The graph server is then ready on localhost:5004

6. Register it with the webapp, which routes graph requests by source set:

   make host-add SERVICE=GRAPH MODEL=gemma-2-2b URL=http://127.0.0.1:5004 SOURCE_SETS=gemmascope-transcoder-16k
   

For example requests, see the Graph Server README.

'I Want to Run/Develop Autointerp Locally'

What This Does + What You'll Get

The autointerp server provides automatic interpretation and scoring of neural network features. It uses EleutherAI's Delphi for generating explanations and scoring.

⚠️ Warning: The Eleuther embedding scorer uses an embedding model only supported on CUDA (it won't work on Mac MPS or CPU)

Steps

1. Ensure you have installed uv

2. Install the autointerp server's dependencies

   make autointerp-install
   

3. Run the autointerp server:

   make autointerp-dev
   

4. Wait for it to load. The autointerp server is then ready on localhost:5003

Using the Autointerp Server

To interact with the autointerp server, you have a few options:

1. Use the OpenAPI spec at apps/autointerp/openapi.json to make calls with any client of your choice, or to generate one. You can get a Swagger interactive spec at /docs after the server starts up. (Set environment variable AUTOINTERP_SERVER_SECRET to localhost-secret, or whatever it's set to in apps/webapp/.env.localhost if you've changed it.)

Doing Local Autointerp Development

• The Pydantic models are the spec: To add or change an endpoint, edit the models under apps/autointerp/neuronpedia_autointerp/schemas/, then run make autointerp-openapi and make webapp-openapi to refresh the committed openapi.json and the webapp's TypeScript types. There is no schema file to edit first. See OpenAPI below and the "Cross-server APIs" section of AGENTS.md.
• No auto-reload: When you change any files in the apps/autointerp subdirectory, the autointerp server will NOT automatically reload. Restart make autointerp-dev to pick up changes.

'I Want to Do High Volume Autointerp Explanations'

This section is under construction.

• Use EleutherAI's Delphi library
• For OpenAI's autointerp, use utils/neuronpedia_utils/batch-autointerp.py

'I Want to Generate My Own Dashboards/Data and Add It to Neuronpedia'

This section is under construction.

TODO: Simplify generation + upload of data to Neuronpedia

TODO: neuronpedia-utils should use Poetry

In this example, we will generate dashboards/data for an SAELens-compatible SAE, and upload it to our own Neuronpedia instance.

1. Ensure you have Poetry installed

2. Upload your SAELens-compatible source/SAE to Hugging Face.

   Example ➡️ https://huggingface.co/chanind/gemma-2-2b-batch-topk-matryoshka-saes-w-32k-l0-40

3. Clone SAELens locally.

   git clone https://github.com/jbloomAus/SAELens.git
   

4. Open your cloned SAELens and edit the file sae_lens/pretrained_saes.yaml. Add a new entry at the bottom, based on the template below (see comments for how to fill it out):

   Example ➡️ https://github.com/jbloomAus/SAELens/pull/455/files

   gemma-2-2b-res-matryoshka-dc:                 # a unique ID for your set of SAEs
     conversion_func: null                       # null if your SAE config is already compatible with SAELens
     links:                                      # optional links
       model: https://huggingface.co/google/gemma-2-2b
     model: gemma-2-2b                           # transformerlens model id - https://transformerlensorg.github.io/TransformerLens/generated/model_properties_table.html
     repo_id: chanind/gemma-2-2b-batch-topk-matryoshka-saes-w-32k-l0-40  # the huggingface repo path
     saes:
     - id: blocks.0.hook_resid_post                 # an id for this SAE
       path: standard/blocks.0.hook_resid_post      # the path in the repo_id to the SAE
       l0: 40.0
       neuronpedia: gemma-2-2b/0-matryoshka-res-dc  # what you expect the Neuronpedia URI to be - neuronpedia.org/[this_slug]. should be [model_id]/[layer]-[identical_slug_for_this_sae_set]
     - id: blocks.1.hook_resid_post                 # more SAEs in this SAE set
       path: standard/blocks.1.hook_resid_post
       l0: 40.0
       neuronpedia: gemma-2-2b/1-matryoshka-res-dc  # note that this is identical to the entry above, except 1 instead of 0 for the layer
     - [...]
   

5. Clone SAEDashboard locally.

   git clone https://github.com/jbloomAus/SAEDashboard.git
   

6. Configure your cloned SAEDashboard to use your cloned modified SAELens, instead of the one in production

   cd SAEDashboard                    # set directory
   poetry lock && poetry install      # install dependencies
   poetry remove sae-lens             # remove production dependency
   poetry add PATH/TO/CLONED/SAELENS  # set local dependency
   

7. Generate dashboards for the SAE. This will take from 30 min to a few hours, depending on your hardware and size of model.

   cd SAEDashboard                    # set directory
   rm -rf cached_activations          # clear old cached data
   
   # start the generation. details for each argument (full details: https://github.com/jbloomAus/SAEDashboard/blob/main/sae_dashboard/neuronpedia/neuronpedia_runner_config.py)
   #     - sae-set = should match the unique ID for the set from pretrained_saes.yaml
   #     - sae-path = should match the id for the sae in from pretrained_saes.yaml
   #     - np-set-name = should match the [identical_slug_for_this_sae_set] for the sae.Neuronpedia from pretrained_saes.yaml
   #     - dataset-path = the huggingface dataset to use for generating activations. usually you want to use the same dataset the model was trained on.
   #     - output-dir = the output directory of the dashboard data
   #     - n-prompts = number of activation texts to test from the dataset
   #     - n-tokens-in-prompt, n-features-per-batch, n-prompts-in-forward-pass = keep these at 128
   poetry run neuronpedia-runner \
        --sae-set="gemma-2-2b-res-matryoshka-dc" \
        --sae-path="blocks.12.hook_resid_post" \
        --np-set-name="matryoshka-res-dc" \
        --dataset-path="monology/pile-uncopyrighted" \
        --output-dir="neuronpedia_outputs/" \
        --sae_dtype="float32" \
        --model_dtype="bfloat16" \
        --sparsity-threshold=1 \
        --n-prompts=24576 \
        --n-tokens-in-prompt=128 \
        --n-features-per-batch=128 \
        --n-prompts-in-forward-pass=128
   

8. Convert these dashboards for import into Neuronpedia

   cd neuronpedia/utils/neuronpedia-utils          # get into this current repository's util directory
   python convert-saedashboard-to-neuronpedia.py   # start guided conversion script. follow the steps.
   

9. Once dashboard files are generated for Neuronpedia, upload these to the global Neuronpedia S3 bucket - currently you need to contact us to do this.

10. From a localhost instance, import your data

Architecture

Here's how the services/scripts connect in Neuronpedia. It's easiest to read this diagram by starting at the image of the laptop ("User").

Requirements

You can run Neuronpedia on any cloud and on any modern OS. Neuronpedia is designed to avoid vendor lock-in. These instructions were written for and tested on macOS 15 (Sequoia), so you may need to repurpose commands for Windows/Ubuntu/etc. At least 16GB RAM is recommended.

Each service runs directly on the host, so install the toolchain for the ones you plan to work on:

Services

Services Are Standalone Apps

By design, each service can be run independently as a standalone app. This is to enable extensibility and forkability.

For example, if you like the Neuronpedia webapp frontend but want to use a different API for inference, you can do that! Just ensure your alternative inference server matches the apps/inference/openapi.json spec, and/or that you modify the Neuronpedia calls to inference under apps/webapp/lib/utils.

Service-Specific Documentation

There are draft READMEs for each specific app/service under apps/[service], but they are heavily WIP. Each service's pyproject.toml or package.json under the same directory lists its dependencies if you want to run or package it yourself.

OpenAPI

For services to communicate with each other in a typed and consistent way, we generate types from OpenAPI — in one direction, from the Python out.

Each Python service's Pydantic models are the source of truth. make <app>-openapi dumps that server's route table to apps/<app>/openapi.json, and make webapp-openapi turns every one of those specs into apps/webapp/lib/api/<app>.d.ts for the webapp to compile against. Both artifacts are committed, and both are guarded: a drift test in each service's own suite catches a stale openapi.json, and .github/workflows/openapi-drift.yml catches stale TypeScript.

So there is no spec file to hand-edit and nothing to publish by hand. To change a wire format, change the Pydantic model and run make openapi, which does both halves for every service you have installed — regenerating the spec but not the TypeScript is the easy mistake, and it surfaces in CI rather than locally.

The neuronpedia-{inference,autointerp}-client packages on npm and PyPI are still published for callers outside this repo, but nothing here imports them and they are committed nowhere. .github/workflows/openapi-publish.yml rebuilds them from the same committed openapi.json, so they are downstream of a wire-format change rather than a step in making one. make sdk-dry-run SERVICE=inference runs that build locally without uploading.

The one thing this cannot cover is streaming: SSE and NDJSON frames are not response bodies, so they never reach a spec. Those are pinned by contract tests instead — see apps/inference/tests/unit/test_lens_frame_contract.py and apps/nla/tests/test_frame_contract.py.

For the full workflow, including which servers are camelCase on the wire and which are deliberately snake_case, see the "Cross-server APIs" section of AGENTS.md.

Monorepo Directory Structure

apps - The six Neuronpedia services: webapp, inference, autointerp, graph, nla, and sparsity. Most of the code is here. utils - Various utilities that help do offline processing, like high volume autointerp, or generating dashboards, or exporting data. webapp-python-client - The hand-written Python SDK for the public API, published to PyPI as neuronpedia.

The interpretability engine that inference, graph and nla run on — hooking, capture, steering, the vLLM backend — is interp-engine, a separate repository published to PyPI. Those three apps pin a release of it; see "interp-engine is a dependency" in AGENTS.md for working on both at once.

Security

Please report vulnerabilities to johnny@neuronpedia.org.

We don't currently have an official bounty program, but we'll try our best to give compensation based on the severity of the vulnerability - though it's likely we will not able able to offer awards for any low-severity vulnerabilities.

Contact / Support

• Slack: join #neuronpedia
• Email: johnny@neuronpedia.org
• Issues: GitHub issues

Contributing

See CONTRIBUTING.md.

The checks that gate a pull request are the ones in make python-lint and npm run lint. You can have the fast ones run on each commit, over the files you changed, with make githooks-install - make webapp-install enables the same .githooks/ hook for you through npm. It is per checkout and optional: see "Checks Before You Commit".

Appendix

'Make' Commands Reference

You can view all available make commands and brief descriptions of them by running make help

Import Data Into Your Local Database

If you set up your own database, it will start out empty - no features, explanations, activations, etc. To load this data, there's a built-in admin panel where you can download this data for SAEs (or "sources") of your choosing.

⚠️ Warning: The admin panel is finicky and does not currently support resuming imports. If an import is interrupted, you must manually click re-sync. The admin panel currently does not check if your download is complete or missing parts - it is up to you to check if the data is complete, and if not, to click re-sync to re-download the entire dataset.

ℹ️ Recommendation: When importing data, start with just one source (like gpt2-small@10-res-jb) instead of downloading everything at once. This makes it easier to verify the data imported correctly and lets you start using Neuronpedia faster.

The instructions below demonstrate how to download the gpt2-small@10-res-jb SAE data.

1. Navigate to localhost:3000/admin.
2. Scroll down to gpt2-small, and expand res-jb with the ▶.
3. Click Download next to 10-res-jb.
4. Wait patiently - this can be a LOT of data, and depending on your connection/CPU speed it can take up to 30 minutes or an hour.
5. Once it's done, click Browse or use the navbar to try it out: Jump To/Search/Steer.
6. Repeat for other SAE/source data you wish to download.

Why an OpenAI API Key Is Needed for Search Explanations

In the webapp, the Search Explanations feature requires you to set an OPENAI_API_KEY. Otherwise you will get no search results.

This is because the search explanations functionality searches for features by semantic similarity. If you search cat, it will also return feline, tabby, animal, etc. To do this, it needs to calculate the embedding for your input cat. We use OpenAI's embedding API (specifically, text-embedding-3-large with dimension: 256) to calculate the embeddings.