This repository contains the code to reproduce all experimental results and figures from arXiv:2506.15871 using the nnsight library. A summary is available at Blog.

For a fast, interactive introduction to the codebase, please see demo.ipynb.

---

Getting Started

Installation

Clone the repository and install the required dependencies before running any scripts:

git clone https://github.com/xl2024/Interpretable-VLM.git
cd Interpretable-VLM
pip install -r requirements.txt

Running Experiments

All execution scripts for experiments and plotting are located in the src/plots/ directory. You can run them as Python modules from the root directory.

Example:

python -m src.plots.pca_1b

---

Project Structure

.
├── configs/
│   └── config.yaml
├── dataset/                  # Test datasets
│   ├── figure_29/            # Image pairs for Figure 29 reproduction
│   └── figure_3/             # PUG-style dataset for Figure 3 reproduction
├── outputs/                  # Experiment results
│   ├── cma/
│   ├── pca/
│   ├── rsa/
│   └── v0/                   # Historic reference outputs
├── src/
│   ├── data/                 # Dataset generation and experiment data records
│   │   ├── cma/
│   │   ├── test_samples/
│   │   ├── generate_pug_dataset.py
│   │   └── synthetic_generator.py
│   ├── math_core/
│   │   └── rsa.py            # Common RSA computations
│   ├── mech_interp/
│   │   ├── cma.py            # Common Causal Mediation tracking & patching
│   │   └── tracer.py
│   ├── model/
│   │   └── loader.py
│   ├── plots/
│   │   ├── cma_*.py          # Scripts to reproduce and plot CMA results
│   │   ├── pca_*.py          # Scripts to reproduce and plot PCA results
│   │   └── rsa_*.py          # Scripts to reproduce and plot RSA results
│   └── utils/
│       ├── hardware.py
│       └── tools.py
├── demo.ipynb                # Quickstart Jupyter Notebook

---

Reproduction Matrix

The following table details which script maps to which figure, alongside their datasets and output directories.

Notes

¹ Generated using src/data/generate_pug_dataset.py. See dataset/figure_3/README.md for generation details.
² Created from running script.
³ pca_new_26_v2.py was used to generate results in the same output directory and pca_new_26.py requires less memory and more time.