PearlyGates

PearlyGates is a research harness for studying causal geometry in transformer residual streams with activation patching.

The current paper studies whether clean/corrupt intervention effects are better understood as isolated layer-localized causes, or as low-rank residual subspaces that are partially shared across related prompt surfaces and can reroute through late-layer mediation bands.

Repository: https://github.com/TheApeMachine/pearlygates

Research Claim

The working hypothesis is deliberately narrow:

Causal mediation is often better modeled as partially shared low-dimensional residual subspaces that are robust across related surfaces but reroutable under intervention.

The project does not claim to identify universal semantic coordinates, a full structural causal model, or a single location where a fact is stored. The current evidence is stronger for low-rank residual transport geometry than for reusable semantic causal coordinates.

What The Harness Measures

Each benchmark defines:

• a clean prompt,
• a corrupt prompt,
• a target token,
• a distractor token.

The harness runs the model on both prompts, patches internal activations from one counterfactual world into the other, and measures restoration at the final sequence position. The main normalized scores are:

• target-logit restoration,
• target-minus-distractor margin restoration.

The final residual stream of the last block is treated as a readout upper bound and excluded from competitive causal layer rankings.

Main Results In The Current Paper

The paper in production/paper/main.tex and production/paper/main.pdf reports:

• high recovery from learned low-rank residual subspace patches,
• near-zero recovery from same-rank random orthonormal controls,
• selective held-out semantic-family transfer under stronger empirical nulls,
• adversarial marginal-scramble controls that constrain semantic-coordinate claims,
• shared/local residual decomposition across related prompt surfaces,
• high-entropy layer profiles rather than single-layer localization,
• layer-band reroute and ablate-and-rescue diagnostics,
• sparse latent and pretrained SAE transfer probes,
• attention-path interventions,
• exploratory worldline surfaces for causal-state rather than factual-recall geometry,
• calibration-aware cross-model replication and basis-alignment diagnostics.

The central interpretive distinction is geometric concentration versus semantic identity. A low-rank basis can restore behavior because it carries a reusable causal coordinate, but it can also restore behavior because it moves the residual stream toward a narrow readout-support region. Much of the project is designed to stress-test that alternative.

Repository Layout

The repository contains both early prototype files and the maintained production package.

.
|-- README.md
|-- Makefile
|-- main.py
|-- frontier_lab.py
|-- milestone_interp.py
|-- benchmark_specs.py
|-- filter/
`-- production/
    |-- pearlygates/
    |-- examples/
    |-- scripts/
    |-- tests/
    |-- docs/
    |-- paper/
    |-- Makefile
    |-- pyproject.toml
    `-- uv.lock

Important paths:

• production/pearlygates/: maintained Python package.
• production/examples/: model and experiment YAML configs.
• production/scripts/: table generation, visualization, basis alignment, and scramble diagnostics.
• production/tests/: unit tests that avoid loading a full model.
• production/docs/: methodology, research program notes, known issues, and next steps.
• production/paper/: LaTeX source, generated tables, bibliography, and compiled PDF.
• filter/: separate causal-worldline extraction prototype.
• root main.py, frontier_lab.py, milestone_interp.py: earlier monolithic prototype entry points preserved for reference.

For new work, use production/.

Installation

The recommended workflow uses uv from the production/ directory:

cd production
uv sync --extra dev --extra sae

For a plain editable install:

cd production
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

For optional pretrained SAE transfer:

pip install -e '.[sae,dev]'

Some model configs use gated Hugging Face checkpoints. Accept the relevant licenses and authenticate before running those configs:

huggingface-cli login

or set HF_TOKEN in the environment.

Quick Start

From the repository root, the top-level Makefile delegates to the production pipeline:

make help
make paper-main

make paper-main runs the main configured experiment, regenerates visualizations and paper tables for the single-model path, and compiles the PDF. It is the best iteration target while editing the paper.

The full paper pipeline is:

make paper

This runs the main frontier suite, visualizations, alternate-model replication tier, cross-model basis alignment, adversarial basis scrambling, table generation, and LaTeX compilation.

Production Commands

Most commands can be run directly from production/.

Inspect a config without loading a model:

uv run pearlygates inspect --config examples/france.yaml

Validate target and distractor tokenization for the active model:

uv run pearlygates validate-tokens --config examples/france.yaml

Run the baseline suite:

uv run pearlygates run --config examples/france.yaml --mode baseline

Run the frontier probes:

uv run pearlygates run --config examples/frontier.yaml --mode frontier

Run the full configured production suite:

uv run --extra sae pearlygates run --config examples/frontier.yaml --mode all

Generate visualizations:

uv run python scripts/visualize_results.py runs/frontier

Regenerate paper tables:

uv run python scripts/generate_paper_tables.py runs/frontier --out-dir paper/tables
uv run python scripts/generate_model_summary.py runs/models --out-dir paper/tables

Run cross-model basis alignment:

uv run python scripts/cross_model_basis_alignment.py runs/models --out-dir paper/tables

Run adversarial basis scrambling:

uv run python scripts/adversarial_basis_scramble.py runs/frontier --out-dir paper/tables

Compile the paper only:

cd production/paper
make all

Model Replication Targets

The production Makefile exposes separate model tiers:

cd production
make paper-models
make paper-models-experimental
make paper-models-bridge

The default required model tier includes:

• Llama 3.2 1B Instruct,
• GPT-OSS-20B through the custom safetensors loader,
• Gemma 2 2B-it text-only replication.

paper-models-experimental keeps an extra text-only Gemma 3 1B-it probe available outside the default tier. paper-models-bridge keeps multimodal TransformerBridge/Gemma configs separate because some installs fail before PearlyGates hooks are available.

Validate or smoke-test one model config:

make model-validate MODEL_CONFIG=examples/models/gpt_oss_20b.yaml
make model-smoke MODEL_CONFIG=examples/models/gpt_oss_20b.yaml

Output Artifacts

Runs write artifacts under each config's output_dir, usually runs/frontier or runs/models.

Common outputs include:

• run.json,
• benchmarks.csv,
• residual sweep CSVs,
• frontier_subspace.csv,
• frontier_subspace_holdout.csv,
• frontier_shared_local.csv,
• frontier_cross_domain_transfer.csv,
• frontier_cross_domain_summary.csv,
• frontier_layer_profiles.csv,
• frontier_layer_profile_distances.csv,
• frontier_layer_band_reroute.csv,
• frontier_layer_band_rescue.csv,
• frontier_pairwise_cosines.csv,
• attention_path_interventions.csv,
• worldline_patch_surfaces.csv,
• saelens_transfer.csv when a compatible pretrained SAE is configured.

Paper tables are generated into production/paper/tables/. Figures are drawn from the visualization PDFs under production/runs/frontier/visualizations/.

Methodological Guardrails

The project treats positive patching results as measurements, not proofs of a semantic mechanism. Main guardrails include:

• rank-matched random subspace controls,
• empirical grouped-basis controls,
• adversarial marginal-scrambled bases,
• held-out semantic-family transfer,
• calibrated cross-model replication,
• layer-profile entropy rather than argmax-only localization,
• moving-window and cumulative layer-band ablations,
• ablate-and-rescue diagnostics,
• explicit skip reasons for thin or inverted clean/corrupt gaps,
• worldline surfaces treated as stress tests, not confirmation of a universal semantic subspace.

The interpretation would be weakened if compositional interventions fail selectivity, if stronger readout-preserving scrambles retain recovery, if fit/holdout worldline transfer fails despite local patchability, or if matched-capacity nonlinear maps dominate the linear SVD/CCA diagnostics.

Testing

Run tests from production/:

cd production
uv run pytest

The tests avoid loading TransformerLens or a full model. They cover normalization logic, config behavior, log parsing, residual-subspace geometry, basis alignment, and worldline utilities.

Paper

The current paper source is:

production/paper/main.tex

The compiled PDF is:

production/paper/main.pdf

To rebuild only the LaTeX paper from existing tables and figures:

cd production/paper
make all

To rebuild the full experiment-backed paper from the repository root:

make paper

Current Status

This is an active research artifact. The paper is scoped as a technical report: the methods and results are intended to be reproducible and falsification-aware, but the benchmark suite is still small and several claims require stronger cross-model and compositional follow-up before they should be treated as settled.