Faithful and Efficient Circuit Discovery in Language Models via Relevance Patching

If you find this code useful, please star the repository. ⭐

🔖 Overview

Activation patching is a standard method in mechanistic interpretability for localizing the components of a model responsible for specific behaviors, but it is computationally expensive to apply at scale. Attribution patching offers a faster, gradient-based approximation, yet suffers from noise and reduced reliability in deep, highly non-linear networks. In this work, we introduce Relevance Patching (RelP), which replaces the local gradients in attribution patching with propagation coefficients derived from Layer-wise Relevance Propagation (LRP). LRP propagates the network's output backward through the layers, redistributing relevance to lower-level components according to local propagation rules that ensure properties such as relevance conservation or improved signal-to-noise ratio. Like attribution patching, RelP requires only two forward passes and one backward pass, maintaining computational efficiency while improving faithfulness. We validate RelP across a range of models and tasks, showing that it more accurately approximates activation patching than standard attribution patching, particularly when analyzing residual stream and MLP outputs in the Indirect Object Identification (IOI) task. For instance, for MLP outputs in GPT-2 Large, attribution patching achieves a Pearson correlation of 0.006, whereas RelP reaches 0.956, highlighting the improvement offered by RelP. Additionally, we compare the faithfulness of sparse feature circuits identified by RelP and Integrated Gradients (IG), showing that RelP achieves comparable faithfulness without the extra computational cost associated with IG.

📢 News

• 2025.09.23 🎉 RelP is accepted as a Spotlight at NeurIPS 2025 Mechanistic Interpretability Workshop.
• 2025.09.01 🚀 We have released the code for RelP.

Quick Start

🛠️ Install

git clone https://github.com/FarnoushRJ/RelP.git
cd RelP/TransformerLens
pip install -e .

🚀 Use

import transformer_lens

# Load a model (eg GPT-2 Small)
model = transformer_lens.HookedTransformer.from_pretrained("gpt2-small")
model.cfg.use_lrp = True  # Use LRP
model.cfg.LRP_rules = ['LN-rule', 'Identity-rule', 'Half-rule']  # LRP rules to be used

# Run the model and get logits and activations
logits, activations = model.run_with_cache("Hello World")

📜 Propagation Rules for Transformers

The following rules are currently implemented and supported.

---

🙏 Acknowledgment

We build on the TransformerLens library, which provides the foundation for much of this code.

🔎 License

This project is distributed under the Apache 2.0 License. It incorporates code and models from third-party sources, which are provided under their own licenses. Please review those licenses before use.

Citation

@misc{jafari2025relp,
      title={RelP: Faithful and Efficient Circuit Discovery via Relevance Patching}, 
      author={Farnoush Rezaei Jafari and Oliver Eberle and Ashkan Khakzar and Neel Nanda},
      year={2025},
      eprint={2508.21258},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2508.21258}, 
}