Mechanistic Interpretability of Filler Tokens & Latent Reasoning

Empirical replication and interpretability laboratory based on:

"Reading Between the Dots: Decoding Hidden Computation across Filler Tokens"
Kaley Brauer, Claudio Mayrink Verdun, Samuel Marks (ICML 2026 Mechanistic Interpretability Workshop)
arXiv:2607.03502

---

Key Questions Addressed

1. Why do filler tokens work?
   Autoregressive Transformers operating under single-pass execution are bounded by constant circuit depth ($\mathbf{TC}^0$). Appending content-free filler tokens (e.g. . . ., 1 2 3, a b c) expands the effective computational DAG depth to $O(L + K)$, transforming serial token generation into deeper compositional capacity.
2. Is it taking advantage of parallelism?
   During prompt prefill, the GPU executes all filler tokens in a single parallel tensor operation, while the lower-triangular causal mask enforces an effective sequential relay of depth $L+K$. The model decomposes multi-hop retrieval into parallel working-memory slots before composition in deep layers.
3. Can we read the hidden computation without CoT?
   Using the Residual Logit Lens ($r = P - \bar{P}$) and Jacobian Workspace Lenses (J-Lens) (camilablank/workspace-lenses), intermediate operands ($A_1, A_2$) and in-context arithmetic ($x \to c_1 x \to y$) can be decoded with 80%–95% accuracy without supervision.

---

Directory Structure

d:/MATS/
├── src/
│   ├── tasks.py                       # 1-fact, 2-fact, system-of-equations dataset generators
│   ├── lens.py                        # Residual Logit Lens (P - P_mean) & J-Lens workspace projection
│   ├── attention.py                   # Attention relay routing analyzer (Q -> Filler -> A)
│   ├── interventions.py               # Causal KV-cache transplant & position-resolved patching
│   ├── decoder.py                     # Unsupervised top-K token aggregator + LLM judge prompt formatter
│   └── plotting.py                    # Publication-quality plotting (heatmaps, uplift, attention flow)
├── notebooks/
│   └── filler_token_mech_interp.ipynb # Local interactive Jupyter notebook
├── colab/
│   └── filler_token_colab.ipynb       # Turn-key Google Colab notebook (T4 / L4 / A100)
├── run_experiments.py                 # CLI driver for automated local experiments
├── generate_notebooks.py              # Generator script for .ipynb notebooks
└── requirements.txt                   # Environment dependencies

---

Quickstart

1. Local Execution (RTX 4060 8GB / CUDA)

Activate the environment and run the automated test suite:

.venv\Scripts\python.exe run_experiments.py --task 1fact --model Qwen/Qwen2.5-1.5B-Instruct --filler_type dots --filler_lengths 0,5,10,20 --num_samples 20 --run_interventions

Outputs and plots are saved in results/:

• behavioral_uplift_1fact.png: Accuracy curve vs. filler length.
• logit_lens_heatmap_1fact.png: Residual logit lens layer $\times$ position heatmap.
• attention_relay_1fact.png: $Q \to \text{filler} \to A$ attention budget displacement.
• kv_rank_shift_1fact.png: Causal donor-answer rank improvement distribution.

2. Google Colab (Cloud GPUs)

Open colab/filler_token_colab.ipynb in Google Colab. The notebook auto-installs dependencies and includes pre-configured experiments for larger models (Qwen-2.5-7B, DeepSeek-R1-Distill-7B, etc.) and precomputed Jacobian Workspace Lenses from camilablank/workspace-lenses.