ArXiv: 2403.00745

🎯 Pitch

When trying to localize a specific behavior in large language models, gradient-based approximations can miss up to 90% of the truly causal components because attention softmax saturation collapses gradient signals and cancellation between direct and indirect effects hides important contributors. The proposed AtP* method fixes these blind spots by recomputing the softmax during gradient computation and zeroing out node contributions during backpropagation, recovering those hidden causal links while remaining orders of magnitude faster than brute-force patching.

1. Executive Summary

This paper introduces AtP*, an improved variant of Attribution Patching (a gradient-based approximation to the expensive practice of exhaustively measuring every model component's causal effect via Activation Patching) that addresses two identified failure modes — attention saturation (where softmax non-linearity collapses gradient information for keys and queries) and cancellation (where direct and indirect effects offset, producing near-zero and thus-missed estimates) — while retaining the scalability that makes it suitable for state-of-the-art LLMs. Evaluating against brute-force Activation Patching on the Pythia model suite (410M to 12B parameters) across several prompt-pair distributions including a factual recall task (CITY-PP) and an indirect object identification task (IOI-PP), the authors demonstrate that AtP significantly outperforms alternative fast approximation methods (Subsampling, Blocks, Hierarchical), with AtP* providing further significant reduction in false negatives, establishing that the method can serve as a reliable prefiltering step for localizing behaviour to fine-grained components — though its strongest gains manifest on single prompt pairs exhibiting crisp, localized circuitry, with the cancellation-correction benefit diminishing when effects are averaged across a broader distribution.

2. Context and Motivation

The Core Problem: Exhaustive Causal Attribution Is Prohibitively Expensive

The fundamental challenge this paper tackles is a computational bottleneck in mechanistic interpretability: identifying which specific components of a transformer model (e.g., attention heads, MLP neurons, individual token-position activations) are causally responsible for a particular model behavior. The gold-standard method for doing this is Activation Patching (also known as causal mediation analysis), where you intervene on a model component by replacing its activation during a "clean" forward pass (where the behavior of interest occurs) with the activation from a "noise" or "corrupted" forward pass (where the behavior is absent), and then measure how much the model's output changes.

This is a direct, causally rigorous attribution. The problem is that doing it exhaustively — sweeping across every component — requires a number of forward passes that scales linearly with the number of components. As the authors note in Section 2.2:

"On state of the art models, computing c(n)c(n) for all nn can be prohibitively expensive as there may be billions or more nodes."

The scale is genuinely staggering for fine-grained attributions. The authors give a concrete example: on Chinchilla 70B, with a prompt of length 1024 tokens, there are approximately 2.7×1092.7 \times 10^9 neuron nodes alone. Each intervention requires a separate forward pass. If each forward pass takes even a fraction of a second, exhaustively verifying every single neuron is a multi-year computational endeavor. This is not a hypothetical bottleneck — it makes certain kinds of attribution (e.g., neuron-level, or edge-level between pairs of components) practically impossible on frontier models.

This matters because, as the introduction highlights, many important model behaviors have been found to be driven by sparse subgraphs within the model — small collections of attention heads or MLP layers that form identifiable circuits (Olsson et al., 2022; Wang et al., 2022; Meng et al., 2023). Finding these circuits requires attribution at fine granularity across all components, but the computational cost of doing so naively prevents this type of analysis from being conducted at scale, particularly on the largest and most interesting models.

Why This Problem Matters

The paper's framing situates this work within the broader project of mechanistic interpretability: as LLMs become "ubiquitous and integrated into numerous digital applications," understanding their internal mechanisms becomes "an increasingly pressing research problem" (Section 1). There are at least four concrete reasons why making fine-grained causal attribution practical — not just possible, but efficient — is significant:

1. Scaling to the frontier. Interpretability work has historically focused on relatively small models (e.g., GPT-2 Small, with ~85M parameters) or on coarse-grained components (e.g., entire layers). But the models that matter for safety, policy, and deployment are much larger, and the most interesting circuits likely operate at finer granularities than whole layers. The authors explicitly note that the cost blowup is worst precisely where interpretability might be most valuable — when attributing behavior to small components like heads or neurons on SoTA models with long context lengths.

2. Enabling comprehensive analysis. Without efficient attribution methods, researchers face a difficult tradeoff: they can either restrict their analysis to a small set of pre-selected components (risking missing important contributors), or they can perform exhaustive activation patching at prohibitive cost. A fast approximation method that reliably surfaces the top contributors would enable genuinely comprehensive analysis — checking every component — at a cost closer to that of targeted analysis.

3. Downstream applications. The paper identifies several applications in Section 5.3 that depend on efficient component attribution: automated circuit finding (automatically identifying sparse subgraphs responsible for behaviors), mask learning (learned probabilistic masks over components, where mask probabilities are updated based on estimated contributions), and targeted steering (identifying components whose activation can be modified at inference time to control model behavior). All of these require iterating or sweeping over many components, making the speedup from approximate methods multiply valuable.

4. Averaging over distributions. The need to average effects across a distribution D\mathcal{D} of prompt pairs (Section 2.1) — rather than patching on a single prompt pair — adds "a potentially large multiplicative factor to the cost" that further motivates efficient methods. If exhaustive patching on a single prompt pair costs ∣N∣|N| forward passes, doing it on a distribution with 120 prompt pairs (as in the IOI distribution studied in Section 4.3) would cost 120×∣N∣120 \times |N| forward passes. This makes distributional analysis essentially impossible with brute-force methods on anything but the smallest models.

Prior Approaches and Where They Fall Short

The paper situates itself against two broad classes of prior work: exact causal attribution methods (which are too expensive) and existing approximation methods (which have uncharacterized failure modes or limited scope).

Exact Methods: Activation Patching and Its Variants

The standard approach, documented extensively in the literature (Olsson et al., 2022; Vig et al., 2020; Meng et al., 2023; Wang et al., 2022; and many others cited in Section 6), is to directly intervene on model components and measure the effect. The paper formalizes this in Section 2.1: for a node nn, the contribution c(n)c(n) is the expected absolute change in a behavioral metric L\mathcal{L} when nn's activation on the clean prompt xcleanx_{\text{clean}} is replaced with its activation on the noise prompt xnoisex_{\text{noise}}, averaged over a distribution D\mathcal{D} of such pairs.

This approach has strong causal foundations — it uses the do-calculus framework from Pearl (2000, 2001) and directly measures the counterfactual impact. But the cost is O(∣N∣)O(|N|) forward passes per prompt pair, making it "prohibitively expensive for SoTA Large Language Models" (Section 1). The paper treats this exact method as the ground truth to approximate, not as a competitor.

Existing Approximation Methods

The paper builds most directly on Attribution Patching (AtP), introduced by Nanda (2022) and related to earlier work on gradient-based pruning by Figurnov et al. (2016) and Molchanov et al. (2017). AtP makes a first-order Taylor expansion of the intervention effect, approximating the change in model output from patching a node with the dot product of the activation difference (n(xnoise)−n(xclean))(n(x_{\text{noise}}) - n(x_{\text{clean}})) and the gradient of the metric with respect to that activation. This reduces the cost from O(∣N∣)O(|N|) forward passes to two forward passes and one backward pass for all nodes on a given prompt pair — an enormous speedup.

However, prior to this work, AtP's reliability was not systematically characterized. The authors note that Nanda (2022) speculated about potential issues (including non-linearity problems with layer normalization and attention softmax), but there was no rigorous study of when and why AtP produces false negatives. Concurrent work by Syed et al. (2023) showed AtP could be useful for automated circuit discovery, but did not analyze its failure modes or propose improvements to address them. The core gap this paper addresses is that AtP's failure modes were unknown and uncharacterized, making practitioners uncertain about whether they could trust its rankings to not miss important components.

The Absence of Alternatives

The paper introduces several alternative fast approximation methods in Section 3.3 — Subsampling (using random subsets of nodes to statistically isolate individual contributions), Blocks (grouping nodes into fixed-size blocks, measuring block-level effects, then traversing high-contribution blocks), and Hierarchical (a recursive version of Blocks) — but notes that these are introduced here for the first time as baselines. Prior to this work, there were essentially no published alternatives to AtP that could accelerate fine-grained activation patching. Iterative methods (directly patching nodes one at a time in some order) were the default, and they scale poorly.

How This Paper Positions Itself

The paper explicitly frames itself not as proposing an entirely new paradigm, but as improving, characterizing, and benchmarking AtP to make it a reliable tool for a specific use case: as a prefiltering step before verification via exact activation patching. As stated in the introduction:

"We propose to accelerate this process by using Attribution Patching (AtP)... as a prefiltering step: after running AtP, we iterate through the nodes in decreasing order of absolute value of the AtP estimate, then use Activation Patching to more reliably evaluate these nodes and filter out false positives — we call this verification."

This is a pragmatic framing. The authors don't claim AtP* eliminates the need for exact activation patching. Rather, they claim it can order the components well enough that exact verification of only the top KK components (for small KK) recovers nearly all the causally important nodes at a fraction of the cost of exhaustive patching. False positives are acceptable because they get caught during verification; false negatives are the real enemy, because they mean causally important components never get verified at all.

The paper's contributions are thus organized around this prefiltering perspective:

  1. Identify failure modes of plain AtP that produce false negatives (Section 3.1).
  2. Propose fixes (QK fix for attention saturation, GradDrop for cancellation) that reduce false negatives while preserving AtP's favorable scaling (Sections 3.1.1, 3.1.2).
  3. Introduce baselines (Subsampling, Blocks, Hierarchical) for the first time to provide a comparative landscape (Section 3.3).
  4. Systematically evaluate all methods against exact ground truth across multiple model sizes and prompt distributions (Section 4).
  5. Provide diagnostics to bound remaining false negatives without exhaustive verification (Section 3.2).

A key nuance in the paper's positioning is that AtP and AtP* are gradient-based methods, which means they make a local linearity assumption: that the metric L\mathcal{L} changes approximately linearly with small perturbations to the activation n(xclean)n(x_{\text{clean}}). When this assumption holds — which the authors find is often the case for fine-grained components like MLP neurons and attention heads, where the intervention is a relatively small perturbation to the full residual stream — AtP performs well. When it breaks down (e.g., due to attention saturation or cancellation between effects), the paper proposes targeted fixes rather than abandoning the gradient approximation entirely. This is a design choice that reflects the paper's commitment to scalability: gradient-based methods cost two forward passes and one backward pass regardless of model size, while alternatives like Subsampling or Blocks scale with the number of components or prompt pairs.

The paper also positions itself relative to two specific distributional choices that affect the interpretation of results:

  • Noising vs. denoising. The paper primarily studies noising (replacing clean activations with noise activations, measuring whether behavior is destroyed), following Chan et al. (2022) and Wang et al. (2022). It briefly touches on denoising (replacing noise activations with clean activations, measuring whether behavior is restored) in Section 5.2 and shows one result suggesting AtP* performs worse under denoising (Figure 14, bottom row), noting this is not yet well-understood.
  • Single prompt pairs vs. distributions. The paper deliberately studies both settings. Single prompt pairs (Section 4.3, clean prompt pairs) are simpler to set up and allow exhaustive ground-truth computation; distributions allow investigation of whether the methods' performance generalizes when effects must be averaged across multiple noise sources. The finding that GradDrop's benefit diminishes on distributions (because cancellation tends to be prompt-pair-specific) is a practical insight for practitioners choosing which variant to use.

The Practical Question This Paper Answers

Ultimately, the paper addresses a very concrete question that mechanistic interpretability researchers face daily: "I want to find which MLP neurons or attention heads are causally important for this behavior on my Pythia/Llama/GPT model. I can't afford to patch them all. Should I use Attribution Patching? And if so, which variant should I use, and how confident can I be that I'm not missing important components?"

By providing the first systematic empirical study of AtP against ground truth, characterizing its failure modes, proposing AtP* as an improved variant, comparing against several alternative fast methods, and providing statistiscal diagnostics, the paper gives researchers evidence-based answers to these questions — with the important caveat (Section 5.4) that the recommendations are "best-substantiated in settings similar to those [they] studied" and that practitioners should "look before you leap" when departing from those settings.

3. Technical Approach

3.1 Reader Orientation

This paper builds a causal attribution system that takes an LLM, a specific behavior (quantified by a metric on its output, e.g., the negative log probability of a target token), and a pair of prompts (one where the behavior occurs, one where it does not), and rapidly ranks every fine-grained model component (e.g., each MLP neuron at each token position, each attention head's query/key/value) by how causally important it is for that behavior. The system replaces the impossibly expensive "patch every single component and measure the effect directly" brute-force approach with a cheap gradient-based prefilter (Attribution Patching, or AtP) that estimates all component rankings from just two forward passes and one backward pass — and then patches only the top-ranked components to verify them, catching any false positives while keeping the total number of patches manageable.

The core problem is a search problem in a vast discrete space (all model components) where the evaluation function (activation patching) is expensive but a cheap surrogate (gradient-based approximation) exists — and the paper's technical contribution is to (1) characterize when and why that surrogate produces false negatives, (2) propose targeted fixes that address those failures without sacrificing the surrogate's favorable O(1) cost scaling, and (3) provide statistical diagnostics that bound the remaining error so practitioners can decide how many components to verify.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components operating in sequence:

  1. Prompt Pair Distribution ($\mathcal{D}$): A set of $(x_{\text{clean}}, x_{\text{noise}})$ pairs. The clean prompt elicits the target behavior; the noise prompt does not. This defines what constitutes the behavior and what counts as an intervention.

  2. Model and Node Partition ($\mathcal{M}$, $N$): The LLM is viewed as a computational graph whose nodes are the atoms of analysis (e.g., attention queries/keys/values, MLP neurons, each at every token position). The choice of which atoms to use — AttentionNodes or NeuronNodes — determines the granularity and total cost.

  3. Attribution Patching Estimator (AtP or AtP*): The core acceleration. Given one $(x_{\text{clean}}, x_{\text{noise}})$ pair, this component runs two forward passes (one on clean, one on noise) and one backward pass (on clean), then computes for every node $n$ a scalar estimate $\hat{c}(n)$ of its true causal contribution $c(n) \in \mathbb{R}$. The estimate is the absolute value of a first-order Taylor approximation to the intervention effect. AtP* includes two modifications (QK fix and GradDrop) that address specific failure modes of plain AtP.

  4. Verification Pipeline: After AtP/AtP* produces a ranked list of nodes (by estimated contribution), the system iterates through this list in descending order, performing exact activation patching on each node (i.e., replacing its clean activation with its noise activation, running a full forward pass, and measuring the metric change). This removes false positives (nodes that AtP ranked highly but whose true effect is small) at a per-node cost of one forward pass.

  5. Diagnostics (optional): After verifying the top $K$ nodes, the system can use Subsampling (Section 3.3, Algorithm 1) to statistically bound the size of any remaining false negatives — nodes whose true effect exceeds some threshold $\theta$ but that were not in the top $K$ AtP* estimates. This provides a confidence level that the important components have been found without requiring exhaustive verification of all nodes.

Information flows linearly: the prompt pair distribution →\to the AtP/AtP* estimator (which produces a ranked list of all nodes) →\to the verification step (which patches top nodes to get exact effects) →\to optionally, the diagnostics step (which bounds the error on unverified nodes). The key interfaces: AtP* takes activations and gradients from the model as inputs and outputs scalar estimates; verification takes a node index and the prompt pair as inputs and outputs the true effect $c(n)$; diagnostics take summary statistics from Subsampling and output confidence thresholds $\theta$ for various confidence levels $1-p$.

3.3 Roadmap for the Deep Dive

Because the method is a pipeline whose correctness depends on each stage's assumptions, I'll explain it in order of information flow rather than conceptual priority:

  1. First, the exact attribution problem and ground-truth definition (Section 2.1, formalized here): What exactly are we trying to approximate? Without a precise definition of $c(n)$ — including the treatment of distributions, absolute value placement, and metric choice — the approximation's error cannot be defined.
  2. Second, vanilla Attribution Patching (Section 2.2): How AtP turns the exact (expensive) computation into a gradient-based one, what the Taylor expansion is approximating, and what implicit assumptions it makes about linearity — because these assumptions are exactly what the two failure modes violate.
  3. Third, the QK fix (Section 3.1.1): The first failure mode (attention saturation) and how recomputing the softmax non-linearity explicitly — rather than approximating it with gradients — fixes the problem while adding minimal compute.
  4. Fourth, GradDrop (Section 3.1.2): The second failure mode (cancellation between direct and indirect effects) and how zeroing gradients at each downstream layer — producing multiple independent estimates and averaging their absolute values — breaks the cancellation.
  5. Fifth, alternative fast methods (Section 3.3): Subsampling, Blocks, and Hierarchical — introduced as baselines against which AtP*'s speed-accuracy tradeoff is measured, and (in the case of Subsampling) providing the statistical machinery for diagnostics.
  6. Sixth, the diagnostic procedure (Section 3.2): How Subsampling's summary statistics feed into Welch's t-test to produce confidence bounds on false negative magnitudes, closing the loop on the prefilter-then-verify pipeline.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical methods paper whose core idea is that the first-order Taylor approximation used in Attribution Patching has two specific, addressable failure modes — attention saturation and direct-indirect cancellation — and that fixing these produces a significantly more reliable prefilter for causal attribution without sacrificing the method's favorable scaling properties.

The Exact Causal Attribution Problem

Before explaining how AtP approximates the contribution $c(n)$, we must understand exactly what it means for a component to be "causally important" — because the approximation's error is measured against this definition, and several design choices (absolute value placement, distribution handling, metric) affect what AtP is trying to estimate.

Model as computational graph. The paper views an LLM $\mathcal{M}$ as a directed acyclic computational graph $(N, E)$ where each node $n \in N$ is a model component (e.g., an attention query vector at a specific head and token position, or an MLP neuron at a specific layer and position), and a directed edge $e = (n_1, n_2) \in E$ exists if the output of $n_1$ is a direct input to the computation of $n_2$. The activation (intermediate computation result) of node $n$ when computing $\mathcal{M}(x)$ on input $x$ is written as $n(x)$.

The choice of $N$ — the granularity of attribution — is a design decision. The paper studies two settings (Section 4.1):

  • AttentionNodes: The query, key, and value vector for each attention head are distinct nodes, as is the pre-linear per-head attention output. Each copy at each token position is a separate node.
  • NeuronNodes: Each individual MLP neuron is a separate node, again per token position.

This matters because the per-node effect sizes, linearity approximations, and total number of nodes differ substantially between settings. AttentionNodes tend to be fewer in number but each has a larger-dimensional activation; NeuronNodes are far more numerous (4–8× the MLP intermediate dimension) but each is a scalar, making the linear approximation potentially more accurate (a scalar perturbation is a smaller change to the full residual stream than a $d_{\text{key}}$-dimensional perturbation).

Intervention effect. For a single prompt pair $(x_{\text{clean}}, x_{\text{noise}})$ sampled from a distribution $\mathcal{D}$, the intervention effect $\mathcal{I}(n; x_{\text{clean}}, x_{\text{noise}})$ of node $n$ is defined as:

I(n;xclean,xnoise):=L(M(xclean∣do(n←n(xnoise))))−L(M(xclean))\mathcal{I}(n; x_{\text{clean}}, x_{\text{noise}}) := \mathcal{L}(\mathcal{M}(x_{\text{clean}} \mid \text{do}(n \leftarrow n(x_{\text{noise}})))) - \mathcal{L}(\mathcal{M}(x_{\text{clean}}))

where $\mathcal{L}: \mathbb{R}^V \to \mathbb{R}$ is a metric quantifying the behavior of interest (e.g., negative log probability of a target token), $\mathcal{M}(x_{\text{clean}})$ is the model's output on the clean prompt without any intervention, and $\mathcal{M}(x_{\text{clean}} \mid \text{do}(n \leftarrow n(x_{\text{noise}})))$ is the model's output when the activation of node $n$ is overwritten with the value it takes on the noise prompt during the clean forward pass.

What it computes: This is the single-prompt-pair, single-node counterfactual: "If I take the clean forward pass but surgically replace only node $n$'s activation with what it would have been on the noise prompt, how much does the behavioral metric change?" A large positive change means the node's clean activation was important for bringing the metric to its clean value; a negative change means the node was actively suppressing the metric relative to the noise condition.

Why this form: This is the standard noising formulation of causal mediation analysis (robins1992; pearl2001). The do-operator notation ($\text{do}(\cdot)$) from Pearl (2000) signals that this is a causal, not merely correlational, intervention: we physically overwrite the node's value rather than conditioning on it. The difference $\mathcal{L}(\text{intervened}) - \mathcal{L}(\text{clean})$ measures the cost of removing the node's clean activation — this is a necessity measure, testing whether the component is required for the behavior. The alternative (denoising) would measure the sufficiency of the node's clean activation to restore the behavior from the noise condition.

Distribution-level contribution. The contribution of a node $n$ is then the expected absolute intervention effect:

c(n):=E(xclean,xnoise)∼D[∣I(n;xclean,xnoise)∣]c(n) := \mathbb{E}_{(x_{\text{clean}}, x_{\text{noise}}) \sim \mathcal{D}} \left[ |\mathcal{I}(n; x_{\text{clean}}, x_{\text{noise}})| \right]

What it computes: The average magnitude of the node's causal effect across all prompt pairs in the distribution. Taking the absolute value inside the expectation means that a node which has a +0.5 effect on half the pairs and a −0.5 effect on the other half gets $c(n) \approx 0.5$, not $c(n) \approx 0$. This is intentional: the authors note in Section 2.2 that "this decreases the chance that estimates across prompt pairs with positive and negative effects might erroneously lead to a significantly smaller estimate."

Why this form: This is a distributional robustness choice. If a node systematically affects the metric in opposite directions on different prompt pairs (a sign-flipping effect), omitting the absolute value would cause cancellation and the node would appear unimportant despite being consistently influential. The paper provides evidence in Appendix B.2 (Figure 10) that such cancellation does occur frequently across the IOI distribution, with cancellation ratios (defined as $1 - |\sum_x \mathcal{I}| / \sum_x |\mathcal{I}|$) often exceeding 0.5 for nodes with large total absolute effects.

The metric $\mathcal{L}$. All reported results in the main experiments use the negative log probability (i.e., cross-entropy loss) of the clean prompt's next token as the metric. For a clean prompt ending with token $t$ (e.g., the completion "Spain" in CITY-PP), $\mathcal{L} = -\log p(t \mid \text{prompt})$. This means an intervention that makes the correct token less probable increases $\mathcal{L}$, producing a positive $\mathcal{I}$. The paper briefly explores other metrics (logit difference, log-odds) in Appendix B.4 (Figure 14) and finds AtP* is "not sensitive to the choice of $\mathcal{L}$" — the node rankings are broadly preserved across metrics.

Vanilla Attribution Patching (AtP)

Attribution Patching approximates the intervention effect $\mathcal{I}$ using a first-order Taylor expansion of $\mathcal{L}$ around the clean activation $n(x_{\text{clean}})$. The key assumption is that the difference $n(x_{\text{noise}}) - n(x_{\text{clean}})$ is small enough that the metric $\mathcal{L}$ can be well-approximated by its linearization at the clean point.

I^AtP(n;xclean,xnoise):=(n(xnoise)−n(xclean))⊤∂L(M(xclean))∂n∣n=n(xclean)\hat{\mathcal{I}}_{\text{AtP}}(n; x_{\text{clean}}, x_{\text{noise}}) := (n(x_{\text{noise}}) - n(x_{\text{clean}}))^{\top} \frac{\partial \mathcal{L}(\mathcal{M}(x_{\text{clean}}))}{\partial n} \Big|_{n = n(x_{\text{clean}})}

where $n(x_{\text{clean}}) \in \mathbb{R}^{d}$ is the clean activation of node $n$ (a vector of dimension $d$ — e.g., $d = d_{\text{key}}$ for a key node, $d = 1$ for an MLP neuron), $n(x_{\text{noise}}) \in \mathbb{R}^{d}$ is the noise activation, and $\frac{\partial \mathcal{L}(\mathcal{M}(x_{\text{clean}}))}{\partial n} \big|_{n = n(x_{\text{clean}})} \in \mathbb{R}^{d}$ is the gradient of the metric $\mathcal{L}$ with respect to the node's activation, evaluated at the clean point.

What it computes: The dot product of the "direction and magnitude of perturbation" (the activation difference) with the "local sensitivity of the metric" (the gradient). If the gradient points in the same direction as the perturbation, large positive dot product — the linear approximation predicts a large positive change in $\mathcal{L}$. If they point in opposite directions, large negative dot product. If they are orthogonal, near-zero — the perturbation doesn't affect the metric to first order. The absolute value of this dot product is what feeds into the distribution-level estimate:

c^AtP(n):=Exclean,xnoise[∣I^AtP(n;xclean,xnoise)∣]\hat{c}_{\text{AtP}}(n) := \mathbb{E}_{x_{\text{clean}}, x_{\text{noise}}} \left[ |\hat{\mathcal{I}}_{\text{AtP}}(n; x_{\text{clean}}, x_{\text{noise}})| \right]

Why this form: This is the standard first-order Taylor expansion of $\mathcal{L}(\mathcal{M}(x_{\text{clean}} \mid \text{do}(n \leftarrow n(x_{\text{noise}}))))$ around $n(x_{\text{clean}})$. If we define $f(n) = \mathcal{L}(\mathcal{M}(x_{\text{clean}} \mid \text{do}(n \leftarrow n)))$ (the metric as a function of the node's value, holding all other activations fixed), then $f(n(x_{\text{noise}})) \approx f(n(x_{\text{clean}})) + \nabla f(n(x_{\text{clean}})) \cdot (n(x_{\text{noise}}) - n(x_{\text{clean}}))$, and $f(n(x_{\text{clean}})) = \mathcal{L}(\mathcal{M}(x_{\text{clean}}))$ by definition. The intervention effect is $f(n(x_{\text{noise}})) - f(n(x_{\text{clean}}))$, so the constant term cancels. This is why the approximation uses only the gradient and the activation difference, not the absolute value of the activation.

The critical computational property is that one backward pass computes the gradient for all nodes simultaneously (by backpropagation through the entire computational graph). Combined with one forward pass on $x_{\text{clean}}$ (to get both the clean activations and to enable the backward pass) and one forward pass on $x_{\text{noise}}$ (to get the noise activations), this produces $\hat{\mathcal{I}}$ estimates for every node $n \in N$ on a given prompt pair at a cost entirely independent of $|N|$. The cost is roughly two forward passes plus one backward pass (3× the cost of a standard inference), compared to $|N|$ forward passes for exhaustive exact patching. For a Pythia-12B model with $\sim 10^5$ attention nodes or $\sim 10^6$ neuron nodes on a moderate-length prompt, this is a 4–6 order of magnitude speedup.

Where the approximation breaks. The paper identifies two specific mechanisms that cause the linear approximation to fail catastrophically for certain nodes — producing estimates $\hat{\mathcal{I}}_{\text{AtP}}$ that are orders of magnitude smaller than the true $\mathcal{I}$, thereby sending those nodes to the bottom of the ranking (false negatives). The next two sections explain these mechanisms and the paper's fixes.

The QK Fix: Addressing Attention Saturation (Section 3.1.1)

The failure mechanism. In attention mechanisms, the query and key vectors feed directly into a softmax non-linearity to produce attention weights. The softmax has the property that when the pre-softmax logits are in a "saturated" regime — meaning one key's dot product with the query is much larger than others — the gradient of the attention probability with respect to that key or query is extremely small (the softmax output is saturated near 1, and the derivative $\sigma(1 - \sigma)$ is near 0). The linear approximation $\hat{\mathcal{I}}_{\text{AtP}}$ multiplies the gradient by the activation difference. If the gradient is near zero because of saturation, the estimate will be near zero even if the activation difference is large and the true non-linear effect (recomputing the softmax with the perturbed key/query) would produce a large change in attention weights and downstream behavior.

Figure 3 in the paper illustrates this geometrically: the linear approximation to the sigmoid (a special case of the softmax when varying a single key relative to a fixed query) is a poor fit when one or both endpoints are in the saturated (flat) region. The consequence is visible in Figure 4 (left): many key and query nodes appear as pronounced false negatives (far below the diagonal in the true-rank-vs-AtP-rank scatter plot), and these false negatives are strongly associated with nodes that produce large changes in attention probability when patched (indicated by color in the plot).

The fix: explicit softmax recomputation. Rather than using the gradient through the softmax, the QK fix explicitly recomputes the attention weights that would result from patching the key or query — keeping the rest of the linear approximation (from attention weights to the final metric) intact. This effectively moves the "linearization boundary" from before the softmax (where the gradient is unreliable due to saturation) to after the softmax (where the relationship between attention weights and downstream effects is more faithfully linear for fine-grained perturbations).

For query nodes, the computation is straightforward (Equation 7): run the model on $x_{\text{clean}}$ and cache the attention keys and weights; run the model on $x_{\text{noise}}$ and cache the noise queries; compute new attention weights $\text{attn}(n)_{\text{patch}}$ by combining the clean keys with the noise queries (for the head that $n$ belongs to); then compute the AtP estimate using the difference in attention weights multiplied by the gradient on the attention weights:

I^AtPfixQ(n;xclean,xnoise):=(attn(n)patch−attn(n)(xclean))⊤∂L(M(xclean))∂attn(n)∣attn(n)=attn(n)(xclean)\hat{\mathcal{I}}_{\text{AtPfix}}^Q(n; x_{\text{clean}}, x_{\text{noise}}) := (\text{attn}(n)_{\text{patch}} - \text{attn}(n)(x_{\text{clean}}))^{\top} \frac{\partial \mathcal{L}(\mathcal{M}(x_{\text{clean}}))}{\partial \text{attn}(n)} \Big|_{\text{attn}(n) = \text{attn}(n)(x_{\text{clean}})}

where $\text{attn}(n) \in \mathbb{R}^T$ is the attention weight vector (over $T$ key positions) for the query node $n$, $\text{attn}(n)_{\text{patch}}$ is the attention weight vector that results from using the noise query with the clean keys, and $\frac{\partial \mathcal{L}}{\partial \text{attn}(n)}$ is the gradient of the loss with respect to those attention weights.

What it computes: Instead of asking "if I change the query vector, how much does the loss change (linear approximation through softmax)?", this asks "if I change the query vector, the attention weights change exactly to these new values (recomputing the softmax non-parametrically); now, how much would the loss change if the attention weights changed by that amount (linear approximation from attention weights onward)?".

Why this form: The softmax is the only non-linearity between the query/key and the attention weights. By computing the exact softmax output for the perturbed query/key, we eliminate the one source of non-linearity that was causing the gradient-based approximation to fail. The remaining approximation (from attention weights to final metric) is a composition of linear projections, residual connections, and non-linearities (GELU, layer norm) that tend to be more locally linear for the small perturbations induced by changing a single head's attention weights — especially because attention weights are already normalized (sum to 1) and the downstream effects are distributed across the residual stream.

For key nodes, the computation is more involved but follows the same principle. The complication is that changing a single key affects the attention weights of all queries in the same head that attend to that key's position (due to causality, this is queries at positions $t_{\text{key}}$ through $T$). For each such query $n_q$, the patched attention weight vector is:

attnpatcht(nq):=attn(nq)(xclean∣do(ntk←ntk(xnoise)))\text{attn}_{\text{patch}}^t(n_q) := \text{attn}(n_q)(x_{\text{clean}} \mid \text{do}(n_t^k \leftarrow n_t^k(x_{\text{noise}})))

where $n_t^k$ is the key node at position $t$. This requires computing how a single key change alters the softmax output for every affected query. Doing this naively costs $O(T^3)$ per head per prompt pair (recomputing attention for all $O(T)$ queries for each of $O(T)$ keys). The paper provides an efficient $O(T^2)$ algorithm (Algorithm 4) that exploits the structure of the softmax: changing one key only affects the logit for that key position in each query, which changes the attention distribution proportionally (the relative weights of all other positions remain fixed). The corrected estimate for key $n_t^k$ is then:

I^AtPfixK(ntk;xclean,xnoise):=∑nq∈queries(ntk)Δtattn(nq)⊤∂L(M(xclean))∂attn(nq)\hat{\mathcal{I}}_{\text{AtPfix}}^K(n_t^k; x_{\text{clean}}, x_{\text{noise}}) := \sum_{n_q \in \text{queries}(n_t^k)} \Delta_t \text{attn}(n_q)^{\top} \frac{\partial \mathcal{L}(\mathcal{M}(x_{\text{clean}}))}{\partial \text{attn}(n_q)}

where $\Delta_t \text{attn}(n_q) = \text{attn}_{\text{patch}}^t(n_q) - \text{attn}(n_q)(x_{\text{clean}})$ and the sum is over all query nodes in the same head that can attend to position $t$.

What it computes: For each key position $t$, sum up (over all queries affected by that key) the AtP-style estimate using the exact attention weight change induced by patching that key, multiplied by the gradient on those attention weights. This correctly captures that one key change can ripple through multiple queries' attention distributions.

Why this form: This is the minimal-perturbation fix: compute exactly the first non-linear function downstream of the key, and use the gradient-based approximation for everything after. It avoids the $O(T^3)$ of fully recomputing all queries for all keys by using the structural property that the softmax with respect to a single key change is a scalar reweighting of the attention distribution (the $\text{onehot}(t) - \text{attn}(n_q)$ term multiplied by a scalar derived from the logit change at position $t$).

Computational cost. The paper states that "the full QK fix requires less than two transformer forward passes" because (1) the query fix costs approximately one attention forward computation (computing attention weights from the cached clean keys and noise queries), and (2) the key fix uses the $O(T^2)$ algorithm which also costs no more than the attention computation itself. Since these are on top of the two forward passes and one backward pass already required for AtP, the total is roughly equivalent to 4–5 forward passes — still independent of $|N|$.

Effect on false negatives. Figure 4 (middle) shows the rank scatter plot after applying QK fix. The cloud of false negatives (points far below the diagonal) among key and query nodes is substantially reduced. The improvement is most dramatic for nodes with large attention probability changes (indicated by coloration), confirming the saturation hypothesis.

GradDrop: Addressing Cancellation Between Direct and Indirect Effects (Section 3.1.2)

The failure mechanism. In a transformer with residual connections, the total effect of patching a node $n$ at layer $\ell$ can be decomposed (following Pearl, 2001) into:

I(n)=Idirect(n)+Iindirect(n)\mathcal{I}(n) = \mathcal{I}_{\text{direct}}(n) + \mathcal{I}_{\text{indirect}}(n)

where $\mathcal{I}_{\text{direct}}(n)$ is the effect that propagates from $n$ directly to the output through all paths that bypass downstream layers (i.e., via the residual stream skip connections) — this is what would happen if all downstream layers were held fixed at their clean values — and $\mathcal{I}_{\text{indirect}}(n)$ is the effect mediated through downstream layers whose computations are altered by the change in $n$'s output (e.g., an attention head in layer $\ell+1$ whose query is computed from the residual stream containing $n$'s contribution).

The cancellation failure occurs when $\mathcal{I}_{\text{direct}}(n) \approx -\mathcal{I}_{\text{indirect}}(n)$ but the two effects have different functional forms (through different numbers of non-linearities), leading the linear approximations $\hat{\mathcal{I}}_{\text{AtP}}^{\text{direct}}$ and $\hat{\mathcal{I}}_{\text{AtP}}^{\text{indirect}}$ to have different approximation errors. If the true total effect $\mathcal{I}(n)$ is large (because direct and indirect effects have large magnitude but opposite sign) but the true cancellation is slightly imperfect while the approximating cancellation is almost perfect, the AtP estimate can be orders of magnitude smaller than the truth — a false negative.

The paper frames this in terms of decomposing the gradient $\frac{\partial \mathcal{L}}{\partial n}$ into a sum of path gradients $\frac{\partial \mathcal{L}^s}{\partial n}$, where each path $s \subseteq \{\text{downstream layers}\}$ corresponds to a set of layers that the signal passes through (versus skipping). The total gradient is $\frac{\partial \mathcal{L}}{\partial n} = \sum_{s \subset \{\ell+1,\dots,L\}} \frac{\partial \mathcal{L}^s}{\partial n}$. The AtP estimate for each path is $\hat{\mathcal{I}}_{\text{AtP}}^s = (n_{\text{noise}} - n_{\text{clean}})^{\top} \frac{\partial \mathcal{L}^s}{\partial n}$, and the total is $\hat{\mathcal{I}}_{\text{AtP}} = \sum_s \hat{\mathcal{I}}_{\text{AtP}}^s$. The path $s = \emptyset$ (the empty set, i.e., skipping all downstream layers) is the direct effect path.

When cancellation occurs, $\sum_{s \neq \emptyset} \hat{\mathcal{I}}_{\text{AtP}}^s \approx -\hat{\mathcal{I}}_{\text{AtP}}^{\emptyset}$, and small approximation errors in the indirect paths (due to non-linearities like GELU and attention softmax in the downstream layers) can push the sum close to zero while the true $\mathcal{I}$ remains large.

The fix: path dropout via gradient zeroing. GradDrop (gradient dropout) disrupts this cancellation by artificially zeroing out the gradient contributions from one downstream layer at a time, creating $L$ modified estimates (one per layer $\ell = 1, \dots, L$):

∂Lℓ∂n:=∂L∂n(M(xclean∣do(nℓout←nℓout(xclean))))\frac{\partial \mathcal{L}^{\ell}}{\partial n} := \frac{\partial \mathcal{L}}{\partial n} \left( \mathcal{M}(x_{\text{clean}} \mid \text{do}(n_{\ell}^{\text{out}} \leftarrow n_{\ell}^{\text{out}}(x_{\text{clean}}))) \right)

where $n_{\ell}^{\text{out}}$ is the contribution of layer $\ell$ to the residual stream across all positions. Setting the gradient at this node to zero (and preventing backpropagation through it) is equivalent to "patching in the clean activation at the output of layer $\ell$" — it removes from the gradient computation all paths that pass through layer $\ell$.

For each dropped layer $\ell$, we compute a modified AtP estimate:

I^AtP+GDℓ(n;xclean,xnoise):=(n(xnoise)−n(xclean))⊤∂Lℓ∂n\hat{\mathcal{I}}_{\text{AtP+GD}}^{\ell}(n; x_{\text{clean}}, x_{\text{noise}}) := (n(x_{\text{noise}}) - n(x_{\text{clean}}))^{\top} \frac{\partial \mathcal{L}^{\ell}}{\partial n}

then aggregate across all layers by averaging absolute values and scaling:

c^AtP+GD(n):=Exclean,xnoise[1L−1∑ℓ=1L∣I^AtP+GDℓ(n;xclean,xnoise)∣]\hat{c}_{\text{AtP+GD}}(n) := \mathbb{E}_{x_{\text{clean}}, x_{\text{noise}}} \left[ \frac{1}{L-1} \sum_{\ell=1}^{L} \left| \hat{\mathcal{I}}_{\text{AtP+GD}}^{\ell}(n; x_{\text{clean}}, x_{\text{noise}}) \right| \right]

What each drop does: When dropping layer $\ell$, any path $s$ that includes $\ell$ is removed from the gradient sum, so $\hat{\mathcal{I}}_{\text{AtP+GD}}^{\ell}(n) = \sum_{s: \ell \notin s} \hat{\mathcal{I}}_{\text{AtP}}^s(n)$. For layers that are not on the critical downstream path of node $n$ (most layers, assuming sparse circuits), $\hat{\mathcal{I}}_{\text{AtP+GD}}^{\ell}(n) \approx \hat{\mathcal{I}}_{\text{AtP}}(n)$ (because all paths that matter remain, and the dropped layer contributed no relevant paths anyway). For layers that are on critical downstream paths, dropping them removes indirect effect contributions, making the remaining estimate closer to the direct effect (plus indirect effects through other layers).

Why the $\frac{1}{L-1}$ scaling? The authors note that the scaling is chosen to avoid changing the direct-effect path's contribution: "the direct-effect path... is otherwise zeroed out when dropping the layer the node is in." Since each node's own layer $\ell(n)$ is always dropped (not in the sum — the sum omits the layer of the node itself), the scaling $\frac{L}{L-1}$ approximately corrects for this. The paper doesn't give the derivation explicitly, but the idea is that on average, a random layer contributes $\frac{1}{L}$ of the total effect, so dropping one layer reduces the estimate by a factor of $\frac{L-1}{L}$ on expectation — multiplying by $\frac{L}{L-1}$ approximately debiases.

Why the absolute-value-then-average aggregation? The cancellation problem is specifically about signed estimates cancelling to near zero. By taking absolute values before averaging across dropped layers, GradDrop ensures that even if the estimates from different dropped layers have opposite signs (because dropping different layers reveals different parts of the effect), their magnitudes are preserved. The average-of-absolute-values is necessarily at least as large as the absolute value of the average, so this is conservative.

Computational cost. GradDrop requires $L$ backward passes (one per layer) from the same intermediate activations on the clean forward pass. Unlike the QK fix (which required extra forward computation), GradDrop only modifies the backward pass — the forward pass remains unchanged. When combined with the QK fix, the corrected attributions $\hat{\mathcal{I}}_{\text{AtPfix}}$ are dot products with the attention weight gradients, so only the modified gradients $\frac{\partial \mathcal{L}^{\ell}}{\partial \text{attn}(n)}$ need to be recomputed for each layer. The total compute for AtP* (QK fix + GradDrop) on a single prompt pair is thus: 2 forward passes + 1 forward-pass-equivalent for QK recomputation + $L$ backward passes.

Connecting to the cancellation hypothesis. The paper provides evidence that GradDrop's false negatives are indeed due to cancellation by computing the direct effect ratio: $c_{\text{direct}}(n) / c(n)$. A high ratio means the direct effect is much larger than the total (net) effect, implying that indirect effects are cancelling most of the direct effect. In Figure 5, the nodes that GradDrop improves most dramatically (highlighted in red) have direct effect ratios of 5.35, 12.2, and 0 (the latter being a node with zero direct effect — i.e., a node whose effect is purely indirect and thus was missed entirely by AtP because it has no direct path to the output). The median direct effect ratio across all nodes that have a direct effect is 0.77, so these false negatives are clear outliers in terms of cancellation.

Distribution-averaged GradDrop. An important practical finding (discussed in Section 4.3 results) is that GradDrop's benefit diminishes when effects are averaged across a distribution of prompt pairs rather than evaluated on a single pair. The authors explain this in Section 4.3: "the cancellation failure mode tends to be sensitive to the particular input prompt pair, and as a result, averaging across a distribution diminishes the benefit of GradDrops." In other words, a node that suffers from cancellation on one prompt pair may not on another (because the sign of the indirect effect depends on the specific inputs to downstream layers), so averaging absolute effects across pairs already provides some robustness to cancellation — GradDrop provides less marginal benefit on top of this. This is a key practical consideration: on distributions, the recommendation (Section 5.4) is to use AtP+QKFix for AttentionNodes and plain AtP for NeuronNodes, omitting GradDrop.

Alternative Fast Methods: Subsampling, Blocks, and Hierarchical (Section 3.3)

The paper introduces three non-gradient-based baseline methods for fast approximate attribution. These serve two purposes: (1) as competitive alternatives against which AtP/AtP* is compared, and (2) as conceptual scaffolding — Subsampling in particular provides the statistical machinery used in the diagnostic procedure.

All three rely on an approximate node additivity assumption: that when intervening on a set of nodes $\eta \subset N$, the measured effect is approximately the sum of individual node effects:

I(η;xclean,xnoise)≈∑n∈ηI(n;xclean,xnoise)\mathcal{I}(\eta; x_{\text{clean}}, x_{\text{noise}}) \approx \sum_{n \in \eta} \mathcal{I}(n; x_{\text{clean}}, x_{\text{noise}})

This assumption is stronger than the linearity assumption in AtP (which only assumes the metric is locally linear in the perturbation to a single node's activation, not that multiple nodes' effects sum linearly). In practice, interaction effects (where patching two nodes together has a different effect than the sum of patching each individually) cause this assumption to be violated, but the paper finds the resulting bias is manageable — Section A.1.1 provides a formal analysis showing that under a pairwise interaction model, the Subsampling estimator's bias scales with $p$ (the inclusion probability) times the sum of interaction effects involving node $n$.

Subsampling

Subsampling creates an approximately unbiased estimator of individual node contributions by randomly intervening on subsets of nodes and using the include/exclude differences.

Procedure (Algorithm 1):

  1. Choose an inclusion probability $p \in (0, 1)$ (the paper sweeps over $\{0.01, 0.03\}$, Section B.5). Choose a total number of samples $m$.
  2. For each sample $k = 1, \dots, m$:
    • Sample a prompt pair $(x_{\text{clean}}, x_{\text{noise}}) \sim \mathcal{D}$.
    • Sample a binary mask over all nodes from $\text{Bernoulli}^{|N|}(p)$. Let $\eta_k^+$ be the set of nodes where the mask is 1; these are "included" (their activations are replaced with noise versions).
    • Run the model on $x_{\text{clean}}$ with $\text{do}(\eta_k^+ \leftarrow \eta_k^+(x_{\text{noise}}))$ and measure the metric, producing a patch effect $\mathcal{I}_k = \mathcal{I}(\eta_k^+; x_{\text{clean}}, x_{\text{noise}})$.
  3. Maintain running sums: for each node $n$, track $\text{runSum}^+_n$ and $\text{count}^+_n$ (the sum and count of $\mathcal{I}_k$ when $n$ was included in the mask), and $\text{runSum}^-_n$ and $\text{count}^-_n$ (when $n$ was not included).

The estimator is then:

I^SS(n):=1∣η+(n)∣∑k:n∈ηk+Ik−1∣η−(n)∣∑k:n∉ηk+Ik\hat{\mathcal{I}}_{\text{SS}}(n) := \frac{1}{|\eta^+(n)|} \sum_{k: n \in \eta_k^+} \mathcal{I}_k - \frac{1}{|\eta^-(n)|} \sum_{k: n \notin \eta_k^+} \mathcal{I}_k

c^SS(n):=∣I^SS(n)∣\hat{c}_{\text{SS}}(n) := |\hat{\mathcal{I}}_{\text{SS}}(n)|

What it computes: The difference between the average effect of a random subset that does include node $n$ and the average effect of a random subset that does not. Under the additivity assumption (no interactions), the first term equals $\mathbb{E}[\sum_{n' \in \eta} \mathcal{I}(n') \mid n \in \eta] = \mathcal{I}(n) + \mathbb{E}[\sum_{n' \neq n} \mathcal{I}(n') \mid n \in \eta]$ and the second term equals $\mathbb{E}[\sum_{n'} \mathcal{I}(n') \mid n \notin \eta]$. Since the inclusion of all other nodes is independent of $n$, the expected sum of other nodes' effects is the same in both conditions, so they cancel, leaving $\mathcal{I}(n)$.

Why this form: This is essentially a Monte Carlo version of Shapley value estimation, adapted to the binary-inclusion setting. The subtraction of the excluded-$n$ average from the included-$n$ average isolates the marginal contribution of $n$ in a way that is unbiased under the additivity assumption. The key advantage over Blocks/Hierarchical is that Subsampling is intrinsically distributional: each sample already averages over both random subsets and random prompt pairs, so the cost per prompt pair decreases as the number of prompt pairs increases (because each sample evaluates a different prompt pair and contributes to the running estimates for all nodes). This is in contrast to Blocks, which would need to be run separately for each prompt pair and then averaged.

Computational cost. Each sample costs one forward pass (to evaluate $\mathcal{I}_k$). The total cost is $m$ forward passes, where $m$ must be large enough that $\text{count}^+_n$ and $\text{count}^-_n$ are sufficient for statistical reliability. In practice, this means $m$ must be a multiple of $1/p$ — if $p = 0.03$ and we want at least ~30 include observations per node, we need $m \geq 30 / 0.03 = 1000$ samples. This is expensive for large $|N|$ and explains why Subsampling generally underperforms AtP on cost-of-verified-recall metrics (Figures 1, 7, 8).

Summary statistics for diagnostics. In addition to the estimates $\hat{c}_{\text{SS}}(n)$, Algorithm 1 tracks $\text{runSquaredSum}^\pm_n$ to compute sample standard deviations $s^\pm_n$. These feed into the diagnostic procedure (Section 3.2, described below).

Blocks

Blocks partitions all nodes into $\lceil |N| / B \rceil$ blocks of (approximately) size $B$. The procedure (Algorithm 2):

  1. Randomly assign each node to a block.
  2. For each block, patch all nodes in that block simultaneously (replacing their clean activations with noise activations) and measure the block-level effect.
  3. Rank blocks by their measured effect magnitude.
  4. Starting with the highest-effect block, verify individual nodes within the block by exact single-node patching, continuing until the verification budget $M$ is exhausted.

Key tradeoff. Small $B$ means many blocks (costly to evaluate all of them) but few nodes to verify per high-effect block. Large $B$ means few blocks (cheap to evaluate) but many nodes to verify per high-effect block (many of which may have zero individual effect — false positives from the block-level sweep). The paper sweeps $B \in \{2, 6, 20, 60, 250\}$ (Section B.5) and selects the best per setting based on the IRWRGM cost metric.

Hierarchical

Hierarchical (Algorithm 3) generalizes Blocks to a recursive tree structure. At the top level, nodes are grouped into $\lceil |N| / B^L \rceil$ top-level blocks (where $L$ is the number of levels and $B$ is the branching factor). Each top-level block is evaluated (all its constituent nodes patched simultaneously). Then, starting with the highest-effect top-level block, its children (sub-blocks at level $L-1$) are evaluated, and high-effect sub-blocks are recursively expanded until individual nodes are reached.

The paper uses $B = 3$ based on a heuristic argument: if exactly one node has non-zero effect, the cost of finding it by traversing a $B$-ary tree is $B \log_B |N| = \frac{B}{\log B} \log |N|$ forward passes, which is minimized at $B = e \approx 2.718$, or $B = 3$ when constrained to integers. The number of levels $L$ is swept from 2 to 12.

Priority queue and delayed evaluation. A subtlety in Algorithm 3 (line 22) is that when a sub-block is enqueued, its priority is set to $\min(\text{blockContribution}, \text{priority of parent})$. This prevents a sub-block from being evaluated before its parent's other siblings, which matters in practice because the implementation batches multiple block evaluations. Without this constraint, a deep high-priority sub-block could be delayed many batches behind other sub-blocks that happened to share a batch with it, increasing the effective cost of finding important nodes.

Scalability limitation. A critical disadvantage of Blocks and Hierarchical relative to AtP and Subsampling is that their cost scales linearly with the size of $\mathcal{D}$'s support. Each block must be evaluated on every prompt pair to produce distribution-level estimates, so if there are 120 prompt pairs (as in the IOI distribution), the number of forward passes for the block evaluation phase is $120 \times (\text{number of blocks})$. This makes them impractical for large distributions and explains why the paper only compares them on single prompt pairs (Figures 1, 7 top row) and not on distributions (Figures 8, 9 bottom rows).

The Diagnostic Procedure: Bounding Remaining False Negatives (Section 3.2)

Even with the QK fix and GradDrop, AtP* provides no guarantee that all important nodes have been found. The diagnostic procedure uses Subsampling's statistical summary to bound the effect magnitude of nodes that weren't in the top $K$ AtP* estimates.

Null hypothesis. For a threshold $\theta > 0$ and a node $n$ not in the top $K$ AtP* estimates, the null hypothesis is $H_0^n: |\mathcal{I}(n)| \geq \theta$ (the node has a true effect at least as large as $\theta$ — we missed it). The alternative is $H_1^n: |\mathcal{I}(n)| < \theta$ (the node's true effect is smaller than $\theta$ — it's fine that we missed it).

Test statistic. From Subsampling, we have:

  • $\bar{i}^+_n$: mean effect of subsets including $n$
  • $\bar{i}^-_n$: mean effect of subsets excluding $n$
  • $s^+_n, s^-_n$: sample standard deviations
  • $\text{count}^+_n, \text{count}^-_n$: sample sizes

The test uses a one-sided Welch's t-test (welch1947), which does not assume equal variances. Under the conservative choice of the simple null hypothesis that $\mathcal{I}(n) = \theta \cdot \text{sign}(\bar{i}^+_n - \bar{i}^-_n)$ (the boundary of the compound null), the test statistic is:

tn=θ−∣iˉn+−iˉn−∣(sn+)2countn++(sn−)2countn−t_n = \frac{\theta - |\bar{i}^+_n - \bar{i}^-_n|}{\sqrt{\frac{(s^+_n)^2}{\text{count}^+_n} + \frac{(s^-_n)^2}{\text{count}^-_n}}}

with effective degrees of freedom $\nu_n$ approximated by the Welch–Satterthwaite equation:

νn=((sn+)2countn++(sn−)2countn−)2(sn+)4(countn+)2(countn+−1)+(sn−)4(countn−)2(countn−−1)\nu_n = \frac{\left(\frac{(s^+_n)^2}{\text{count}^+_n} + \frac{(s^-_n)^2}{\text{count}^-_n}\right)^2}{\frac{(s^+_n)^4}{(\text{count}^+_n)^2 (\text{count}^+_n - 1)} + \frac{(s^-_n)^4}{(\text{count}^-_n)^2 (\text{count}^-_n - 1)}}

The p-value for node $n$ is $p_n = \mathbb{P}_{T \sim t_{\nu_n}}(T > t_n)$, the probability of observing a t-statistic at least this large if the null were true.

Aggregate p-value. The hypothesis that any node outside the top $K$ has true effect $\geq \theta$ is a compound null: $H_0 = \bigvee_{n \notin \text{Top}_K} H_0^n$. The p-value for this compound null is $\max_{n \notin \text{Top}_K} p_n$ (the maximum individual p-value, because the null is true if any of the individual nulls is true).

Confidence bound. To obtain an upper confidence bound with confidence level $1 - p_{\text{target}}$, the procedure inverts this: find the lowest $\theta$ for which $\max_{n \notin \text{Top}_K} p_n \leq p_{\text{target}}$. This $\theta_{\min}(p_{\text{target}})$ is the smallest effect size for which we can reject (at level $p_{\text{target}}$) the hypothesis that any unverified node exceeds it. In other words: we are $(1 - p_{\text{target}}) \times 100\%$ confident that all nodes outside the top $K$ have true effect $< \theta_{\min}$.

What this buys. For a given compute budget spent on diagnostic samples, the diagnostics provide progressively tighter bounds (Figure 6). The left subplots show the true effect of the highest-ranked unverified nodes (red) and the bound $\theta_{\min}$ at three confidence levels (90%, 99%, 99.9%) for varying sample sizes. As the sample budget increases, the bound drops — eventually below the true effect of the largest false negative, at which point we can be confident we haven't missed anything important. The paper shows this works well in practice: for IOI-PP (Figure 6a), the bound finds the "true biggest false negative reasonably early," while for the full IOI distribution (Figure 6b), where there is no large false negative, "we progressively keep gaining confidence with more data."

Why Welch's t-test? The paper uses Welch's t-test rather than Student's t-test because the inclusion and exclusion subsets have different sample sizes ($\text{count}^+_n \approx p \cdot m$, $\text{count}^-_n \approx (1-p) \cdot m$) and potentially different variances (effects conditional on $n$ being included may have different variability than effects conditional on $n$ being excluded, due to interactions). Welch's test does not assume equal variances and adjusts the degrees of freedom accordingly.

Summary of Design Choices and Their Justifications

  • Two forward passes + one backward pass for AtP rather than a cheaper alternative (e.g., straight-through estimator, input gradients): This is the standard backpropagation-derived first-order Taylor expansion, which is both principled (exact when the function is linear) and implementable with existing autodiff infrastructure without modification.

  • Absolute value inside the distribution average rather than outside: addresses sign-cancellation across prompt pairs, which Appendix B.2 shows is a real phenomenon on the IOI distribution.

  • QK fix recomputes only the softmax rather than full attention or further layers: the softmax is identified as the specific non-linearity causing the gradient approximation to fail, and recomputing only one function keeps the computational overhead low (less than one additional forward pass).

  • GradDrop drops one layer at a time and averages absolute values rather than dropping multiple layers or using signed averaging: dropping one layer at a time minimizes the perturbation to the estimate (most layers are unrelated to a given node, so dropping them doesn't change the estimate), and averaging absolute values prevents sign-cancellation across dropped layers from recreating the original problem.

  • Subsampling used for both estimation and diagnostics rather than a separate diagnostic-only procedure: reusing the same samples amortizes the cost and ensures the diagnostic $\bar{i}^\pm_n$ quantities are directly relevant to the estimation $\hat{c}_{\text{SS}}(n)$.

  • Welch's t-test with a max-p-value aggregation rather than a more sophisticated multiple-testing correction: the max-p-value is the valid p-value for the compound null "any node exceeds $\theta$" (since it's a union of events), and the paper prioritizes conservatism (not falsely declaring confidence) over statistical power.

  • IRWRGM cost metric rather than cost-at-fixed-K: This avoids the metric being dominated by small or large $K$ and provides a single scalar for hyperparameter selection. The inverse-rank weighting ($1/K$) ensures that finding the top-10 nodes and finding the top-1000 nodes contribute roughly equally to the metric (since $\sum_{K=1}^{1000} 1/K \approx \log(1000)$), and the geometric mean ($\exp(\mathbb{E}[\log(\text{cost})])$) ensures that cost ratios rather than absolute differences drive the comparison.

  • The $L/(L-1)$ scaling in GradDrop corrects for the omission of the node's own layer from the set of dropped layers, approximately debiasing the estimate for the common case where a node's effect is distributed roughly equally across downstream layers. The derivation is heuristic but the experimental results validate it (GradDrop does not systematically overestimate or underestimate).

4. Key Insights and Innovations

Innovation 1: Characterizing and Classifying AtP's Failure Modes as Systematically Addressable, Not Fundamental

Before this paper, the dominant assumption about Attribution Patching's reliability was essentially binary and anecdotal: it worked well enough in some cases (Nanda, 2022; Syed et al., 2023 used it successfully for circuit discovery) and was suspected to fail in others (Nanda hypothesized layer-norm non-linearity could cause problems), but there was no taxonomy of how it fails, why it fails, or whether those failures could be fixed without sacrificing the method's speed. Practitioners faced a trust problem: AtP was fast, but without knowing its failure modes, they couldn't assess whether a given analysis was missing critical components or how to improve reliability beyond just "generate more samples."

This paper makes a fundamentally diagnostic contribution: it decomposes AtP's approximation error into two mechanistically distinct failure classes — attention saturation (gradients vanish through softmax non-linearity when attention is peaked, producing false negatives for keys and queries) and direct-indirect cancellation (gradients from different downstream paths cancel in the scalar dot product, producing near-zero estimates for nodes whose true effect is large). This is not mere taxonomy. By identifying specific computational subgraphs where the linear approximation breaks (attention softmax for saturation, residual-stream path summation for cancellation), the paper transforms AtP from a "hope it works" heuristic into a diagnosable and improvable estimator — you can check whether your prompt pair produces saturated attention (Figure 4's color coding does this visually), and you can assess whether cancellation is plausible by computing direct-effect ratios (as in Figure 5's highlighted nodes with ratios of 5.35 and 12.2).

The intellectual move here parallels what happened in numerical analysis when people stopped asking "is this algorithm accurate?" and started asking "under what condition numbers does this algorithm lose precision, and can we precondition to fix it?" The paper doesn't just observe that AtP has false negatives; it identifies the conditioning problems (saturated softmax, cancelling path gradients) that cause precision loss and proposes targeted preconditioners (explicit softmax recomputation, path dropout) that address them without changing the asymptotic scaling. This reframes the problem from "approximation vs. exact" to "where does the linear approximation hold, and can we rewrite the function to make it hold in more places?" — a more productive framing that invites further preconditioning ideas (e.g., the MLP saturation analog discussed in Appendix C.2.4) rather than abandonment of gradient-based methods.

The significance extends beyond AtP itself. The failure modes the paper identifies — saturation of non-linearities causing gradient collapse, and cancellation across summation nodes in computational graphs — are generic to any gradient-based attribution method applied to deep networks (Integrated Gradients, DeepLIFT, Gradient × Input all face the saturation problem; any method that summarizes a vector gradient into a scalar importance score faces cancellation when components have opposite signs). By naming and fixing these for AtP specifically, the paper provides a template for diagnosing similar problems in related methods. The finding that GradDrop's benefit diminishes on distributions (Section 4.3) is itself a conceptual contribution: it reveals that cancellation is often prompt-pair-specific, so distributional averaging already provides some robustness — a non-obvious interaction between the estimator's structure and the evaluation setting that practitioners need to understand when choosing which variant to deploy.

Innovation 2: The Prefilter-Then-Verify Pipeline as a Principled Regression Test for Attribution Methods

Attribution Patching is typically presented as a replacement for activation patching — an approximation you use when you can't afford the real thing. This paper makes a subtle but important shift: it frames AtP not as a replacement but as a ranking prefilter whose output feeds a verification step. The framing matters because it changes the evaluation criterion. Instead of asking "how accurate are the absolute values of AtP's estimates?" (which the paper explicitly does not evaluate — Section 5.1 notes they "do not present evidence about how closely the estimated effect magnitudes... match the ground truth"), it asks "does AtP rank the most causally important nodes near the top, so that verifying the top K finds most of the large-effect nodes?"

This is a regression-test framing for causal attribution: AtP is the inexpensive screening tool, verification is the confirmatory gold standard, and the relevant metric is recall of top true contributors within a fixed verification budget. This is intellectually significant because it decouples two problems that are often conflated: (1) accurately estimating effect magnitudes, which is hard and requires handling all non-linearities faithfully, and (2) identifying which components are worth examining, which only requires a good relative ordering. The paper shows that (2) is substantially easier than (1) — AtP's rankings are much better than its magnitude estimates — and designs the method (most notably GradDrop's absolute-value-then-average aggregation) to optimize for ranking rather than magnitude calibration.

This framing also provides a principled way to compare fundamentally different approximation methods on a single axis: cost of verified recall. Gradient-based AtP, statistical Subsampling, and hierarchical tree-search Blocks/Hierarchical operate on entirely different principles, but they all produce node rankings, and they can all be evaluated by how many forward passes it takes to find the top K true contributors. The paper's cost-of-verified-recall plots (Figures 1, 7, 8) are conceptually analogous to precision-recall curves in information retrieval — they show the tradeoff between computational cost and completeness of discovery — and the IRWRGM cost metric (Section 4.2) provides a principled scalar summary that weights all rank regimes equally. This evaluation framework is a methodological contribution independent of which method performs best: it gives future work a standard benchmark for comparing new fast attribution methods.

The diagnostic procedure (Section 3.2) completes this framing by addressing the "when can I stop verifying?" question. Using Subsampling's statistical summaries to bound the effect size of unverified nodes transforms AtP* from a heuristic prefilter into a statistically principled screening procedure: you verify the top K nodes, then use the diagnostics to state (with specified confidence) that all remaining nodes have effect below some threshold θ. This is the first method in the mechanistic interpretability literature to provide rigorous false-negative guarantees for approximate causal attribution — not guarantees that the approximation is always accurate, but guarantees that if there's a large missed node, the diagnostics will detect it with high probability. The paper shows this works in practice (Figure 6: the bound finds the biggest false negative on IOI-PP, and progressively tightens on IOI where there are no large false negatives), providing a practical "look before you leap" tool for practitioners.

Innovation 3: Gradient Path Dropout as a General Approach to Breaking Cancellation in Summation-Based Estimates

GradDrop is introduced as a fix for a specific AtP failure mode, but the underlying idea — disrupting cancellation between terms in a sum by independently removing individual terms and aggregating absolute values — is a conceptual contribution that generalizes beyond this paper. The problem GradDrop addresses is not specific to transformers or language models: whenever you approximate a complex function f(x) = g(x) + h(x) (with g and h having different non-linearity structures) using a single linearization, the approximation error can cause the estimated f(x) to be near zero even when the true f(x) is large, if g(x) ≈ -h(x). This is the cancellation problem.

Prior approaches to similar problems (e.g., in Shapley value estimation or feature importance with correlated features) typically either (a) accept the cancellation as ground truth (treating the net effect as the quantity of interest, even if it hides large offsetting contributions), or (b) report decomposed effects separately (direct and indirect effects, as in Pearl 2001's mediation framework), which requires defining and computing the decomposition explicitly. GradDrop takes a third approach: it approximates the decomposition without needing to compute it, by intervening on the computation graph (zeroing gradients at each downstream layer) to create multiple estimates that each omit a different subset of indirect paths, then averaging their absolute values to recover the magnitude that cancellation would hide.

The key conceptual move is that the layers being "dropped" don't need to correspond to anatomically meaningful pathways — they just need to create a set of estimates whose absolute values don't cancel. The paper's analysis in Appendix A.2.2 formalizes this: each dropped layer ℓ produces an estimate that includes all paths except those through ℓ, and the aggregation (1/(L-1)) * sum_ℓ |estimate_ℓ| is bounded below by (L - |K| - 1)/(L - 1) * |total_estimate| (where K is the set of layers that node n's effect actually depends on) and above by (L - |K| - 1)/(L - 1) * |total_estimate| + (|K|/(L-1)) * sum_s |path_estimate_s|. For nodes where a few paths dominate and nearly cancel, the second term is much larger than the first, and GradDrop recovers the true magnitude. For typical nodes without strong cancellation, the estimate approximately equals the plain AtP estimate.

This is a fundamental insight about how to make gradient-based attribution robust to cancellation in networks with residual connections — a structural property shared by ResNets, Transformers, and many modern architectures. The approach of "path dropout during backpropagation, absolute value aggregation" could be applied to any gradient-based attribution method (Input × Gradient, Integrated Gradients, DeepLIFT) applied to any residual network, not just AtP on transformers. The paper doesn't explore this generalization, but the Appendix C.2 analysis of edge-AtP* costs with and without GradDrop shows the authors are thinking in these terms: GradDrop's overhead (L backward passes) can be amortized and shared across nodes, making it practical for larger attribution tasks.

The finding that GradDrop's benefit diminishes on distributions (Section 4.3, Figures 8, 9) is itself an important refinement of this insight: it shows that cancellation is often sample-specific — a node whose direct and indirect effects cancel on prompt pair A may not cancel on prompt pair B — so distributional averaging already serves a similar "path dropout" function by averaging over different cancellation patterns. This explains why AtP+QKFix (without GradDrop) is the recommendation for distributional AttentionNodes settings (Section 5.4): the distribution provides its own robustness to cancellation, and GradDrop's extra cost (L backward passes per prompt pair) doesn't buy enough additional benefit to justify the overhead. This is a practically valuable finding that results from the interaction between the path-dropout idea and the distributional evaluation setting — a non-obvious design implication that could only be discovered empirically.

Innovation 4: First Systematic Empirical Mapping of Speed-Accuracy Tradeoffs for Causal Attribution at Scale

Prior to this work, the mechanistic interpretability literature had no systematic evidence about how much faster approximate attribution methods were than brute force, how their accuracy degraded with model scale, prompt type, or node granularity, or which method to prefer under which circumstances. Individual papers used individual methods (usually AtP or direct patching) and reported qualitative judgments about whether results were sensible, but there was no head-to-head comparison against ground truth across multiple models, node types, and prompt distributions. This made it impossible for practitioners to make evidence-based method choices.

This paper provides the first comprehensive empirical calibration of the speed-accuracy Pareto frontier for causal attribution in LLMs. The scale of the undertaking is significant: four model sizes (Pythia-410M, 1B, 2.8B, 12B), two node granularities (AttentionNodes and NeuronNodes), four prompt-pair settings (CITY-PP, IOI-PP, RAND-PP, and two distributions IOI and A-AN), and 5-7 methods (Iterative, AtP, AtP+QKfix, AtP*, Subsampling, Blocks, Hierarchical) — all evaluated against exactly computed ground-truth contributions obtained by exhaustive activation patching. This represents a massive ground-truth computation effort that makes the comparison possible and credible.

The empirical patterns that emerge constitute new knowledge about the structure of causal attribution problems:

  1. AtP is genuinely much better than alternatives for fine-grained attribution. Across essentially all settings, AtP-based methods dominate the cost-of-verified-recall curves (Figures 1, 7, 8) and relative-cost metrics (Figures 2, 9). This is not a foregone conclusion — Subsampling could have been competitive on distributions, Blocks could have won on single prompt pairs with non-linear effects — but the evidence strongly favors gradient-based ranking for the fine-grained setting.

  2. The degree of AtP's advantage varies meaningfully with model scale and prompt type. On CITY-PP with NeuronNodes (Figures 2, 12), AtP methods achieve ~1.1-1.5× oracle-relative cost versus 2-5× for alternatives — a substantial but not overwhelming gap. On RAND-PP with AttentionNodes (Figures 8, 9), the gap narrows further, with AtP* still leading but the alternatives closer than on cleaner prompt pairs. This variation tells practitioners that AtP's advantage is robust but not universal, and that the "cleanliness" of the circuit (how localized the true effects are) matters for gradient-based methods.

  3. GradDrop's benefit is setting-dependent in a predictable way. On single prompt pairs (Figures 1, 7), AtP* (which includes GradDrop) outperforms AtP+QKfix. On distributions (Figures 8, 9), the two are essentially tied — GradDrop adds cost without improving recall. This pattern is consistent with the paper's explanation (distributional averaging provides its own cancellation robustness) and gives practitioners a clear decision rule: use GradDrop on single prompt pairs where cancellation is a risk; omit it on distributions where the cost isn't justified.

  4. Alternative methods have specific niches where they outperform. On the A-AN distribution with NeuronNodes (Figure 8c), Subsampling performs surprisingly well, approaching AtP's performance at higher verification budgets. On RAND-PP AttentionNodes with smaller models (Figure 9a-b), Blocks is competitive. These findings prevent the paper's overall recommendation ("use AtP*") from being dogmatic — it recognizes that method choice depends on the specific setting, and it provides the evidence base for those choices.

This empirical contribution matters because mechanistic interpretability is an engineering practice as much as a science. Practitioners need to know whether spending a day implementing AtP* will save them a month of compute, and whether the savings hold for their specific model and prompt type. By providing concrete cost numbers (e.g., "AtP* finds the top 100 attention nodes in Pythia-12B on IOI-PP using ~10³ forward passes versus ~10⁵ for brute force" — readable from Figure 1b), the paper translates a methodological innovation into an actionable cost model. This is the kind of calibration that turns a technique from "interesting idea" into "standard tool."

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper uses four distinct prompt-pair settings, none of which is a pre-existing benchmark dataset with predefined train/test splits. All experiments use author-constructed prompt pairs designed to elicit specific model behaviors. The single prompt pairs (CITY-PP, IOI-PP, RAND-PP) consist of one clean and one noise prompt each; the distributions (IOI, A-AN) consist of 120 and 100 prompt pairs respectively, constructed by systematically varying names or objects in the prompt templates. Ground truth is computed by exhaustive activation patching on every node in every setting, which itself constitutes the "test set" — there is no held-out evaluation split, because the goal is to measure how well each method recovers the known ground-truth ranking of node contributions within the same setting.

  • Base models. All experiments use decoder-only transformer language models from the Pythia suite (Biderman et al., 2023), specifically Pythia-410M, Pythia-1B, Pythia-2.8B, and Pythia-12B. The Pythia family was chosen because it spans nearly two orders of magnitude in parameter count (from 410M to 12B) while sharing architecture and training data, enabling the authors to study how method performance scales with model size without confounding architectural or data-distribution changes. The models are standard GPT-style autoregressive transformers, making the results likely transferable to similar architectures (GPT-Neo, Llama), though the paper acknowledges in Section 5.1 that transfer to "SotA–scale models or models that significantly deviate from the standard decoder-only transformer architecture" is unverified. All models are used pretrained from the Pythia release without additional fine-tuning.

  • Metrics. The primary behavioral metric ℒ is the negative log probability (cross-entropy loss) of the clean prompt's next-token target. For example, in CITY-PP where the clean prompt ends with "Country: Spain," ℒ measures −log p("Spain" | prompt). The intervention effect ℐ(n) is then the increase in this loss when node n is patched — a positive value means the node's clean activation was important for predicting the correct token. The ground-truth contribution c(n) for a node is the average of |ℐ(n; x_clean, x_noise)| across all prompt pairs in the distribution (or simply |ℐ(n)| for single prompt pair settings). The evaluation metric for comparing methods is cost of verified recall: given a method's ranked list of nodes, verification iterates through nodes in descending order, measuring true c(n) for each, and the metric is the total number of forward passes (estimation cost + verification cost) required to find the K nodes with the largest true effect, for varying K. Scalar aggregation uses IRWRGM cost (inverse-rank-weighted relative geometric mean), which is the ratio of a method's area under the cost-of-verified-recall curve (weighted by 1/K and averaged geometrically) to the oracle's area. This metric equally weights all orders of magnitude of K (e.g., finding top-10 and top-1000 matter equally) and measures relative performance on a log scale.

  • Baselines. The paper evaluates six methods in total. Iterative: direct activation patching of nodes in a random uninformed order (for single prompt pairs) or alternating between sampling new unmeasured prompt pairs for each node and verifying the highest-average-effect nodes (for distributions — a more sophisticated variant that interleaves estimation and verification). AtP (Attribution Patching): vanilla first-order Taylor approximation as formulated by Nanda (2022), requiring two forward passes and one backward pass per prompt pair. AtP+QKfix: AtP with the query/key attention recomputation fix (Section 3.1.1). AtP*: AtP with both QKfix and GradDrop (Section 3.1.1 and 3.1.2). Subsampling: the subset-sampling method described in Algorithm 1 with inclusion probability p (swept from {0.01, 0.03}) and fixed total sample count m. Blocks: the fixed-block-size method (Algorithm 2) with block sizes swept from {2, 6, 20, 60, 250}. Hierarchical: the recursive tree-search method (Algorithm 3) with branching factor B = 3 and number of levels swept from 2 to 12. All baselines except AtP are novel to this paper (Section 3.3). The Iterative method for distributions is also described here for the first time.

  • Compute accounting. All costs are measured in forward passes. One generation of a full set of output logits from a single prompt counts as one forward pass. A backward pass (required for gradient computation in AtP-based methods) is counted as costing approximately one forward pass — the paper does not specify an exact FLOPs-equivalent but treats forward and backward passes as roughly equal in cost, which is standard for transformer models (backpropagation is typically ~2× the FLOPs of a forward pass, but the paper's simplified accounting treats them as 1:1). The QK fix's extra computation (explicit softmax recomputation, the O(T²) key-correction algorithm) is costed at "less than two transformer forward passes" (Section 3.1.1). GradDrop's L extra backward passes are counted as L additional forward-pass-equivalents. For Subsampling, each random subset evaluation costs one forward pass. For Blocks and Hierarchical, each block evaluation costs one forward pass. Verification cost is one forward pass per node per prompt pair — for single prompt pairs, verifying K nodes costs K forward passes; for distributions, verifying K nodes costs K × |𝒟| forward passes (one per node per prompt pair). The total cost for a method to find the top K true nodes is the sum of its estimation cost (all forward passes used to produce the initial node rankings, before any verification) and its verification cost (the forward passes spent verifying nodes in ranked order until the top K true nodes have been verified). For the Iterative baseline, estimation and verification are interleaved, so the distinction is less clear, but the total forward passes are counted the same way. The oracle verification cost (the diagonal line in cost-of-verified-recall plots) assumes nodes are verified in exact descending order of true effect — it represents the best possible performance any ranking method could achieve, requiring estimation cost of zero and verification cost of exactly K forward passes.

  • Cross-validation / statistical protocol. There is no cross-validation in the traditional machine-learning sense, because there is no model training or hyperparameter tuning on train/test splits. Hyperparameter selection for the non-AtP methods (Subsampling's p and m, Blocks' block size, Hierarchical's number of levels) is done by selecting the configuration that produces the lowest IRWRGM cost for each setting, using 5 random seeds per configuration (Section B.5). Since the IRWRGM cost is computed against the ground truth (which is known for these settings), this is effectively "oracle hyperparameter tuning" — the best hyperparameters are selected based on the same data used for evaluation. This is a reasonable choice because the goal is to compare methods at their best, not to evaluate generalization of hyperparameter choices. AtP-based methods have no hyperparameters (the QK fix and GradDrop are deterministic given the model and prompt pairs). The statistical protocol for the diagnostic procedure (Section 3.2) uses Welch's t-test with the Welch–Satterthwaite degrees-of-freedom approximation and a max-p-value aggregation across all unverified nodes, with confidence levels of 90%, 99%, and 99.9% reported (Figure 6).

Main Quantitative Results

AtP Significantly Outperforms All Alternative Fast Methods

The headline finding across all settings is that AtP-based methods dominate the cost-of-verified-recall curves, finding the most causally important nodes using substantially fewer forward passes than Subsampling, Blocks, Hierarchical, or Iterative approaches.

Single prompt pairs. On CITY-PP with NeuronNodes and Pythia-12B (Figure 1a), AtP* finds the top 10 nodes using approximately 30 forward passes (versus ~300 for Subsampling, ~200 for Blocks, and ~150 for Hierarchical). By the time 100 nodes are found, AtP* requires roughly 200 forward passes, while Subsampling needs ~1,200 and Iterative (random order) needs ~3,000. The gap is even larger on IOI-PP with AttentionNodes and Pythia-12B (Figure 1b): AtP* finds the top 10 nodes in roughly 20 forward passes versus ~400 for Subsampling and ~500 for Hierarchical; finding the top 100 nodes costs AtP* roughly 150 forward passes versus ~3,000 for Subsampling.

Translated to the IRWRGM relative cost metric (Figure 2): on CITY-PP NeuronNodes, AtP* achieves approximately 1.3× the oracle cost, AtP+QKfix achieves roughly 1.5×, vanilla AtP achieves roughly 1.8×, while Subsampling and Blocks are in the 2–5× range, Hierarchical is around 3×, and Iterative is approximately 7× the oracle cost. On IOI-PP AttentionNodes, AtP* achieves roughly 1.2× oracle, AtP+QKfix roughly 1.3×, vanilla AtP roughly 1.6×, versus 2–4× for the alternatives.

Single prompt pair, relaxed recall criterion. When the requirement is relaxed to finding 90% rather than 100% of the top true nodes (Figure 7), the curves shift but the relative ordering is preserved. On CITY-PP NeuronNodes (Figure 7a), AtP* finds 90% of the top 1,000 nodes in roughly 1,200 forward passes, versus ~3,000 for the next-best alternative. On IOI-PP AttentionNodes (Figure 7b), AtP* reaches 90% recall of the top 128 nodes in approximately 130 forward passes — note that at this point the total number of attention nodes is roughly 400-500 for this prompt length, so 90% recall means finding about 115 of the 128 most important nodes, demonstrating that AtP*'s false negatives are "a small minority of nodes" (Section 4.3).

Random prompt pair (RAND-PP). The advantage of AtP* narrows but persists. On RAND-PP with NeuronNodes and Pythia-12B (Figure 8a), AtP* still leads, with roughly 1.5× oracle cost versus 2–3× for alternatives. On RAND-PP with AttentionNodes (Figure 8b), the gap is smaller — AtP* achieves around 2× oracle cost, while Blocks and Hierarchical are in the 2.5–4× range. The relative-cost aggregation across models (Figure 9a-b) shows that AtP* is consistently the best method on RAND-PP, but its advantage is less dramatic than on CITY-PP or IOI-PP, suggesting that "the strong performance of AtP/AtP* isn't reliant on the clean prompt using a particularly crisp circuit" but that "cleaner" circuits do amplify its advantage.

Distributions. On the IOI distribution (120 prompt pairs) with AttentionNodes and Pythia-12B (Figure 8d), only AtP-based methods and Subsampling are compared (Blocks and Hierarchical scale poorly to distributions). AtP+QKfix achieves roughly 1.8× oracle cost per prompt pair, compared to roughly 2.5× for Subsampling and roughly 3.5× for vanilla AtP. AtP* (with GradDrop) performs essentially identically to AtP+QKfix on this distribution, consistent with the paper's observation that "averaging across a distribution diminishes the benefit of GradDrops." On the A-AN distribution (100 prompt pairs) with NeuronNodes (Figure 8c), the margin is narrower: AtP achieves roughly 1.5× oracle cost, AtP* is similar, Subsampling reaches roughly 1.8× at higher verification budgets, and Iterative is around 3×.

The relative-cost aggregation across models and distributions (Figure 9c-d) confirms these patterns: on IOI AttentionNodes, AtP+QKfix achieves roughly 1.4× oracle cost averaged across models, versus roughly 2.5× for Subsampling and 2.8× for vanilla AtP. On A-AN NeuronNodes, AtP achieves roughly 1.3× oracle, AtP* roughly 1.4×, Subsampling roughly 1.7×, and Iterative roughly 3.5×.

The QK Fix Provides the Majority of AtP*'s Improvement; GradDrop Helps on Single Prompt Pairs

Attention saturation is the dominant failure mode for attention nodes. The rank scatter plots in Figure 4 provide direct evidence. In Figure 4 (left) showing vanilla AtP ranks versus true ranks on CITY-PP, there is a large cluster of false negatives (nodes far below the diagonal) concentrated among keys and queries, and their coloration (indicating maximum difference in attention probability when patching) shows that most of these false negatives are nodes whose patching produces large attention weight changes — precisely the saturation failure mode. After applying the QK fix (Figure 4, middle), this cluster largely disappears: the false negative rate among keys and queries drops substantially, and the overall scatter is tighter around the diagonal. Adding GradDrop on top (Figure 4, right — AtP*) provides further improvement for a smaller set of nodes, visible as the remaining attention-node outliers moving closer to the diagonal.

GradDrop specifically fixes cancellation false negatives. Figure 5 shows this for NeuronNodes on CITY-PP. The rank scatter compares plain AtP (orange crosses) and AtP+GradDrop (red circles). Three specific neurons are highlighted as major false negatives under plain AtP that are substantially corrected by GradDrop — their direct effect ratios are 5.35, 12.2, and 0 (no direct effect), compared to a median of 0.77 for nodes that have any direct effect. The ratio of 12.2 means the direct effect is more than 12 times larger than the net total effect, indicating strong cancellation by indirect effects. The ratio of 0 means the node's effect is purely indirect (it has no direct path to the output) — plain AtP estimates it near zero, but the true effect is substantial, and GradDrop recovers it by including estimates from dropped layers that leave the indirect effect intact while zeroing out the direct effect.

Cost-of-verified-recall shows GradDrop's benefit is setting-dependent. On single prompt pairs (Figures 1, 2, 7), AtP* (QKfix + GradDrop) consistently outperforms AtP+QKfix by a modest but visible margin — typically reducing the IRWRGM cost by 0.1–0.3 oracle-relative units. On distributions (Figures 8c-d, 9c-d), the two are essentially identical: AtP+QKfix and AtP* curves overlap almost completely. This is explained in Section 4.3: "the cancellation failure mode tends to be sensitive to the particular input prompt pair, and as a result, averaging across a distribution diminishes the benefit of GradDrops." On single pairs, where a node happens to suffer cancellation on that specific pair, GradDrop rescues it; on distributions, the cancellation is less likely to align across many pairs, so absolute-value averaging already provides robustness, and GradDrop's L extra backward passes don't buy additional improvement commensurate with their cost.

GradDrop's upfront cost is visible in aggregate metrics. Figures 2 and 9 show IRWRGM costs. Because GradDrop costs L extra backward passes before any verification begins, its estimation cost is higher than AtP+QKfix. When GradDrop doesn't find enough additional true positives to offset this upfront cost (as on distributions or on settings where false negatives are already rare), its relative cost metric is worse than AtP+QKfix's, even though its per-node estimates may be more accurate. The paper notes this explicitly in Figure 2's caption: "GradDrop (difference between AtP+QKfix and AtP*) comes with a noticeable upfront cost and so looks worse in this comparison while still helping avoid false negatives as shown in Figure 1." This is a genuine tradeoff, not a failure — for applications where missing even one large node is unacceptable (safety-critical circuit analysis), the 0.2 oracle-relative cost premium of AtP* is worth paying for the additional recall; for bulk screening where cost efficiency is paramount and a few missed nodes are tolerable, AtP+QKfix is sufficient.

Methods Do Not Transfer Equally Across Models and Prompt Types

Model scale effects. The relative-cost aggregations in Figures 2 and 9 show consistent patterns across Pythia sizes, but the magnitude of AtP*'s advantage varies. On CITY-PP NeuronNodes (Figure 2a), AtP*'s relative cost is roughly 1.1–1.3× oracle for all model sizes, while Subsampling degrades from roughly 2× on 410M to roughly 5× on 12B — suggesting Subsampling scales worse with node count. On IOI-PP AttentionNodes (Figure 2b), AtP* is roughly 1.1–1.2× oracle across models, while Blocks ranges from 1.5× on 1B to 3× on 12B. The full cost-of-verified-recall curves for all model sizes and settings are shown in Appendix Figures 12 and 13, and they generally show AtP* maintaining a clear advantage at every scale.

Crisp circuits versus random prompts. On the carefully constructed CITY-PP and IOI-PP, AtP* achieves oracle-relative costs in the 1.1–1.5× range (Figures 1, 2). On RAND-PP (randomly chosen prompts from The Pile, with no intentional circuit structure), AtP*'s relative cost increases to 1.5–2.0× (Figures 8a-b, 9a-b), and the gap over alternative methods narrows. This suggests that AtP's gradient approximation works better when the true causal structure is sparse and localized — edges are cleaner, non-linearities are less entangled — and degrades slightly (but not catastrophically) when effects are more distributed. The fact that AtP* still leads on RAND-PP is important evidence that its performance isn't an artifact of cherry-picked prompts, but the reduced advantage is a meaningful caveat for practitioners working with less-structured prompt pairs.

Node granularity effects. The paper deliberately separates results by node type. On AttentionNodes (Figures 1b, 2b, 8b, 8d), AtP+QKfix is essential — vanilla AtP performs substantially worse (e.g., roughly 2.8× oracle on IOI distribution versus 1.4× for AtP+QKfix, Figure 9c). On NeuronNodes (Figures 1a, 2a, 8a, 8c), the QK fix is irrelevant (neurons don't feed into attention softmax), and vanilla AtP already performs well — AtP*'s GradDrop provides a small additional benefit on single pairs but not on distributions. This is consistent with the paper's Section 5.4 recommendation: use AtP* for AttentionNodes on single pairs, AtP+QKfix for AttentionNodes on distributions, and AtP for NeuronNodes on distributions.

Diagnostics Can Reliably Bound False Negative Magnitudes

Figure 6 shows the diagnostic procedure applied to Pythia-12B on two settings: IOI-PP (single pair, Figure 6a) and the IOI distribution (Figure 6b). On IOI-PP, the diagnostic with a sampling budget of roughly 600 forward passes achieves 99% confidence that the largest false negative (outside the top 1024 AtP* nodes) has effect below roughly 0.08 — and the true largest false negative is at roughly 0.06, so the bound is slightly conservative but correct. With 2,000 forward passes of diagnostic budget, the 99% confidence bound tightens to approximately 0.04, below the true largest false negative — at this point the diagnostics would correctly signal that no important nodes remain unverified.

On the IOI distribution (Figure 6b), where there is no large false negative to find (the true effects of nodes outside the top 1024 are all very small, below 0.02), the diagnostics "progressively keep gaining confidence with more data." With 50 forward passes, the 90% confidence bound is roughly 0.08; with 200 passes, the 99.9% bound tightens to roughly 0.03. Note the cost difference: on IOI-PP (single pair), diagnostic samples cost one forward pass each; on IOI distribution, each diagnostic sample costs one forward pass per prompt pair (120 forward passes per sample), but the paper reports costs in "per prompt pair" units, so the diagnostic budget is effectively 120× larger in absolute terms. The fact that the distributional diagnostic converges faster per prompt pair is consistent with Subsampling's intrinsic distributional averaging.

Ablation Studies and Robustness Checks

Metrics other than negative log probability (Appendix B.4, Figure 14): AtP*'s rankings are "not sensitive to the choice of ℒ." Rank scatter plots for Pythia-12B on IOI-PP and IOI using three metrics — negative log probability (the main metric), logit difference (difference in logit between correct and incorrect tokens), and log-odds (log ratio of correct to incorrect token probabilities) — show "qualitatively similar patterns." The paper does not report quantitative correlation numbers between the rankings, but the scatter plots (Figure 14) show tight diagonal structure for all three metrics in the noising setting. However, in the denoising setting (bottom row of Figure 14), the scatter degrades notably — AtP* ranks are visibly less correlated with true ranks under denoising. The authors note they "do not have a satisfactory explanation for this observation" (Section B.4), but it suggests that AtP* transfers less reliably to the denoising use case, where the intervention replaces noise activations with clean ones rather than vice versa.

Noising versus denoising (Section 5.2, Appendix B.4, Figure 14 bottom row): The paper primarily evaluates noising (replacing clean with noise activations) and provides only preliminary evidence for denoising. The bottom row of Figure 14 shows that AtP*'s rank accuracy degrades in the denoising setting for IOI — the scatter is more diffuse, and the false-negative region (bottom right) is more populated. The authors speculate that "the lower-right subplot (log-odds denoising) is similar to the lower-middle one (logit-diff denoising) because IOI produces a bimodal distribution over the correct and alternate next token," but they do not investigate the degradation further. This is a meaningful limitation for practitioners who prefer denoising (as in Meng et al., 2023; Lieberum et al., 2023) — the paper's strong results for noising do not automatically transfer.

True effect distribution (Appendix D, Figure 17): The distribution of true node effects c(n) across models and prompt settings is highly skewed — a small number of nodes have large effects, while the vast majority have effects near zero. This is consistent with the sparse-subgraph hypothesis that motivates the work and explains why a good ranking method (that puts the few large-effect nodes near the top) can dramatically reduce verification costs. Figure 17 shows this pattern is consistent across all four model sizes and across all three single-prompt-pair settings (CITY-PP, IOI-PP, RAND-PP). The skewness is more pronounced for larger models (12B has a sharper elbow than 410M) and for AttentionNodes compared to NeuronNodes (attention nodes have a more extreme heavy-tailed distribution).

Cancellation across a distribution (Appendix B.2, Figure 10): The cancellation ratio — defined as $1 - |∑_x ℐ(n; x)| / ∑_x |ℐ(n; x)|$ — measures how much positive and negative effects cancel when effects are summed across prompt pairs before taking the absolute value. A ratio near 1 means near-perfect cancellation (the node's effect flips sign across pairs and cancels out). Figure 10 shows cancellation ratios for nodes in the IOI distribution across the four Pythia model sizes, binned by percentile of total absolute effect ∑_x |ℐ|. For nodes with the largest total absolute effects (95th percentile and above), cancellation ratios of 0.5–0.9 are common — meaning these important nodes have effects that would be severely underestimated if the absolute value were taken outside rather than inside the expectation. This empirically validates the paper's design choice in Equation 5 to place the absolute value inside the distribution average.

Frozen layer normalization for AtP (Appendix C.1, Figure 15): The paper analyzes but does not empirically evaluate a variant of AtP that holds layer normalization scaling factors fixed during the backward pass (i.e., treating the denominator as constant, as recommended by Nanda, 2022, for patching residual-stream nodes). The analysis (Figure 15) shows that this variant reduces approximation error compared to standard AtP when the cosine similarity between clean and noise activations is less than 1/2 (angle > 60 degrees), but increases error when the activation difference is small (cosine similarity near 1). Since the nodes studied in this paper (attention queries/keys/values, MLP neurons) produce relatively small perturbations to the residual stream, the standard AtP (with varying layer norm denominator) is appropriate — but for future work on coarser nodes (full residual streams, entire layers), freezing the denominator would likely be beneficial. This is an example of the paper providing design guidance even for settings it does not directly evaluate.

Edge-AtP extensions (Appendix C.2, Table 2, Figure 16): The paper analyzes but does not empirically evaluate edge-level attribution patching (where the effect of an edge from node n₁ to node n₂ is estimated rather than the effect of node n₂ alone). Table 2 provides formulas for the per-token per-layer-pair quadratic cost of edge-AtP variants across different edge types (Output→Value, Output→Query/Key, MLP→MLP, etc.) and AtP variants (vanilla, QKfix, GradDrop, AtP*, and "long" versions for large sequence lengths). Figure 16 plugs in the Pythia model dimensions and shows that costs vary over several orders of magnitude depending on the variant and edge type. The paper recommends a specific configuration for edge-AtP*: include QKfix and MLPfix (analogous to QKfix but for MLP non-linearity saturation when n₂ is an MLP node) but omit GradDrop (because edge attribution already decomposes paths, and the L backward passes of GradDrop would be extremely expensive when multiplied by the edge count). This analysis, while not empirically validated, provides a concrete roadmap for scaling attribution patching to edge-level analysis, which the paper identifies as providing "more fine-grained information about which paths in the computational graph matter" (Section 5.2).

ReST^EM revision model (Appendix K, Figure 16): This appears to be a reference from the example template that does not exist in the actual paper. There is no Appendix K or Figure 16 showing revision model results — the paper does not discuss revision models at all. The correct reference for Appendix K is the edge-AtP costs Figure 16 (which exists and is described above). The actual paper has no reinforcement learning or self-improvement experiments.

Subsampling hyperparameter robustness (Appendix B.5): The Subsampling inclusion probability p was chosen from {0.01, 0.03}, and the number of samples m was chosen from power-of-2 numbers of batches. The IRWRGM-best p is not reported, so the sensitivity of Subsampling performance to p is unknown. Similarly, Blocks' block size was swept over {2, 6, 20, 60, 250} and Hierarchical's number of levels over 2–12, but the selected values per setting are not reported — only the best configuration's performance is shown. This makes it difficult to assess whether the alternative methods' underperformance relative to AtP is robust or an artifact of suboptimal hyperparameter ranges.

Critical Assessment

Central Claim 1: AtP significantly outperforms all other investigated methods for approximating activation patching, with AtP* providing further significant improvement

What the experiments demonstrate: The cost-of-verified-recall curves (Figures 1, 7, 8) consistently show AtP-based methods requiring fewer forward passes than Subsampling, Blocks, Hierarchical, or Iterative to find the top-K true nodes, across all four model sizes, both node granularities, and all prompt settings (single pairs and distributions). The IRWRGM cost metrics (Figures 2, 9) quantify this as a ~1.2–1.8× oracle-relative cost for AtP-based methods versus 2–5× for alternatives on single pairs, and ~1.3–1.8× versus 1.7–3.5× on distributions.

What the experiments do NOT demonstrate: The comparison is against baselines introduced in this same paper with no prior optimization history. Subsampling, Blocks, and Hierarchical are reasonable strawmen — they represent obvious non-gradient-based approaches — but they are not state-of-the-art methods refined by a community. It is possible that a more sophisticated version of Subsampling (e.g., using importance sampling to focus budget on high-AtP-score nodes, or using adaptive p rather than fixed) would close the gap. Similarly, the Iterative baseline for distributions (interleaving sampling and verification) is novel here and has not been optimized. The paper's claim that "AtP significantly outperforms all other investigated methods" is true of the methods as implemented, but the word "all" overstates the generality — only 5 alternative methods (all baseline-simple) are tested.

Additionally, all hyperparameters for the alternative methods were selected by oracle tuning (picking the configuration that minimizes IRWRGM cost against the known ground truth for the same setting). This gives the alternatives an unfair advantage relative to a practitioner who would not have ground truth available. The fact that AtP* still dominates even against oracle-tuned alternatives strengthens the conclusion, but it also means the reported gaps are upper bounds on the practical advantage — with realistic hyperparameter selection (which would be worse), the alternatives would look even worse.

Genuine weaknesses: The IRWRGM metric equally weights all K from 1 to |N|, which may not align with practitioner preferences. A researcher who only cares about the top 5 nodes will experience different relative performance than one who needs the top 500. The paper provides disaggregated curves (Figures 1, 7, 8) which allows readers to assess their specific K of interest, but the headline "AtP* achieves 1.3× oracle cost" is an average that obscures variation across rank regimes. In several settings (e.g., IOI-PP Figure 1b at very low K ~ 5-10), the gap between AtP* and alternatives is smaller than at K ~ 50-100.

Central Claim 2: Two failure modes (attention saturation and cancellation) produce significant false negatives in vanilla AtP; the QK fix and GradDrop address these while retaining scalability

What the experiments demonstrate: The existence of the attention saturation failure mode is directly supported by Figure 4 (left vs. middle), where the false-negative cluster among keys and queries is strongly associated with nodes whose patching produces large attention probability changes (coloration), and this cluster largely disappears with the QK fix. The existence of the cancellation failure mode is supported by Figure 5, where specific neurons with extreme direct-effect ratios (5.35, 12.2, 0) improve substantially with GradDrop. The scalability claim is supported by the fact that AtP*'s cost does not scale with |N| — the estimation cost is fixed (two forward passes + one backward pass + QKfix overhead + L backward passes for GradDrop) regardless of node count.

What the experiments do NOT demonstrate: The paper does not provide a systematic decomposition of how many false negatives each failure mode causes, or what fraction of the total false-negative rate is attributable to each. Figure 4 shows qualitatively that the QK fix eliminates a large cluster of false negatives, and Figure 5 highlights three specific cancellation cases, but there is no quantitative breakdown (e.g., "saturation accounts for 73% of false negatives on IOI-PP attention nodes; cancellation accounts for 15%"). This makes it difficult for practitioners to assess whether one fix is more critical than the other for their specific setting.

The paper also does not demonstrate that the two failure modes are exhaustive — i.e., that there are no other common causes of AtP false negatives beyond saturation and cancellation. AtP* still has some false negatives (points below the diagonal in Figures 4 right and 5 red circles), and their causes are not analyzed. The paper's implicit claim is that AtP* reduces false negatives to a "small minority of nodes" (Section 4.3, regarding Figure 7b), but it does not claim to eliminate them entirely. The diagnostic procedure is designed to bound the remaining false negatives, which is the paper's acknowledgment that AtP* is not perfect.

Notable missing analysis: The paper identifies attention saturation as a problem for keys and queries but does not empirically investigate the analogous problem for MLP neurons — where the GELU non-linearity could similarly saturate and cause gradient collapse. Appendix C.2.4 briefly discusses an "MLP fix" analogous to the QK fix for edge-AtP but does not empirically evaluate it or assess whether neuron-level AtP false negatives are partly due to GELU saturation. This is a significant gap given that NeuronNodes is one of the two primary node partitionings studied.

Central Claim 3: Subsampling diagnostics can bound remaining false negatives with statistical confidence

What the experiments demonstrate: Figure 6 shows that on two specific settings (IOI-PP and the IOI distribution with Pythia-12B), the diagnostic procedure produces upper confidence bounds that correctly capture or approach the true largest false negative effect size, and these bounds tighten with increased sampling budget. The Welch t-test machinery (Section 3.2) is correctly applied, and the max-p-value aggregation across all unverified nodes is a valid procedure for the compound null hypothesis.

What the experiments do NOT demonstrate: Two settings is a very limited evaluation of a statistical procedure that the paper proposes as a general diagnostic tool. The diagnostic's performance has not been characterized across model sizes, prompt types, node granularities, or numbers of verified nodes K. Key questions are unanswered: Does the diagnostic remain well-calibrated when the number of unverified nodes is very large (millions)? Does it remain calibrated when the Subsampling inclusion probability p is far from 0.5 (the paper uses p = 0.01–0.03, meaning count⁺_n is very small and the normal approximation underlying the t-test may be unreliable)? Does it work when interaction effects (violating the additivity assumption) are strong? The paper's additivity analysis (Appendix A.1.1) shows that interaction effects produce bias in the Subsampling estimator proportional to p times the sum of interaction effects — but the diagnostic's t-test assumes unbiased estimation, so interaction effects could cause the diagnostic to be overconfident (bound too tight, suggesting smaller missed effects than are actually present).

A structural concern: The diagnostic uses Subsampling to bound false negatives from AtP*, but Subsampling's own estimator may suffer from different failure modes (interaction effects, high variance with small p). If Subsampling also underestimates a node's effect, the diagnostic would be overconfident — it would bound the underestimated effect and falsely reassure the practitioner. The paper does not discuss or test this "second-order" failure mode of the diagnostic procedure.

Claim Implicit in the Paper's Framing: AtP* enables practical causal attribution at previously infeasible scales

What the experiments demonstrate: For Pythia-12B, AtP* reduces the cost of finding the top 100 attention nodes on IOI-PP from roughly 3,000 forward passes (brute force) to roughly 150 (Figure 1b) — a 20× speedup. For NeuronNodes on CITY-PP, the speedup is similar: finding the top 1,000 neurons costs roughly 1,200 forward passes with AtP* versus roughly 3,000–10,000 for alternatives (Figure 1a). These are genuine reductions that make certain analyses feasible in hours rather than days, and on larger models (70B, 175B, 530B) where brute force is completely infeasible, AtP* could be the difference between possible and impossible.

What the experiments do NOT demonstrate: The paper's largest model is Pythia-12B, which is 1–2 orders of magnitude smaller than frontier models at the time of writing (LLaMA-2 70B, GPT-3 175B, PaLM 540B). The claim that AtP* is "suitable for state-of-the-art LLMs" (as implied in the abstract and introduction) is an extrapolation from the Pythia scaling curve (410M → 1B → 2.8B → 12B). The scaling plots (Figures 2, 9) show AtP*'s relative performance is stable or slightly improving with model size, which is encouraging, but there is no physical law guaranteeing this trend continues to 70B+ — non-linearities could become more pronounced, attention could saturate more heavily, or gradient magnitudes could change in ways that degrade the approximation.

What a stronger demonstration would require: A FACTS-style input prompt or a held-out set of GSM8K-style problems with ground-truth neuron-level attributions computed via exhaustive patching (or a verified subset). Alternatively, a real-world case study where AtP* identifies a known circuit (e.g., the IOI circuit in Wang et al., 2022) with high recall and precision on a model where brute-force verification is feasible for confirmation but would have been prohibitively expensive without the prefilter.

Key Missing Experiments and Ablations

  1. No systematic study of AtP*'s sensitivity to prompt length. All experiments use prompts of length ~10–20 tokens. The paper's cost model treats prompt length T as a factor only in the attention computation (the O(T²) key correction algorithm cost), but the accuracy of the gradient approximation could also degrade with longer prompts (more layers of non-linearity between early nodes and the output, larger total residual stream signal relative to individual node perturbations). This is particularly relevant because the paper's motivation emphasizes long-context attribution (the Chinchilla 70B example with 1024 tokens).

  2. No study of how node effect sparsity affects method performance. The paper shows that true effects are heavy-tailed (Figure 17), but doesn't analyze whether methods perform differently on settings with more versus less sparse effects — for instance, whether Subsampling becomes relatively more competitive when effects are dense (many nodes have moderate effects) because its estimator averages over many nodes efficiently, while AtP maintains its advantage when effects are extremely sparse.

  3. No controlled experiment varying the "cleanliness" of the prompt pair. The paper uses CITY-PP and IOI-PP (designed to elicit clean circuits) and RAND-PP (random) as two points on a spectrum, but doesn't systematically vary properties of the prompt pair (e.g., noise prompt's similarity to clean prompt, complexity of the circuit involved) and measure how AtP*'s accuracy degrades. This makes it difficult to predict AtP*'s performance on a new prompt pair without running the expensive ground-truth computation.

  4. No comparison to human-identified circuits as an external validity check. Concurrent work by Syed et al. (2023) showed that AtP-based edge attribution patching could automatically recover circuits that matched human-interpretability findings. This paper evaluates only against the ground truth of exhaustive activation patching (an internal consistency check) but does not validate that the nodes identified as causally important by AtP* correspond to the circuits identified by human researchers. The paper acknowledges this limitation explicitly in Section 5.1: "we did not compare to past manual interpretability work to check whether our methods find the same nodes to be causally important as discovered by human researchers."

  5. No runtime or memory profiling. The paper measures cost in "forward passes" but does not report actual wall-clock times, GPU memory requirements, or the practical feasibility of running L backward passes on a 12B model (which requires storing L intermediate activation sets or recomputing them). For practitioners deciding whether to adopt GradDrop, knowing whether it fits in memory on their hardware is as important as the theoretical forward-pass cost.

Summary Assessment

The experimental evaluation is thorough within its defined scope — the paper systematically compares multiple methods across multiple models, node types, and prompt settings against exactly computed ground truth, and the results consistently favor AtP*/AtP-based methods over the introduced baselines. The evidence for the existence and successful mitigation of the two identified failure modes is solid (Figures 4, 5), though the quantitative contribution of each mode to total error is not decomposed. The evidence for AtP*'s practical utility is strong for the specific regime tested (Pythia models up to 12B, short prompts, clean and moderately clean circuits, fine-grained nodes) but should not be extrapolated to substantially different regimes (much larger models, much longer prompts, coarse-grained or residual-stream attribution, denoising rather than noising) without additional validation. The diagnostic procedure is correctly designed and works on the two tested examples, but its generality is largely unproven — two demonstrations do not constitute a reliability guarantee, especially given the interaction-effects concern that the paper itself raises (Appendix A.1.1) without testing. The paper's recommendations in Section 5.4 appropriately qualify the domains of applicability and urge practitioners to "look before you leap" when departing from the studied settings, which is an honest reflection of the evidence's scope.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For — and It May Dominate the Budget

The assumption or constraint. The entire compute-optimal framework conditions on an estimate of prompt difficulty, which is obtained by generating 2,048 samples per question and averaging either accuracy (oracle bins) or PRM final-answer scores (predicted bins). The authors explicitly acknowledge this cost is not included in any reported budget (Section 3.2):

"estimating difficulty in this way still incurs additional computation cost during inference… our experiments do not account for this cost largely for simplicity"

The consequence. As the authors themselves note, 2,048 samples per question is "more expensive than the inference budgets we consider" — the largest test-time budgets studied are 256–512 generations, meaning difficulty estimation consumes 4–8× MORE compute than the actual inference strategy being evaluated. The headline claim of "more than 4× better efficiency over best-of-N" (Section 1) is computed after difficulty is known, without amortizing the cost of learning it. In any realistic deployment, the total cost would be difficulty estimation PLUS strategy execution, and the former could dominate. In the worst case — many short questions where difficulty changes between queries — each query would incur the full 2,048-sample overhead, making the approach less efficient than a uniform best-of-N strategy, not more.

What evidence exists in the paper. The paper itself raises this concern in Section 3.2: "This is an exploration-exploitation tradeoff… we view this as a important avenue for future work." The predicted-bin variant (using PRM scores rather than ground-truth accuracy for difficulty estimation) partially mitigates the data requirement (no ground-truth labels needed) but does NOT mitigate the compute requirement — it still needs 2,048 samples per question. There is no experiment measuring the total wall-clock cost including difficulty estimation, no ablation where difficulty is estimated from fewer samples, and no comparison to a uniform strategy that uses those 2,048 samples as part of its inference budget rather than as a separate overhead. The paper's cost curves (Figures 4, 8) show the performance of the strategy given the difficulty estimate, not the performance of the system that must generate the difficulty estimate.

Mitigation status. The paper frames this as future work (Section 8): "pretraining or finetuning models to directly predict difficulty of a question." An alternative — adaptive difficulty estimation where the first few samples serve double duty as both difficulty assessment and candidate generation — is mentioned but not implemented. Until this gap is closed, the reported efficiency gains should be understood as an upper bound on what is achievable if difficulty can be estimated cheaply, not as a realized deployment gain. Any practitioner attempting to deploy this method would need to solve the difficulty estimation problem separately, and their measured efficiency would be strictly worse (potentially much worse) than the paper's reported numbers.

Hard Problems Remain Essentially Unsolved — Test-Time Compute Cannot Substitute for Missing Capability

The assumption or constraint. The paper's framework assumes the base model's pass@1 on a given problem is non-trivially above zero — that is, the model already can produce the correct answer at some non-zero rate through random sampling. The compute-optimal policy then amplifies this existing capability. As the authors state in the Section 7 takeaway box:

"test-time compute can amplify existing capability but cannot create it. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help."

The consequence. For difficulty bin 5 — the hardest 20% of MATH problems for the PaLM 2-S* model — the paper shows near-zero improvement across ALL methods and ALL budgets. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for both beam search and best-of-N weighted, even at 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy regardless of the sequential-to-parallel ratio at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% for both revisions and PRM search — below the ~14× larger model's greedy performance at all three R values.

This is not a minor edge case. It means that for problems genuinely outside the base model's training distribution — novel reasoning patterns, compositional generalization, problems requiring skills the model didn't acquire during pretraining — test-time compute provides NO benefit. The entire compute-optimal framework only applies to problems within the model's approximate capability range. This is a fundamental capability bound: test-time compute is a selection mechanism (finding correct answers that already exist in the distribution) and a refinement mechanism (polishing nearly-correct answers), not a generation mechanism (creating correct answers from scratch). For frontier models tackling genuinely novel problems, this distinction is critical, and the paper provides no path around it.

What evidence exists in the paper. The bin 5 results are consistent and unambiguous: Figures 3 (right, bin 5), 7 (right, bin 5), and 9 (bin 5 lines) show flat or near-flat performance. The FLOPs-matched comparison quantifies the gap explicitly: on hard problems with R ≪ 1 and revisions, test-time compute shows a +21.6% advantage over the larger model (meaning the larger model is also bad, but the test-time compute at least doesn't make things worse), while at R ≫ 1 with PRM search, the gap is −52.9% relative disadvantage (meaning the larger model substantially outperforms the smaller model with test-time compute). The paper is transparent about this limitation — it's not hidden — but the framing in the introduction ("more than 4× better efficiency") risks misleading readers into thinking the gains apply uniformly, when they are concentrated on easy-to-medium problems.

Mitigation status. Not addressed. The paper identifies this boundary but does not propose methods to extend test-time compute to harder problems. The obvious future direction — combining test-time compute with retrieval, tool use, or other external knowledge sources that might provide the missing capabilities — is not discussed. The paper's contribution is to characterize when test-time compute works, not to expand where it works.

The ~14× Larger Model Baseline Is Not Compute-Optimally Trained — and It Uses No Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal scaling (Hoffmann et al., 2022), where data and parameters are scaled equally. The authors acknowledge this explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model is evaluated with greedy decoding only — no majority voting, no best-of-N, no search, no revisions.

The consequence. Both choices make the pretraining baseline weaker than it could be. A Chinchilla-optimal model trained with ~14× more total FLOPs (scaling both parameters AND data) would likely outperform a parameter-only-scaled model, reducing or reversing the reported advantages of test-time compute. Similarly, giving the larger model even a modest test-time compute budget (say, best-of-8 or best-of-16) would create a much more realistic baseline — after all, the question is not "small model + test-time compute vs. large model + nothing," but "small model + test-time compute vs. large model + test-time compute" under a fixed total budget.

The paper's finding that test-time compute with a smaller model can outperform a ~14× larger model (e.g., +27.8% relative improvement on easy questions with revisions at R ≪ 1) should therefore be interpreted as an upper bound on the advantage — tested against a deliberately weakened baseline. Against a compute-optimally trained larger model with its own test-time compute allocation, the advantage would narrow, potentially reversing on medium and hard problems.

What evidence exists in the paper. The paper explicitly states the parameter-only scaling choice (Section 7) and notes that greedy decoding is used for the larger model. It does NOT provide an ablation where the larger model is Chinchilla-optimally trained, or where the larger model is given any test-time compute budget. A relevant sensitivity analysis would be: if the larger model used best-of-8 or best-of-16 (consuming a small fraction of its total FLOPs budget), how much of the reported advantage would disappear? This analysis is not performed.

Mitigation status. The paper frames this as future work (Section 8): "analyzing compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally." The ~14× larger model with test-time compute is not discussed as a baseline, which is a notable omission given the paper's own framing of the training-inference tradeoff. A practitioner deciding whether to invest in test-time compute or larger pretraining runs needs to know the answer against a fair baseline, and this paper provides only a one-sided comparison.

Single Benchmark, Single Model Family — the Difficulty-Dependent Patterns May Not Generalize

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* (and one ~14× parameter-scaled variant) as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an assertion, not a finding.

The consequence. Several aspects of the results could be model-specific or benchmark-specific:

  • The PRM's quality and over-optimization behavior depend on the base model's output distribution. A model with different calibration properties, different error patterns, or different training data might produce a PRM with different scaling characteristics — potentially changing the difficulty thresholds at which beam search over-optimizes, or altering which search algorithm is optimal at which budget.

  • The revision model's training depends on the base model's in-context learning capabilities and its ability to produce revisions when conditioned on previous incorrect answers. Different model families (e.g., Llama, GPT, Gemini) might have different in-context revision behaviors, and the specific edit-distance pairing strategy (Section 6.1) might transfer differently.

  • The MATH benchmark consists of competition-level math problems requiring multi-step symbolic reasoning. It is unclear whether the difficulty-dependent patterns (beam search HURTING easy problems, revisions HELPING easy problems, minimal gains on hard problems) generalize to other reasoning domains: code generation (HumanEval, MBPP), logical reasoning (ARC, PrOntoQA), scientific QA, or multi-step planning.

  • The difficulty bins are defined relative to PaLM 2-S*'s pass@1 rates. The same MATH problems would fall into different difficulty bins for a stronger or weaker model, potentially changing which strategies are compute-optimal. A practitioner using a different model family cannot simply adopt the paper's strategy-per-bin mapping — they would need to recompute difficulty bins and cross-validate strategies for their specific model, which is expensive.

Because the test set has only 500 questions, split into quintiles of ~100 each, and further split by two-fold cross-validation, the compute-optimal policy is selected based on ~50 questions per fold per bin. The paper does not report confidence intervals on the compute-optimal scaling curves (Figures 4, 8), making it difficult to assess whether the observed efficiency gains are statistically reliable or could shift substantially with a different set of 500 MATH questions.

What evidence exists in the paper. The paper provides no cross-model or cross-benchmark validation. The four Pythia model sizes used in the main study (410M to 12B) share architecture and training data, and the scaling trends observed (Figures 2, 9) are within this single family. The difficulty-dependent strategy selection (e.g., best-of-N for easy, beam search for medium) is derived entirely from PaLM 2-S* on MATH, with no evidence it transfers.

Mitigation status. The paper acknowledges the limitation implicitly by recommending practitioners "look before you leap" (Section 5.4) and stating their recommendations are "best-substantiated in settings similar to those [they] studied." There is no proposed methodology for efficiently transferring the strategy selection to new models or benchmarks without recomputing the full ground-truth-validated analysis.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate — and the Mitigation Is a Patch, Not a Solution

The assumption or constraint. The revision model is fine-tuned on trajectories where all in-context answers are incorrect followed by a correct target (Section 6.1). At test time, however, the model may encounter a correct answer in its context (produced during an earlier revision step) and — having never been trained on what to do when the current answer is already correct — will often "revise" it into an incorrect answer.

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach."

The consequence. A chain of sequential revisions is not monotonically improving. Each revision step carries a substantial risk of destroying a correct answer and replacing it with an incorrect one. The paper mitigates this by using majority voting or verifier-based selection across the entire revision chain — picking the best answer from any point in the chain rather than always taking the last revision. However, these are post-hoc fixes that don't address the underlying problem: the revision model doesn't know how to recognize when it should stop revising.

This has two practical consequences that are underappreciated in the paper's optimistic framing:

  1. The effective length of useful revision chains is limited. Even though Figure 6 (left) shows pass@1 at each step improving through ~15-20 revisions and staying in the 23-25% range out to 64 steps, the selected answer (via verifier or majority) must come from somewhere in the chain. If correct answers are being generated at some steps and then reverted at others, the verifier must be good enough to distinguish which steps are correct — introducing dependency on verifier quality that is not accounted for in the pass@1 trajectory.

  2. The revision model's training protocol is fragile. The ReST^EM experiment (Appendix K, Figure 16) — where an attempt to further optimize the revision model using on-policy data collection caused performance to degrade substantially — suggests that the revision training is sensitive to the data generation procedure in ways that are not fully understood. Practitioners attempting to replicate the revision pipeline on their own models may encounter similar instability, especially if they modify the data collection process.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. The ReST^EM failure is reported in Appendix K, Figure 16: "additional sequential revisions substantially hurt performance" with the ReST^EM-trained model, and at 256 generations, fully sequential performance drops to ~33.5% compared to ~38.5% at the optimal ratio. The paper speculates that "on-policy data collection exacerbates spurious correlations in revision data" but does not investigate further.

Mitigation status. The paper uses within-chain selection (majority voting or verifier-based) to handle the reversion problem, but acknowledges it as a limitation only implicitly — by describing the need for the selection mechanism — rather than framing it as a fundamental flaw in the revision training approach. Future work on training the model to recognize when no revision is needed (e.g., by including "correct → correct" trajectories in training data) is not discussed. The ReST^EM negative result is reported but not explained, and no guidance is provided for practitioners on how to avoid similar training instabilities.

No Accounting for Latency or Wall-Clock Time — Sequential Strategies Assume Unlimited Parallelism

The assumption or constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores latency — the wall-clock time required to produce a final answer. This matters because the compute-optimal policy discovered by the paper often recommends sequential strategies for easy problems (e.g., long revision chains, Section 6) and beam search for medium problems (Section 5.3), both of which are inherently serial.

The consequence. A sequential strategy that allocates 128 generations as 64 sequential revisions × 2 parallel chains takes roughly 64× longer wall-clock time than a parallel strategy that runs 128 best-of-N samples simultaneously (assuming sufficient hardware). For latency-sensitive applications — interactive chatbots, real-time code assistants, on-device deployment where model inference is the bottleneck — the sequential-heavy strategies recommended by the compute-optimal policy may be impractical regardless of their FLOPs efficiency. Even beam search, which is parallel within each beam expansion step, requires multiple sequential rounds of generation (up to 40 as configured in the paper), while best-of-N is embarrassingly parallel.

The FLOPs-matched comparison (Section 7) further complicates this: the R = D_inference / D_pretrain analysis assumes the relevant metric is total FLOPs across all inference queries, but a practitioner with a fixed latency budget (e.g., "must respond within 2 seconds per query") may find that the smaller model + test-time compute violates the latency constraint even if it hits the FLOPs budget.

What evidence exists in the paper. The paper does NOT mention latency or wall-clock time anywhere. All cost metrics are in forward passes or generations, with no discussion of whether those generations can be parallelized, what hardware assumptions are needed, or how the sequential-to-parallel ratio affects end-to-end response time. Figure 7's optimal ratio analysis (sweeping the sequential-to-parallel ratio at fixed total generation budget) treats all ratios as equivalent in cost, when in practice a higher sequential ratio implies higher latency.

Mitigation status. Not addressed. The paper does not mention latency as a consideration, does not include latency-constrained Pareto curves, and does not discuss the tradeoff between sequential and parallel strategies in wall-clock terms. For practitioners deploying interactive systems, this is a critical gap: the paper's recommendation to "use sequential revisions on easy problems" (Section 5.4) may be correct in FLOPs but wrong in wall-clock time if parallelism is available.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper transforms Attribution Patching from a heuristic that "seems to work in some cases" into a diagnosable, improvable, and statistically-auditable estimator whose failure modes are understood, addressed, and bounded. This is a significant shift for mechanistic interpretability practice — not because AtP is a new idea (the core approximation dates to Nanda, 2022), but because prior to this work, no one knew when AtP's rankings could be trusted, what types of components it systematically missed, or whether those misses could be fixed without sacrificing the method's speed. The field's default was trust-but-verify-exhaustively, which defeated the purpose of having a fast approximation. This paper provides the conceptual machinery — failure mode taxonomy, targeted preconditioners, statistical diagnostics — to replace that default with verify-only-the-top-K-and-bound-the-rest, making genuinely comprehensive attribution on large models practical for the first time.

The shift is best understood as moving AtP from a point estimate to a screening procedure. Individual AtP scores are never interpreted as accurate effect magnitudes — the paper explicitly does not evaluate magnitude calibration (Section 5.1) — but rather as a ranking that feeds a downstream verification step. This reframes the evaluation criterion from "how close is ĉ(n) to c(n)?" (a hard problem requiring global linearity) to "does the top-K ranking contain the most important nodes?" (an easier problem requiring only local ordinal consistency). The paper's cost-of-verified-recall metric (Figures 1, 7, 8) and IRWRGM scalar (Figures 2, 9) operationalize this framing and provide a benchmark template that future fast-attribution methods can adopt. This is a methodological contribution to how the field evaluates attribution approximations — not just to which method wins.

The paper also resolves a latent tension in the literature that practitioners experienced but could not articulate. Nanda (2022) speculated that AtP might fail for coarser nodes or when layer norm scaling is important. Syed et al. (2023) showed AtP worked well for circuit discovery on specific tasks. Individual researchers reported mixed experiences — sometimes AtP found the right heads, sometimes it missed obvious contributors. These conflicting anecdotes are now explicable: cancellation false negatives are prompt-pair-specific (Section 4.3: GradDrop's benefit "diminishes on distributions"), so single-pair studies could produce false negatives that multi-pair studies would average out; attention saturation produces systematic misses concentrated in keys and queries (Figure 4), so studies focusing on attention heads would encounter these failures more than studies focusing on MLPs. The paper provides a unified diagnostic framework — check for attention probability changes (Figure 4 coloration), compute direct-effect ratios (Figure 5), run Subsampling diagnostics (Figure 6) — that converts "AtP sometimes fails" into "here's how to check whether AtP failed in your specific setting." This is a practical advance that makes AtP a tool practitioners can use with eyes open rather than a black box they must trust blindly.

On the specific question of which research directions become more or less attractive:

More attractive: Building better preconditioners for gradient-based attribution. The paper shows that targeted fixes to specific non-linearities (the softmax in QK fix, the path-cancellation structure in GradDrop) can substantially improve reliability without abandoning the gradient approximation's O(1) scaling. This invites analogous fixes for other non-linearities the paper didn't address: GELU saturation in MLP neurons (discussed but not empirically evaluated in Appendix C.2.4), layer normalization scaling for residual-stream attribution (analyzed in Appendix C.1 but not tested), and RMSNorm interactions in non-Pythia architectures. The edge-AtP analysis in Appendix C.2 provides a concrete cost model for extending these fixes to edge-level attribution, making that previously prohibitive analysis scale tractably. Research into automated preconditioning — detecting which non-linearities a given node's effect passes through and selectively fixing them — becomes natural given the paper's decomposition of approximation error by computational subgraph.

Less attractive: Developing entirely new non-gradient-based fast attribution methods for fine-grained components. The paper's systematic comparison (Section 4) shows that Subsampling, Blocks, and Hierarchical — which represent obvious non-gradient approaches — are consistently and substantially outperformed by AtP-based methods across all settings tested (model sizes, node types, prompt distributions). The gap is 2–4× in oracle-relative cost (Figures 2, 9) and is stable or widening with model scale. While more sophisticated non-gradient methods might close this gap, the paper sets a high empirical bar — any new method should be compared against AtP* on cost-of-verified-recall curves, and the evidence suggests gradient-based approximations are genuinely well-suited to the fine-grained regime where individual node perturbations are small relative to the full residual stream.

Unchanged but clarified: The importance of verifier quality for test-time compute scaling (from the reference example — this paper does not address verifiers). The paper's contribution is to make AtP reliable enough that researchers can confidently attribute behaviors to components, which in turn enables the circuit-discovery and steering applications discussed in Section 5.3. The "attribution bottleneck" — the inability to efficiently identify which components matter — has been a drag on multiple interpretability research directions. By substantially widening this bottleneck for fine-grained attribution, the paper accelerates all downstream research that depends on knowing which components are causally important.

A subtle but important landscape change: the paper makes statistical rigor in component attribution feasible. The diagnostic procedure (Section 3.2, Figure 6) provides false-negative confidence bounds using Welch's t-test on Subsampling statistics. Prior to this work, the only way to be confident a component wasn't important was to patch it exhaustively. Now, a practitioner can state: "we are 99% confident that all nodes outside our verified top 1024 have true effect below 0.03" — a statement that was previously impossible to make without exhaustive verification. This transforms attribution from an engineering exercise ("we patched what we could afford") into a statistical task ("we've bounded what we might have missed"). This is a conceptual step toward making mechanistic interpretability a falsifiable scientific practice rather than a collection of case studies — you can now quantify how confident you are that you've found the important components, and that confidence level can be tuned with compute budget.

Follow-Up Research This Work Enables

Efficient verification of edge-level circuits on frontier models using edge-AtP* with storage optimizations. The paper provides a complete cost model for edge-AtP* variants in Appendix C.2 (Table 2, Figure 16) but does not empirically validate any of them. A direct follow-up would implement edge-AtP* (QKfix + MLPfix, omitting GradDrop per the paper's recommendation) on Pythia-12B or a comparable model, evaluate cost-of-verified-recall against ground-truth edge contributions on a known circuit (e.g., the IOI circuit from Wang et al., 2022), and measure whether the storage explosion from premultiplied parameter matrices (which the paper notes can be (L-1) * d_neurons / (4 * d_resid) times larger than the MLP weights) is practical with contemporary GPU memory. The specific claimed bottleneck is that "storage may need to be considered carefully" and "it may be worth considering ways to only find the largest estimates... rather than full estimates for all edges." A strong follow-up would implement thresholded edge-AtP* — only computing and storing edge estimates above some quantile of the node-level AtP* estimates — and measure the recall of important edges versus the memory savings. This would determine whether edge-level attribution on models with millions of neurons is feasible or whether the storage requirement makes it impractical without further algorithmic innovation.

Training a lightweight difficulty predictor to close the difficulty-estimation cost gap. The paper's most significant unaddressed practical limitation is that difficulty estimation costs 2,048 samples per question — more than the inference budgets being optimized. The paper explicitly calls for "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). A concrete experiment: train a small classifier (e.g., a distilled version of the PRM, or a lightweight probe on the base model's intermediate representations) that takes only the question text and predicts the difficulty quintile. Measure: (1) what classifier accuracy is needed for the compute-optimal policy's performance to remain within 5% of oracle-bin performance? (2) How does the total cost (classifier inference + strategy execution) compare to the current 2,048-sample overhead? The paper's Figure 4 shows that predicted-bin performance closely tracks oracle-bin performance when the prediction uses PRM scores averaged over 2,048 samples — a strong follow-up would show how many samples are actually needed, and whether a learned classifier can match that performance with orders-of-magnitude fewer forward passes. An alternative adaptive approach: use the first 8–16 generated samples for both difficulty estimation (via PRM score variance or consistency) AND as candidates for the final answer selection, then allocate the remaining budget using the compute-optimal policy. This would amortize difficulty estimation into the solution process and could be compared directly against the paper's static difficulty estimation on the cost-of-verified-recall metric extended to include estimation costs.

Stress-testing AtP* on models with different architecture families and substantially longer contexts. The paper evaluates only on Pythia models (GPT-style decoder-only transformers) with prompt lengths of ~10–20 tokens. The motivation emphasizes long-context attribution (the Chinchilla 70B example with 1,024 tokens producing 2.7 × 10^9 neuron nodes), but no experiments test whether AtP*'s gradient approximation degrades with context length — where more layers of non-linearity separate early-token nodes from the output, potentially amplifying approximation error. A direct stress test: evaluate AtP* against ground truth on Pythia-12B with prompts of systematically increasing length (16, 64, 256, 1024 tokens), measuring IRWRGM cost at each length. Hypothesis: AtP*'s relative cost degrades with length because the perturbation from a single node becomes a smaller fraction of the total residual stream signal, making the gradient approximation more accurate (favorable) but the metric more sensitive to accumulated approximation errors across many layers (unfavorable) — the net effect is unknown. Additionally, evaluate AtP* on a non-Pythia architecture (e.g., Llama-2 with its different normalization placement, or a mixture-of-experts model where the routing function introduces additional non-linearities). The paper explicitly acknowledges this limitation (Section 5.1: "we cannot rule out qualitatively different behavior without further evidence, especially on SotA–scale models or models that significantly deviate from the standard decoder-only transformer architecture"), making this a high-priority validation experiment.

Characterizing when Subsampling diagnostics remain calibrated under interaction effects. The paper's diagnostic procedure (Section 3.2) uses Subsampling estimates ī⁺_n - ī⁻_n as approximately unbiased estimators of individual node effects ℐ(n). However, Appendix A.1.1 proves that under a pairwise interaction model, this estimator has bias p × Σ_{n'≠n} σ_{n,n'} — proportional to the inclusion probability times the sum of interaction effects involving node n. This means that if a node has strong interactions with other nodes, Subsampling will systematically overestimate or underestimate its effect, and the diagnostic t-test (which assumes the estimator is unbiased under the null) could be miscalibrated. A specific experiment: construct synthetic ground-truth data by defining node effects and pairwise interaction terms, measure the diagnostic's actual false-negative rate (how often the confidence bound fails to cover the true largest false negative) as a function of interaction strength, and determine the interaction magnitude at which the diagnostic becomes unreliable. The paper's Figure 6 shows the diagnostic works on two real examples, but the interaction strength in those examples is unknown — this experiment would establish the diagnostic's safe operating envelope. If interaction effects are found to cause significant miscalibration, a natural extension is to incorporate interaction-aware estimators (e.g., Shapley-value sampling, or higher-order inclusion probabilities) into the diagnostic while preserving its computational efficiency.

Combining AtP* with sparse autoencoder features to scale attribution to disentangled representations. Section 5.3 discusses the potential of using AtP* to attribute behavior to sparse autoencoder (SAE) features rather than native model components (neurons, attention heads). SAEs typically have 10–50× more features than the MLP neuron count, making exhaustive patching even more prohibitive. However, SAE features are also sparse — on a given forward pass, only 10–20 features are active out of thousands — which reduces the effective node count and may make gradient-based approximations more accurate (since the perturbation from switching off a single active feature is a small change to the SAE reconstruction). A concrete experiment: train an SAE on Pythia-12B MLP activations following the recipe of Bricken et al. (2023), then apply AtP* to attribute behavior to SAE features on the CITY-PP and IOI-PP prompts, measuring cost-of-verified-recall against ground truth obtained by exhaustively patching active SAE features. Compare: (1) Does AtP*'s accuracy on SAE features match its accuracy on native neurons? (2) Do the identified SAE features correspond to semantically interpretable concepts that are also causally important? This would provide the first evidence on whether gradient-based attribution transfers to the SAE setting — where the "node" is not a native model component but a learned feature in a disentangled basis — and would inform whether the community's investment in SAEs is compatible with efficient causal attribution.

Using AtP* for targeted inference-time interventions validated by behavioral metrics. Section 5.3 suggests AtP* could identify single nodes whose activation can be modified at inference time to steer model behavior, producing "more localized interventions with less impact on the rest of the model's computation" than current activation steering methods (Turner et al., 2023; Zou et al., 2023). A specific experiment: on the IOI task, use AtP* to identify the top-K attention nodes (keys, queries, values, outputs) causally implicated in the model's prediction of the indirect object. Then, intervene on these nodes by amplifying or suppressing their activations (e.g., scaling the value vectors, or adding a bias to the attention weights) and measure the change in model accuracy on held-out IOI examples. Compare the specificity of the intervention (does it affect only IOI performance, or does it degrade general language modeling?) against existing activation steering methods. The paper's advantage would be in the targeting — AtP* can identify specific heads and token positions whose causal role in the behavior is quantified, enabling more surgical interventions than contrastive-activation-addition approaches that modify entire residual-stream directions. A strong result would show that AtP*-guided interventions achieve comparable behavioral change with smaller side effects on unrelated capabilities.

Practical Applications and Downstream Use Cases

Circuit discovery on frontier models at previously-impossible granularity. The most direct application: a researcher who wants to understand how a specific behavior (e.g., factual recall for country-capital relationships, indirect object identification, sentiment classification) is implemented in a large model can now run AtP* to rank all attention heads and MLP neurons by causal importance, verify the top few hundred, and be statistically confident (via diagnostics) that they haven't missed any large-effect components — all in a few hours on a single GPU for a 12B model, versus the weeks or months that exhaustive patching would require. For a Pythia-12B model on IOI-PP, Figure 1b shows AtP* finding the top 100 attention nodes in roughly 150 forward passes. Extrapolating to a 70B model with similar architecture: if the number of nodes scales approximately linearly with parameters (heads × layers × tokens), AtP*'s estimation cost remains fixed (two forward passes + one backward pass + QKfix overhead), while the verification cost for the top K nodes scales as K × (forward pass cost of larger model). The speedup relative to brute force thus increases with model size, making AtP* increasingly valuable on frontier models where brute force is completely infeasible for fine-grained attribution. The paper's diagnostic procedure means the researcher can stop verifying when the confidence bound drops below their threshold of interest — e.g., "we're 99% confident we've found all nodes with effect > 0.01 on the metric" — rather than verifying arbitrarily many nodes until budget runs out.

Targeted model editing via component-level interventions identified by AtP*. The paper shows AtP* can efficiently identify which MLP neurons or attention heads are causally responsible for specific factual associations (CITY-PP, Section 4.3) or behavioral patterns (IOI-PP). A practitioner who wants to edit a model's knowledge — e.g., update a factual association from "Barcelona is in Spain" to a corrected version, or remove a biased association — can use AtP* to identify the minimal set of components that need modification, then apply localized edits (e.g., rank-one updates to MLP weight matrices, or bias terms added to attention outputs) to those specific components. This would produce more targeted edits than full-model fine-tuning or representation-level interventions, potentially preserving the model's general capabilities better. The key advantage this paper provides is speed: a practitioner can test multiple candidate edit targets (e.g., "which neurons encode the Spain-Barcelona relationship?") using AtP* rapid sweeps before committing to an editing approach, and the diagnostics can confirm that no obvious contributors were missed. Without AtP*, this exploratory phase would be prohibitively expensive on large models.

Large-scale automated analysis of model behavior across prompt distributions for safety auditing. The paper's distributional results (IOI, A-AN, Figures 8c-d, 9c-d) show that AtP+QKfix and AtP perform well when effects are averaged across many prompt pairs. A safety auditor who needs to understand whether a model exhibits systematic undesirable behavior (e.g., gender bias in pronoun prediction, sycophancy in opinion-referencing prompts, or compliance with harmful requests) can construct prompt-pair distributions that isolate the behavior, run AtP* across all components to identify which circuits are consistently implicated, and verify only the top candidates. The Subsampling diagnostic provides rigorous confidence bounds on the effect sizes of unverified components, which is particularly important in a safety context where false negatives (missing a component that contributes to harmful behavior) are more costly than false positives (wasting verification budget on a component that turns out to be unimportant). The paper's finding that AtP* is "not overly sensitive to the noise distribution" (Section 5.2, referencing the RAND-PP results) suggests the method is robust to variation in how the noise prompts are constructed, which is practically important when the auditor doesn't have perfectly matched minimal-pair prompts.

Efficient development of sparse autoencoder-based interpretability tools with causal validation. Sparse autoencoders (SAEs) are emerging as a promising approach for finding disentangled, interpretable features in LLM activations (Bricken et al., 2023; Cunningham et al., 2023). However, establishing that SAE features are causally relevant — not just correlated with input patterns — requires activation patching on SAE features, which is even more expensive than native-component patching due to the larger number of features. AtP* could serve as the causal-validation engine for SAE-based interpretability: after training an SAE and identifying features that appear semantically meaningful, run AtP* (adapted to treat SAE features as nodes, as proposed in Section 5.3) to rank features by causal importance for specific behaviors, verify the top candidates, and use diagnostics to bound the importance of the rest. This would enable researchers to move from "this feature activates on mentions of Barcelona" (correlational) to "this feature's activation at this token position is causally necessary for the model to predict Spain" (causal) — a crucial distinction for building interpretability tools that support model editing, steering, or safety verification. The paper's analysis of edge-AtP* (Appendix C.2) would be directly relevant here, since understanding SAE feature circuits requires attributing effects to feature-to-feature edges, not just individual features.