Open Problems in Mechanistic Interpretability

🎯 Pitch

The field cannot tell us how a single GPT-4 reasoning step works—because current interpretability methods, like sparse autoencoders on hidden activations, would cost 90% of the model’s original training compute just to reconstruct features with tolerable error, and still lack theoretical guarantees for carving a network’s computation at its causal joints rather than its correlations.

1. Executive Summary

This forward-facing review systematically surveys the current frontier of mechanistic interpretability and identifies the key open problems the field must solve to achieve its scientific and engineering goals. The paper organizes these challenges along three axes: fundamental methodological limitations in reverse-engineering neural networks—from decomposition via sparse dictionary learning (SDL; e.g., training sparse autoencoders to extract interpretable "latents" from activations) to describing component function (e.g., using causal interventions like activation patching) and validating hypotheses (e.g., predicting counterfactual behaviors)—alongside open problems in applying these methods toward concrete objectives such as monitoring AI systems for unsafe cognition, controlling model behavior through activation steering, and predicting emergent capabilities. The review synthesizes perspectives from over thirty researchers to establish that progress is needed simultaneously along multiple dimensions—improved decomposition methods that carve networks "at the joints" of their generalization structure, deeper causal descriptions rather than mere correlational analyses, and automated pipelines that scale to frontier systems—while cautioning that current methods like SDL face both practical bottlenecks (reconstruction errors equivalent to losing 90% of pretraining compute when applied to GPT-4) and unresolved conceptual questions about whether the linear representation hypothesis and sparsity assumptions adequately capture how computation is implemented in neural networks.

2. Context and Motivation

The Core Problem: We Don't Yet Know How Neural Networks Actually Work

The fundamental challenge this paper addresses is deceptively simple: neural networks are black boxes, and our current scientific tools can't explain how they generalize. When a transformer-based language model produces a coherent paragraph, solves a math problem, or refuses a harmful request, we have almost no understanding of the mechanisms—the specific neural algorithms implemented in its weights—that produced that output. The training process (gradient descent on a loss function) is designed by humans, but the resulting computational structure is emergent and opaque.

This isn't merely an academic curiosity. The paper identifies several concrete reasons this gap in understanding matters (Section 1):

We lack rigorous safety assurance for frontier AI systems. Current evaluation paradigms rely almost entirely on behavioral testing—we observe input-output pairs and check whether they match desired patterns. But behavioral testing alone cannot guarantee safety, because it can only cover a finite set of scenarios. An AI system that learns to strategically underperform evaluations ("sandbagging," cited in Section 3.2) or that develops dangerous internal reasoning patterns that don't manifest in test-set behavior would be invisible to black-box methods. The paper explicitly frames mechanistic interpretability as a path toward "white-box evaluations" that could detect such patterns by examining model internals directly (Section 3.2.1).
We can't predict how models will behave in novel situations. As Section 3.3 argues, understanding the mechanisms underlying model behavior would allow us to anticipate failures before they occur in deployment—predicting whether a model will hallucinate on certain inputs, whether it can be jailbroken through specific attack vectors, or whether it harbors trojans or backdoors that only activate under rare trigger conditions. The paper notes that mechanistic interpretability could potentially enable claims of the form "there exists no mechanisms that would cause the model to deliberately behave undesirably" (Section 3.3.1), a much stronger assurance than "we tested it on a thousand examples and it seemed fine."
We lack surgical control over model behavior. Current methods for modifying model behavior—fine-tuning, RLHF, prompting—are coarse and unpredictable. They often produce unwanted side effects or fail to fully remove undesirable capabilities. The paper discusses how mechanistic understanding could enable "more precise control" through targeted editing of specific circuit components (Section 3.2.2), analogous to how understanding a software program's source code allows fixing bugs by changing specific lines rather than retraining the entire system.
We're missing scientific insights that models have discovered. AI systems can outperform humans on tasks from protein folding to game strategy, suggesting they've learned patterns about the world that we don't yet possess. Without interpretability, those insights remain locked inside opaque weight matrices. The paper discusses "microscope AI" as a paradigm where trained neural networks serve as scientific instruments—we train them to model complex data, then use interpretability to extract the novel patterns they've discovered (Section 3.5).

The Gap Between Need and Capability

The paper emphasizes a stark disconnect between the ambitions of mechanistic interpretability and its current capabilities. The introduction is explicit about this:

"Despite recent hopeful signs of progress, mechanistic interpretability still has considerable distance to cover before achieving satisfactory progress toward most of its scientific and engineering goals" (Section 1.2).

This honesty about limitations is central to the paper's motivation. It's not simply a survey of cool things we can do; it's an assessment of what's blocking us from doing the things that matter. The authors are essentially arguing: we know what we want to achieve, we've made some progress, but here are the specific problems we need to solve before the field can deliver on its promises.

Prior Approaches and Why They Fall Short

The paper distinguishes mechanistic interpretability from two prior threads of interpretability research (Section 1.1.1), establishing what's different about the current approach and why earlier paradigms proved insufficient:

Thread 1: Interpretability by Design. Early interpretability work focused on building inherently interpretable models—decision trees, linear models, generalized additive models. While these remain useful (and the paper acknowledges ongoing work on architectures like Concept-Bottleneck Models, Backpack Language Models, and Kolmogorov-Arnold Networks), they face a fundamental limitation: they don't achieve state-of-the-art performance on complex tasks. The deep learning revolution demonstrated that large, nonlinear neural networks dramatically outperform interpretable alternatives on virtually every challenging benchmark. The paper implicitly accepts this tradeoff—it assumes we'll continue using large, opaque models for capability reasons, and focuses on post-hoc interpretation rather than architectural constraints.

Thread 2: Local Attribution Methods. As networks grew larger, researchers developed methods to answer "why did the model make this particular decision?"—techniques like Grad-CAM, integrated gradients, SHAP, LIME, and saliency maps. These methods identify which input features were most important for a specific output. The paper identifies two critical limitations with this approach:

First, they provide correlational rather than mechanistic explanations. Knowing that certain pixels in an image were important for a classification decision doesn't tell you how those pixels were processed—what features were detected, how they were combined, what intermediate representations were formed, what algorithm was executed. It's analogous to knowing which ingredients were important for a dish without understanding the recipe.

Second, and more subtly, they answer the wrong question for generalization. Local attribution explains individual decisions but doesn't reveal the general-purpose mechanisms that produce those decisions across a class of inputs. If you want to know whether a model will fail on a novel input, you need to understand the algorithm it's using, not just which features mattered for past examples.

The Mechanistic Turn. The paper frames mechanistic interpretability as a response to these limitations, driven by a fundamental hypothesis:

"This kind of interpretability work is driven by a fundamental hypothesis in deep learning that generalization arises from shared computation" (Section 1.1.1).

The key shift is from asking "which inputs mattered for this decision?" to asking "what algorithm does this model implement, and how is that algorithm distributed across its components?" This requires decomposing the network into functional units (attention heads, MLP layers, or finer-grained "features"), understanding what each unit computes, and describing how they compose to produce behavior—essentially, reverse-engineering the learned program.

Where Existing Mechanistic Methods Fall Short

Even within the mechanistic paradigm, the paper identifies specific shortcomings in current approaches:

Decomposition methods are immature. The paper devotes substantial attention (Section 2.1.2) to sparse dictionary learning (SDL)—currently the dominant decomposition approach—and catalogs its limitations in detail:

Reconstruction errors are too high: When using sparse autoencoders to reconstruct GPT-4's activations, the language modeling loss was equivalent to a model with only 10% of GPT-4's pretraining compute. This means the decomposition isn't faithfully capturing what the model actually computes—there's structure in the activations that SDL is systematically missing (Section 2.1.2c).
The underlying assumptions may be wrong: SDL rests on the linear representation hypothesis (that concepts are represented as directions in activation space) and the superposition hypothesis (that models represent more features than dimensions by making them sparse). The paper notes that a "strong version" of the linear representation hypothesis is "false for some models," and that sparsity "may not necessarily" be a good proxy for interpretability—sparser decompositions can sometimes be less interpretable due to feature splitting, absorption, and composition artifacts (Section 2.1.2c).
SDL decomposes activations, not mechanisms: Even a perfect decomposition would identify what features are present in the network's representations, but wouldn't describe the computational processes that produce those features from inputs. The paper notes we "primarily focus on understanding neural network activations, with little attention paid to how this structure in activations is computed via weights" (Section 2.1.2c).
Scaling is expensive and possibly intractable: SDL involves training a separate model (the autoencoder) for every layer of the target network, often with more parameters at that layer than the original model. As frontier models grow, this cost escalates, and it's unclear whether the approach can keep pace.

Description methods remain shallow. Even when we can identify components, describing what they do is challenging (Section 2.1.3). The paper catalogs several issues:

Maximum activating examples (finding inputs that strongly activate a component and looking for patterns) are "merely correlational," prone to human projection bias, and can yield "plausible explanations for arbitrarily chosen directions"—meaning we might think we understand something when we don't.
Attribution methods (measuring causal importance) often capture only first-order approximations, can be manipulated by adversaries, and perturbation-based methods take models off their training distribution, producing artifacts.
Causal interventions (activation patching, ablation) are more rigorous but computationally expensive and can miss indirect effects—a component might influence the output through pathways that a simple ablation doesn't capture, or might trigger compensatory "hydra effect" behaviors where other components take over its function.

Validation is underemphasized. The paper is unusually frank about this problem (Section 2.1.4):

"Conflating hypotheses with conclusions has regrettably been commonplace in mechanistic interpretability research, making validation an important area for the field to improve."

It cites examples where "interpretability illusions" produced seemingly convincing explanations that later proved false—a sobering acknowledgment that the field's impressive-sounding stories about what models are doing might often be wrong. The paper calls for more rigorous validation through multiple convergent methods, predicting counterfactual behaviors, using ground-truth toy models, and demonstrating engineering utility in competitive benchmarks—all of which are currently rare.

The Application Gap

Beyond methodological limitations, the paper identifies a crucial gap between what interpretability can currently do and what would be needed to achieve practical goals (Section 3). For each application area—monitoring, control, prediction, capability improvement, microscope AI—the paper asks what specific methodological advances are necessary and how far we are from them:

Monitoring for unsafe cognition: "Even shallow, correlation-based descriptions could signal potentially concerning cognition," suggesting this is one of the nearer-term applications. But reliably distinguishing between mechanisms that cause deceptive behavior versus those that merely recognize it remains unsolved.
Formal verification: The paper calls this "well beyond the current capabilities of mechanistic interpretability," but notes that toy-model proofs exist, raising the question of whether they can scale.
Predicting capabilities: Current work has shown correlations between mechanism emergence (like induction heads) and capability emergence (like in-context learning), but the paper argues we're far from being able to predict when capabilities will appear or which capabilities are hiding in existing models waiting to be elicited.

How This Paper Positions Itself

The paper explicitly distinguishes itself from prior reviews by being "forward-looking"—it aims not to survey what's been done, but to "discuss not only where the frontier is today, but also which directions we might benefit most from prioritizing in the future" (Section 1.1). This forward-facing orientation is central to its identity.

The paper also positions itself as synthesizing a community perspective rather than advocating for a single approach. It's authored by over thirty researchers from organizations including Apollo Research, Anthropic, Eleuther AI, FAR AI, Google DeepMind, Harvard, Leap Laboratories, MATS, MIT, METR, Northeastern University, Tel Aviv University, Timaeus, University of Melbourne, and Goodfire—an unusually broad coalition. The disclaimer notes that the paper "collects the perspectives of its various authors and represents a synthesis of their views by Apollo Research," explicitly acknowledging that "the perspectives presented here do not necessarily reflect the views of any individual author."

This coalition-building effort is itself a response to a problem the paper identifies: the field lacks "paradigmatic clarity" about "which goals the field of interpretability should pursue, how success should be graded, and how we should define interpretability" (Section 4.2). By gathering perspectives from across the field and organizing open problems systematically, the paper aims to provide the shared framework that has been missing—not resolving all disagreements, but making them explicit and structured enough that researchers can coordinate around solving them.

The paper's scope is deliberately broad but bounded: it focuses on problems that require mechanistic understanding specifically, distinguishing itself from interpretability work that aims only to explain individual decisions or provide local attributions. And within mechanistic interpretability, it covers methods (Section 2), applications (Section 3), and socio-technical concerns (Section 4), arguing that progress is needed on all three fronts simultaneously:

"The goals, applications, and methods of mechanistic interpretability do not exist in a vacuum. Like any scientific field, they lie within a broader societal context" (Section 1.2).

This tripartite structure—methods, applications, and societal context—reflects the paper's thesis that solving any one category in isolation won't be sufficient. Better decomposition methods without clearer application goals might produce elegant science that doesn't deliver practical benefits. Ambitious application goals without methodological advances are empty promises. And technical progress without attention to governance, communication, and philosophical foundations risks misuse, hype, or misalignment with societal needs.

3. Technical Approach

3.1 Reader Orientation

This paper is not a system paper proposing a single method, but rather a forward-facing review that systematically catalogs the current methodological toolkit of mechanistic interpretability and identifies the open problems preventing those tools from achieving the field's goals. The "system" being described is the conceptual framework and methodological pipeline that mechanistic interpretability researchers use to reverse-engineer neural networks—a three-step cycle of decomposition, description, and validation—along with the alternative concept-based approach that searches for predefined human-interpretable ideas within model internals. The paper's core idea is that progress in mechanistic interpretability requires simultaneous advances along multiple axes (better decomposition, deeper causal descriptions, rigorous validation, automation), and that these methodological improvements must be evaluated against concrete engineering goals rather than pursued in isolation.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's technical content is organized around two complementary methodological paradigms, each with its own workflow:

Paradigm 1: Reverse Engineering (Section 2.1) — identifying the roles of network components by decomposing the network first, then figuring out what each piece does:

Decomposition Module: Takes a trained neural network as input, breaks it into simpler constituent parts (either architectural components like neurons/attention heads, or learned components like sparse autoencoder latents), and outputs a set of components that are hypothesized to be the "atomic units" of computation.
Description Module: Takes each component and generates hypotheses about its functional role—either what causes it to activate (using maximum activating examples, attribution methods, or feature synthesis) or what downstream effects it has (using logit lens, causal interventions, or steering experiments).
Validation Module: Takes the hypotheses from the description module and tests them through convergent methods—predicting activations on new inputs, explaining failure cases, handcrafting replacement components, testing on ground-truth models, or demonstrating downstream engineering utility.

The pipeline is iterative: failed validation returns to either improve the decomposition (Step 1) or revise the functional description (Step 2).

Paradigm 2: Concept-Based Interpretability (Section 2.2) — identifying network components for predefined roles by starting with human concepts and searching for their neural correlates:

Concept Definition: A human researcher specifies a concept of interest and creates a labeling function that assigns concept values to input data.
Probing Module: A simple classifier (typically linear) is trained to predict the concept labels from the network's hidden activations. If successful, the learned probe direction identifies where that concept is represented.
Causal Verification (optional but recommended): Counterfactual interventions test whether the identified representation is causally involved in the network's use of that concept, or merely correlated.

These paradigms are complemented by two cross-cutting themes: circuit discovery pipelines (Section 2.3) that proceduralize the reverse-engineering workflow for specific tasks, and automated interpretability methods (Section 2.4) that use language models or other algorithms to replace manual researcher labor in description and validation steps.

3.3 Roadmap for the Deep Dive

I will explain the technical content in the following order, which mirrors how a researcher would encounter these methods in practice:

The Reverse Engineering Framework (Section 2.1 overall): The three-step cycle that structures most mechanistic interpretability work—decomposition, description, validation—and why each step is necessary given that AI systems use representations and algorithms that differ from human cognition.
Step 1: Decomposition Methods (Section 2.1.2): The current state-of-the-art (sparse dictionary learning with sparse autoencoders) and its unresolved problems, because decomposition is the foundation that all downstream interpretation depends on.
Step 2: Description Methods (Section 2.1.3): How researchers generate hypotheses about what causes components to activate and what effects they produce, plus the limitations of current correlational and attribution-based approaches.
Step 3: Validation Methods (Section 2.1.4): How hypotheses are tested, why validation is underemphasized in current practice, and the role of model organisms and benchmarks.
Concept-Based Interpretability (Section 2.2): The alternative paradigm of probing for predefined concepts, its relationship to reverse engineering, and its specific challenges around causal relevance.
Circuit Discovery as a Proceduralized Pipeline (Section 2.3): How the above steps are combined in practice for task-specific analysis, and where current pipelines break down.
Automation (Section 2.4): How individual steps are being automated using language models and gradient-based approximations, and what's still missing.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a forward-facing review paper whose core technical contribution is a systematic taxonomy of methodological approaches in mechanistic interpretability and a careful enumeration of their unresolved problems, rather than the introduction of a novel algorithm or system.

The Two Paradigms: Reverse Engineering vs. Concept-Based Interpretability

The paper structures its methodological survey around a fundamental distinction between two ways of approaching interpretability (Section 2, Figure 1):

Reverse engineering starts with the components of the network (neurons, attention heads, SAE latents) and asks: "What does this component do?" It is bottom-up: decomposition precedes functional understanding.

Concept-based interpretability starts with human-defined concepts (sentiment, part-of-speech, truthfulness, "refusal to answer harmful requests") and asks: "Where in the network is this concept represented?" It is top-down: the role is specified first, and the search finds which components implement it.

The paper treats these as complementary rather than competing. Reverse engineering is necessary because AI systems may use unfamiliar concepts that humans wouldn't think to search for (Section 2.1.1): "To grasp the potentially alien cognition of these models, we must develop methods to uncover and understand the previously unknown concepts and mechanisms implemented within them." Concept-based interpretability is useful when we have specific safety-relevant or scientifically interesting concepts we want to monitor or control. The two approaches can inform each other: concept-based probes can generate hypotheses about component function, and reverse-engineered descriptions can reveal concepts to probe for.

Why Reverse Engineering Is Necessary: The Alien Cognition Argument

The paper devotes Section 2.1.1 to justifying why reverse engineering—as opposed to simply assuming models think like humans—is essential. This is not a methodological detail but a foundational premise:

Empirical evidence that models and humans reason differently:

A model 1% the size of GPT-3 outperforms humans on next-token prediction tasks (Shlegeris et al., 2024), while state-of-the-art multimodal LLMs struggle with causal reasoning tasks that four-year-old children master (Kosoy et al., 2023).
Image models trained on ImageNet rely more heavily on texture than shape when classifying objects (Geirhos et al., 2019)—a strategy humans rarely use—and exploit dataset-specific correlations like recognizing fish by the fingers of fishermen holding them (Brendel & Bethge, 2019).
Even on simple algorithmic tasks like modular addition, a small transformer learned a Fourier transform strategy that researchers only understood retrospectively (Nanda et al., 2023a)—the model found a mathematical structure that wasn't anticipated by its designers.

Implication for methodology: If models use algorithms and representations that differ from human intuition, then interpreting them requires methods that can discover computational structure rather than merely verify the presence of expected patterns. This is the core motivation for the decomposition-first approach of reverse engineering.

Decomposition: Carving Neural Networks at Their Joints

The paper's treatment of decomposition (Section 2.1.2) is the most technically detailed portion of the methods discussion, reflecting the field's current focus on this as the foundational unsolved problem.

The Neuron Doctrine and Its Failure

The simplest decomposition—interpreting individual neurons—was inspired by the "Neuron Doctrine" in neuroscience, which posits that single neurons are the fundamental functional units of the nervous system. Early interpretability work studied individual neuron responses (Erhan et al., 2009; Zhou et al., 2015; Bau et al., 2020) and individual convolutional filters (Olah et al., 2017a, 2020a; Cammarata et al., 2020).

However, researchers discovered that individual neurons are polysemantic: they respond to multiple, seemingly unrelated features. The paper cites evidence for this in both artificial networks (Wei et al., 2015; Nguyen et al., 2016c; Olah et al., 2017a) and biological neural recordings (Churchland & Shenoy, 2007; Rigotti et al., 2013). The same problem afflicts attention heads, which also exhibit polysemanticity (Janiak et al., 2023). Furthermore, individual layers are too coarse-grained—work on model editing shows that intervening on entire layers produces unwanted side effects (Meng et al., 2022b; Wang et al., 2023), and some representations appear to span multiple layers (Yun et al., 2021; Lindsey et al., 2024).

The failure of architectural decomposition motivates the search for alternative bases—sets of directions in activation space that correspond to the "true" computational primitives.

Dimensionality Reduction Methods and Their Limitations

Early alternative decomposition methods applied unsupervised dimensionality reduction to collections of hidden activations collected from diverse inputs. The intuition: if structure in the activations corresponds to structure in the computation, then principal components or similar decompositions should reveal the functional units.

The paper lists commonly used techniques including Principal Component Analysis, Singular Value Decomposition (Hollinsworth et al., 2024; Marks & Tegmark, 2024; Huang et al., 2024a; Bushnaq et al., 2024), and non-negative matrix factorization (Olah et al., 2018; Voss et al., 2021; Cammarata et al., 2020).

The critical limitation: dimensionality reduction cannot identify more directions than there are dimensions. If a layer has 768 dimensions, PCA can find at most 768 principal components. The superposition hypothesis (discussed below) suggests models represent more features than they have dimensions, meaning dimensionality reduction fundamentally cannot recover the full set of computational primitives.

The paper notes these techniques are "no longer predominant methods used for mechanistically decomposing language models" (Section 2.1.2b), having been largely superseded by sparse dictionary learning.

The Superposition Hypothesis

The superposition hypothesis (Elhage et al., 2021) provides the theoretical motivation for sparse dictionary learning. The paper explains it through three interconnected ideas (Figure 3):

The Linear Representation Hypothesis. This hypothesis observes that, despite neural networks being highly nonlinear functions, their internal representations exhibit strikingly linear structure. High-level concepts appear to be encoded as directions (vectors) in activation space, such that the composition of multiple concepts corresponds to addition of their feature vectors, and the intensity of a concept corresponds to the magnitude (scale) of its vector.

The paper is careful to note that the "strong version" of this hypothesis—that all concepts are linearly represented—is "false for some models" (Section 2.1.2c, citing Black et al., 2022; Csordás et al., 2024). The weaker version—that some concepts are linearly represented—is supported by the empirical success of linear probes, activation steering, and sparse autoencoders.

Superposition. If concepts are represented as directions, and the number of concepts a model needs to represent exceeds the dimensionality of its activation space, the model faces a compression problem. The superposition hypothesis states that models solve this by representing features in a sparse, overlapping manner: each individual feature activates rarely (sparsity), and multiple features share the same dimensions (superposition), with the model exploiting the fact that when one feature is active, others are typically inactive to avoid interference.

This is why dimensionality reduction fails: a 768-dimensional space might represent thousands or tens of thousands of features in superposition, but PCA can only recover 768 orthogonal directions.

The SDL Solution. Sparse Dictionary Learning (SDL) attempts to recover the overcomplete basis of features from activations that are linear combinations of sparse feature vectors. Because the encoder is nonlinear, it can (in principle) learn to detect when a particular feature is active even when it's mixed with others in the activation vector.

Sparse Autoencoders: The Dominant SDL Implementation

The paper describes Sparse Autoencoders (SAEs) as the most popular SDL variant (Section 2.1.2c). The architecture and training procedure are:

Architecture. An SAE is a two-layer neural network inserted at a specific layer of the target model:

Encoder: Takes the hidden activation vector $x \in \mathbb{R}^d$ (where $d$ is the model dimension, e.g., 768 for GPT-2 small) and maps it to a much higher-dimensional latent space $z \in \mathbb{R}^m$ where $m \gg d$ (e.g., $m = 16 \times d$ or larger). The encoder applies a linear transformation followed by a nonlinearity (typically ReLU) to produce sparse latent activations.
Decoder: Takes the latent activations $z$ and maps them back to the original dimension $\hat{x} \in \mathbb{R}^d$ via a linear transformation (a matrix of learned "dictionary" directions). The goal is $\hat{x} \approx x$ .
Sparsity constraint: The encoder activations are trained to be sparse—most latent dimensions should be zero (or near-zero) for any given input. This is enforced through an L1 penalty on the latent activations added to the reconstruction loss:

$\mathcal{L}_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \lambda \sum_{i=1}^{m} |z_i|$

where $x$ is the original activation vector, $\hat{x}$ is the reconstruction, $z_i$ is the activation of latent $i$ , and $\lambda$ is a hyperparameter controlling the sparsity penalty strength.

What it computes: the mean-squared error between the original activation and the SAE's reconstruction, plus a penalty proportional to the sum of absolute latent activations. The reconstruction term ensures the latents collectively capture the information in the activation vector; the sparsity term ensures each latent activates only for a narrow range of inputs, encouraging the decoder directions to align with individual features.

Why this form: the L1 penalty is the standard convex relaxation for encouraging sparse solutions. Alternative sparsity-inducing objectives exist (e.g., TopK activation where only the k-largest latents are kept and the rest zeroed), which the paper references (Gao et al., 2024; Rajamanoharan et al., 2024) as potentially better-performing variants.

Variants. The paper mentions several SAE variants without detailed equations:

Transcoders (Dunefsky et al., 2024): Instead of reconstructing the input activation, these reconstruct the output of the MLP or attention layer that processes that activation. This is motivated by the idea that features should be defined by their computational role, not just their informational content.
Crosscoders (Lindsey et al., 2024): These can take inputs from and reconstruct activations across multiple layers simultaneously, addressing the fact that features may be distributed across layers rather than confined to a single one.
Gated SAEs and other architectural variants are mentioned as ongoing work (Rajamanoharan et al., 2024).

SDL Training Data and Procedure

The paper does not provide explicit training hyperparameters for SAEs, as it is surveying methods rather than reporting new experiments. However, it describes the general procedure:

Data collection: Run the target model on a corpus of text (typically the pretraining corpus or a diverse web crawl) and collect the hidden activations at the target layer.
Training: Train the SAE to reconstruct these activations with sparsity. The SAE typically has more parameters at that layer than the original model (since the latent space is overcomplete), meaning training SAEs for all layers of a large model is computationally expensive.
Interpretation: After training, each latent is interpreted by examining which input contexts cause it to activate (see Section 2.1.3a below) and, optionally, studying its decoder direction's effect on model outputs via the logit lens (see Section 2.1.3b below).

Practical and Conceptual Limitations of SDL

The paper enumerates eight specific problems with SDL, which I'll detail because they constitute the primary open methodological problems the field faces:

1. Reconstruction errors are too high. The paper reports the starkest quantitative finding: "When a sparse dictionary with 16 million latents was inserted into GPT-4, the language modeling loss was equivalent to a model with only 10% of GPT-4's pretraining compute" (Gao et al., 2024). In GPT-2 small, using SAE reconstructions decreased performance by 10% on task-specific data and 40% on the full distribution (Makelov et al., 2024). This means the SAE is systematically missing structure in the activations. The paper notes that "error nodes" (Marks et al., 2024) attempt to account for the discrepancy but "contain 'everything else', making them an inadequate solution" (Section 2.1.2c). Engels et al. (2024b) found the errors are not random—their direction and norm can be linearly predicted from the original activation—suggesting a systematic failure rather than irreducible noise.

2. SDL is expensive to apply to large models. Training an SAE for every layer of a frontier model requires "more parameters at that layer than the original model does" and is "relatively expensive to train compared to the original model" (Section 2.1.2c). The paper notes there are no public attempts to apply SDL to every vector space in a frontier model.

3. SDL assumes the linear representation hypothesis in nonlinear models. The entire approach rests on the idea that the encoder can recover linear feature directions from nonlinear combinations. But the paper acknowledges that neural networks "could potentially use highly nonlinear representations" (Section 2.1.2c, citing Black et al., 2022; Engels et al., 2024b; Kirch et al., 2024). If concepts are represented nonlinearly, an SAE trained with a linear decoder will fail to capture them regardless of dictionary size.

4. Sparsity is not a good proxy for interpretability. The SAE objective optimizes for sparse latent activations under the assumption that sparser = more interpretable. But the paper documents counterexamples: "feature splitting" (Bricken, 2023) where a single true feature gets divided across multiple SAE latents, "feature absorption" (Chanin et al., 2024) where latents blend multiple unrelated features, and "composition" artifacts (Till, 2024). With sufficient optimization pressure toward sparsity, these artifacts can increase—sparser solutions can be less interpretable. The paper suggests alternative proxies like minimum description length (Ayonrinde et al., 2024) but notes this remains unresolved.

5. SDL leaves feature geometry unexplained. SAEs produce a set of latent directions—effectively a "bag of features" with no structure relating them. However, evidence suggests that "the geometric arrangement of features in relation to each other seems to reflect semantic and functional structure" (Engels et al., 2024a; Gurnee & Tegmark, 2024; Park et al., 2024b). If understanding networks requires understanding why certain features are close together in activation space while others are orthogonal, SAEs don't provide this information. The paper raises the possibility that "if only local geometric relationships between features need to be understood"—which features interact with which—"understanding networks with a 'bag of features' approach may be more feasible" (Section 2.1.2c). But if global geometry matters, current methods are fundamentally limited.

6. SDL cannot be straightforwardly applied to all architectures. The paper identifies two specific challenges: attention head superposition (Jermyn et al., 2023; Janiak et al., 2023), where features may be distributed across separate attention heads rather than across neurons within a single layer, and cross-layer superposition (Yun et al., 2021; Lindsey et al., 2024), where features span multiple layers. It is "not immediately obvious how to decompose representations distributed across attention heads with SDL" (Mathwin et al., 2024; Wynroe & Sharkey, 2024), and "nor is it straightforward to translate cross-layer distributed representations into causal descriptions of neural network mechanisms" (Section 2.1.2c).

7. SDL describes activations, not mechanisms. The paper makes a crucial conceptual distinction: "SDL identifies directions in activation space. Being activations, they only interact with the network's mechanisms, but are not the mechanisms themselves" (Section 2.1.2c). Even a perfect SAE tells you what features are present in the network's representations, but not how the weights compute those features from previous-layer activations. Understanding the mechanisms requires additional post-hoc analysis—tracing how SDL latents connect across layers, which circuits they participate in—which "can be labor intensive, computationally expensive, or data set dependent" (Huben et al., 2024; Bricken, 2023; Riggs et al., 2023; Marks et al., 2024). The paper cites Chughtai & Bushnaq (2025) for the observation that "we primarily focus on understanding neural network activations, with little attention paid to how this structure in activations is computed via weights."

8. SDL latents may not contain the concepts needed for downstream use cases. The paper gives the example of Kantamneni et al. (2024), who found a single SAE latent that outperformed official dataset labels on a grammatical acceptability task. But "more often than not, a sparse set of latents that encode some useful concept of interest do not exist" (Section 2.1.2c). The paper lists several hypotheses for why: the SAE might capture how the model "thinks about" the concept, which differs from the human definition; the training distribution might be too narrow (Kissane et al., 2024c found that SAEs trained on pretraining data lack good latents for "refusing harmful requests" while those trained on chat data possess them); or the dictionary might be too small. Importantly, the paper notes that "the latents identified depend on the data set used to train them"—an undesirable property for a method "initially hoped to be capable of identifying the fundamental units of computation in neural networks" (Section 2.1.2c).

The Need for Theoretical Foundations

The paper identifies a deeper problem underlying all these limitations: the field lacks solid theoretical foundations for what a "feature" even is (Section 2.1.2d). Despite being the central object of study, "satisfying formal definitions are elusive and conceptual foundations are not yet established." The paper is candid about this:

"Without solid conceptual foundations, it remains unclear whether the superposition hypothesis, which underpins the SDL paradigm, is fundamentally valid or merely pragmatically useful."

Alternative theoretical frameworks are mentioned but have not been productively connected to interpretability practice: spline theory (Balestriero & richard baraniuk, 2018), the neural tangent kernel (Jacot et al., 2018), singular learning theory (Watanabe, 2009; Wei et al., 2023). The paper calls for establishing these connections as a path toward "carving trained networks at their joints," which would both improve post-hoc decomposition and suggest ways to train networks that are "pre-carved."

Intrinsic Interpretability: Building Easier-to-Decompose Models

Section 2.1.2e considers the alternative to post-hoc decomposition: training models to be interpretable from the start. The paper lists several approaches without detailed equations:

Discrete codes: Forcing activations to use a discrete codebook (Tamkin et al., 2025) rather than continuous vectors.
Sparser activation functions: Using TopK (Makhzani & Frey, 2013; Bills et al., 2023) or SoLU (Elhage et al., 2022) in place of standard activations to reduce superposition.
Mixture of Experts (MoE): Using sparsely-gated experts (Shazeer et al., 2017; Fedus et al., 2022; He, 2024) where "with a large enough number of experts, and with sufficient activation sparsity, experts may become individually interpretable."
Weight sparsity: L0 regularization (Louizos et al., 2018), pruning (Mozer & Smolensky, 1988; Frankle & Carbin, 2019; Mocanu et al., 2018), or magnitude pruning with finetuning (Han et al., 2015).
Modularity constraints: Brain-inspired training that embeds neurons in geometric space and encourages local connections (Liu et al., 2023), or gradient routing that applies data-dependent masks (Cloud et al., 2024).
Linearized architectures: Removing layer norm (Heimersheim, 2024) or using input-switched affine transformations (Foerster et al., 2017) to simplify linear analysis.
Mathematically analyzable components: Bilinear MLP activations (Sharkey, 2023; Pearce et al., 2024) that admit different kinds of analysis than standard ReLU networks.

The paper notes a key tension: "Many attempts to incentivize interpretable activations directly so far have not been competitively performant, and have also allowed 'superposition to sneak through', mitigating benefits" (Section 2.1.2e). The trade-off between interpretability and performance remains unresolved.

Description: Generating Hypotheses About Component Function

Once the network is decomposed into components (whether neurons, attention heads, or SAE latents), the next step is to describe what each component does. The paper divides description methods into two categories: those that explain what causes a component to activate (upstream, input-facing) and those that explain what effects a component has (downstream, output-facing). Figure 5 provides a visual schema: the component of interest sits in the middle, with upstream causes on the left and downstream effects on the right.

Explaining What Causes Components to Activate (Section 2.1.3a)

Method 1: Highly activating dataset examples. The simplest approach: collect inputs on which the component's activation is strongest, then look for common patterns across those inputs. This can be done by humans visually inspecting text or images, or by language models automatically generating descriptions (Section 2.4).

The paper identifies three problems:

Human projection bias: Because "the method relies on human prior beliefs," interpreters may "project their human understanding onto models that may, in fact, be using unfamiliar concepts" (Section 2.1.3a). We see what we expect to see.
Interpretability illusions: Bolukbasi et al. (2021) demonstrated that "bias in data sets can create misleading explanations even when top activating examples are selected from real data sets." Depending on the dataset from which examples are drawn, "human annotators identified dramatically different meanings for given directions in the activation space of BERT."
Plausible explanations for arbitrary directions: Szegedy et al. (2014) showed it's possible to find convincing patterns for any randomly chosen direction in activation space. This means "plausible explanations based on highly activating data set examples cannot be solely relied upon to identify the basic units of computations in neural networks—other methods are needed to accurately identify them." The threat is particularly concerning because Geirhos et al. (2025) showed it's possible to "develop adversarial models that deliberately yield misleading feature visualizations."

A critical limitation: these are "merely correlational explanations... rather than causal explanations" (Section 2.1.3a). They show what inputs accompany the component's activity, not what causes it.

Method 2: Attribution methods. These attempt to measure causal importance of upstream variables on a downstream target (like a component's activation). The paper lists two categories:

Gradient-based methods: Simonyan et al. (2014a), Nguyen et al. (2016b), Selvaraju et al. (2019), Sundararajan et al. (2017), Wang et al. (2024d). These compute the gradient of the target with respect to the input and use it as a measure of importance.
Perturbation-, sampling-, or ablation-based methods: Fong & Vedaldi (2017), Vig et al. (2020), Geiger et al. (2020), Ghorbani & Zou (2020), Meng et al. (2022b), Chan et al. (2022a), Nanda (2023a). These systematically modify or remove components and observe the effect.

Problems identified:

First-order approximation: "On a theoretical level, many gradient-based methods identify only a first-order approximation of the ideal attribution, which is sometimes a poor approximation" (Watson, 2022).
Model independence failure: Adebayo et al. (2018) showed that "some gradient-based methods identify attributions that are independent both of the model and of the data generating process"—the attributions don't change when you randomize the model weights, indicating they're not measuring anything real.
Adversarial manipulability: "An adversary can train a model or perturb an input to reveal any attribution map" (Dombrowski et al., 2019; Ghorbani et al., 2019; Heo et al., 2019; Kindermans et al., 2019; Slack et al., 2020; Zhang et al., 2020). If attributions can be arbitrarily manipulated, they can't be trusted as faithful explanations.
Off-distribution behavior: Perturbation methods "take models off their training distribution and elicit unusual behavior" (Feng et al., 2018; Molnar et al., 2021; Hooker et al., 2021; Molnar et al., 2024; Freiesleben & König, 2023; Slack et al., 2021). If we're trying to understand what the model does on its normal operating distribution, testing it on unnatural ablated states may give misleading results.

Method 3: Feature synthesis. This approach "attempts to synthesize inputs that maximize the activation of a component subject to some regularization, such as consistency with a generative model (Nguyen et al., 2016a; 2017) or total variation distance (Mahendran & Vedaldi, 2014)" (Section 2.1.3a). The idea is to generate the "prototypical" input for a feature rather than relying on dataset examples.

Problems: Some work suggests "natural dataset examples may serve interpretation better" (Zimmermann et al., 2021; Borowski et al., 2021), and current methods "struggle to identify trojans" (Casper et al., 2023a)—they may fail to surface rare but important activation patterns.

Explaining the Downstream Effects of Components (Section 2.1.3b)

Method 1: The logit lens and its variants. The logit lens (Nostalgebraist, 2020) is a technique for projecting intermediate representations into the model's output vocabulary space to see what the representation "would predict" if it were the final layer:

Take an intermediate residual stream activation vector $h \in \mathbb{R}^d$ from some layer.
Apply the model's unembedding matrix $W_U \in \mathbb{R}^{V \times d}$ (where $V$ is vocabulary size) to produce logits over the vocabulary: $\text{logits} = W_U h$ .
Apply softmax to get a probability distribution over tokens.

This answers: "If the model were to output a token based solely on this intermediate representation (ignoring all subsequent layers), what would it predict?"

Direct logit attribution (Geva et al., 2021, 2022b,a; Dar et al., 2023) generalizes this: apply the unembedding matrix to any vector in the model's residual stream—MLP outputs, attention outputs, or even the decoder weights of SAE latents (Bricken, 2023; Bloom & Lin, 2024a) or the components of gradients (Katz et al., 2024).

The tuned lens (Belrose et al., 2023a; Yom Din et al., 2024) improves decoding accuracy by adding a trainable affine or linear transformation before the unembedding matrix. This compensates for the fact that intermediate representations are not directly optimized to produce output logits. The paper notes a tradeoff: the tuned lens is better at decoding but "at the cost of less faithfully representing when the model has completed computation" (Section 2.1.3b)—it may artificially "read out" information that the model hasn't finished processing.

Causal interpretation: The paper frames the logit lens in causal terms (Pearl, 2009): unembedding a residual stream vector measures the direct effect of that vector on the output—what happens if we bypass all subsequent layers. However, it "cannot measure the indirect effect, the effects resulting from the influence that the embedding has on the hidden activations of subsequent layers" (McGrath et al., 2023). To capture total effects (direct + indirect), causal intervention methods are needed.

Method 2: Causal interventions. These methods modify ("patch") the value of a network component during a forward pass and observe the effect on downstream components or the final output. The paper lists a family of related techniques:

Ablation: Zeroing out or otherwise deleting activations entirely.
Activation patching: Replacing an activation with its value from a different ("corrupt") input while keeping everything else the same.
Causal mediation analysis: A specific statistical framework for decomposing total effects into direct and indirect components (Vig et al., 2020).
Causal tracing: Identifying which components are necessary for factual recall by systematically ablating and measuring effect (Meng et al., 2022a).
Interchange interventions: Replacing activations with those from inputs where a specific concept has been altered, to test whether that component encodes the concept (Geiger et al., 2020).

The paper gives a concrete example of activation patching: "by patching activations from the corrupt prompt 'the capital of Italy is' into the clean prompt 'the capital of France is', we can observe the effect on the output ('Rome' vs. 'Paris'). This tells us which component values are relevant for the differing output between the two prompts, but not the information that remains consistent (e.g. the fact that the answer is a city)" (Section 2.1.3b, footnote 5).

Path patching (Goldowsky-Dill et al., 2023) is a more surgical variant: instead of modifying a component and observing its effect on the entire rest of the network, path patching isolates the effect of one component on one other component by creating a "path" from source to target while leaving other pathways unmodified.

Causal scrubbing (Chan et al., 2022a) is a generalization that tests hypotheses about any given connection between a set of network components.

Distributed alignment search (Geiger et al., 2024c) uses causal interventions as supervision to learn the subspace representing a particular concept, rather than assuming it's a single direction. The idea: find a linear subspace such that intervening on the concept in the input produces a corresponding change in this subspace, and intervening on the subspace produces a corresponding change in the output.

Learning masks over components: De Cao et al. (2020), Csordás et al. (2021), Davies et al. (2023) use differentiable masks learned through optimization to identify which components are relevant for a task, rather than testing them one by one.

Computational cost problem: "A causal intervention typically requires a forward pass of the model. This may make performing one for every network component in large models, long contexts, or when using finer-grained components such as sparse autoencoder latents prohibitively expensive" (Section 2.1.3b). For a model with thousands of components and a task requiring hundreds of examples, full causal analysis is intractable.

Approximations: To address the cost, the paper mentions gradient-based approximations including:

Attribution patching (Nanda, 2023a; Syed et al., 2024): Uses gradients to estimate the effect of ablating a component without actually doing the forward pass.
AtP* (Kramár et al., 2024): An improved variant of attribution patching.
Integrated gradients (Sundararajan et al., 2017): Accumulates gradients along a path from a baseline input to the actual input.

These are acknowledged to be approximations—"attribution patching uses gradients, which only yield a first-order approximation of the effect of ablating components" (Section 2.3)—but are considered necessary for scaling to large models and many components.

Method 3: Observing effects through sequential behavior. Rather than analyzing static representations, these methods observe what happens when a component's activity is modified and the model generates text:

Activation steering (Rimsky et al., 2024; Turner et al., 2024): Adding a fixed vector to an intermediate activation during inference to influence the model's behavior. If adding the vector for "honesty" makes the model more truthful, that's evidence the vector represents (or causally influences) honesty-related computation. This technique is discussed more extensively under applications (Section 3.2.2).
Patchscopes (Ghandeharioun et al., 2024; Chen et al., 2025; Watkins, 2023; Huang et al., 2024b; Kharlapenko et al., 2024): A method where activations from one forward pass are "patched" into a different forward pass (potentially in a different model) designed to elicit information about the original activation. For example, an activation from a reasoning task might be patched into a model processing a prompt asking "what concept does this represent?" to get a natural language description.
Chain-of-thought reading: Simply reading the model's generated reasoning steps. The paper notes this is "not entirely faithful to the model's underlying decision-making process" (Agarwal et al., 2024; Atanasova et al., 2023; Turpin et al., 2023; Lanham et al., 2023; Ye & Durrett, 2022)—models can generate plausible-sounding reasoning that doesn't reflect their actual computation. The paper suggests "incorporating model internals into chain-of-thought training" as a future direction to incentivize faithfulness.

Validation: Testing Whether Our Hypotheses Are Correct

Section 2.1.4 addresses what the paper frames as perhaps the field's most neglected step: confirming that the descriptions we generate are actually true. The opening is unusually direct:

"Conflating hypotheses with conclusions has regrettably been commonplace in mechanistic interpretability research, making validation an important area for the field to improve."

The core question: "Does the hypothesis make good predictions about the neural network's behavior?" (Section 2.1.4). The paper emphasizes that "testing the hypothesis often requires multiple approaches"—no single validation method is sufficient, because any method can fail in ways that produce misleadingly positive results.

Validation Method 1: Using a different description method than was used to generate the hypothesis. The simplest form of validation is convergent evidence: if you hypothesized a component's function based on maximum activating examples, check whether causal interventions on that component produce the expected behavior. If they disagree, the hypothesis is invalidated, and you return to an earlier step in the reverse engineering cycle (Figure 2).

Validation Method 2: Predicting activations from natural language descriptions. If you've described a component's function in natural language, that description should enable you (or a language model) to predict how strongly the component will activate on novel inputs it wasn't originally interpreted on. The paper cites Hernandez et al. (2022), Bills et al. (2023), Shaham et al. (2025), and Juang et al. (2024) as examples of this approach. The prediction accuracy can be quantified, providing a metric for description quality.

Validation Method 3: Predicting counterfactual behaviors. Good explanations should enable predictions about what happens under interventions: "For instance, if we ablate or activate particular network components, we should be able to predict specific downstream effects on other components" (Section 2.1.4). If the description says "this component detects curves," ablating it should selectively impair curve-related tasks but leave other vision tasks intact. If it impairs everything, the description is too vague to be useful.

Validation Method 4: Predicting and explaining failures or adversarial examples. Hilton et al. (2020) validated their interpretation of a deep RL agent by explaining specific failure cases and hallucinations. Carter et al. (2019), Casper et al. (2023c), Mu & Andreas (2020), and Hernandez et al. (2022) used interpretability to handcraft adversarial examples that exploited the model's specific computational strategy. Being able to design inputs that break the model in predicted ways is strong evidence you understand how it works.

Validation Method 5: Handcrafting replacement components. Cammarata et al. (2020) validated their interpretation of a curve detector in a CNN by building a simple handcrafted detector that replaced the original component while preserving overall model function. This is the strongest form of validation in some sense: if you can build it, you understand it.

Validation Method 6: Testing on ground truth. If a model has known ground-truth mechanisms (because it was handcrafted or compiled from a symbolic program), interpretability methods can be evaluated by checking whether their descriptions match the known ground truth. The paper cites Conmy et al. (2023) validating their circuit discovery tool on a simple model implementing a known algorithm, and discusses benchmarks built from compiled programs (Lindner et al., 2023; Weiss et al., 2021; Thurnherr & Scheurer, 2024; Gupta et al., 2024) and interchange intervention training (Gupta et al., 2024; Geiger et al., 2022).

Validation Method 7: Achieving engineering goals. The paper argues that the highest standard of validation is demonstrating that interpretability enables useful interventions that wouldn't be possible otherwise. For example:

Templeton et al. (2024) showed that manually editing an LLM based on SAE interpretations produced "predictable high-level changes in its behavior."
Marks et al. (2024) showed an interpretability tool helping humans debug a classifier.
Farrell et al. (2024) used unlearning experiments to demonstrate that SAE latents don't perfectly match human concepts.

The paper is careful to distinguish between "achieving competitive methods" versus cherry-picked demonstrations. It notes that "the practice of conducting evaluations using non-cherry-picked tasks remains relatively uncommon" and that current interpretability tools "have not proven to be consistently useful" when evaluated against appropriate baselines (Adebayo et al., 2020; Denain & Steinhardt, 2023; Casper et al., 2022b; Hase et al., 2023; Durmus et al., 2024).

Model Organisms

The paper proposes "model organisms" as a strategy to facilitate validation (Section 2.1.4a). The analogy is to biology, where researchers study a few extensively characterized species (E. coli, fruit flies, mice, macaques) to develop tools and insights that transfer to other organisms.

Desiderata for a mechanistic interpretability model organism: "open source, easy and cheap to use, representative of a broad range of systems and phenomena, have a replicable training process with open source training data, and have multiple instances with different random seeds" (Sharkey et al., 2022a).

The paper notes that current practice has mostly used models meeting only some criteria: a small transformer trained on modular addition (Nanda et al., 2023a) and GPT-2 (Radford et al., 2018). The advantage: model organisms allow "cross-checking results with previous work, enabling stronger validation of hypotheses" (Section 2.1.4a).

A risk the paper identifies: "Studying solely model organisms, instead of more directly pursuing engineering goals, risks merely making true statements about neural network structure, rather generating insights that are of immediate practical benefit." An additional risk: focusing on small toy models "risks incentivizing research and methods that fail to generalize to more safety-relevant real-world settings" (Section 2.1.4a).

Benchmarks for Interpretability Methods

Beyond validating individual hypotheses, the paper discusses benchmarks for evaluating entire interpretability methods (Section 2.1.4b):

Ground-truth benchmarks: Models with known explanations—either compiled from symbolic programs (Lindner et al., 2023; Weiss et al., 2021; Thurnherr & Scheurer, 2024; Gupta et al., 2024) or trained to have specific structure via Interchange Intervention Training (Geiger et al., 2022)—can serve as testbeds where the ground truth is known.

Method-specific benchmarks:

Decomposition quality (Huang et al., 2024a; Makelov et al., 2024)
Description generation (Schwettmann et al., 2023)
Natural language explanation testing (Huang et al., 2023)

Concept-Based Interpretability: The Alternative Paradigm

Section 2.2 describes the alternative to reverse engineering: starting with a human-defined concept and searching for its neural correlate.

Probing

A probe is a classifier trained to predict a concept label from the hidden activations of a frozen model (Section 2.2.1). The procedure:

Define a concept and create a labeling function: For each input in a dataset, assign a label indicating the concept's value. For binary concepts (e.g., "is the sentiment positive?"), labels are 0/1; for continuous concepts, labels are real-valued.
Train a simple classifier: Typically a linear model (logistic regression or linear regression) that takes the hidden activation at some layer as input and predicts the concept label. If the probe is linear and achieves high accuracy, we say the concept is linearly represented at that layer, and the learned weight vector identifies the direction corresponding to the concept.
Interpret the learned direction: The weight vector (or, in the binary case, the difference between positive and negative class means) becomes the hypothesized "concept vector." This vector can be used for downstream applications like steering or editing.

Alternative probing approaches mentioned:

Concept Activation Vectors (CAVs) (Kim et al., 2018a): Training a linear classifier to distinguish between concept-present and concept-absent examples, then using the learned direction as the concept vector.
Information-theoretic probing (Voita & Titov, 2020): Measuring mutual information between representations and concepts.
Structural probing (Hewitt & Manning, 2019): Testing whether syntactic structure (like parse tree depth) is linearly encoded in embeddings.
Contrast-Consistent Search (CCS) (Burns et al., 2023): Probing for an axis corresponding to positive/negative propositions by enforcing probabilistic consistency conditions, without explicit concept labels. The paper notes this still requires "the construction of data sets with clear positive and negative cases" (Section 2.2.2).

Two Main Challenges with Probing

Challenge 1: Probes need carefully chosen data for well-defined concepts (Section 2.2.2). Creating the labeling function "may require substantial human effort to define a data set for a single concept." More fundamentally, probing "can only identify concepts that we were already looking for, rather than reveal unexpected features in the network." This is the central limitation relative to reverse engineering: probes find what you search for, not what the model actually uses.

A potential solution: developing "methods that automatically develop data sets for probing and concept localization" (Shaham et al., 2025), which blurs the line between probing and reverse engineering.

Challenge 2: Probes detect correlations, not causal variables (Section 2.2.3). This is the most theoretically important limitation. The paper explains:

"Training a probe to associate a concept with specific hidden activations does not necessarily imply that those activations causally mediate how that concept is used by the network, or even if the network uses the concept at all."

From an information-theoretic perspective, a sufficiently powerful probe measures the mutual information between the representation and the concept (Hewitt & Liang, 2019; Pimentel et al., 2020). But mutual information is symmetric and correlational—it tells you that the representation contains information about the concept, not that the model uses that information in its computations. The paper cites evidence that "probes can be successfully trained on hidden activations that lack any causal connection to the output" (Ravichander et al., 2021; Geiger et al., 2024b; Elazar et al., 2021; Belinkov, 2022b).

Improving causal relevance:

Counterfactual data: Use pairs of inputs where the concept is intervened upon (e.g., an image with a dog vs. the same image with the dog replaced by a cat) and ensure the probe direction captures the difference caused by that intervention (Elazar et al., 2021; Mueller, 2024; Geiger et al., 2024b).
Distributed alignment search (Geiger et al., 2024c; Wu et al., 2023; Huang et al., 2024a): Uses counterfactual pairs as supervision to learn a causal subspace.
Causal probing (Guerner et al., 2024): Explicitly incorporates causal assumptions into the probing framework.
Concept erasure methods (Ravfogel et al., 2020, 2022; Elazar et al., 2021; Belrose et al., 2023b): Remove the concept information from representations and check whether the model's behavior changes. If erasing the concept doesn't change behavior, the concept was merely correlated.

The paper acknowledges that "at times, it might be acceptable for probes to identify merely correlated hidden activations if the correlations generalize to the test distribution" (Section 2.2.3). For some applications (monitoring, anomaly detection), knowing the model has information about a dangerous concept is useful even if that information isn't causally used.

Concept-Based Intrinsic Interpretability

Section 2.2.4 briefly notes the possibility of incorporating concepts into training: Concept-Bottleneck Models (Koh et al., 2020) constrain the model to explicitly predict concepts as an intermediate step, and Interchange Intervention Training (Geiger et al., 2022) enforces specific causal structures. The tradeoff, as with other intrinsic interpretability approaches, is performance.

Circuit Discovery: A Case Study in Proceduralizing Interpretability

Section 2.3 provides a concrete example of how the above methods are combined into a pipeline, using the popular "circuit discovery" paradigm (Wang et al., 2023; Hanna et al., 2023; Heimersheim & Janiak, 2023). The goal is to identify and describe the subgraph of model components responsible for a specific task.

Step 1: Task Definition. Choose a task the model can perform and a dataset on which it performs that task. The paper notes this is "a concept-based step, since the definition of the task was based on how human researchers define a task distribution" (Section 2.3). This is a limitation: the decomposition is with respect to human-defined tasks, which may not correspond to the model's natural task decomposition.

Step 2: Decomposition. The network is modeled as a Directed Acyclic Graph (DAG) where nodes are components (typically architectural: attention heads and MLP layers, more recently SAE latents) and edges represent information flow between them.

Step 3: Identify task-relevant vs. task-irrelevant subgraphs. Causal interventions (typically activation patching) are used to determine which nodes and edges are important for the task. The paper mentions two specific approaches:

Iterative activation patching (Wang et al., 2023; Chan et al., 2022a; Lieberum et al., 2023): Systematically test each component or edge, keeping those that affect task performance.
Integrated gradients (Marks et al., 2024): Use gradient-based attribution to identify important components without individual forward passes.

Step 4: Iterative description-validation loop. For each node or edge in the task-relevant subgraph, researchers hypothesize its function and test those hypotheses through custom experiments. The paper notes "this step is less formulaic than previous steps. Researchers rely on their intuition" (Section 2.3). Only after multiple iterations of hypothesis refinement is a satisfactory description reached. The paper cites Chan et al. (2022b) for a detailed account of this process on the induction task and Nanda (2023b) for a meta-description.

Step 5: Final validation. Circuits are evaluated on three criteria (Wang et al., 2023):

Faithfulness: How closely does the circuit's behavior (when run in isolation, with other components ablated) match the full model's behavior?
Minimality: Are any nodes in the identified subgraph unnecessary for the task?
Completeness: Are any nodes outside the identified subgraph actually important for the task?

The paper notes that Shi et al. (2024) developed formal statistical hypothesis tests for evaluating these criteria.

Problems with the Circuit Discovery Pipeline

The paper identifies several issues that limit current circuit discovery work:

1. Task definition is concept-based. This means circuits "provide a good approximation of the average case performance on the dataset, but a poor one for any individual data point" (Miller et al., 2024). The paper suggests "it might be worth learning the task decomposition instead" (Haani et al., 2024)—automatically discovering the tasks the model naturally partitions its computation into.

2. Network decomposition methods are flawed. If the underlying decomposition (architectural components or SAE latents) doesn't carve the network at its joints, then even a perfect circuit identification procedure will find subgraphs that don't correspond to the true computational structure.

3. Circuit faithfulness is low. Miller et al. (2024) show that existing faithfulness metrics "depend on the causal intervention implementation used" and are "misleading when applied to several complex end-to-end circuits." Chan et al. (2022b, 2023) found that even simple early circuits were unfaithful.

4. Scalable methods are only approximate. Attribution patching (Syed et al., 2024) was designed to identify candidate important components without testing each one individually, but "uses gradients, which only yield a first-order approximation of the effect of ablating components" (Wu et al., 2024c; Molchanov et al., 2017). It's unclear whether this approximation is good enough for complex circuits.

5. Backup and negative behavior. Circuit discovery algorithms focused on maximizing task performance may miss "backup" behaviors (where ablating one component causes another to compensate, concealing its true importance) and "negative" behaviors (where components actively suppress incorrect outputs). McGrath et al. (2023) and McDougall et al. (2024) provide empirical evidence for these phenomena.

6. Streetlight interpretability. The paper uses this evocative term to describe the tendency to study tasks that are "simple to define and study mechanistically" (Section 2.3). This gives "a misleading impression of the level of difficulty involved in implementing circuit discovery for any arbitrary task." The paper cites Nanda et al. (2023b) as an example where studying arbitrary circuits proved substantially harder than expected.

Automation: Scaling Interpretability Through AI Assistance

Section 2.4 discusses efforts to automate steps of the interpretability pipeline, motivated by the need to scale to large models and reduce manual researcher effort.

Automating Feature Description and Validation

The paper identifies this as the most successfully automated step. The approach:

Input: A set of highly activating dataset examples for a component (neuron or SAE latent).
Automated description: A language model is prompted to examine these examples and generate a natural language description of what pattern they share. This has been applied to neurons in image models (Hernandez et al., 2022), neurons in language models (Bills et al., 2023), and SAE latents (Huben et al., 2024; Bricken, 2023; Juang et al., 2024).
Automated validation: The natural language description is tested by having a human or model use it to predict the component's activation on new examples. Prediction accuracy serves as a quantitative measure of description quality.

The paper notes that "while imperfect, these methods for interpretation hypothesis generation and validation might be improved by automating the generation of inputs to test the interpretation hypotheses" (Huang et al., 2023), or by "defining more rigorous statistical tests" (Bloom & Lin, 2024b). Future directions include "descriptions of feature effects, relationships between features (Bussman et al., 2024), or how components interact during runtime produce behavior" (Section 2.4).

Automating Circuit Discovery

ACDC (Automated Circuit DisCovery) (Conmy et al., 2023) automates the identification of task-relevant subgraphs (Step 3 in the circuit discovery pipeline). Several works have improved its efficiency (Syed et al., 2024; Kramár et al., 2024; Marks et al., 2024).

The paper is careful to note the limitations: ACDC-like approaches "assist in identifying relevant subgraphs for a pre-defined task, but do not automate important subsequent steps, such as describing the functional role of subgraph components" (Section 2.4). Full automation of current pipelines would not yield satisfactory explanations, because the underlying methods (decomposition, description) are themselves inadequate. The paper cites Marks et al. (2024) as "one attempt at this using leading decomposition and description methods," implying it was not fully successful.

The paper concludes this section with a cautionary note about misaligned AI systems: "Should we take steps to mitigate potentially misaligned AI systems sabotaging AI automated interpretability?" (Appendix A.1.4, Question 4). This reflects a concern that if automated interpretability tools become widely used for safety evaluation, models might learn to generate misleading interpretations to evade detection—a form of interpretability adversarial attack.

4. Key Insights and Innovations

Innovation 1: Mechanistic Interpretability Is Defined by Its Commitment to Explaining Generalization, Not Individual Decisions

The paper makes a crucial taxonomic move that the field had been lacking: it cleanly separates mechanistic interpretability from the broader interpretability literature by what question it asks. Prior categorizations (causal vs. correlational, supervised vs. unsupervised, local vs. global) blurred this boundary, but this paper draws a sharp line: the defining question of mechanistic interpretability is "How did my model solve this general class of problems?" — not "Why did my model make this particular decision?" (Section 1.1.1).

This is fundamental rather than incremental because it changes what counts as a valid explanation. Local attribution methods (integrated gradients, SHAP, LIME) can answer the "why this decision?" question but fail the generalization test: knowing which pixels were important for classifying one image doesn't tell you the algorithm the model uses across all images. The paper traces this commitment to "a fundamental hypothesis in deep learning that generalization arises from shared computation" (Section 1.1.1), grounding mechanistic interpretability in a claim about how neural networks actually work rather than just what questions researchers happen to find interesting.

The significance extends beyond taxonomy. This framing resolves why earlier interpretability paradigms proved insufficient for safety goals: attributing individual decisions doesn't let you predict behavior in novel situations, doesn't reveal latent dangerous capabilities that haven't manifested in test data, and doesn't support claims of the form "there exists no mechanism that would cause the model to deliberately behave undesirably." By tying interpretability to generalization, the paper implicitly argues that only mechanistic understanding — understanding the algorithm, not the inputs — can provide the assurances that high-stakes deployment requires.

Innovation 2: The Field Systematically Underinvests in Validation, and This Creates a Credibility Crisis for Purported Interpretations

This is the paper's sharpest negative result, and arguably its most important conceptual contribution. Rather than proposing a new method, the paper diagnoses a systemic failure mode: researchers routinely "conflate hypotheses with conclusions" (Section 2.1.4), generating plausible-sounding descriptions of component function without rigorously testing whether those descriptions actually predict or control model behavior. The paper catalogs evidence that this isn't merely a theoretical concern: Bolukbasi et al. (2021) showed that interpretations can depend on which dataset examples are shown to annotators; Szegedy et al. (2014) showed that convincing explanations can be generated for arbitrary directions in activation space; Geirhos et al. (2025) demonstrated that models can be adversarially designed to produce misleading visualizations.

This is a fundamental critique because it targets the field's central epistemic practice. If descriptions cannot be trusted, then the entire enterprise — decomposing networks into components, assigning functions to those components, building circuits from those functions — rests on sand. The paper doesn't merely note that validation is hard; it argues that current validation practices are systematically inadequate and that "plausible" and "true" are routinely confused in published work.

The evidence for this claim is structural, not anecdotal. The paper observes that circuit discovery, the field's most prominent pipeline, has produced circuits that are demonstrably unfaithful (Chan et al., 2022b, 2023; Miller et al., 2024). It notes that interpretability tools "have not proven to be consistently useful" when evaluated against appropriate baselines on non-cherry-picked tasks (Section 2.1.4). And it highlights the "streetlight interpretability" problem: researchers study tasks that are easy to define mechanistically, giving the misleading impression that methods work when they may fail catastrophically on arbitrary tasks or individual datapoints.

The paper's proposed solution — convergent validation through multiple independent methods, ground-truth benchmarks, model organisms, and competitive engineering baselines — is less novel than the diagnosis itself. But the diagnosis has practical bite: it implies that much published interpretability work may be describing artifacts of the interpretation method rather than properties of the model, and that the field needs to invest substantially more in validation infrastructure before it can credibly claim to understand even small models, let alone frontier systems.

Innovation 3: The Dominant Decomposition Paradigm (SDL) Rests on a Stack of Assumptions That May Be Collectively Unsound

This is the paper's most technically specific diagnostic insight. Rather than rejecting sparse dictionary learning outright, the paper systematically unpacks the chain of assumptions it depends on — the linear representation hypothesis (concepts are directions), the superposition hypothesis (models represent more features than dimensions by making them sparse), the sparsity-as-interpretability-proxy assumption (sparser = more interpretable), and the bag-of-features assumption (feature geometry doesn't matter) — and shows that each link in this chain has empirical counterevidence or unresolved theoretical questions.

The innovation is not that SDL has problems — individual limitations had been noted in prior work — but the paper's architectural argument that these are not independent bugs to be patched individually. They form a connected structure: if the linear representation hypothesis is false in its strong form (as the paper notes it is for some models), then linear-decoder SDL fundamentally cannot recover the true features regardless of dictionary size or training technique. If sparsity is not a good proxy for interpretability (as feature splitting and absorption demonstrate), then optimizing for sparsity may actively push SDL away from human-interpretable features. If feature geometry matters (as mounting evidence suggests), then a decomposition that produces an unstructured bag of independent directions is missing essential structure. And if SDL discovers at best compositions of true features rather than the true features themselves, then interpreting SDL latents as "the fundamental units of computation" (as the field initially hoped) is a category error.

The quantitative evidence the paper marshals is damning: "When a sparse dictionary with 16 million latents was inserted into GPT-4, the language modeling loss was equivalent to a model with only 10% of GPT-4's pretraining compute" (Section 2.1.2c). This isn't a small gap — it means 90% of the model's effective compute is not captured by the largest SDL decomposition attempted. And the errors are not random noise: Engels et al. (2024b) found they can be linearly predicted, suggesting systematic structure that SDL is blind to.

The significance of this diagnosis is that it redirects research attention. If SDL's problems are architectural rather than merely implementational, then incremental improvements to SAE training (bigger dictionaries, better sparsity penalties, improved architectures) may hit fundamental ceilings. The field may need entirely different decomposition paradigms — ones that do not assume linear representations, that account for feature geometry, that directly decompose weights and mechanisms rather than just activations, or that build interpretability into the training process itself (the "intrinsic interpretability" research program). The paper's careful enumeration of what's broken in the current paradigm provides a roadmap for what any successor approach must address.

Innovation 4: Progress in Mechanistic Interpretability Must Be Evaluated Along Multiple Independent Axes, Not a Single Metric

The paper introduces a diagnostic framework in Section 3.1 that decomposes progress into orthogonal dimensions: decomposition quality vs. description depth, extent of network coverage (single feature vs. circuit vs. whole model), breadth of task distribution analyzed, and whether understanding is post-hoc vs. during training. Each application goal — monitoring for unsafe cognition, controlling behavior, predicting capabilities, microscope AI — requires progress along different combinations of these axes to different degrees.

This is conceptually novel because it rejects the field's implicit assumption that progress is unidimensional — that "better interpretability" means the same thing regardless of what you want to do with it. The paper shows this concretely: for monitoring applications, "even shallow, correlation-based descriptions could signal potentially concerning cognition" (Section 3.1); for formal verification of safety properties, deep mechanistic understanding of the entire network across all task distributions is probably required; for microscope AI (extracting novel scientific insights), a different set of axes entirely may matter (the ability to discover novel concepts rather than merely verify known ones).

The framework's power is that it converts vague disagreements about "are we making progress?" into specific, testable questions: "progress toward which application goal along which axis compared to what baseline?" It also provides a structure for prioritizing research: if the field's most urgent goal is monitoring for deception, then improving decomposition methods (to identify all relevant mechanisms) may be more important than improving description depth (to understand exactly how each mechanism works), because detecting the presence of a deception-related feature is sufficient for raising an alarm even if you don't fully understand its causal role. The framework makes these tradeoffs explicit in a way prior work had not.

Innovation 5: Mechanistic Interpretability Has a Distinctive Comparative Advantage in Uncovering Internal Cognition That Black-Box Methods Cannot Access

The paper makes a strategic argument about where mechanistic interpretability should focus its efforts, grounded in an analysis of the field's unique epistemic position. Section 3.2.1 argues that "the majority of other areas of machine learning already focus on controlling or steering the behavior of AI systems to alter input-output behavior" — but only mechanistic interpretability can address questions about how and why models produce that behavior. This implies the field should prioritize problems where this comparative advantage matters most: "auditing for unsafe cognition, debugging unexpected behavior, and monitoring systems in deployment."

This is a meta-scientific claim about research strategy, not a technical finding. But it has practical implications for what problems the field studies. If mechanistic interpretability competes on behavioral prediction or control, it may lose to simpler black-box methods (fine-tuning, prompting, RLHF) that don't require understanding internals. But if it focuses on detecting internal patterns that have no behavioral signature — sandbagging, hidden deceptive reasoning, latent dangerous capabilities that haven't been elicited — it addresses problems that no other approach can solve.

The paper provides concrete examples of this advantage: using internals to detect when models are intentionally underperforming on evaluations (van der Weij et al., 2024), distinguishing between features that merely recognize deceptive behavior vs. mechanisms that generate it, and providing "mechanistic anomaly detection" (Christiano, 2022; Johnston et al., 2024) that flags when a model takes actions for abnormal reasons even without understanding what those reasons are. These applications exploit the fact that internal representations carry information about model cognition that is not necessarily visible in outputs.

The strategic implication is clear but understated: the field should be wary of benchmarking itself against applications where black-box methods already work well, because interpretability methods are likely to be more expensive and less reliable for those use cases. The distinctive value proposition of mechanistic interpretability is access to information that behavior alone cannot provide, and the field's research agenda should be structured accordingly.

5. Experimental Analysis

Evaluation Methodology

This paper is a forward-facing review that synthesizes existing research rather than presenting new experimental results. Consequently, it does not report original experiments in the traditional sense. However, it cites empirical findings from the literature extensively to support its claims about the current state and limitations of mechanistic interpretability methods. The "experimental analysis" in this review takes the form of evidence synthesis across published work, and the paper's central arguments about what's broken in current methods are grounded in specific quantitative findings that it draws from the literature.

Because the paper's contribution is a diagnostic framework rather than a new empirical result, I will structure this section around the key empirical claims the paper cites as evidence for its diagnosis of the field's open problems, treating each major methodological area (SDL decomposition, circuit faithfulness, probing causal relevance, etc.) as a locus of empirical assessment. Where the paper reports specific numbers from prior work, I will attribute them to the original source while noting how the review marshals them as evidence.

Important caveat: The paper does not introduce a new dataset, model, or controlled experiment of its own. The "metrics" are the performance measures reported in the cited works (reconstruction error, accuracy, faithfulness scores, etc.). The "baselines" vary by the original study. The paper's analytical contribution is in organizing and interpreting these results, not in generating them.

Main Quantitative Results

I organize this section around the key empirical findings the paper uses to motivate its open problems, grouped by methodological area. Each subsection corresponds to a major limitation of current methods that the paper documents with quantitative evidence from prior work.

The Failure of Architectural Decomposition: Neurons and Attention Heads Are Polysemantic

The paper's argument that natural architectural components do not provide interpretable decompositions rests on empirical findings of polysemanticity — individual neurons and attention heads responding to multiple, seemingly unrelated features. The paper cites converging evidence from both artificial and biological networks (Section 2.1.2a):

In artificial neural networks: Wei et al. (2015), Nguyen et al. (2016c), and Olah et al. (2017a) documented that individual neurons in vision models respond to mixtures of features rather than single interpretable concepts. The paper notes this finding "echoes earlier theoretical work that suggested representations used by neural networks do not necessarily align with the activation of individual neurons" (Hinton, 1981).
In biological networks: Churchland & Shenoy (2007), Rigotti et al. (2013), Mante et al. (2013), and Raposo et al. (2014) observed similar mixed selectivity in neural recordings.
For attention heads: Janiak et al. (2023) found that "attention heads also exhibit polysemanticity," and Jain & Wallace (2019) and Pruthi et al. (2020) showed that "studying the attention patterns of models can often be misleading."
Individual layers: Meng et al. (2022b) and Wang et al. (2023) provided evidence that "this level is too coarse-grained to robustly carve the network at its joints."

The paper does not report a single quantitative metric for polysemanticity (e.g., average number of distinct concepts per neuron), but the cumulative weight of evidence across multiple model types and architectures supports its conclusion that architectural components do not constitute the "atomic units" of neural computation.

SDL Reconstruction Error: Quantifying the Information Lost by Current Decompositions

The paper's strongest quantitative critique of SDL comes from reported reconstruction errors, which measure how much model performance degrades when original activations are replaced with their SDL reconstructions. Two specific findings are highlighted (Section 2.1.2c):

Gao et al. (2024) on GPT-4: "When a sparse dictionary with 16 million latents was inserted into GPT-4, the language modeling loss was equivalent to a model with only 10% of GPT-4's pretraining compute." This means replacing the model's true activations with SAE reconstructions degrades performance to the level of a model trained with 90% less compute — a severe loss of information.
Makelov et al. (2024) on GPT-2 small: "Using reconstructions from sparse autoencoders decreased GPT-2 small performance by 10% when trained on task-specific data, and 40% when trained on the full distribution." The gap between task-specific (10%) and full-distribution (40%) degradation suggests SAEs are better at capturing features relevant to narrow tasks than the full breadth of representational content.

The paper uses these findings to argue that SDL "reconstruction errors are too high" to serve as a faithful decomposition for downstream interpretability work. If 40% of GPT-2's effective computation is lost, or 90% of GPT-4's pretraining compute is not captured, then interpretations based on SAE latents may miss the majority of what the model actually computes.

Feature Splitting, Absorption, and Composition: Evidence That Sparsity Undermines Interpretability

The paper documents specific failure modes where optimizing for sparsity produces less interpretable decompositions, contradicting the assumption that sparsity is a reliable proxy for interpretability (Section 2.1.2c):

Feature splitting (Bricken, 2023): "A single true feature gets divided across multiple SAE latents." Instead of one latent cleanly representing "elephant," the concept gets distributed across several latents that collectively encode it, making interpretation harder not easier.
Feature absorption (Chanin et al., 2024): Latents blend multiple unrelated features, so a single latent might activate for elephants, the letter "e," and concepts related to memory, making its activation pattern uninterpretable.
Composition artifacts (Till, 2024): With sufficient optimization pressure toward sparsity, SAE latents can represent combinations of features rather than primitive features, so interpreting them as "atomic" is a category error.

The paper notes that "it is debated whether feature splitting is actually a problem," indicating ongoing disagreement in the field, but the existence of these artifacts undermines the claim that sparser decompositions are automatically more interpretable.

Circuit Faithfulness: Quantifying How Poorly Discovered Circuits Approximate Model Behavior

The paper reports that circuit discovery — the field's most prominent pipeline for producing mechanistic explanations — yields circuits that are demonstrably unfaithful, meaning they do not closely approximate the full model's behavior even when other components are ablated (Section 2.3):

Chan et al. (2022b, 2023): "Simple early circuits were found to be unfaithful" — the initial circuits discovered for induction, considered a paradigmatic success of mechanistic interpretability, did not actually reproduce model behavior when run in isolation.
Miller et al. (2024): "Existing measures of faithfulness depend on the causal intervention implementation used, and further demonstrate that such metrics are misleading when applied to several complex end-to-end circuits." This is a double critique: not only are circuits unfaithful, but the metrics used to evaluate faithfulness are themselves unreliable.
Within-task variance: Miller et al. (2024) also showed that "the within-task variance of model performance across the distribution of data points in a task is large, implying that the circuit provides a good approximation of the average case performance on the dataset, but a poor one for any individual data point." This means circuit-level explanations may be valid on average but fail to capture the model's behavior on specific inputs — a serious limitation for safety applications where worst-case behavior matters.

Probing Detects Correlation, Not Causation: Evidence of the Causal Relevance Gap

The paper marshals several lines of evidence that linear probes — a widely used tool for identifying concepts in neural networks — detect correlations rather than causally relevant representations (Section 2.2.3):

Theoretical argument: From an information-theoretic perspective, "an arbitrarily powerful probe measures the mutual information between a hidden representation and a concept" (Hewitt & Liang, 2019; Pimentel et al., 2020). But mutual information is symmetric — it tells you the representation contains information about the concept, not that the model uses that information.
Empirical demonstrations: Ravichander et al. (2021), Geiger et al. (2024b), Elazar et al. (2021), and Belinkov (2022b) showed that "probes can be successfully trained on hidden activations that lack any causal connection to the output, only localizing correlated hidden activation vectors."
Interpretability illusions: Bolukbasi et al. (2021) demonstrated that "bias in data sets can create misleading explanations even when top activating examples are selected from real data sets." Depending on the dataset, "human annotators identified dramatically different meanings for given directions in the activation space of BERT" — the same activation direction received different interpretations based on which examples were shown.

The paper treats this as a fundamental limitation: even highly accurate linear probes do not, by themselves, demonstrate that the model uses the probed concept in its computations, and naive interpretation of probe results is a source of "interpretability illusions."

SDL Latent Utility: Mixed Evidence for Practical Usefulness

The paper reports mixed results on whether SDL latents are useful for downstream tasks (Section 2.1.2c):

Positive example: Kantamneni et al. (2024) "found a single latent whose activation pattern was more accurate than official dataset labels on the NLP task GLUE CoLA" (Warstadt et al., 2019). This suggests that in some cases, SAE latents capture structure that even human annotators miss.
Negative assessment: "More often than not though, a sparse set of latents that encode some useful concept of interest do not exist." The paper lists several hypotheses for this failure: the SAE might capture the model's internal representation, which differs from the human concept; the training distribution might be too narrow (Kissane et al., 2024c "found that SAEs trained on pretraining data generally do not have good latents for the concept of 'refusing' harmful user requests, while SAEs trained on chat formatted data do"); or the dictionary size might be insufficient.

Automation Results: LMs Can Describe Features, But Descriptions Are Imperfect

The paper describes the current state of automated feature description using language models (Section 2.4):

Feasibility demonstrated: Hernandez et al. (2022) for image model neurons, Bills et al. (2023) for language model neurons, and Huben et al. (2024), Bricken (2023), and Juang et al. (2024) for SAE latents have shown that language models can generate plausible natural language descriptions of component function from maximum activating examples.
Validation approach: These descriptions are validated by "assessing how effectively a human or model can use them to predict the activation of a feature in a given data set example, or predict where a feature is active within a single image or text excerpt." Prediction accuracy serves as a quantitative metric for description quality.
The paper notes these methods are "imperfect" and suggests improvements: "automating the generation of inputs to test the interpretation hypotheses by ensuring that generations activate the interpreted feature" (Huang et al., 2023), or "defining more rigorous statistical tests" (Bloom & Lin, 2024b). The paper does not report specific accuracy numbers from automated description studies.

Ablation Studies and Robustness Checks

Since this paper does not present original experiments, traditional ablation studies are absent. However, the paper cites findings that function as robustness checks on the field's core assumptions, and I report these as analogous to ablations:

The linear representation hypothesis's scope: The paper notes that "some works have shown that the strong version [that all concepts are linearly represented] is false for some models" (Black et al., 2022; Csordás et al., 2024) — a finding that functions as an "ablation" of the strongest form of the assumption underlying SDL. The weak version (some concepts are linearly represented) "is supported by the successes of linear probes, activation steering, and the success of sparse autoencoders in finding seemingly interpretable latents" (Section 2.1.2c), suggesting the assumption is useful but must be applied with caution.

SDL performance depends critically on training data distribution: Kissane et al. (2024c) found that SAEs trained on pretraining data lack good latents for the concept of "refusing harmful requests," while SAEs trained on chat-formatted data possess them. This demonstrates that SDL decomposition is not distribution-invariant — the features that an SAE recovers depend on what data it was trained to reconstruct, which undermines the hope that SDL identifies a universal, task-independent set of computational primitives.

Reconstruction errors are not random: Engels et al. (2024b) "found that these reconstruction errors are not purely random, as much of the direction of the error and its norm can be linearly predicted from the initial activation vector." This finding — that errors are structured rather than noise — suggests that SAEs are systematically missing certain types of representational structure, not just failing due to capacity limitations. It also "implies potential solutions" — if errors are predictable, it might be possible to train models to capture what current SAEs miss.

Circuit faithfulness metrics are themselves unreliable: Miller et al. (2024) showed that "existing measures of faithfulness depend on the causal intervention implementation used" — a finding analogous to showing that a proposed metric fails a sanity check. If faithfulness scores change based on how you perform the ablation (which is supposed to be a methodological detail, not a substantive variable), then the metric is not measuring a stable property of the circuit.

Concept-based interpretability can identify examples where models fail to use probed concepts: Elazar et al. (2021) and related concept erasure work demonstrate that removing probed concept information from representations does not necessarily change model behavior — a critical negative result showing that the presence of linearly decodable information does not imply that information is causally used.

Automated interpretability can be validated through prediction: The paper notes that the success of automated description methods is measured through their predictive power — "how effectively a human or model can use [the descriptions] to predict the activation of a feature" — providing a quantitative, reproducible metric rather than relying solely on subjective plausibility judgments. However, the paper does not report how high these prediction accuracies actually are, which is a notable omission for assessing the state of the art.

Critical Assessment

Because this paper is a review rather than an experimental contribution, the relevant question is not "do the experiments support the paper's claims?" but rather "does the evidence the paper cites support its diagnostic claims about the field's open problems?" I assess this for the paper's major diagnostic arguments.

Does the Evidence Support the Claim That SDL Reconstruction Errors Are a Critical Bottleneck?

The paper's most quantitative critique of SDL rests on two findings: 90% effective compute loss when applying SAEs to GPT-4 (Gao et al., 2024), and 40% performance degradation on GPT-2 small for full-distribution reconstruction (Makelov et al., 2024). These numbers are genuinely striking and support the claim that current SDL methods lose substantial information. However, several caveats are worth noting:

The paper does not report what "language modeling loss equivalent to a model with only 10% of GPT-4's pretraining compute" means in absolute accuracy terms. Pretraining compute and downstream task performance have a sublinear relationship (the scaling laws literature shows diminishing returns), so this may correspond to a smaller accuracy drop than intuition suggests. The paper does not provide the absolute loss values or downstream benchmark scores.

The 40% degradation for GPT-2 small is on "the full distribution," but the paper does not specify whether this means next-token prediction perplexity or some other metric. Task-specific degradation of 10% versus full-distribution degradation of 40% is an interesting gap, but without knowing what tasks were tested, it's hard to assess how concerning this is for mechanistic interpretability specifically (as opposed to the SAE's utility as a compression method).

The paper does not report whether scaling SAEs further (beyond 16 million latents for GPT-4) might close the reconstruction gap. The argument that errors are "too high" is implicitly a claim that they won't converge to acceptable levels with more compute, but this is not demonstrated — it's possible that even larger dictionaries or improved training would substantially reduce error. The Engels et al. (2024b) finding that errors are structured rather than random suggests room for improvement through better methods.

Crucially, the paper does not establish that perfect reconstruction is necessary for useful interpretability. It's possible that SAEs capture the most interpretable structure while the reconstruction error contains information that is important for performance but not for human understanding (e.g., low-level texture details, statistical regularities). The paper assumes, without direct evidence, that reconstruction fidelity and interpretability are tightly coupled.

Does the Evidence Support the Claim That Sparsity Is Not a Good Proxy for Interpretability?

The paper's argument here relies on documented phenomena — feature splitting, absorption, composition — that show sparsity optimization can produce less interpretable results. This evidence is suggestive but has limitations:

The cited work is primarily qualitative or case-study-based, not systematic. The paper does not report how frequently feature splitting or absorption occurs across SAE latents — are these rare edge cases or common failure modes? Without prevalence data, it's unclear whether these phenomena are theoretical curiosities or practical barriers.

"Interpretable" is itself not rigorously defined. The paper notes that the field lacks "clarity around the nature of features" (Section 2.1.2d), so claims that sparsity undermines interpretability are operating without a clear definition of what interpretability means. If interpretability means "maps cleanly to a single human concept," then feature splitting is a problem. But if interpretability means "predictively useful for downstream tasks," it may not be — a split feature could still be useful even if it doesn't match human conceptual categories.

The paper mentions alternative proxies like minimum description length (Ayonrinde et al., 2024) but does not provide quantitative comparisons showing MDL does better than sparsity. The criticism of sparsity is well-argued, but the alternatives remain speculative.

Does the Evidence Support the Claim That Circuit Faithfulness Is Fundamentally Limited?

The paper's strongest evidence here comes from Miller et al. (2024), which showed that (a) circuits provide poor approximations for individual datapoints, and (b) faithfulness metrics depend on implementation details of the intervention method. These findings directly challenge the field's standard validation procedure. However:

The paper generalizes from a small number of studied circuits. Miller et al. (2024) studied "several complex end-to-end circuits" — the paper does not specify how many, and it's possible that different circuits or different models would show different faithfulness properties. The field has produced relatively few complete end-to-end circuits, so the evidence base is thin.

The within-task variance finding is serious but may be addressable. If circuits approximate average behavior well but individual datapoints poorly, this could mean: (a) the circuit discovery method finds the right components but the wrong interactions (solvable with better methods), or (b) there is no stable circuit — the model uses different mechanisms for different inputs within the same task (a more fundamental limitation). The paper does not distinguish these possibilities.

The paper does not report specific faithfulness numbers (e.g., "circuit achieves 80% of full-model accuracy on task X"). Without quantitative baselines, it's hard to assess whether faithfulness is "low" in an absolute sense or just lower than the field had hoped.

Does the Evidence Support the Claim That Probes Detect Correlation Rather Than Causation?

This claim is well-supported theoretically and empirically. The existence of "probing on hidden activations that lack any causal connection to the output" is a clean demonstration that probe accuracy alone does not imply causal relevance. The paper cites multiple independent demonstrations of this phenomenon (Ravichander et al., 2021; Geiger et al., 2024b; Elazar et al., 2021; Belinkov, 2022b), and the information-theoretic argument is sound.

However, the paper does not quantify how frequently this occurs. Are most high-accuracy probes causally irrelevant, or only probes trained on certain kinds of concepts or certain layers? Without prevalence data, the practical significance is unclear — the phenomenon exists, but it might be rare enough that probes remain useful as hypothesis-generation tools with appropriate follow-up validation.

Overall Assessment: The Diagnostic Claims Are Well-Argued But Rest on an Incomplete Evidence Base

The paper's central contribution is a diagnosis of what's holding mechanistic interpretability back, and the evidence it marshals is genuinely concerning: SDL loses substantial information, circuits are unfaithful, sparsity can undermine interpretability, and probes detect correlation without implying causation. Each of these problems is real and documented.

However, the paper's framing tends toward the pessimistic — it emphasizes limitations without equally weighing recent progress. The same SDL methods that lose 40% of GPT-2's performance also produce latents that "are highly interpretable" by human judgment (Huben et al., 2024; Bricken, 2023), and single latents can outperform human labels on specific tasks (Kantamneni et al., 2024). Circuits that are unfaithful by one metric may still capture genuine computational structure (the paper itself notes that faithfulness metrics are unreliable). The paper acknowledges this tension implicitly but does not resolve it — it catalogs problems without providing a framework for trading off different types of failure or assessing which limitations are showstoppers versus acceptable imperfections.

The most significant gap in the paper's evidence base is the absence of systematic benchmarks across multiple models, architectures, and tasks. The paper cites findings from individual studies on individual models (GPT-2, GPT-4, BERT, specific vision CNNs) without a unified evaluation framework. This makes it difficult to assess whether the documented problems are universal features of neural networks or artifacts of specific interpretations of specific models. The paper's own proposal — developing model organisms, standardized benchmarks, and ground-truth testbeds — is itself a response to this gap, implicitly acknowledging that the current evidence is more suggestive than conclusive.

Finally, the paper does not present counterarguments or alternative interpretations of the negative findings it cites. The SDL reconstruction error might be acceptable if the lost information is not important for interpretability. Circuit unfaithfulness might be a problem of current methods rather than a fundamental limitation. Feature splitting might be benign if the split features collectively represent the concept in a structured way. By presenting only the critical evidence without engaging with these counterarguments, the paper makes a strong case for concern but a weaker case for the specific degree of alarm it conveys.

6. Limitations and Trade-offs

1. Difficulty Estimation Cost Is Unaccounted For, Potentially Dominating the Inference Budget

The assumption or constraint. The entire compute-optimal framework rests on knowing each prompt's difficulty before allocating the inference budget. The paper's method for estimating difficulty — generating 2048 samples per question and averaging the base model's pass@1 rate (oracle) or the PRM's final-answer score (predicted) — is "extraordinarily expensive," as the authors acknowledge in Section 3.2: "our experiments do not account for this cost largely for simplicity." Generating 2048 samples per question consumes more compute than the largest test-time budgets studied (256–512 generations).

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter — potentially making the compute-optimal approach more expensive than simply running best-of-N with a large fixed budget on every question. The paper frames difficulty estimation as an "exploration-exploitation tradeoff" (Section 3.2) but does not quantify where the crossover point lies or provide a method for cheaply approximating difficulty.

What evidence exists in the paper. Section 3.2 acknowledges the problem explicitly ("estimating difficulty in this way still incurs additional computation cost during inference"), and the paper notes that the predicted difficulty method (using PRM scores rather than ground-truth labels) still requires the full 2048 samples. Figure 4 and Figure 8 show compute-optimal scaling curves that exclude estimation cost. There is no ablation studying how performance degrades if difficulty is estimated from fewer samples (e.g., 16, 64, 256), and no experiment measuring total FLOPs including estimation.

Mitigation status. The paper does not attempt to mitigate this limitation, beyond suggesting future work: "Cheap difficulty estimation" is listed as a key open problem in Section 8, mentioning the possibility of training a lightweight classifier to predict difficulty from the question text alone, or using adaptive estimation that amortizes difficulty assessment into the problem-solving process. But no such method is developed or evaluated. Until this gap is closed, the 4× figure should be understood as an upper bound on achievable efficiency under an oracle difficulty model, not a realized deployment gain.

2. Single Benchmark, Single Model Family: Findings May Not Generalize Beyond MATH with PaLM 2-S*

The assumption or constraint. All experiments use the MATH benchmark (500 test questions, high-school competition math) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. The paper acknowledges, in its conclusion (Section 8), that extending to "other domains and modalities" is future work.

The consequence. Several aspects of the paper's central findings could be model-specific or domain-specific:

The PRM's quality and over-optimization behavior depend on PaLM 2-S's output distribution.* A model with different calibration properties, different typical error patterns, or different base accuracy on MATH might exhibit entirely different difficulty-dependent scaling curves — the qualitative pattern that beam search hurts on easy problems but helps on medium ones might not replicate.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. PaLM 2-S* may be particularly amenable to this training procedure in ways that other models (e.g., Llama, Claude, GPT-4) are not.
MATH is exclusively symbolic reasoning with closed-form answers. It is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions dominating easy problems, no method helping on the hardest bin) generalize to code generation, logical reasoning, scientific QA, or tasks requiring factual recall rather than multi-step inference. The paper's core finding — that the optimal strategy depends on difficulty — might be universal while the specific mapping from difficulty to optimal strategy is entirely domain-specific, requiring re-derivation for each new task family.

What evidence exists in the paper. The paper provides no cross-model or cross-benchmark results. The test set is 500 questions, split into five difficulty quintiles of ~100 each, further split by two-fold cross-validation — meaning the compute-optimal policy is selected based on ~50 questions per fold per bin. No confidence intervals are reported on the compute-optimal scaling curves, making it impossible to assess whether the observed strategy differences across difficulty bins are statistically reliable at this sample size.

Mitigation status. The paper does not attempt to mitigate this limitation. It frames single-model single-benchmark evaluation as a deliberate scoping choice (MATH provides clean correctness signals for PRM training and difficulty estimation), but does not discuss how findings might transfer. The absence of even a second benchmark (e.g., GSM8K, a different math dataset) is notable. The paper's suggested future work includes "Replicating the study on code generation (e.g., HumanEval, MBPP), logical reasoning (e.g., ARC, FOLIO), scientific QA, and open-ended generation tasks" (Section 8), implicitly acknowledging the current scope is narrow.

3. The Revisions and Search Mechanisms Are Never Combined, Leaving the Core Synergy Untested

The assumption or constraint. The paper studies two complementary axes — PRM-guided search (Section 5) and iterative revisions (Section 6) — as independent mechanisms. They are never deployed together. Section 8 explicitly acknowledges this gap: "we did not experiment with PRM tree-search techniques in combination with revisions."

The consequence. This is a significant gap because the paper's own framework (Section 2) treats these as complementary: revisions improve the proposal distribution (generating better candidates), while PRM search improves candidate selection (finding the best among generated candidates). The paper demonstrates (Figures 4, 8) that each mechanism individually provides ~4× efficiency gains over best-of-N, but these gains are on different difficulty regimes — revisions dominate on easy problems, search dominates on medium problems. Combining them could yield:

Beam search over revision model outputs: Using the revision model as the proposal distribution within beam search, where each beam step conditions on previous (potentially rejected) branches as context, potentially producing higher-quality candidates at each step.
PRM-guided revision: Using the PRM's per-step scores to decide when to revise versus when to restart from scratch, rather than blindly generating long revision chains.
Hierarchical strategies: Using PRM search to select among revision chains (at the chain level) while using revisions to refine within each chain, which the paper's sequential + parallel analysis (Figure 7) suggests would be beneficial.

The current results therefore represent a lower bound on what a fully integrated system could achieve. The paper's headline claims about compute-optimal scaling (4× efficiency gains, outperforming a ~14× larger model on easy-to-medium problems) might significantly understate what's possible with combined mechanisms.

What evidence exists in the paper. There is no experiment combining PRM search with the revision model. The FLOPs-matched comparison (Section 7, Figure 9) treats revisions and PRM search as separate strategies and compares each independently against the larger model — there is no "revisions + PRM search" bar. The paper's own difficulty-bin analyses (Figures 3 right, 7 right) show that the mechanisms have complementary strengths, which is precisely the pattern that would motivate combining them, but this combination is never tested.

Mitigation status. The paper does not attempt to mitigate this limitation, beyond listing it as future work ("Combining search and revisions" in Section 8). The separation may have been driven by practical constraints — the PRM is trained on base model outputs and the paper demonstrates (Appendix J, Figure 15a) that it does not transfer well to revision model outputs due to distribution shift, requiring retraining an ORM specifically for revisions. Combining the two would likely require training a PRM on revision model outputs, which would be a non-trivial engineering effort. But the absence of even a preliminary experiment (e.g., using the revision-specific ORM to do best-of-N selection over sequential revision chains) leaves the central synergy untested.

4. The ~14× Larger Model Baseline Is Weak: Not Compute-Optimally Trained and Uses Only Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 pits PaLM 2-S* with compute-optimal test-time strategies against a model with approximately 14× more parameters under greedy decoding with no additional test-time compute. The paper acknowledges that this scales parameters only, not training data — following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal training (Hoffmann et al., 2022) where both parameters and data are scaled equally. Section 7 states: "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. This makes the pretraining baseline weaker than it needs to be, potentially inflating the apparent advantage of test-time compute:

A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, since the latter is undertrained by compute-optimal standards. The paper's headline finding — that test-time compute can substitute for pretraining compute — may be weaker or absent against a properly compute-optimal larger model.
The 14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search, no revisions. This is an extremely weak baseline given that the paper's central argument is about the value of test-time compute. A fairer comparison would give the larger model a modest test-time compute budget (e.g., best-of-8 or best-of-16, which is standard in practice) and compare that against the smaller model with its full compute-optimal budget. The paper's FLOPs matching accounts for the larger model's higher per-token inference cost, but does not consider whether the larger model could also benefit from test-time compute strategies — which, given the paper's own results showing these strategies help, it almost certainly would.
The paper does not report how much of the 14× parameter model's compute budget is pretraining versus inference. If the larger model's total FLOPs are dominated by pretraining and only a small inference budget remains, the comparison is essentially testing "can we make a small model better by spending a lot on inference?" — which is a different question than "should we allocate our total compute to pretraining or inference?"

What evidence exists in the paper. Section 7 and Figure 9 present the FLOPs-matched comparison, with Figure 1 providing bar charts of relative improvement. The paper is transparent about the parameter-only scaling choice. However, there is no ablation showing how results change if the larger model is also given test-time compute (e.g., best-of-N weighted with N=4, 8, 16), and no comparison against a Chinchilla-optimal 14× model. The paper also does not decompose total FLOPs into pretraining vs. inference components for the larger model, making it difficult to assess whether the test-time compute advantage is primarily coming from the pretraining savings (which scale as 6ND) or the inference savings per token.

Mitigation status. The paper acknowledges the parameter-only scaling issue explicitly and frames it as future work. The authors argue this choice is "representative of a canonical approach to scaling pretraining compute" (Section 7) — essentially, that most deployed large models are not Chinchilla-optimal, so the comparison is practically relevant even if theoretically suboptimal. This is a reasonable defense, but it means the headline claim "test-time compute can substitute for pretraining" should be qualified as "test-time compute on a specific small model can match a parameter-scaled (but not compute-optimally trained) larger model on easy-to-medium problems when the larger model uses no test-time compute of its own." The paper does not test the stronger, arguably more interesting claim: that for a fixed total FLOPs budget, it is better to allocate more to inference and less to pretraining even when both regimes are compute-optimal.

5. Verifier Over-Optimization Is a Hard Ceiling, Not a Solved Problem by the Compute-Optimal Policy

The assumption or constraint. The paper identifies verifier over-optimization — where aggressive search finds solutions that score highly under the PRM but are actually incorrect — as a central limiting factor on test-time compute scaling. Evidence includes: beam search degrading easy-problem performance at high budgets (Figure 3, right), lookahead search paradoxically performing worst overall (Figure 3, left), and qualitative examples of degenerate outputs like repetitive low-information steps and overly short solutions (Appendix M). The compute-optimal policy mitigates this by routing easy problems away from aggressive search, but it does not eliminate the underlying problem.

The consequence. The compute-optimal approach is fundamentally bounded by verifier quality. Even on medium-difficulty problems where beam search is the recommended strategy, over-optimization limits the scaling ceiling — the beam search curves in Figure 3 (right, bins 3–4) flatten and, in some cases, begin declining well before the budget is exhausted. This means:

The compute-optimal policy is specific to the verifier quality achieved with the paper's Monte Carlo rollout training procedure (Appendix D). If a better PRM were available (e.g., trained with more on-policy data, better calibrated, adversarially robust), the optimal strategy per difficulty bin would change — and the overall scaling ceiling would be higher. The paper does not explore how verifier improvements would shift the difficulty-dependent strategy mapping.
On hard problems (bin 5), no method helps regardless of budget — but it's unclear whether this is because the base model truly cannot produce correct solutions (capability ceiling) or because the PRM cannot recognize correct solutions when they occur (verifier ceiling). If the latter, improving the verifier could unlock gains on harder problems.
The paper's framework does not provide a way to detect or predict over-optimization during deployment. The compute-optimal policy is a static lookup table — for a given difficulty bin and budget, it always uses the same strategy. It does not monitor for signs of over-optimization mid-search (e.g., PRM scores becoming uncorrelated with actual correctness, or solution diversity collapsing) and adjust accordingly.

What evidence exists in the paper. The over-optimization evidence is strong and multifaceted: quantitative (Figures 3, 4), qualitative (Appendix M), and consistent across search methods. The paper explicitly names this as a central bottleneck: "improving verifier robustness is the key bottleneck for further scaling test-time compute, not improving search algorithms" (Section 5.3, takeaway box). However, the paper does not attempt to measure the verifier quality ceiling — it does not correlate PRM scores with ground-truth correctness across difficulty bins or search budgets, which would quantify how much room for improvement exists.

Mitigation status. The paper does not attempt to solve the over-optimization problem beyond the compute-optimal routing strategy. It identifies robust verifier training as a key research direction (Section 8: "Robust verifiers resistant to over-optimization"), mentioning adversarial training, ensemble verification, and constrained search methods as possibilities. But none are tested. The compute-optimal framework, as presented, is a mitigation strategy (avoid pushing search too hard where the verifier is unreliable) rather than a solution to the underlying verifier quality problem.

6. The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate and the Mitigation Is a Patch, Not a Fix

The assumption or constraint. The revision model, trained only on sequences where all in-context answers are incorrect (followed by a correct target), has no training signal for what to do when the current answer is already correct. At test time, approximately 38% of correct answers get "revised" back to incorrect ones (Section 6.1). The paper's mitigation is to use majority voting or verifier-based selection across the entire revision chain, picking the best answer from any step rather than always taking the last revision. This is a post-hoc selection patch, not a solution to the model's inability to recognize when no revision is needed.

The consequence. This limitation has several practical implications:

The effective sequential budget is reduced. If 38% of correct answers are lost to reversion and must be recovered by selection across the chain, the model is wasting ~38% of its sequential steps on destructive revisions. The benefits of sequential revision (Figure 6) would be substantially larger if the model could recognize correct answers and stop.
The selection mechanism introduces a new failure mode. Majority voting or verifier-based selection can fail — if the model produces mostly incorrect answers with one correct one, and the verifier misranks the correct answer lower than an incorrect but verifier-pleasing answer, the system selects the wrong answer. The paper does not report how often the selection mechanism fails to recover correct answers that were produced in the chain but not selected.
The 38% figure is likely architecture- and training-procedure-dependent, but without understanding the root cause, it's unclear whether this problem would be better or worse with different base models, different revision training data constructions, or different chain lengths.
Latency is inflated: Each sequential step takes wall-clock time, and if 38% of steps are actively harmful, the latency cost of reaching a correct answer (including recovery steps) is higher than the FLOPs-based generation budget suggests.

What evidence exists in the paper. The paper reports the 38% figure in Section 6.1, identifying it as "A significant practical issue." The mitigation (within-chain selection) is evaluated in Figure 6, and the compute-optimal revision results (Figure 8) include this selection mechanism. However, the paper does not report an ablation showing what performance would be without reversion — i.e., a counterfactual where the model stops revising when it reaches a correct answer — which would quantify the ceiling loss from this problem.

Mitigation status. The paper's mitigation (within-chain selection) is acknowledged as a patch: "A more principled solution — such as training the model to recognize when no revision is needed — is not explored" (Section 6.1). The negative result with ReST^EM (Appendix K, Figure 16), where additional optimization made revision performance worse, suggests that addressing this through better training is non-trivial and may require fundamentally different data construction procedures. The paper does not attempt to characterize the types of correct answers that get revised (are they systematically different from those that survive?) or to develop a lightweight correctness detector that could be used to truncate chains early.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new method or empirical result. It is a diagnostic intervention rather than a technical contribution. Its impact, if the field takes it seriously, would be to shift mechanistic interpretability from a collection of loosely coordinated research threads toward a more disciplined scientific and engineering enterprise with clearer standards of evidence, shared infrastructure, and explicit prioritization among competing objectives.

The magnitude of this shift depends on uptake. Unlike a new model or algorithm, a review paper cannot force change — it can only make the case that current practices are insufficient and offer a framework for improvement. The paper's unusual coalition of authors (over thirty researchers from organizations spanning academia, industry labs, independent research groups, and governance-focused nonprofits) gives it a degree of legitimacy that a single-lab perspective would lack. But the paper is careful to note its limitations: it represents "a synthesis of their views by Apollo Research" and "the perspectives presented here do not necessarily reflect the views of any individual author" — acknowledgments that the field's disagreements are real and not fully resolved even among the paper's own authors.

The paper reframes mechanistic interpretability's central challenge from "how do we understand neural networks?" to "what would constitute adequate evidence that we understand a neural network, and how far are we from producing it?" This is a shift from method development to epistemic rigor — not "can we do X?" but "if we claim we've done X, how would we know we're right?"

The paper resolves several tensions in prior literature by making them explicit rather than by providing new data:

"Are SAEs working?" becomes a set of specific, falsifiable questions: What reconstruction fidelity is necessary for what applications? Is sparsity the right optimization target? Does the linear representation hypothesis hold strongly enough in practice to support the approach? The paper doesn't answer these but makes it impossible to ignore them.
"Are circuits faithful?" the paper shows cannot be answered with current metrics, which are themselves unreliable (Miller et al., 2024). The resolution is not a new metric but an acknowledgment that the field doesn't yet know how to measure its central object of study.
"Should we prioritize reverse engineering or concept-based approaches?" is reframed as a false dichotomy — the paper argues both are necessary, they serve different goals, and the real question is how to combine them (e.g., using concept-based probes to generate hypotheses that reverse engineering validates causally).
"Is interpretability a science or an engineering discipline?" is left unresolved, but the paper argues that conflating the two leads to confusion about what counts as progress. Scientific understanding of small models may not translate to engineering control of large ones, and engineering demonstrations on cherry-picked examples may not constitute scientific understanding.

Research directions that become more attractive after this paper:

Rigorous validation infrastructure: Benchmarks with known ground truth (compiled from symbolic programs, interchange intervention training), model organisms with standardized evaluation protocols, and head-to-head comparisons against non-interpretability baselines on real-world tasks.
Theoretical foundations for features: If "feature" has no formal definition, arguing about whether SAEs recover "true features" is vacuous. The paper highlights connections to singular learning theory, causal abstraction, and other formalisms that could provide the missing definitions.
Decomposition methods that don't assume linearity: The paper's systematic critique of the linear representation hypothesis suggests that methods assuming linear features may hit fundamental ceilings, opening space for nonlinear or geometrically structured alternatives.
Mechanism-level (weight-level) interpretability: The observation that SDL "describes activations, not mechanisms" points toward work that directly decomposes weight matrices or parameter-level structure rather than just representational content.
Intrinsic interpretability: If post-hoc decomposition of trained networks is fundamentally limited, building interpretability into the training process becomes more attractive, even at some performance cost.
Automated validation, not just automated description: Current automated interpretability focuses on generating descriptions; the paper argues we need automation for testing those descriptions — generating counterfactuals, predicting interventions, checking consistency.

Research directions that become less attractive (or at least harder to justify without additional validation):

Pure scaling of current SDL methods without addressing the underlying assumptions. If sparsity isn't a reliable proxy for interpretability, making SAEs bigger and sparser may not help.
Circuit discovery on architectural components without demonstrating that those components are the right decomposition. If neurons and attention heads are polysemantic, circuits built from them may describe artifacts of the architecture rather than computational primitives.
Isolated demonstrations of interpretability on cherry-picked tasks. The paper's "streetlight interpretability" critique suggests that the field has been systematically overestimating its capabilities by studying only the easiest cases.

Follow-Up Research This Work Enables

A unified evaluation suite for mechanistic interpretability methods built from compiled ground-truth models. The paper identifies the absence of standardized benchmarks with known ground truth as a critical bottleneck. A concrete follow-up would construct a suite of transformer models that implement known algorithms — compiled from symbolic programs (extending Tracr/Lindner et al., 2023; or RASP/Weiss et al., 2021) — at increasing scales and complexities, covering tasks like modular arithmetic, sorting, graph algorithms, and multi-step reasoning. Each model would have a verified ground-truth description of its computational structure. The suite would then evaluate representative decomposition methods (SAEs, PCA, non-negative matrix factorization), description methods (automated interpretability via LMs, maximum activating examples, logit lens), and circuit discovery pipelines (ACDC, attribution patching, sparse feature circuits) by measuring how closely their outputs match the known ground truth. Key metrics: (a) do discovered features correspond to ground-truth features? (b) do discovered circuits recover the true computational graph? (c) how does performance degrade as model scale / task complexity increases? This directly addresses the paper's concern that "interpretability illusions" can go undetected without ground truth, and provides a controlled environment for isolating whether current methods fail due to flawed assumptions about superposition and linearity (which would manifest even on these simple models) or due to scaling and complexity (which would only appear on frontier systems).

Quantifying how much reconstruction error matters for specific downstream applications. The paper reports that SAE reconstruction incurs a ~90% effective compute loss on GPT-4 (Gao et al., 2024) and 40% performance degradation on GPT-2 small (Makelov et al., 2024), but frames this as a problem without establishing how much fidelity different applications require. A targeted study would take a single model (e.g., Gemma 2B or Llama 3 8B, chosen for reproducibility) and train SAEs at multiple dictionary sizes and sparsity levels, producing a sweep of reconstruction fidelities. For each fidelity level, evaluate performance on concrete downstream tasks that the paper identifies as application goals: (a) monitoring — can SAE-based detectors identify when the model is engaging in deceptive reasoning (using a sleeper-agents-style setup like Hubinger et al., 2024)? (b) control — can SAE-based steering vectors (using the method of Turner et al., 2024 or Rimsky et al., 2024) reliably modify behavior, and does steering precision degrade with reconstruction error? (c) editing — can SAE-based unlearning (Farrell et al., 2024) remove specific knowledge, and does it fail when reconstruction is poor? (d) microscope AI — can SAE latents recover known scientific patterns embedded in synthetic data? This would replace the paper's binary "reconstruction is too high" claim with a function mapping reconstruction fidelity to application utility, identifying minimum viable reconstruction thresholds and whether current methods meet them.

Interchange intervention training as a bridge between intrinsic and post-hoc interpretability. The paper mentions Interchange Intervention Training (IIT; Geiger et al., 2022) as a method for training models with known causal structure but does not explore its implications for validating post-hoc methods. A concrete experiment: train a series of models with IIT to enforce specific causal abstractions (e.g., a hierarchical composition of modular subtasks), then apply standard post-hoc interpretability pipelines (SAE decomposition + circuit discovery) to these models. Because the ground-truth causal structure is known (it was enforced during training), this provides a controlled test of whether existing methods can recover it. Key variables to manipulate: the complexity of the enforced causal structure, the extent to which it departs from human-intuitive task decompositions, and whether the model naturally develops superposition "around" the enforced structure. This tests the paper's implicit claim that intrinsic and post-hoc interpretability are complementary rather than competing — if post-hoc methods succeed at recovering IIT-enforced structure, it validates their use on arbitrary trained models; if they fail even when structure is known to exist, it demonstrates fundamental limitations that intrinsic approaches may circumvent.

A systematic study of feature geometry and its functional significance. The paper raises the question of whether understanding feature geometry (the relative positions of feature directions in activation space) is necessary for interpreting neural networks, or whether a "bag of features" approach suffices. A concrete investigation would take an SAE decomposition of a model (e.g., Gemma 2B with Gemma Scope SAEs; Lieberum et al., 2024) and systematically measure: (a) whether geometrically proximate SAE latents participate in the same circuits (measured via patching or attribution), (b) whether intervening on a feature's neighbors produces side effects that are predictable from geometry, and (c) whether features that are functionally related but geometrically distant are connected through specific weight matrix structure that compensates for the distance. The finding that local geometry matters (nearby features interact more) would indicate that current SAEs are discarding essential structure and that decomposition methods must preserve or recover geometric relationships. The finding that global geometry can be ignored (only which features exist matters, not where they are) would validate the bag-of-features approach and suggest that reconstruction error from SAEs may not be geometrically structured in functionally relevant ways — a finding that would partially mitigate the paper's concern about "feature geometry unexplained."

An adversarial robustness benchmark for interpretability methods. The paper warns that automated interpretability could be sabotaged by adversarial models that "deliberately yield misleading feature visualizations" (citing Geirhos et al., 2025). A direct follow-up would construct a benchmark where models are trained to exhibit specific deceptive behaviors: (a) a model that uses feature X for task performance but represents X in a way that SAEs or automated description tools misidentify as feature Y, (b) a model with a hidden trojan that activates only under rare triggers and whose associated features are systematically absent from SAE dictionaries trained on standard data distributions, and (c) a model whose chain-of-thought outputs are unfaithful to its internal reasoning in predictable, interpretability-detectable ways. The benchmark would then measure whether current interpretability methods can detect these deceptions, and under what conditions (dictionary size, training distribution breadth, automated description prompt design) they succeed or fail. This directly addresses the paper's concern about "misaligned AI systems sabotaging AI automated interpretability" and provides concrete evidence about whether current methods would detect deliberately hidden capabilities — a question of direct relevance to the monitoring applications discussed in Section 3.2.

Replicating the SDL critique across architectures: Do the same problems appear in state-space models and mixture-of-experts architectures? The paper's critique of SDL is almost entirely based on transformer models (mostly GPT-2, GPT-4, and BERT variants). The paper explicitly calls for extending interpretability methods to broader model families (Section 3.6) and lists SSMs (Gu & Dao, 2024) and MoE (Shazeer et al., 2017; Fedus et al., 2022) as architectures where method transferability is unknown. A concrete study would train SAEs with identical hyperparameters on comparable-scale transformer, Mamba/SSM, and MoE models trained on the same data, then measure: (a) reconstruction error as a function of dictionary size and sparsity, (b) feature splitting / absorption prevalence (using the methods of Bricken, 2023 and Chanin et al., 2024), (c) automated description quality (using LM-based labeling and predictive accuracy), and (d) whether the linear representation hypothesis holds equivalently across architectures (do linear probes achieve similar accuracy for the same concepts?). This would test whether SDL's limitations are specific to the transformer architecture's representational geometry (e.g., residual stream structure) or are universal properties of deep networks. If SSMs show dramatically lower reconstruction error or more monosemantic features, it would suggest that architectural choices can mitigate the problems the paper diagnoses — a finding with direct implications for intrinsic interpretability research.

Practical Applications and Downstream Use Cases

Structuring safety cases for frontier AI systems using the paper's difficulty-progress matrix. The paper's framework (Section 3.1) decomposes progress into independent axes: decomposition quality, description depth, extent of network coverage, breadth of task distribution, and post-hoc vs. during-training understanding. For organizations developing frontier models and constructing safety cases (as Anthropic's Responsible Scaling Policy and OpenAI's Preparedness Framework require), this matrix provides a template for specifying what kind of interpretability evidence would be sufficient for particular safety claims. For instance, a claim that "the model does not contain mechanisms for long-horizon deceptive planning" might require: moderate decomposition quality (to identify all major feature groups), high description depth (to distinguish planning-relevant features from benign ones), extensive network coverage (whole-model analysis, not just a few circuits), narrow task distribution (only planning-relevant contexts), and post-hoc understanding (sufficient for a static model). A weaker claim — "the model shows no internal signatures of deception on evaluation datasets" — might require only shallow descriptions and limited task coverage. This turns the paper's diagnostic into a practical tool for safety case engineering, specifying at what point interpretability evidence becomes adequate for regulatory or internal governance purposes rather than treating "better interpretability" as an undifferentiated goal.

Guiding investment in interpretability infrastructure for AI auditing organizations. The paper's analysis of which applications require which methodological advances (Section 3) provides a decision framework for organizations like the UK AI Safety Institute, METR, or Apollo Research that conduct audits of AI systems. The key practical insight is that monitoring for unsafe cognition is likely the nearest-term achievable application because it can tolerate shallow, correlation-based descriptions (Section 3.2.1: "even shallow, correlation-based descriptions could signal potentially concerning cognition"), while formal verification and prediction of emergent capabilities require advances along essentially every axis and are not near-term achievable. This suggests a concrete investment strategy: prioritize building infrastructure for white-box evaluation using current imperfect methods (e.g., SAE-based detectors for specific concerning behaviors like sandbagging or sycophancy), while funding longer-term research on the decomposition and description problems that block more ambitious goals. Without this prioritization, auditing organizations risk overinvesting in methods that cannot yet deliver on their intended use cases (e.g., attempting formal verification with inadequate tools) while underinvesting in methods that could provide partial but actionable safety evidence today.

Selecting model organisms for community-wide research coordination. The paper's call for model organisms (Section 2.1.4a) is actionable immediately. The field currently lacks consensus on which models to study deeply, leading to fragmented results that cannot be cross-validated. A concrete step: the community could standardize on a small set of models meeting the paper's criteria — "open source, easy and cheap to use, representative of a broad range of systems and phenomena, have a replicable training process with open source training data, and have multiple instances with different random seeds" (Sharkey et al., 2022a). Candidates include: (a) Pythia models (Biderman et al., 2023) which provide multiple scales with identical training data and multiple seeds per scale, (b) OLMo models (Groeneveld et al., 2024) with fully open training pipelines, and (c) Gemma models with openly available SAEs (Lieberum et al., 2024). Standardizing on these would enable the validation practices the paper advocates — cross-checking results across independent research groups, building cumulative understanding, and testing whether methods that work on one seed or scale transfer to others — that are currently impossible due to methodological and model fragmentation. The paper's diagnosis that validation is the field's weakest link implies that infrastructure for coordinated validation may be higher-impact than any single new interpretability technique.

Designing evaluation protocols that distinguish interpretability method failures from model capability limitations. The paper notes that on the hardest questions (difficulty bin 5), no method improves performance (Figures 3, 7, 9), but it cannot distinguish whether this is because the base model cannot produce correct answers (capability ceiling) or because the PRM/verifier cannot recognize them (evaluation ceiling). For organizations deploying interpretability-based monitoring or control in high-stakes settings, this distinction matters enormously: if the failure is capability-limited, monitoring can detect that the model is out of its depth and escalate to human review; if the failure is evaluation-limited, the model might be producing dangerous outputs that the monitoring system cannot recognize. A practical protocol would separately measure: (a) the base model's pass@k (can it ever produce correct outputs?), (b) the verifier's AUROC on distinguishing correct from incorrect outputs the model can produce, and (c) the verifier's calibration on out-of-distribution or adversarially constructed inputs that the model would never naturally generate. This protocol, which the paper does not propose but which its analysis implies is necessary, would allow deployment decisions to be conditioned on measured failure modes rather than assuming methods work uniformly across difficulty regimes.