Partially Rewriting a Transformer in Natural Language

🎯 Pitch

Replacing a feedforward network’s encoder with an LLM that predicts neuron activations from auto-generated explanations causes a loss increase no worse than zeroing out the layer entirely. This reveals that current automated explanations are too vague, triggering on roughly a third of tokens instead of the precise contexts where features should fire.

1. Executive Summary

This paper attempts to partially rewrite the feedforward network of an LLM using natural language explanations, measuring whether these explanations faithfully capture the model's internal computation. Using Pythia 160M with a sparse transcoder (a wider MLP with sparsely activating neurons that approximates an MLP layer's function) and a sparse autoencoder on the residual stream, the authors generate automated explanations for each latent feature, then replace the encoder with an LLM-based simulator that predicts each feature's activation from its explanation and surrounding context. After applying quantile normalization to calibrate the simulator's systematically overconfident predictions, the increase in cross-entropy loss is statistically indistinguishable from simply zeroing out the MLP output entirely—the model performs comparably to a Pythia checkpoint trained on only 10–15% of the full corpus. This result establishes that existing automated explanations are insufficiently specific, producing far too many false positives (average specificity around 80% on 32,768 latents), and that rewriting a model in natural language succeeds only when explanations correctly identify the contexts where a feature is not active, not merely where it fires.

2. Context and Motivation

The Core Problem: We Can Find Interpretable Features, But Are They Faithful?

The central question this paper tackles is deceptively simple: if we extract interpretable features from a neural network and write natural language explanations for what they do, can we replace the original computation with a language model that reads those explanations and predicts feature activations — while preserving the original model's behavior? This is not merely an evaluation exercise. It is a test of whether our explanations actually capture what the network computes, or whether they are just plausible-sounding stories that correlate with activation patterns without describing the true underlying function.

The problem sits at the intersection of two distinct challenges in mechanistic interpretability. The first is feature extraction: decomposing a dense, polysemantic neural network into a set of more interpretable, ideally monosemantic features. The second is explanation generation: producing natural language descriptions of what those features represent. Both have seen significant progress in recent years. But there is a gaping hole between them: we have no rigorous way to determine whether a generated explanation is faithful — whether it actually describes the computation the network performs, rather than merely being correlated with it. This paper proposes a direct test: if the explanation is faithful, an external LLM reading only that explanation (plus the input context) should be able to reproduce the feature's activation pattern, and inserting those predicted activations back into the model should preserve the model's output.

This matters because the entire project of mechanistic interpretability is premised on the idea that we can understand neural networks by decomposing them into human-interpretable pieces. If our best automated explanations fail to enable even a partial rewrite of a single MLP layer in a 160M-parameter model — as this paper finds — then the field's current methods are not producing explanations that capture the network's actual computation. The explanations are, at best, shallow descriptions of activation contexts rather than functional specifications.

Why This Problem Is Important

The microscope AI alignment agenda. The paper explicitly situates itself within the "microscope AI" framework (Hubinger, 2019), which proposes an alternative to deploying AI systems directly: instead of running the model, we analyze its learned representations and computations to extract actionable insights for humans, then use those insights without needing to execute the original network. This is not an abstract philosophical position — it is a concrete alignment proposal. If we could fully rewrite a frontier model in natural language, we could inspect its reasoning, verify its safety properties, and potentially distill its capabilities into a form that cannot harbor hidden objectives or deceptive alignment. The paper's title — "Partially Rewriting a Transformer in Natural Language" — signals this ambition directly.

The authors connect this to imitative generalization (Barnes, 2021), the idea that we could jointly optimize a network and its human-interpretable annotations to maximize their joint prior likelihood, effectively building models that are interpretable by construction rather than requiring post-hoc analysis. The current paper can be understood as a diagnostic: before we can build interpretable-by-construction models, we need to know whether our interpretation tools are good enough to describe what existing models actually compute. The negative result — that current explanations fail even at the level of a single layer — suggests the field is far from this goal.

The faithfulness problem in interpretability. A persistent methodological challenge in mechanistic interpretability is distinguishing between explanations that are correlated with a feature's behavior and explanations that are causally faithful. An explanation like "this neuron activates on text about sports" might achieve high correlation with activation patterns while missing critical details about exactly which sports, in which contexts, with which syntactic or semantic modifiers. Standard evaluation metrics — such as the simulation and detection scores used in Paulo et al. (2024) — measure whether an explanation helps predict activations on held-out examples, but they do not directly test whether the explanation captures the feature's causal role in the network's computation. Replacing the encoder with an LLM simulator and measuring the downstream effect on the model's output is a causal test: if the explanation faithfully describes the feature's function, the simulated activations should produce the same output as the original activations when fed through the decoder.

Practical implications for interpretability research. If the current generation of automated explanations cannot support even a partial rewrite of a single layer in a small model, this redirects research effort. It tells the community that incremental improvements to existing pipelines — slightly better scoring functions, slightly larger explanation models — are unlikely to close the gap. Something more fundamental is missing: the explanations lack the specificity to distinguish when a feature is active versus inactive. This finding has immediate practical consequences for how interpretability researchers should allocate effort, pointing toward contrastive methods, higher-resolution feature descriptions, and evaluation protocols that penalize false positives as heavily as they reward true positives.

Where Prior Approaches Fall Short

Sparse autoencoders find interpretable features but don't guarantee faithful explanations. SAEs (Cunningham et al., 2023) have become the dominant tool for extracting interpretable features from LLM activations. The architecture is straightforward: an encoder compresses activation vectors into a sparse, higher-dimensional latent space via a learned overcomplete basis with an L1 sparsity penalty (or, more recently, a TopK constraint as in Gao et al., 2024), and a decoder reconstructs the original activation from these sparse latents. When trained successfully, individual latent dimensions often correspond to interpretable concepts — specific words, syntactic patterns, semantic categories, or higher-level abstractions.

Recent work has scaled SAEs dramatically: Gao et al. (2024) trained them on GPT-4, and Templeton et al. (2024) trained them on Claude 3 Sonnet, finding features corresponding to complex concepts like "inner conflict," "sycophancy," and "deception." This scaling demonstrated that sparse decompositions can extract interpretable structure from frontier models. But — and this is the gap the current paper exploits — finding features and explaining them are different tasks. The existence of interpretable-looking features does not guarantee that our natural language descriptions of those features are sufficiently precise to reproduce their behavior. An SAE latent might activate on "text about animals" in 80% of cases, but the remaining 20% — where it activates on non-animal text or fails to activate on animal text — might be where the functionally important computation lives. Standard interpretability metrics don't capture this precision.

Transcoders offer a more direct path to rewriting, but the explanation bottleneck remains. Dunefsky et al. (2024) introduced sparse transcoders as an alternative to SAEs specifically designed for understanding feedforward network (MLP) layers. Rather than reconstructing residual stream activations, a transcoder is trained to directly predict the output of an MLP layer given its input, using a sparse latent bottleneck with the same architecture as an SAE:

$f(x) = W_2 \cdot \text{TopK}(W_1 x + b_1) + W_{\text{skip}} x + b_2$

The key advantage for the rewriting agenda is that a transcoder can entirely replace the original MLP in the model. If the transcoder approximates the MLP well, and if its latents are interpretable, and if we can simulate those latents from explanations, then we have rewritten that component of the model in natural language. The transcoder architecture makes the rewriting experiment clean: the decoder ( $W_2$ , the skip connection, and the bias) remains unchanged, and we only replace the encoder ( $W_1$ , the part that produces sparse activations from the input). This is a more principled decomposition than attempting to explain raw MLP neurons, which are known to be highly polysemantic (Elhage et al., 2022; Gurnee et al., 2023).

However, transcoders inherit the same fundamental limitation as SAEs for the rewriting task: the quality of the rewrite depends entirely on the quality of the explanations. The transcoder gives us a set of latents that are more interpretable than raw neurons, but it does nothing to ensure that our natural language descriptions of those latents are functionally adequate.

Automated interpretability pipelines produce explanations, but their faithfulness is unvalidated. Bills et al. (2023) pioneered the approach of using an LLM to generate natural language explanations of individual neurons, showing GPT-4 context windows where a neuron activated and asking it to describe the pattern. Paulo et al. (2024) scaled this to millions of features across SAE latents, building an automated pipeline that: (1) collects activating and non-activating examples for each latent, (2) prompts an LLM to generate an explanation from these examples, and (3) evaluates the explanation using "simulation" and "detection" scores that measure how well another LLM can predict the latent's activations on held-out examples given the explanation.

These evaluation scores are the state of the art for measuring explanation quality, but they have a critical blind spot: they test correlation, not causal faithfulness. A high simulation score means the predictor LLM's activations correlate with the true activations in a statistical sense, but it does not test whether those predicted activations, when inserted into the model, produce the same downstream behavior as the original activations. The detection score — which is what the current paper uses to select "top scoring" latents for partial rewriting — measures whether an LLM can classify whether a latent is active or not, given the explanation and context. But even perfect detection doesn't guarantee functional faithfulness: knowing that a feature fires is not the same as knowing exactly how strongly it fires, or understanding the subtle contextual factors that modulate its activation level.

The current paper's central insight is that the rewriting experiment closes this gap. If an explanation perfectly captures a feature's function, then an LLM reading that explanation should produce activations that — when decoded through the same $W_2$ matrix — produce the same MLP output, and therefore the same model behavior. Any deviation measures the explanation's unfaithfulness.

Prior rewriting attempts are largely aspirational. The concept of rewriting neural networks in interpretable form appears in several theoretical proposals (the microscope AI framework, imitative generalization) but has seen almost no empirical realization. This is understandable: it requires solving feature extraction, explanation generation, and faithful simulation simultaneously. The current paper is, to the authors' knowledge, the first to attempt even a partial rewrite — a single MLP layer in a small model — and to quantify the results against meaningful baselines (zero ablation, random latent substitution). This makes the negative result particularly valuable: it establishes a concrete performance floor and identifies the specific failure mode (insufficient specificity) that future work must address.

How This Paper Positions Itself

The paper positions itself not as a solution but as a diagnostic: a methodology for measuring whether our explanations are good enough for the task that mechanistic interpretability ultimately aims to accomplish. The key framing move is to treat the rewriting experiment as an evaluation protocol for explanation quality, rather than as an end in itself.

This is a significant departure from how the interpretability community typically evaluates explanations. Standard practice is to measure explanation quality using held-out prediction accuracy — can a separate model predict a feature's activations given the explanation? The paper argues, implicitly, that this metric is insufficient because it doesn't test whether the explanation captures the functional role of the feature. A predictor might achieve high accuracy by learning spurious correlations (e.g., "this feature activates on text containing the word 'the'" would be highly predictive but functionally vacuous), but those predictions would fail to preserve model behavior when inserted back into the network.

By measuring the downstream effect on the model's cross-entropy loss for next-token prediction, the paper evaluates explanations on a criterion that cannot be gamed by shallow correlations: do the simulated activations produce the same output distribution as the original activations? This is a causal fidelity test. The paper compares performance against two critical baselines:

Zero ablation: simply setting the entire MLP output to the zero vector. This represents the performance of a model with the layer completely removed. If simulated activations cannot outperform zero ablation, then the explanations provide essentially no useful information — the simulator's predictions are no better than guessing "this feature is always inactive."
Random latent substitution: replacing the activations of randomly selected latents with their simulated counterparts. If the "top scoring" latents (those with the highest detection scores from Paulo et al., 2024) do not substantially outperform random selection, then the interpretability scores are not selecting for features whose explanations are functionally faithful.

The paper's theoretical contribution is not a new explanation method or a better feature extraction technique, but rather an experimental protocol that operationalizes the question "are our explanations faithful?" in a way that produces a clear, quantitative answer. The answer, in this case, is "no, not yet" — but the protocol itself is reusable and scalable. Future explanation methods can be benchmarked using the same rewriting experiment, creating a feedback loop between explanation quality and functional faithfulness.

The paper also positions itself as a bridge between two largely disconnected research threads: the automated interpretability work that generates explanations at scale (Bills et al., 2023; Paulo et al., 2024) and the transcoder/SAE work that extracts sparse features (Cunningham et al., 2023; Dunefsky et al., 2024; Templeton et al., 2024). By requiring both high-quality features and high-quality explanations simultaneously, the rewriting experiment creates pressure to improve both components jointly rather than optimizing each in isolation.

3. Technical Approach

3.1 Reader Orientation

The system being built is a partially rewritten transformer: a Pythia 160M language model where one feedforward network (MLP) layer's internal computation has been replaced by an LLM-based simulator that reads natural language explanations of each feature and predicts their activation values, which are then decoded through the original learned weights to produce the layer's output. This solves the problem of evaluating explanation faithfulness by creating a causal test — if the explanations truly capture what each feature computes, an external LLM reading only those explanations should be able to reproduce the original activations well enough to preserve the model's next-token prediction behavior.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a pipeline:

Pythia 160M base model (Biderman et al., 2023) — a 160-million-parameter transformer language model. The target for rewriting is its sixth-layer MLP specifically (and, in a parallel experiment, the residual stream at that same layer).
Sparse transcoder (or sparse autoencoder for the residual stream experiment) — a wider MLP with a sparsity constraint (TopK, k=32) trained to approximate the target MLP's output function. The transcoder decomposes the dense MLP computation into 32,768 sparse latent features, each with an associated decoder direction vector in the matrix $W_2$ . This produces ground-truth activations that the simulator must match.
Automated interpretability pipeline (Paulo et al., 2024) — takes the transcoder latents as input, samples contexts where each latent fires, and prompts Llama 3 Instruct 8B to generate a single-sentence natural language explanation for each latent. Also produces detection scores measuring how well each explanation predicts activation on held-out examples.
LLM-based simulator — a separate instance of Llama 3 Instruct 8B that receives, for each latent and each text context, the latent's natural language explanation plus the surrounding text tokens. It predicts a number from 0 to 9 indicating how strongly that latent should activate on the final token of the context. This yields a vector of 32,768 predicted activation values per token.
Post-processing and evaluation — the simulator's raw predictions undergo quantile normalization (a per-latent monotonic transformation that matches the marginal distribution of predicted activations to the true activation distribution), then a TopK sparsity constraint (k=32) is applied. The resulting vector is decoded through the transcoder's $W_2$ matrix, skip connection, and bias to produce the MLP output, which replaces the original MLP output in Pythia. Cross-entropy loss on next-token prediction is measured over 10K prompts.

Information flows as follows: text tokens enter Pythia layer 6 → the transcoder encoder ( $W_1 x + b_1$ ) produces ground-truth sparse activations (used only for evaluation, not for the rewrite) → in parallel, for each of the 32,768 latents, the LLM simulator reads that latent's explanation plus the text context and outputs a predicted activation → quantile normalization calibrates these predictions per-latent to match the true marginal distribution → TopK enforces sparsity → the calibrated activations are decoded through $W_2$ and summed with the skip connection and bias → the resulting vector replaces the original MLP output → the rest of Pythia's layers process this vector normally → cross-entropy loss is computed on the final token prediction.

3.3 Roadmap for the Deep Dive

First, the sparse transcoder architecture and training (Section 3.4.1), since it defines the feature representation that everything else depends on — including its functional form, sparsity mechanism, skip connection, loss function, and training hyperparameters.
Second, the automated explanation pipeline (Section 3.4.2), which generates the natural language descriptions that the simulator will use — covering how activating contexts are sampled, how explanations are generated, and how detection scores quantify explanation quality.
Third, the LLM-based simulator (Section 3.4.3), which is the core of the rewriting experiment — how it is prompted, what it predicts, and how its raw outputs are collected across all latents and contexts.
Fourth, quantile normalization (Section 3.4.4), which is the critical calibration step without which the rewriting completely fails — including why raw predictions are miscalibrated, how the quantile normalizer is estimated, and the sample-size dependence.
Fifth, the partial rewriting procedure (Section 3.4.5), which mixes ground-truth and simulated activations for the top-scoring versus randomly selected latents, enabling the analysis of how explanation quality correlates with rewriting success.
Sixth, the evaluation protocol (Section 3.4.6), which measures cross-entropy loss on held-out text and compares against the zero-ablation and random-substitution baselines.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical diagnostic paper whose core idea is that replacing a neural network component's encoder with an LLM-based simulator — and measuring the downstream effect on the model's behavior — provides a causal test of whether natural language explanations faithfully capture the component's computation. The paper does not propose a new explanation method; it proposes a methodology and applies it to existing explanations to reveal their inadequacy.

3.4.1 Sparse Transcoder Architecture and Training

The transcoder is the feature extraction mechanism that defines what the explanations describe. Without a transcoder, one would have to explain raw MLP neurons, which are known to be highly polysemantic — each neuron fires in numerous semantically unrelated contexts (Elhage et al., 2022; Gurnee et al., 2023). The transcoder provides a sparse, higher-dimensional latent space where individual dimensions are more likely to correspond to coherent, interpretable concepts. This is the same motivation behind sparse autoencoders on the residual stream, but applied specifically to the MLP layer's function.

Target of approximation. The transcoder is trained to approximate the output of the MLP in layer 6 of Pythia 160M (Biderman et al., 2023). Layer 6 is a specific choice — the authors do not explicitly justify why this layer rather than, say, layer 3 or layer 10, but the practical constraint is that replacing a single MLP already degrades performance substantially (to the level of a Pythia checkpoint trained on only 25% of the data), and rewriting that replacement in natural language degrades it further. Replacing all MLP layers simultaneously would render the model "completely unusable," so a single representative layer is chosen as a testbed.

Functional form. The transcoder has the following architecture:

$f(x) = W_2 \cdot \text{TopK}(W_1 x + b_1) + W_{\text{skip}} x + b_2$

where $x$ is the input activation vector to the MLP (a vector in Pythia 160M's residual stream dimension), $W_1$ is the encoder weight matrix (mapping from the input dimension to the 32,768-dimensional latent space), $b_1$ is the encoder bias, $\text{TopK}$ is the sparsity-enforcing activation function that zeroes out all but the k=32 largest entries in the pre-activation vector, $W_2$ is the decoder weight matrix (mapping from the 32,768-dimensional latent space back to the MLP's output dimension), $W_{\text{skip}}$ is a learned linear skip connection, and $b_2$ is a learned bias.

What it computes: given an input vector $x$ from the residual stream at layer 6, the transcoder first projects $x$ into a 32,768-dimensional latent space via $W_1$ and adds bias $b_1$ , producing a pre-activation vector. It then applies the TopK operator, which retains only the 32 largest entries and sets all others to zero, producing a sparse latent vector $z$ with exactly 32 non-zero values. This sparse vector $z$ is projected back to the original output space via $W_2$ , producing the primary reconstruction. The skip connection $W_{\text{skip}} x$ passes the input through linearly, and the bias $b_2$ is added. The final output $f(x)$ is the sum of the sparse reconstruction and the skip-path output.

Why this form: the architecture is identical to that of a standard sparse autoencoder (encoder + sparsity + decoder), repurposed to predict MLP outputs rather than reconstruct inputs. The skip connection is a novel addition that the authors find "improves its ability to approximate the original MLP at no cost to interpretability scores." Without the skip connection, the transcoder must route all information through the sparse bottleneck — including linear transformations that the original MLP might compute efficiently. The skip connection offloads any approximately linear component of the MLP's function from the sparse latents, allowing the latents to focus on genuinely non-linear, potentially interpretable transformations. This is an important design insight: separating the MLP's computation into a linear component (handled by the skip connection) and a non-linear sparse component (handled by the TopK pathway) may produce more interpretable latents than forcing everything through the bottleneck.

Sparsity mechanism: TopK. Rather than using an L1 penalty on the latents (the traditional SAE approach from Cunningham et al., 2023), the transcoder uses the TopK activation function proposed by Gao et al. (2024) with $k = 32$ . The TopK operator sorts all entries in the pre-activation vector and zeros out all except the 32 largest values. This enforces exact sparsity — exactly 32 latents are active per token, no more, no less — rather than the soft sparsity induced by L1 regularization, which only encourages most latents to be near-zero. The choice of $k = 32$ is not explicitly justified but is likely chosen to balance reconstruction fidelity (more active latents = better approximation) against interpretability (fewer active latents per token = less superposition and easier attribution).

Training loss and procedure. The loss function is pure mean squared error (MSE) between the transcoder's output and the original MLP's output:

$\mathcal{L} = \frac{1}{n} \sum_{i=1}^{n} \| f(x_i) - \text{MLP}(x_i) \|_2^2$

where $x_i$ is the input to the MLP at token position $i$ , $f(x_i)$ is the transcoder's output, $\text{MLP}(x_i)$ is the original MLP's output, and $n$ is the batch size.

What it computes: for each token in each training sequence, the transcoder processes the same input as the original MLP and produces a predicted output vector. The MSE loss penalizes the squared Euclidean distance between the predicted and actual MLP outputs, summed over all output dimensions. Lower MSE means better reconstruction fidelity.

Why this form: MSE is the standard reconstruction objective for autoencoders because it corresponds to maximum likelihood under a Gaussian noise model. Crucially, there are no auxiliary loss terms — no sparsity penalty, no regularization on the weights, no commitment loss. All sparsity is enforced structurally through the TopK operator rather than through optimization. This simplifies training (no need to tune a sparsity coefficient) and guarantees exact sparsity levels rather than the soft, approximate sparsity of L1-based approaches. The downside is that the sparsity level $k$ is a fixed hyperparameter rather than an adaptive quantity, but for a controlled experimental setup this is an advantage.

Training hyperparameters. The paper states: "We train over the first 8B tokens of Pythia's training corpus, the Pile (Gao et al., 2020), using the Adam optimizer (Kingma, 2014), a sequence length of 2049, and a batch size of 64 sequences." This means each training step processes $64 \times 2049 = 131,136$ tokens concurrently. Adam is the standard adaptive optimizer for transformer-scale models. The specific learning rate, betas, and weight decay are not reported in the main text, which is an omission — these matter for reproduction. The 8B token training budget is a significant fraction of Pythia 160M's total training data (which is trained on the full Pile), suggesting that achieving good approximation quality requires substantial data.

Initialization strategy. Both $W_2$ (the decoder) and $W_{\text{skip}}$ are zero-initialized, and $b_2$ is initialized to the empirical mean of the MLP outputs over the training data. This means that at the start of training, the transcoder is a constant function: regardless of the input, it outputs the average MLP output vector. The TopK pathway contributes nothing initially because $W_2$ is zero, and the skip connection contributes nothing because $W_{\text{skip}}$ is zero. The transcoder must learn, through optimization, to deviate from this constant baseline to capture the input-dependent variation in the MLP's output. This initialization strategy ensures that learning starts from a stable, well-defined baseline and that the sparse pathway and skip connection develop incrementally.

Sparse autoencoder for the residual stream. In a parallel experiment, the authors train a standard sparse autoencoder on the residual stream of the same layer 6, "with the same training conditions as the transcoder, but without a skip connection." The SAE's architecture is the standard encoder-decoder form: $\text{SAE}(x) = W_2 \cdot \text{TopK}(W_1 x + b_1) + b_2$ , where the input $x$ is the residual stream activation vector and the output is a reconstruction of that same vector. This SAE serves as an alternative feature extraction method — instead of decomposing the MLP's function, it decomposes the residual stream's information content. The rewriting experiment then replaces the SAE encoder (the part that produces sparse latents from the residual stream) with the LLM simulator and decodes through the SAE's $W_2$ .

Dimension matching. The transcoder has 32,768 latent dimensions. This is substantially larger than the MLP's input and output dimensions (Pythia 160M has a hidden dimension of 768, and the MLP's intermediate dimension is 3072). The overcomplete latent space (32,768 > 3072) is what enables the sparse decomposition: with more available dimensions than the intrinsic dimensionality of the MLP's output manifold, the transcoder can assign distinct latent directions to distinct concepts without forcing superposition. The factor of roughly 10× expansion (32,768 / 3072 ≈ 10.7) is a typical ratio in SAE work; Gao et al. (2024) used similar expansion factors for their GPT-4 SAEs.

3.4.2 Automated Explanation Pipeline

Once the transcoder is trained, each of its 32,768 latent dimensions produces an activation value for every token in every text processed by the model. The goal of the explanation pipeline is to produce a single-sentence natural language description of what each latent dimension represents — what pattern in the input text causes it to fire.

Pipeline source. The authors "use the automated interpretability pipeline released by Paulo et al. (2024) to generate explanations and scores for transcoder and SAE latents." This pipeline has three stages: (1) collecting activating examples for each latent, (2) prompting an LLM to generate an explanation from those examples, and (3) evaluating the explanation using simulation and detection scores. The current paper modifies only the detection stage for their rewriting experiment, changing the task from simple detection ("is this latent active?") to graded prediction ("how strongly is this latent active?" — what they call "single token simulation").

Example collection. For each of the 32,768 latent dimensions, the pipeline samples text contexts from the Pile where that latent fires strongly (producing high activation values). The specific number of positive and negative examples is not stated in the main text, but Figure 3's caption indicates that detection scores are computed "over 100 positive and 100 negative samples" per latent. These 200 examples per latent (totaling $32,768 \times 200 = 6,553,600$ example contexts across all latents) form the "evidence" that the explanation-generating LLM sees.

Explanation generation. An LLM (likely Llama 3 Instruct 8B, though the paper does not specify which model generates the explanations versus which model performs the simulation — Paulo et al., 2024 uses GPT-4 for explanation generation but the current paper may use Llama based on the acknowledgments of using Coreweave computing resources) receives the activating contexts and is prompted to "summarize or otherwise find patterns in the activations and output a simple, single sentence explanation for that latent." The output is a natural language string, e.g., something like "this feature activates on tokens that are part of mathematical expressions" or "this feature activates on the word 'the' when it follows a verb."

Detection scoring. For each latent's explanation, the pipeline evaluates how well it enables another LLM to detect whether the latent is active on held-out contexts. Specifically (Paulo et al., 2024, page 5): given the explanation and a text context, the LLM predicts whether the latent's activation exceeds some threshold. The detection score is a measure of classification accuracy — it captures how well the explanation distinguishes activating from non-activating contexts. The current paper uses these detection scores to rank latents by interpretability, selecting the "top scoring" latents for partial rewriting experiments (Figure 3, labeled "Top scoring").

The detection score's role in the paper. Detection scores serve as the paper's proxy for explanation quality: latents with higher detection scores should, in principle, have more faithful explanations. The partial rewriting experiment tests this directly — if detection scores capture faithfulness, then replacing high-scoring latents should preserve model performance better than replacing low-scoring latents (which the "random sampling" condition implicitly does, since random selection includes both high- and low-scoring latents equally). The fact that top-scoring substitution performs no better than zero ablation (Figure 3) suggests that even the best detection-scored explanations lack functional faithfulness.

Fuzzing scores as an alternative. Appendix Figure A4 shows that "fuzzing scores" — another metric from Paulo et al. (2024) that measures how well an explanation survives perturbations to the examples — also predict sensitivity and specificity of the simulator. This provides convergent evidence that the automated pipeline's metrics correlate with simulator performance, even though the absolute performance is poor.

3.4.3 The LLM-Based Simulator

The simulator is the component that replaces the transcoder's encoder ( $W_1 x + b_1$ ). Instead of computing pre-activations from the input vector via a learned linear transformation, the simulator reads the natural language explanation of each latent and the text context, and directly predicts how strongly that latent should activate.

Architecture of the simulation. The simulator is a separate instance of Llama 3 Instruct 8B (Dubey et al., 2024), an 8-billion-parameter instruction-tuned language model. It is not fine-tuned on the transcoder activations — it is used zero-shot, relying entirely on its pre-trained language understanding capabilities to interpret the explanation and map it onto the text context. This is a critical design choice: if the simulator required fine-tuning, one could argue that it is learning to predict activations from spurious statistical patterns in the training data rather than genuinely understanding the explanation. The zero-shot setting forces the simulator to rely on the explanation's semantic content.

Prompt structure. The simulation prompt is reproduced in full in Appendix A. It tells Llama:

"You are an intelligent and meticulous linguistics researcher. You will be given a certain explanation of a feature of text, such as 'male pronouns' or 'text with negative sentiment' and examples of text that contains this feature. Some explanations will be given a score from 0 to 1. The higher the score the better the explanation is, and you should be more certain of your response (positive or negative). These features of text are normally identified by looking for specific words or patterns in the text. There are many features associated with a single token, and sometimes the feature is related with the previous token or context. Your job is to identify how much the last token, which is marked between << and >>, represents the feature."

The prompt then instructs the model to output "a integer between 0 and 9, where 0 corresponds to no relation to the explanation and 9 to a strong relation." It emphasizes that "most of the tokens should have no relation" and that intermediate values should be used sparingly — "the ones that are related, should more likely be given 1 than 2, 2 than 3, and so on. Only give a 9 if the description exactly matches the token."

What this prompt does: it frames the simulation task as a linguistics research activity, provides the latent's natural language explanation, marks the target token with << >> delimiters, and asks for a graded relevance judgment on a 0–9 integer scale. The prompt includes calibration instructions (encouraging low values, discouraging unjustified high values) that attempt to counteract the LLM's known tendency toward overconfidence.

How predictions are collected. For each of the 32,768 latents, and for each of the 10K evaluation prompts (for the transcoder) or 1K prompts (for the SAE), the LLM receives the latent's explanation plus the text context and outputs a number from 0 to 9. The paper states: "We record the probability that Llama assigns to each of the ten numbers, and compute the expected value." This means they do not simply take the argmax token (the single most likely integer) — they use the full probability distribution over $\{0, 1, \ldots, 9\}$ output by Llama's softmax and compute the weighted average:

$\text{predicted\_activation}_{\text{raw}} = \sum_{v=0}^{9} v \cdot P(\text{output} = v | \text{explanation}, \text{context})$

This expected value is a continuous scalar representing the simulator's "best guess" for how strongly the latent activates, accounting for uncertainty across the integer scale.

Scale of the simulation. The paper states: "Simulating activations over 10K prompts requires individually prompting a model for 32768 latents, for a total of 327 million predictions, an expensive endeavor." This is not a trivial computation — 327 million forward passes through an 8B-parameter model, each producing a single expected-value activation for one latent on one context. This computational cost limits the number of prompts to 10K (for the transcoder) and 1K (for the SAE, which was presumably run later with a smaller budget), and it prevents scaling to the 100K prompts that the authors speculate would improve quantile normalization quality.

The calibration problem. The raw predictions from Llama are systematically miscalibrated. Figure 1 (left) shows the distribution of predicted activations before normalization, and the text states: "the predictor model systematically over-predicts high activation values by multiple orders of magnitude." This means that when the simulator sees an explanation that describes a concept present in the text, it disproportionately outputs high values (7–9), even if the true latent activation is moderate or zero. Conversely, when the concept is absent, it may still output non-zero values from spurious correlations. This calibration failure is the central practical challenge that quantile normalization (Section 3.4.4) addresses.

Why the over-prediction happens. The paper does not analyze the root cause in depth, but it can be understood as a consequence of the prompt design and LLM priors. The prompt asks "how much does the last token represent the feature?" — but the LLM has no training signal for what "represents" means in the context of transcoder activation values, which are continuous quantities produced by a geometrically meaningful projection ( $W_1 x + b_1$ ) that bears no inherent relationship to the 0–9 integer scale. The LLM's natural language understanding of "represents" is semantic and categorical, while the transcoder's activation is geometric and graded. The simulator defaults to mapping semantic relevance to high numerical values because, in most linguistic tasks, "being relevant" is a strong binary signal, not a finely graded quantity.

3.4.4 Quantile Normalization

Quantile normalization is the critical calibration step that makes the rewriting experiment work at all. Without it, the model's performance degrades catastrophically — Figure A5 shows that unnormalized predictions produce cross-entropy loss "equivalent to barely training the model at all." With it, the rewritten model's loss becomes comparable to (though not better than) zeroing out the MLP output entirely.

What quantile normalization does. For each individual latent dimension $i$ (index from 1 to 32,768), quantile normalization applies a monotonic transformation $T_i$ to the raw predicted activation $\hat{a}_i$ such that the marginal distribution of the transformed values $T_i(\hat{a}_i)$ matches the marginal distribution of the true activation values $a_i$ (computed directly from the transcoder). Formally, if $F_i$ is the cumulative distribution function (CDF) of the true activations and $G_i$ is the CDF of the raw predicted activations, the quantile normalizer is:

$T_i(\hat{a}) = F_i^{-1}(G_i(\hat{a}))$

where $G_i(\hat{a})$ is the percentile of the predicted activation within its empirical distribution, and $F_i^{-1}$ maps that percentile back to the corresponding value in the true activation distribution.

What it computes, operationally: for a given latent $i$ , the procedure is: (1) collect a large sample of raw predicted activations $\hat{a}_i$ (from the simulator on the evaluation prompts) and a large sample of true activations $a_i$ (from the transcoder on a separate, much larger dataset of 10M tokens); (2) estimate the empirical CDF of each sample; (3) for each new predicted activation $\hat{a}_i$ , find its percentile rank in the predicted distribution; (4) output the true activation value at that same percentile rank. If the raw prediction is at the 95th percentile of all predictions for this latent, the normalized value will be the 95th percentile of all true activations for this latent, regardless of the absolute numerical values.

Why this form: quantile normalization is an optimal transport map under a variety of cost functions (Santambrogio, 2015), meaning it is the monotonic transformation that minimizes the expected squared error between the transformed predictions and the true values subject to the constraint that the marginal distributions match exactly. Unlike linear rescaling (which only corrects mean and variance), quantile normalization corrects the entire shape of the distribution — it handles skew, kurtosis, and multi-modality. This matters because the raw predictions are not just shifted or scaled; they are qualitatively different in distribution (over-predicting high values by orders of magnitude, as Figure 1 left shows). Linear rescaling cannot fix this.

The key role of specificity. The paper explains why quantile normalization works — and what it costs — through the concepts of sensitivity and specificity. A latent is "active" if its true activation is non-zero (after TopK, only 32 of 32,768 latents are active per token). The simulator's raw predictions have low specificity: they predict many latents as active that are actually inactive. With 32,768 latents and TopK=32, only 0.098% of latents are active per token. If the simulator has specificity of 99% — meaning it correctly identifies 99% of inactive latents as inactive — it will still misclassify 1% of the 32,768 inactive latents as active, producing approximately 320 false positives. That is 10 times more than the 32 true positives we want.

Quantile normalization addresses this by zeroing out low-percentile predictions. Because the true activation distribution for most latents has a large spike at exactly zero (corresponding to the 32,768 - 32 = 32,736 latents that are inactive per token), the lower percentiles of the true distribution are zero. When the quantile normalizer maps low predicted percentiles to the corresponding true percentiles, those predictions become exactly zero. This "significantly increases the specificity," as the paper states. But it comes at a cost: it also "significantly decreases the sensitivity," because some correctly predicted active latents (true positives) that happened to receive lower raw predictions will also be zeroed out if they fall below the threshold.

Tradeoff and why it caps performance. The quantile normalizer enforces the correct marginal distribution, which means it enforces the correct prior probability of each latent being active. But it does not (and cannot) correct for which specific latents are predicted active on which specific tokens. If the simulator's ranking of latents per token were perfect — if it always assigned higher raw scores to truly active latents than to inactive ones — then quantile normalization would produce zero error. The fact that performance only reaches the zero-ablation baseline (Figure 3) means the simulator's per-token rankings are no better than random after accounting for the marginal distribution. The explanations don't provide information about which latents fire on which tokens beyond what you could infer from the baseline activation frequencies alone.

Sample size dependence. The paper emphasizes that the quality of quantile normalization depends heavily on the sample size used to estimate the predicted activation distribution. They compute true activation quantiles from 10M tokens but can only afford 10K tokens for the predicted activations (due to the 327 million LLM predictions required). Appendix B explores this: "If we use only 1K samples instead of 10K samples, the mismatch between the normalized and the empirical distribution grows larger (Figure A1). We then expect that a larger number of samples would lead to a better convergence."

The mismatch arises because activations are sparse and heavy-tailed: a latent fires strongly on only a tiny fraction of tokens, so estimating its full distribution requires seeing enough of those rare events. With 10K samples, many latents will have only a handful of non-zero values in the empirical sample, making the CDF estimate noisy. The paper demonstrates this concretely: using only 1K prompts to estimate the quantile normalizer leads to "much worse CE loss when performing substitution" (Figure A2).

Bias correction discussion. The paper notes a subtle statistical issue: "while the empirical CDF is an unbiased estimator for the true CDF, the empirical inverse CDF (or quantile function) is biased for the true inverse CDF." This means that the quantile normalizer, which uses the empirical inverse CDF, systematically differs from the true population quantile function due to finite-sample effects. The authors mention plans to experiment with bias-corrected estimators (Hyndman & Fan, 1996) in a future draft, but the current version does not implement this. In practice, since they have 1000× more data for the true activation quantiles (10M tokens) than for the predicted activation quantiles (10K tokens), the dominant source of error is likely the finite-sample variance in the predicted CDF rather than the bias in the true inverse CDF.

Using the empirical prompt distribution. An interesting variant shown in Appendix Figure A3 uses the empirical distribution of true activations over the evaluation prompts themselves (rather than the 10M-token corpus) as the target for quantile normalization. This significantly improves the rewritten model's performance, and the authors hypothesize that with a large enough sample (e.g., 100K prompts), the prompt-specific empirical distribution would converge to the corpus distribution, making this the preferred approach. The improved performance likely occurs because matching the activation distribution exactly on the evaluation prompts reduces the variance from distribution mismatch, even though the underlying quantile estimates are noisier.

3.4.5 Partial Rewriting Procedure

The paper does not only rewrite the entire transcoder encoder at once; it also experiments with partial rewriting, where only a subset of the 32,768 latents are simulated and the rest retain their ground-truth activations from the transcoder. This enables analysis of how the fraction of rewritten latents affects performance and whether selecting latents by interpretability score produces better results than random selection.

Two selection methods. For a given fraction $f$ of latents to rewrite (e.g., 20%, 40%, 60%, 80%, 100%):

Top scoring: select the $f \times 32768$ latents with the highest detection scores from the automated interpretability pipeline (Paulo et al., 2024). These are the latents whose explanations were judged most predictive of activation on held-out examples, and should therefore be the easiest for the simulator to predict accurately. The remaining $(1-f) \times 32768$ latents use their ground-truth transcoder activations.
Random sampling: select $f \times 32768$ latents uniformly at random from all 32,768 latents. This serves as a control — if top-scoring selection provides no advantage over random selection, then detection scores are not identifying latents with functionally faithful explanations.

Mixing procedure. For each token, the system produces a hybrid activation vector: for the selected latents (either top-scoring or random), the activation is the quantile-normalized simulator prediction; for the non-selected latents, the activation is the ground-truth transcoder activation (computed from $\text{TopK}(W_1 x + b_1)$ ). This hybrid vector — which may have more or fewer than 32 non-zero entries depending on how simulator predictions interact with the ground-truth activations — is then passed through the decoder $W_2$ , skip connection, and bias to produce the MLP output.

The role of partial rewriting in evaluating explanations. The partial rewriting results in Figure 3 reveal a crucial pattern: when replacing top-scoring latents (green bars), the cross-entropy loss increase is comparable to zero ablation (blue bars) at all fractions. When replacing randomly selected latents (orange bars), the loss increase is worse than zero ablation at all fractions — meaning the simulator's predictions for random latents are actively harmful, degrading performance beyond simply removing the latents' contributions entirely.

This has a precise interpretation: for the top-scoring latents, the simulator's predictions are approximately as useful as setting those latents to zero. Neither helps nor hurts beyond the baseline of removing the information. For randomly selected latents, the simulator's predictions are worse than zero — they inject noise that the decoder misinterprets, producing outputs that actively damage the model's predictions. This implies that even the "best" explanations produce simulated activations that contain essentially no information about the correct activation values beyond what you would get by ignoring the explanation entirely and setting everything to zero.

3.4.6 Evaluation Protocol

The evaluation measures how much the model's behavior degrades when the transcoder's encoder is replaced by the simulator. The metric is cross-entropy loss for next-token prediction on held-out text, compared against baselines.

Evaluation data. For the transcoder experiments, the authors "sample chunks of text from the Pile and gather latents from the transcoder evaluated on the last token of each chunk," using 10K prompts. For the SAE experiments, they use 1K prompts (likely a smaller sample due to the computational cost of simulation). The Pile (Gao et al., 2020) is Pythia's training corpus, so the evaluation is on in-distribution text — this tests whether the explanations capture the features as they operate in their natural domain, not on out-of-distribution generalization.

Baselines reported.

Unmodified Pythia 160M: the original model's cross-entropy loss on the same prompts. For the set of prompts used, this is reported as $3.19 \pm 0.09$ nats per token (Figure 3 caption and Figure A5 caption).
Transcoder-replaced MLP (no simulation): the model where the original MLP is replaced by the transcoder running its own encoder (ground-truth activations from $\text{TopK}(W_1 x + b_1)$ ). This represents the performance ceiling for the rewriting experiment — the best the simulation could possibly achieve, since the transcoder's reconstruction is imperfect. The paper reports that this degrades performance "to that of an early Pythia checkpoint — namely one that was trained on only 25% of the data."
Zero ablation: the model where the entire MLP output of layer 6 is replaced with the zero vector. This represents the performance of simply removing the layer's contribution. If the simulator cannot outperform this baseline, then the explanations provide no useful information — the predictions are no better than assuming every feature is always inactive.
Random substitution at each fraction: explained in Section 3.4.5 above.

Confidence intervals. Figure 3 reports "bar heights represent the median value of the absolute difference, because the distribution is heavy-tailed, and error bars are 95% confidence intervals computed using bootstrapping." The use of medians rather than means suggests that cross-entropy loss across prompts has a long right tail — some prompts experience catastrophic degradation while most are moderately affected. Bootstrapping provides non-parametric confidence intervals, appropriate given the non-normal distribution. The $\pm 0.09$ error on the baseline Pythia loss (3.19 nats) is a standard error or bootstrap CI; the text does not specify which.

Why cross-entropy loss? Cross-entropy loss measures the model's uncertainty in next-token prediction: lower loss means the model assigns higher probability to the correct next token. It is a continuous, differentiable metric that aggregates the model's behavior across the entire vocabulary, making it sensitive to subtle changes in the output distribution that accuracy metrics (like exact match) might miss. For a rewriting experiment, this sensitivity is important — even if the rewritten model still predicts the correct token most of the time, elevated cross-entropy indicates that the probability distribution has become less concentrated on the correct answer, meaning the model is less "certain" and potentially more likely to fail on harder tokens. The metric also enables direct comparison to Pythia checkpoints trained on subsets of the data, providing an interpretable scale: "loss equivalent to a model trained on X% of the data" is a meaningful benchmark.

Computational cost of evaluation. The evaluation requires, for each of the 10K prompts, running 32,768 separate LLM predictions (one per latent), each requiring the full context window to be processed by Llama 3 Instruct 8B. This is 327 million forward passes of an 8B-parameter model, which the authors describe as "an expensive endeavor." This cost is the reason for the limited sample size — the paper acknowledges that scaling to 100K prompts would likely improve quantile normalization and potentially change the results, but the computational budget does not permit it. This is an important practical limitation of the methodology, and it means the current results should be interpreted with appropriate caution about finite-sample effects.

4. Key Insights and Innovations

Innovation 1: Reframing Explanation Evaluation as a Causal Rewriting Test

The paper's most distinctive conceptual move is not the specific rewriting architecture but the evaluation philosophy it embodies. Prior work on automated interpretability — Bills et al. (2023), Paulo et al. (2024), and the broader SAE explanation literature — evaluates explanations using held-out prediction accuracy: can a separate model predict a feature's activations given its explanation? This is fundamentally a correlational test. An explanation can achieve high simulation or detection scores by capturing surface-level statistical patterns ("this feature activates on text containing the word 'the'") without describing the feature's functional role in the network's computation.

The field's implicit assumption has been that improving these correlational metrics would eventually yield faithful explanations. This paper challenges that assumption directly by proposing a different kind of test entirely: can the explanation be used to replace the original computation while preserving the model's output behavior? This is a causal fidelity test. The explanation is not evaluated on whether it correlates with activations in held-out data — it is evaluated on whether, when inserted into the model's forward pass as a substitute for the learned encoder, the downstream probability distribution over next tokens remains unchanged.

This reframing matters because it changes the target. In the correlational paradigm, an explanation that achieves 90% accuracy at predicting whether a feature fires is considered good. In the causal paradigm, that 10% error rate translates into wrong activations being decoded through the same $W_2$ matrix that the original model learned to interpret, producing MLP outputs that diverge from the original and cascading errors through subsequent layers. The paper demonstrates this concretely: even with quantile normalization correcting the marginal distribution, the simulator's per-token ranking of which latents should fire is insufficiently accurate to outperform the zero-ablation baseline (Figure 3). The explanations provide no functionally useful information beyond the baseline activation frequencies.

This is not an incremental improvement to explanation evaluation — it is a fundamental shift in the success criterion. The paper operationalizes the microscope AI agenda's core question: "do we understand this component well enough to rewrite it?" By providing a quantitative protocol and demonstrating that current explanations fail this test, the paper establishes a new bar that future interpretability work must clear. It also reveals something non-obvious about the correlational metrics: detection scores do correlate with simulator sensitivity and specificity (Figure 4, Figure A4), yet even the highest-scoring latents fail the causal test. This suggests that correlational metrics may be measuring something real but insufficient — a finding that would be invisible without the rewriting experiment.

Innovation 2: Identifying Specificity as the Critical Bottleneck, Not Just Activation Prediction

Prior work on automated interpretability focuses almost entirely on positive prediction: can the explanation help a model identify contexts where the feature is active? The simulation and detection scores in Paulo et al. (2024) are both framed around this question. The implicit assumption is that if an explanation correctly describes what makes a feature fire, then an LLM reading that explanation should be able to detect when the feature fires on new text.

This paper reveals that the complementary failure mode — false positives — is the actual bottleneck for functional faithfulness. The transcoder has 32,768 latents with only 32 active per token (TopK, k=32). This means 99.9% of latents are inactive at any given token. A simulator with 99% specificity — correctly identifying 99% of inactive latents as inactive — would still produce approximately 320 false positives per token, swamping the 32 true positives. The paper reports that the raw simulator has specificity around 80%, meaning it predicts roughly 6,400 latents as active per token — 200 times more than the correct number. Figure 1 (left) shows this visually: the raw predicted activation distribution massively over-represents high values compared to the true distribution, which is heavily concentrated near zero with sparse spikes.

This is a conceptual reframing of the explanation quality problem. It is not enough for an explanation to say "this feature fires on mathematical expressions" — it must also implicitly encode all the contexts where it does NOT fire, which is vastly more information. A feature that activates on "the word 'the'" is firing on roughly 7% of all tokens; a good explanation must enable the simulator to distinguish that 7% from the other 93%, not just identify instances of "the." The paper shows that current explanations fail primarily on this negative side: they are too broad, too generic, insufficiently contrastive. Quantile normalization partially compensates by zeroing out low-percentile predictions (Figure 1, right), which enforces the correct prior probability of each feature being active. But this is a distributional correction, not a semantic one — it helps with aggregate statistics but cannot fix per-token errors in which specific features are predicted active.

This finding has direct implications for future interpretability research. If specificity is the bottleneck, then improvements should focus on contrastive explanation methods — generating explanations that explicitly distinguish a feature from similar but distinct features, or that specify negative constraints ("this feature fires on mathematical expressions involving integrals, but NOT on expressions involving only arithmetic"). The paper explicitly calls for "using contrast pairs of highly similar features to bring out additional details" (Section 5), making the diagnosis actionable. This shifts the research agenda from "better positive example collection" toward "better negative/distractor example collection," which is not obvious from correlational metrics alone.

Innovation 3: A Rigorous Negative Result That Establishes a Performance Ceiling

In many fields, negative results are undervalued — they are harder to publish and less celebrated than positive ones. This paper is a deliberate, carefully constructed negative result, and its value lies in the precision and baselining of the failure. The key finding is not merely "the rewriting doesn't work well" — it is that the rewriting performs statistically indistinguishably from zeroing out the entire MLP output (Figure 3, blue bars vs. green bars). This is a specific, quantitative claim with strong baselines.

The zero-ablation baseline is crucial. Zeroing out an MLP layer removes all its contribution — it is equivalent to saying "we have no information about what this layer computes, so we set it to the identity function's additive contribution (zero)." If the simulator's predictions cannot outperform this baseline, then the explanations provide literally no useful information about the MLP's function. The simulator might as well be guessing randomly, and quantile normalization is merely enforcing the correct prior (how often each feature fires) without adding any per-token signal about which features should fire when.

This is a stronger statement than "explanations are imperfect" — it is "explanations, in their current form, contain essentially zero functional information." The paper quantifies this: the rewritten model's loss matches a Pythia checkpoint trained on only 10-15% of the data, which is what you get by removing the layer entirely. The gap between this and the transcoder baseline (which matches a checkpoint trained on 25% of the data) is the information lost due to explanation unfaithfulness, and it accounts for the entire performance degradation beyond what the transcoder's imperfect reconstruction already cost.

The random-substitution baseline (orange bars in Figure 3) adds further precision: substituting randomly selected latents performs worse than zero ablation, meaning the simulator's predictions for low-scoring latents are actively harmful — they inject noise that the decoder misinterprets. This establishes that detection scores do capture something real (top-scoring latents are less harmful than random ones), but not enough to reach functional usefulness.

This kind of rigorous negative result serves the field by establishing a clear performance ceiling. Future work on explanation quality can use this same protocol to measure progress: if a new explanation method achieves rewriting performance that significantly exceeds zero ablation, that is evidence of genuine improvement. Without this baseline, it would be impossible to distinguish truly better explanations from ones that merely achieve higher correlational scores through overfitting to the evaluation metric.

Innovation 4: Unifying the Transcoder and SAE Paradigms Under a Common Rewriting Protocol

The paper applies the same rewriting methodology to both a transcoder (approximating an MLP function) and a sparse autoencoder (reconstructing residual stream activations), treating them as interchangeable feature extraction backends. This is novel because prior work has treated transcoders and SAEs as distinct tools for different purposes — transcoders for understanding MLP function (Dunefsky et al., 2024), SAEs for understanding residual stream representations (Cunningham et al., 2023; Templeton et al., 2024). The rewriting protocol does not care which decomposition is used; it only requires that the component can be expressed as an encoder (producing sparse latents) followed by a decoder (mapping latents back to the original space), and that the encoder can be replaced by an LLM simulator.

This unification is significant because it suggests a general evaluation framework for interpretability methods that is independent of the specific decomposition technique. Any method that produces interpretable latents with an encoder-decoder structure — transcoders, SAEs, linear probes, PCA-based decompositions, clustering-based features — can be plugged into the rewriting protocol. The quality of the explanations is then measured by the same downstream metric (cross-entropy loss on next-token prediction), enabling direct comparison across methods. This is analogous to how standardized benchmarks (ImageNet, GLUE, etc.) enabled comparison across different model architectures in ML, but applied to the interpretability domain where such standardization has been notably absent.

The paper's finding that both the transcoder and SAE experiments yield similar results — rewriting performance comparable to zero ablation — provides convergent evidence that the bottleneck is explanation quality, not the feature extraction method. If one backend had performed dramatically better than the other, that would suggest that transcoders produce more "explainable" features than SAEs (or vice versa). The fact that both fail similarly suggests the problem lies in the explanation generation and simulation pipeline, not in the feature representation. This is a useful negative result: it tells the field not to chase marginal improvements in feature extraction as a path to better explanations, but to focus directly on explanation specificity and simulation fidelity.

5. Experimental Analysis

Evaluation Methodology

Dataset. All evaluations use text chunks sampled from the Pile (Gao et al., 2020), which is also Pythia 160M's training corpus. This means evaluation is on in-distribution text — the features are tested in their natural operating domain, not on out-of-distribution generalization. For the transcoder experiments, the authors use 10K prompts (text chunks); for the SAE experiments, they use 1K prompts (a smaller sample due to the computational cost of simulating all 32,768 latents through an 8B-parameter LLM for each prompt). The model's task is next-token prediction on the last token of each chunk.
Base model(s). The target model is Pythia 160M (Biderman et al., 2023), a 160-million-parameter autoregressive transformer language model. The specific layer targeted for rewriting is the sixth-layer MLP, and for the residual stream experiment, the residual stream activations at that same layer. The choice of a small model is pragmatically forced: replacing a single MLP with a transcoder already degrades performance to that of an early Pythia checkpoint (trained on ~25% of the data), and rewriting that transcoder in natural language degrades it further. Rewriting all MLP layers simultaneously "would likely cause the model to become completely unusable" (Section 3). The simulation is performed by Llama 3 Instruct 8B (Dubey et al., 2024), used zero-shot without fine-tuning on the transcoder activations — this is essential to the claim that the explanation, not learned spurious correlations, drives the prediction.
Metrics. The primary metric is cross-entropy loss for next-token prediction, measured in nats per token. Cross-entropy measures the negative log-probability the model assigns to the correct next token: lower loss means the model is more confident in the correct prediction. The paper reports both absolute loss values and increase in loss relative to the unmodified Pythia baseline. The baseline Pythia loss on the evaluation prompts is reported as 3.19 ± 0.09 nats per token (Figure 3 and Figure A5 captions). The paper reports median values of absolute loss differences rather than means, because "the distribution is heavy-tailed" (Figure 3 caption), and uses 95% confidence intervals computed via bootstrapping. This is important: some prompts experience catastrophic degradation while most are moderately affected, and medians are more robust to these outliers.
Baselines. The paper compares against several carefully chosen baselines, each serving a distinct purpose:
1. Unmodified Pythia 160M: the original model without any transcoder or SAE inserted. This represents the upper bound on performance — no rewriting can improve upon this, only degrade less.
2. Transcoder-replaced MLP (ground-truth activations): the model where the original MLP is entirely replaced by the transcoder running its own learned encoder (TopK(W_1 x + b_1)). The transcoder's output is not a perfect reconstruction of the original MLP output, so this baseline quantifies the transcoder's approximation error — the performance ceiling for any simulation, since the simulator can at best match the transcoder's own activations. The paper reports this degrades performance "to that of an early Pythia checkpoint — namely one that was trained on only 25% of the data" (Section 3). The precise loss value for this baseline is not reported numerically but serves as the reference point against which all simulation results are implicitly compared.
3. Zero ablation: the model where the entire MLP output of layer 6 is replaced with the zero vector. This is the most critical baseline. Zeroing out the MLP removes all its contribution — it is equivalent to having no information about what the layer computes. If the simulator's predictions cannot reduce loss below this baseline, then the explanations provide no useful functional information about the MLP's computation. The simulator is, at best, reproducing the marginal activation frequencies (via quantile normalization) without adding per-token signal about which features should fire when.
4. Random latent substitution: for partial rewriting experiments (explained below), randomly selecting which latents to simulate rather than selecting the highest-scoring ones. This tests whether the automated pipeline's detection scores identify latents with explanations that are more functionally faithful than average. If top-scoring substitution does not outperform random substitution, then the scores are not measuring functionally relevant explanation quality.
5. Pythia checkpoints at varying training fractions: the paper calibrates performance degradation by comparing to Pythia checkpoints trained on subsets of the full training data. For instance, the rewritten model's loss is described as equivalent to "a Pythia checkpoint trained on only 10-15% of the full training corpus" or "barely training the model at all" (for unnormalized predictions, Figure A5). These provide an intuitive scale: "the model behaves as if it had seen only X% of its training data."
No prior work on the rewriting task exists, so these baselines are constructed specifically for this paper rather than drawn from literature.
Generation budget / compute accounting. The paper does not use a "generation budget" concept typical of inference-time scaling work. Instead, the relevant resource constraint is the number of simulated latent predictions: for each of the 10K (transcoder) or 1K (SAE) evaluation prompts, the LLM simulator must produce predictions for all 32,768 latents. This requires 327 million forward passes of Llama 3 Instruct 8B for the full transcoder experiment (10K × 32,768), which the authors describe as "an expensive endeavor." This computational cost is the primary bottleneck limiting the evaluation sample size and the quality of the quantile normalization (which improves with more samples, as discussed in Appendix B). The paper does not account for this simulation cost in the loss metric — cross-entropy is measured only on the rewritten model's forward pass, not including the cost of generating the simulated activations. This is reasonable for an evaluation methodology paper, but would matter for any practical deployment of the rewriting approach.
Cross-validation / statistical protocol. There is no cross-validation in the traditional machine learning sense, because this is not a training procedure — the simulator is used zero-shot. Statistical reliability is assessed through bootstrapped confidence intervals on the median loss increase (Figure 3), computed over the 10K or 1K prompts. The paper does not report standard train/validation/test splits for the explanation pipeline (the explanations are generated on positive/negative examples collected from the Pile and evaluated on separate held-out prompts), but the detection scores are computed on 100 positive and 100 negative held-out samples per latent (Figure 3 caption), providing a degree of separation between explanation generation and evaluation. The quantile normalizer uses 10M tokens for true activation statistics and 10K tokens for predicted activation statistics — these are distinct from the evaluation prompts to avoid circularity, though the paper does not explicitly state this.

Main Quantitative Results

The central empirical finding spans all experiments: replacing the transcoder or SAE encoder with the LLM simulator, even after careful calibration via quantile normalization, produces cross-entropy loss statistically indistinguishable from simply zeroing out the MLP output entirely (Figure 3). The remainder of the quantitative results decompose this headline finding into component analyses.

Full Rewriting vs. Baselines

Transcoder full rewrite (Figure 3, leftmost group). When all 32,768 transcoder latents are replaced with simulated counterparts (100% substitution, green bars), the cross-entropy loss increase is approximately equal to the zero-ablation baseline (blue bars at 100%). Both conditions produce a model whose loss corresponds to a Pythia checkpoint trained on only 10-15% of the full training corpus. The precise numerical loss values are not reported, but Figure 3 shows the bars overlapping within the bootstrapped 95% confidence intervals, meaning the difference between simulated rewrite and zero ablation is not statistically significant at the 100% substitution level.

SAE full rewrite (Figure 3, rightmost group). The pattern replicates for the residual stream SAE at the same layer: full substitution of SAE latents with simulated activations produces loss comparable to zero ablation. The SAE evaluation uses only 1K prompts (vs. 10K for the transcoder), so the confidence intervals are wider, but the qualitative result is the same. This convergent finding across two different feature extraction methods (transcoder for MLP function, SAE for residual stream) suggests the bottleneck is explanation quality rather than the specific decomposition technique.

Raw (unnormalized) predictions are catastrophically worse (Figure A5). Without quantile normalization, the rewritten model's performance degrades to "equivalent to barely training the model at all" — substantially worse than zero ablation. This is visible in Figure A5: the unnormalized loss increase bars are dramatically higher than the normalized ones in Figure 3 for every substitution fraction. This quantifies the severity of the calibration problem described in Figure 1 (left): the raw simulator predicts high activation values by "multiple orders of magnitude" more frequently than the true distribution, flooding the decoder with massive false positive activations that destroy the model's behavior.

Partial Rewriting: Top-Scoring vs. Random Selection

Top-scoring latent substitution (green bars, Figure 3). When replacing only the highest-scoring latents (those with the best detection scores from Paulo et al., 2024), the loss increase tracks the zero-ablation baseline remarkably closely across all substitution fractions (20%, 40%, 60%, 80%, 100%). This means that even for the supposedly best-explained features, the simulator's predictions contribute no functional information beyond what you would get by setting those latents to zero. The green bars and blue bars overlap within confidence intervals at every fraction for both the transcoder (left) and SAE (right) experiments.

Random latent substitution (orange bars, Figure 3). When replacing randomly selected latents, the loss increase is consistently worse than zero ablation at all fractions for both the transcoder and SAE. This means the simulator's predictions for average-quality latents are actively harmful — they inject noise that the decoder misinterprets, producing MLP outputs that damage the model's predictions more than simply removing the layer's contribution entirely. This gap between top-scoring (green) and random (orange) substitution demonstrates that detection scores do capture something real about explanation quality (the best explanations produce less harmful predictions than random ones), but not enough to reach the zero-ablation threshold, let alone to preserve useful information.

Interpreting the partial rewriting pattern. The monotonic increase in loss with substitution fraction for both selection methods — and the fact that top-scoring substitution only matches, never beats, zero ablation — tells a precise story. The simulator's predictions contain essentially no mutual information with the true activation values, once the marginal distribution is corrected by quantile normalization. The quantile normalizer enforces the correct prior (how often each feature fires), but the per-token assignment of which features fire when is essentially random relative to the ground truth. The "top-scoring" latents are those where this random assignment happens to be less harmful — perhaps because their decoder directions (W_2 columns) have smaller norms or are less coupled to other features — not because the explanations enable better per-token predictions.

Explanation Quality Metrics: Detection Scores, Specificity, and Sensitivity

Detection scores predict sensitivity and specificity (Figure 4). Binning latents by their detection scores (from Paulo et al., 2024) reveals that higher-scoring explanations correspond to higher specificity (the simulator's ability to correctly identify when a latent is inactive) and higher sensitivity (the simulator's ability to correctly identify when a latent is active). Figure 4 shows this relationship: as detection score increases, both sensitivity and specificity rise. The paper reports that "on average the current automatic latent explanation setup has a specificity of around 80%" across all latents, which is "much lower than what would be required to do this task."

Why 80% specificity is catastrophically insufficient. The arithmetic is brutal: with 32,768 latents and only 32 active per token (TopK, k=32), exactly 99.9% of latents are inactive at any given token. An 80% specificity means the simulator correctly identifies 80% of the 32,736 inactive latents as inactive, but misclassifies the remaining 20% as active. That is 0.20 × 32,736 ≈ 6,547 false positives per token, swamping the 32 true positives by a factor of over 200. Even 99% specificity would produce ~327 false positives — 10 times the correct number. The paper emphasizes this arithmetic to drive home why quantile normalization is necessary: it zeroes out low-percentile predictions, effectively forcing specificity closer to the required ~99.9% by mapping the lower percentiles of the predicted distribution to zero (since the true distribution's lower percentiles are zero). But this comes at the cost of also zeroing out many true positives (reduced sensitivity), which is why performance only reaches the zero-ablation baseline.

Fuzzing scores show a similar pattern (Figure A4). As an alternative metric to detection scores, fuzzing scores (which measure how well an explanation survives perturbations to the example contexts in Paulo et al., 2024) also predict sensitivity and specificity of the simulator. Higher fuzzing scores correspond to better simulator performance. This convergent evidence across two different evaluation metrics from Paulo et al. (2024) strengthens the claim that the automated pipeline's scores measure something real about explanation quality — just not enough for functional faithfulness.

Quantile Normalization and Sample Size Effects

Normalization is essential but insufficient (Figure A5 vs. Figure 3). The comparison between Figure A5 (unnormalized) and Figure 3 (normalized) is the paper's clearest demonstration of the calibration problem's severity. Without normalization, the rewritten model's loss is far worse than zero ablation and the model becomes nearly useless. Quantile normalization recovers performance to the zero-ablation level, but cannot push beyond it. This means that the information in the simulator's raw predictions is sufficient to recover the correct marginal distribution of each latent's activations, but insufficient to recover the correct per-token pattern.

Sample size critically affects normalization quality (Figure A1, Figure A2, Figure A3). This is a significant practical finding with implications for anyone attempting to replicate or extend the methodology. The paper uses 10K prompts for the predicted activation distribution and 10M tokens for the true activation distribution. Figure A1 shows that using only 1K prompts increases the mismatch between the normalized and empirical activation distributions. Figure A2 demonstrates that this mismatch translates directly into worse performance: "Using only 1K samples to compute the quantile normalization function leads to a much worse CE loss when performing substitution." Figure A3 shows that using the empirical distribution of true activations over the evaluation prompts themselves (rather than the larger 10M-token corpus) as the normalization target improves results for both the 1K and 10K sample sizes.

Extrapolation to larger samples. The authors hypothesize that "were we able to have done prediction over 100k samples, the results of substituting using the empirical distribution over the prompts or over the larger 10M token sample would be closer, and if the trend holds, substituting the predictions would be better than zeroing out the MLP." This is a speculative extrapolation based on the observed trend that larger samples produce better-calibrated normalizers and better performance. However, the paper does not demonstrate this scaling trend beyond 10K samples, so it remains possible that the asymptote is at or below the zero-ablation level. The key uncertainty is whether the simulator's per-token ranking of latents contains any genuine signal beyond the marginal frequencies, once enough samples are available to accurately estimate the marginals. The current evidence is agnostic on this point.

Ablation Studies and Robustness Checks

Skip connection in transcoder architecture: The paper includes a linear skip connection (W_skip x) in the transcoder but does not ablate it (compare with and without skip connection). The authors state that the skip connection "improves its ability to approximate the original MLP at no cost to interpretability scores" and "leave a deeper analysis of the skip connection for future work" (Section 2). This is a notable omission — the skip connection's effect on rewriting fidelity is unknown. If the skip connection handles primarily linear transformations that are not explained by the latents, then the rewriting experiment is only testing explanations of the non-linear residual, which may be an easier or harder target. A skip-connection ablation would clarify whether the explanation failure is in the linear or non-linear component of the MLP's function.

TopK sparsity level (k=32): The paper uses a fixed k=32 for the TopK operator (Section 2). There is no ablation across different sparsity levels. The sparsity level matters for the specificity arithmetic — with k=64, the false positive tolerance would double; with k=16, it would halve. It is possible that a different sparsity level would change the difficulty of the simulation task, but this is not explored. The choice of 32 appears driven by convention (Gao et al., 2024) rather than optimization for this specific task.

Detection score vs. fuzzing score for latent selection: Both metrics from Paulo et al. (2024) — detection scores (Figure 4) and fuzzing scores (Figure A4) — predict simulator sensitivity and specificity. The paper uses detection scores for the top-scoring selection in Figure 3, but does not compare against selecting by fuzzing score. If fuzzing scores identified a different set of "best" latents that performed better, that would suggest the detection metric is not optimal. This comparison is not run.

Explanation generation model: The paper does not specify which model generates the explanations (the automated pipeline from Paulo et al., 2024). Different explanation-generation models (GPT-4 vs. Llama 3 vs. Claude) might produce explanations of different quality, affecting the results. The simulation model is Llama 3 Instruct 8B. An ablation using a different simulator model (e.g., GPT-4, Claude, or a fine-tuned model) would test whether the failure is in the explanations themselves or in Llama 3's ability to interpret them. This is not done.

Contrast pairs or alternative explanation strategies: The paper mentions in Section 5 that "new techniques are needed to make explanations more specific, for instance using contrast pairs of highly similar features to bring out additional details." This is forward-looking; no experiments with contrastive explanations are reported. This means the paper establishes a baseline for current explanation quality but does not demonstrate whether specific proposed improvements actually help.

SAE vs. transcoder comparison: The parallel experiments on the transcoder (MLP function) and SAE (residual stream) at the same layer provide a robustness check on the feature extraction method. The fact that both yield qualitatively identical results (rewriting performance ≈ zero ablation) suggests the bottleneck is explanation quality, not the decomposition. However, the SAE experiment uses only 1K prompts (vs. 10K for the transcoder), so the comparison is not perfectly matched. The wider confidence intervals on the SAE results make the zero-ablation equivalence less precise.

Bias correction for quantile normalization: The paper discusses but does not implement bias-corrected estimators for the quantile function (Hyndman & Fan, 1996), noting this as planned future work (Section 2.1). This is an acknowledged limitation — the current quantile normalizer may be suboptimal due to finite-sample bias in the inverse CDF estimate. However, since the true activation quantiles use 10M tokens (where bias should be negligible), and the predicted activation quantiles are the bottleneck, improved bias correction would primarily help the predicted side, where sample sizes are small. The magnitude of this potential improvement is unknown.

Critical Assessment

Claim 1: "The model's increase in loss is statistically similar to entirely replacing the sparse MLP output with the zero vector."

What the experiments demonstrate: Figure 3 shows that at 100% substitution of top-scoring latents, the green bars (simulated rewrite) overlap with the blue bars (zero ablation) within bootstrapped 95% confidence intervals, for both the transcoder and SAE experiments. This is a well-supported claim for the specific setup — Pythia 160M layer 6, 32,768 latents, TopK=32, quantile-normalized Llama 3 8B simulations, 10K prompts, explanations from Paulo et al. (2024)'s pipeline.

What the experiments do NOT demonstrate: Whether this equivalence holds (a) with larger sample sizes for the predicted activation distribution (where the authors speculate performance might eventually exceed zero ablation), (b) with different sparsity levels, (c) with different explanation generation methods, (d) on different layers of Pythia, (e) on different model architectures or scales, or (f) on out-of-distribution text. The claim is specific to the particular configuration tested.

Genuine weakness: The 10K prompt limit is a computational constraint, not a scientific choice. The authors explicitly acknowledge that larger samples might change the result, making this a conditional finding. If the trend in Figure A3 extrapolates — larger predicted samples → better normalization → lower loss — then the zero-ablation equivalence is an artifact of finite sample size rather than a fundamental limit. The paper cannot distinguish these possibilities with the current data.

Claim 2: "Explanations are not specific enough."

What the experiments demonstrate: The specificity problem is mathematically demonstrated (80% specificity → ~6,500 false positives per token), visually shown in Figure 1 (raw predictions massively over-predict high values), and causally linked to performance through the quantile normalization analysis (normalization, which improves specificity by zeroing low-percentile predictions, recovers performance from catastrophic failure to zero-ablation level). Figure 4 and Figure A4 show that higher-scoring explanations have better specificity, but even the best remain insufficient.

What the experiments do NOT demonstrate: Whether specificity is the ONLY problem, or whether sensitivity is equally limiting once specificity is fixed. Quantile normalization trades off sensitivity for specificity — it improves the latter by zeroing predictions below a percentile threshold, but this also zeros out true positives. The paper does not isolate these effects to determine whether perfect specificity (e.g., through a better explanation method that achieves 99.9% specificity natively) would be sufficient to exceed zero ablation, or whether sensitivity would then become the bottleneck. This is a genuine unknown in the current results.

Genuine strength: The specificity arithmetic is compelling and likely robust to experimental variations. With 32,768 latents and k=32, the base rate of activation is 0.098%. Any non-trivial false positive rate will dominate. This argument does not depend on the particular model, layer, or explanation method — it is a structural challenge for any approach that must predict a sparse subset from a large vocabulary of features.

Claim 3: "Detection scores are predictive of the specificity and sensitivity of an explanation."

What the experiments demonstrate: Figure 4 shows monotonic relationships between detection score bins and both sensitivity and specificity. Figure A4 shows the same for fuzzing scores. This is a robust within-experiment correlation.

What the experiments do NOT demonstrate: Whether detection scores predict functional faithfulness as distinct from simulator accuracy. The top-scoring latents in Figure 3 produce simulations that are less harmful than random latents (they match zero ablation rather than underperforming it), which is consistent with higher specificity/sensitivity reducing the noise injected into the decoder. But they do not produce simulations that carry useful information. Detection scores may be measuring something that correlates with explanation precision without capturing the specific details needed for functional replacement. This is a subtle point: a metric can be "predictive" in a correlational sense while being "insufficient" in a causal sense, and the paper demonstrates exactly this.

Missing Experiments That Would Strengthen the Paper

Fine-tuned simulator: The simulator (Llama 3 8B) is used zero-shot. Fine-tuning the simulator on transcoder activations — even with a small amount of data — would calibrate its predictions and potentially improve per-token ranking. This would test whether the bottleneck is the explanation content (which fine-tuning cannot fix if the explanations lack the necessary information) or the simulator's ability to map explanations to activation values (which fine-tuning could improve). The zero-shot design is principled (it ensures the simulator relies on the explanation's semantic content, not learned spurious patterns), but a fine-tuned baseline would help disentangle these effects.

Explanation quality manipulation: The paper does not experiment with deliberately degraded or improved explanations beyond what the automated pipeline produces. For instance, one could replace explanations with random strings, empty strings, or deliberately wrong descriptions to establish a lower bound on explanation usefulness. Conversely, one could manually write high-quality explanations for a subset of latents (perhaps the top-100 best-scoring ones) to test whether human-written explanations outperform automated ones. Neither direction is explored, leaving open the question of whether the failure is in the automated pipeline specifically or in the fundamental difficulty of the task.

Layer-wise analysis: Only layer 6 is tested. Different layers of a transformer are known to have different functional specializations (e.g., early layers handle local syntax, later layers handle global semantics). The rewriting difficulty may vary systematically by layer, and testing only one layer leaves this dimension unexplored. Layer 6 is an intermediate layer in Pythia 160M (which has 12 layers total), and its features may be of intermediate complexity — neither the simplest nor the hardest to explain.

Scale analysis: The paper uses Pythia 160M, a small model by current standards. Whether the specificity problem becomes better or worse at scale is unknown. Larger models may have more interpretable features (as suggested by Templeton et al., 2024) or more complex, entangled features that are harder to explain. The methodology is applicable to larger models in principle, but the computational cost of simulation (327M LLM predictions for 10K prompts on 32,768 latents) would scale linearly with the number of latents, which likely increases with model size. This creates a practical barrier to testing at scale.

What the paper does NOT claim, appropriately: The authors do not claim that mechanistic interpretability is impossible or that explanations can never be faithful. They present a negative result about the current state of automated explanation quality, using a specific evaluation protocol, and they frame future work on improving explanations (contrast pairs, higher specificity) as the path forward. The paper's contribution is methodological (the rewriting test) and diagnostic (identifying specificity as the bottleneck), not nihilistic. This restraint strengthens the paper: it identifies a clear problem with a clear metric, enabling future work to measure progress.

6. Limitations and Trade-offs

6.1 The Quantile Normalization Bottleneck: Finite-Sample Dependence and Unclear Asymptotic Behavior

The assumption or constraint. The entire rewriting experiment depends on quantile normalization to calibrate the simulator's raw predictions. If the quantile normalizer is poorly estimated, the rewritten model's performance degrades catastrophically — to the level of "barely training the model at all" (Figure A5). The paper uses 10K prompts to estimate the predicted activation distribution, while the true activation distribution is estimated from 10M tokens — a 1000× asymmetry. The authors explicitly acknowledge this gap: "We expect that... were we able to have done prediction over 100k samples, the results... would be closer, and if the trend holds, substituting the predictions would be better than zeroing out the MLP" (Appendix B).

The consequence. The paper's headline finding — that rewriting performance is "statistically similar to entirely replacing the sparse MLP output with the zero vector" (Abstract) — is conditional on the 10K-prompt sample size for the predicted distribution. This is not merely a caveat; it means the central result may be an artifact of insufficient data rather than a fundamental limitation of explanation quality. The quantile normalizer estimated from 10K samples produces a mismatch with the true activation distribution (Figure A1), and this mismatch directly degrades performance (Figure A2). If the simulator's per-token ranking of which latents to activate contains genuine signal beyond the marginal frequencies, that signal would only become apparent with enough samples to accurately estimate the predicted CDF — particularly for rare, high-activation events where most latents spend the vast majority of their time at or near zero. The paper cannot distinguish between "the simulator has no per-token signal" and "the simulator has signal that is drowned out by finite-sample quantile normalizer noise."

What evidence exists in the paper. Appendix B provides the key data. Figure A1 shows that with only 1K prompts, the mismatch between normalized and empirical distributions is visibly larger than with 10K. Figure A2 demonstrates that performance degrades substantially when using only 1K samples: the CE loss increase is much worse with 1K samples than with 10K. Figure A3 shows that using the empirical distribution over the evaluation prompts themselves as the normalization target (rather than the 10M-token corpus) improves results for both 1K and 10K cases — but still does not exceed the zero-ablation baseline. The trend across these figures suggests that larger sample sizes improve performance, and the authors extrapolate that at 100K samples the simulator might finally outperform zero ablation. But 100K samples × 32,768 latents = 3.27 billion LLM predictions, which is computationally prohibitive with current resources. The extrapolation is speculative: the performance-vs-sample-size curve could plateau below the zero-ablation line, or it could cross it. The paper provides no data to distinguish these scenarios.

Mitigation status. The paper does not resolve this limitation. The authors mention plans to experiment with bias-corrected estimators for the population quantiles (Hyndman & Fan, 1996) in a future draft (Section 2.1), but this is not implemented in the current version. Bias correction would help the true activation quantiles (where the dominant error is bias, not variance, because 10M tokens provide extensive data), but the bottleneck is the predicted activation side, where the sample size is three orders of magnitude smaller and the dominant error is variance. Bias correction alone cannot fix insufficient sample size. The paper also suggests that using the empirical prompt distribution as the target (Figure A3) is a promising direction, but does not scale it beyond 10K prompts. Fundamentally, the methodology's computational cost creates a hard barrier: evaluating whether the central result is robust requires 3.27 billion additional LLM forward passes, which the authors did not have the budget to perform. This makes the paper's headline claim provisional — a lower bound that may tighten with more compute, but whose ceiling is unknown.

6.2 Single Layer, Single Model, Single Architecture

The assumption or constraint. All experiments target layer 6 of Pythia 160M, a 160-million-parameter model with 12 transformer layers. The paper does not test rewriting on any other layer, any other model, or any other architecture. Section 3 justifies this pragmatically: "Replacing a single MLP with a transcoder increases the model's cross-entropy loss to that of an early Pythia checkpoint... Rewriting any part of the transcoder in natural language will necessarily degrade the model's performance even further. Consequently, we focus on rewriting a single MLP block of Pythia 160M, since rewriting all MLP blocks simultaneously would likely cause the model to become completely unusable."

The consequence. The paper's findings may not generalize to other contexts, and the direction of the generalization error is unclear. Different layers of a transformer are known to have different functional specializations: early layers tend to handle local syntactic features, middle layers handle semantic composition, and late layers handle task-specific output preparation. Layer 6 in a 12-layer model sits at the boundary between early and middle processing. Features at this layer may be of intermediate complexity — neither trivially simple surface patterns nor deeply abstract semantic concepts. Whether explanations are more or less faithful at other layers is unknown, and the specificity bottleneck may be more or less severe depending on the layer's functional role. For example, if late-layer features are more abstract and hard to describe, specificity might be even worse; if early-layer features are simpler and more local, specificity might be better. Without layer-wise measurements, the paper cannot distinguish whether the explanation failure is uniform across the network or concentrated in particular functional regimes.

Similarly, Pythia 160M is a small model by current standards. Templeton et al. (2024) found that features in Claude 3 Sonnet (a much larger model) could represent complex, coherent concepts like "inner conflict" and "sycophancy." It is plausible that larger models have more interpretable features because they allocate more capacity to representing distinct concepts, reducing superposition and making individual latents more monosemantic. If so, explanations of larger-model features might be more faithful, and the rewriting experiment might succeed where it fails on Pythia 160M. Alternatively, larger models might have more subtle, context-dependent features that are harder to capture in single-sentence explanations, making the specificity problem worse. The architecture also matters: Pythia uses a standard GPT-style decoder architecture; models with different architectural choices (mixture-of-experts, different activation functions, different normalization schemes) might produce features with different interpretability properties.

What evidence exists in the paper. The paper provides two data points relevant to generalization: (1) the transcoder experiment on the MLP of layer 6, and (2) the SAE experiment on the residual stream of the same layer 6. The fact that both yield the same qualitative result — rewriting performance indistinguishable from zero ablation — provides convergent evidence that the failure is not specific to the transcoder decomposition. But this is evidence within a single layer of a single model. No cross-layer, cross-model, or cross-architecture comparisons are made. The paper offers no empirical basis for predicting how the results would change in different settings.

Mitigation status. The authors do not attempt to address this limitation. They do not frame it as a limitation in the discussion (the focus is on explanation quality, not model-specific factors), and they do not propose multi-layer or multi-model experiments as future work. The constraint is implicitly accepted: the computational cost of the simulation (327 million LLM predictions for a single layer on 10K prompts) makes scaling to multiple layers or larger models prohibitively expensive with current resources. This is a practical but significant limitation — the paper establishes a methodology and a baseline result, but provides no evidence about the methodology's behavior across the space of models and layers where it would need to work for the rewriting agenda to succeed.

6.3 The Simulator Is Zero-Shot: Explanations and Simulator Capability Are Not Disentangled

The assumption or constraint. The LLM simulator (Llama 3 Instruct 8B) is used zero-shot, without any fine-tuning on the transcoder activations or explanation-to-activation mappings. The paper's design is principled: if the simulator were fine-tuned, it could learn spurious correlations between surface features of the text and activation patterns that bypass the explanation entirely, undermining the claim that the explanation drives the prediction. As the simulator prompt states (Appendix A): "Your job is to identify how much the last token... represents the feature [described in the explanation]." The zero-shot design ensures that the explanation is the only source of information about what each feature does.

The consequence. The experimental design cannot distinguish between two failure modes: (1) the explanations are poor — they don't contain enough information for even a perfect simulator to predict activations — and (2) the simulator is poor — the explanations are adequate, but Llama 3 Instruct 8B lacks the capability to map natural language descriptions to the precise geometric activation values that the transcoder encoder produces. These have very different implications. If the bottleneck is explanation quality (failure mode 1), the solution is better explanation generation methods (contrast pairs, higher-resolution descriptions, explicit negative constraints, etc.). If the bottleneck is simulator capability (failure mode 2), the solution is better simulator design (fine-tuning, different model architectures, direct regression training on activation prediction rather than zero-shot prompting).

The paper's framing strongly implies failure mode 1, attributing the results to explanations that are "not specific enough" (Section 4.1) and calling for "new techniques... to make explanations more specific" (Section 5). But the evidence is equally consistent with failure mode 2: perhaps the explanations do contain sufficient information about when features fire, but an 8B-parameter instruction-tuned model prompted in a zero-shot linguistics-researcher persona cannot extract that information and map it to a 0–9 integer scale that — after quantile normalization — reproduces the transcoder's geometrically meaningful activation values. The transcoder's activation for a given latent is the result of a specific linear projection (W_1 row) followed by TopK competition across all 32,768 latents. This is a fundamentally geometric quantity. Expecting an LLM to predict it from a single-sentence description, without any training on what activation magnitudes mean in this context, is an extraordinarily demanding task. The fact that the simulator achieves specificity of only ~80% (Section 4.1) could reflect either explanation inadequacy or simulator inadequacy.

What evidence exists in the paper. The paper provides no direct evidence to disentangle these failure modes. The simulator is never compared against alternative architectures, never fine-tuned on a subset of latents to measure the achievable prediction accuracy given the existing explanations, and never evaluated on simpler synthetic features where ground-truth explanation quality can be controlled. The quantile normalization results (Figures 1, A1, A2, A3) show that the simulator's raw predictions contain some signal — the marginal distribution can be recovered — but whether the per-token ranking error is due to explanation vagueness or simulator miscalibration is unknown. Figure 4 and Figure A4 show that higher-scoring explanations produce better simulator sensitivity and specificity, which is consistent with explanation quality being a factor, but this is a correlational relationship that could also arise if "better" explanations happen to describe concepts that Llama 3 happens to be better at recognizing. A fine-tuned regression baseline (e.g., train a small probe on top of Llama 3's representations to predict activation values given the explanation and context) would establish an upper bound on what the explanations enable, distinguishing content limitations from interface limitations. No such baseline is provided.

Mitigation status. The paper does not address this confound. The zero-shot design choice is defended implicitly through the research goal (evaluating explanation faithfulness without introducing learned spurious correlations), but the alternative hypothesis — that the simulator is the bottleneck — is never tested. Future work could resolve this by: (a) fine-tuning the simulator on a held-out set of latents and measuring whether prediction accuracy improves to a level that would enable functional rewriting; (b) using a stronger base model as the simulator (e.g., Llama 3 70B or GPT-4); (c) training a dedicated regression model that takes the explanation embedding and text context as input and directly predicts activation values, bypassing the zero-shot language interface. Until such experiments are done, the paper's attribution of the failure to explanation quality specifically (as opposed to simulator quality or the interface between them) remains an assumption rather than an established finding.

6.4 The Rewriting Experiment Only Tests a Single MLP; Real Rewriting Requires Composing Multiple Layers

The assumption or constraint. The paper rewrites a single MLP block in a 12-layer transformer and leaves the other 11 layers untouched. Section 3 states that rewriting all MLP blocks simultaneously "would likely cause the model to become completely unusable" because "replacing a single MLP with a transcoder increases the model's cross-entropy loss to that of an early Pythia checkpoint — namely one that was trained on only 25% of the data." The rewriting experiment uses a transcoder that already degrades performance; the simulation degrades it further to 10–15% training-data equivalent. Compounding degradation across 12 layers would be catastrophic.

The consequence. A single-layer rewriting test may be substantially easier than multi-layer rewriting — or substantially harder. The direction is unclear, and both possibilities undermine strong conclusions from the current experiment. If single-layer rewriting is easier, the paper's negative result is even more damning — if we cannot rewrite one layer, we certainly cannot rewrite all of them. But if single-layer rewriting is harder, the result may be pessimistic. Here is the argument for "harder": in a multi-layer network, features are distributed and redundant. When you zero out one MLP layer, other layers partially compensate because they have learned to be robust to noise and variation in intermediate representations during training. The zero-ablation baseline (which the simulated rewrite matches) removes the layer entirely, and the model survives with loss equivalent to an early checkpoint. If you rewrote all layers simultaneously with even moderately faithful simulations, the errors might compound — but they might also cancel. The model's representations are high-dimensional and redundant; small errors in one layer's output might be corrected by subsequent layers, just as they are during normal training. A full-network rewrite might be possible even if no single layer's rewrite, tested in isolation, provides useful information.

Conversely, the specificity problem is amplified when composing multiple rewritten layers. Each layer contributes a stream of false positive activations (the ~6,500 incorrectly activated latents per token from the 80% specificity simulator), and these errors propagate forward. Even if quantile normalization controls the marginal distribution at each layer independently, the joint distribution across layers is uncontrolled, and errors can compound nonlinearly. The paper's single-layer result might overestimate multi-layer rewriting feasibility for this reason.

The key point is that the single-layer test does not cleanly isolate explanation quality from the interaction between layers. A feature that appears poorly explained in isolation might be perfectly adequate when composed with downstream layers that filter, sharpen, or reinterpret its output. Or a feature that appears adequately explained in isolation might have subtle interactive effects that only manifest when combined with rewritten upstream and downstream components. The rewriting test, as currently constructed, cannot distinguish these cases.

What evidence exists in the paper. None. The paper does not attempt multi-layer rewriting, does not analyze how errors in the simulated MLP output propagate through subsequent layers, and does not measure whether the zero-ablation baseline's relatively good performance (loss equivalent to 25% training data) is due to the model's robustness to single-layer removal or due to genuine redundancy that would also make simulation errors more tolerable. The paper provides no theoretical analysis of error propagation in composed rewritten layers.

Mitigation status. The authors acknowledge the single-layer limitation implicitly (Section 3: "rewriting all MLP blocks simultaneously would likely cause the model to become completely unusable") but do not propose solutions or partial experiments. One could imagine intermediate experiments: rewriting two adjacent layers, measuring how errors compose, or analyzing the representational similarity between the original model's activations and the rewritten model's activations at each subsequent layer to see whether errors amplify or attenuate. The paper does not pursue these directions. The limitation is fundamentally about the gap between the experimental setup and the goal it serves. The rewriting test is proposed as a methodology for evaluating explanation faithfulness in service of the microscope AI agenda — fully rewriting a model in natural language. But the test only evaluates one component in isolation, and the relationship between single-component faithfulness and full-model faithfulness is uncharacterized. This limits the paper's ability to claim that its findings diagnose the difficulty of the full rewriting problem, as opposed to the difficulty of a particular isolated test.

6.5 In-Distribution Evaluation Only; No Test of Generalization or Adversarial Robustness

The assumption or constraint. All evaluation is performed on text chunks sampled from the Pile (Gao et al., 2020), which is the same corpus used to train Pythia 160M and the dataset from which the transcoder's training tokens were drawn. This is in-distribution evaluation: the model is tested on the same kind of text it was trained on, where its features are operating in their natural statistical environment. The paper does not evaluate the rewritten model on out-of-distribution text, adversarial inputs, or systematically perturbed contexts designed to stress-test the explanations.

The consequence. The rewriting test may overestimate explanation faithfulness because in-distribution evaluation allows the simulator to exploit statistical regularities that do not reflect genuine understanding of the explanation. Consider a latent that activates on "the word 'bank' when used in a financial context." On in-distribution text from the Pile, financial contexts might be identifiable through simple lexical cues (presence of words like "money," "loan," "account") that correlate with but do not constitute the feature's true function. The simulator could achieve reasonable prediction accuracy by detecting these correlated cues, without understanding the explanation's semantic content. The rewriting experiment would then show decent performance, not because the explanation is faithful, but because the evaluation distribution is statistically easy.

Out-of-distribution testing would expose this. Adversarially constructed text — where financial-cue words appear in non-financial contexts, or where "bank" appears in financial contexts without the usual lexical cues — would reveal whether the simulator truly understands the explanation or is relying on spurious correlations. If the rewritten model's performance degrades more sharply than the original model's on such inputs, the explanation is not capturing the feature's robust functional behavior. The paper provides no such stress test.

Conversely, in-distribution evaluation might underestimate explanation faithfulness if the simulator fails on common-but-hard cases that would be easier on simpler, out-of-distribution text. The Pile contains a wide range of text types (academic papers, code, web forums, books), and the transcoder's features must handle this diversity. If explanations are adequate for the "typical" case but fail on rare edge cases that happen to appear in the evaluation set, the measured performance might be worse than what the explanations actually support for the bulk of the distribution.

What evidence exists in the paper. None. The evaluation uses standard Pile text chunks with no systematic variation in domain, style, or difficulty. The paper does not report performance broken down by text type (e.g., code vs. prose vs. mathematical text), does not construct contrastive or adversarial evaluation sets, and does not analyze which kinds of contexts produce the largest simulation errors. The detection scores from Paulo et al. (2024) are computed on held-out Pile examples, so even the explanation quality metrics are in-distribution.

Mitigation status. The paper does not address this limitation and does not propose out-of-distribution evaluation as future work. The omission is standard for mechanistic interpretability research — most SAE and transcoder work evaluates on in-distribution data from the same corpus used for training — but it matters for the rewriting test specifically because the test's purpose is to evaluate explanation faithfulness, not model performance. If the goal is to determine whether explanations capture what the network "really" computes, testing only on the training distribution conflates "the explanation describes the feature's behavior on typical inputs" with "the explanation describes the feature's computational function." The latter requires generalization testing that the paper does not provide. For a practitioner considering whether to rely on such explanations for safety-critical applications (as envisioned by the microscope AI agenda), the lack of robustness evaluation is a significant gap: an explanation that works on the training distribution but fails under distribution shift provides a false sense of understanding.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new evaluation paradigm for mechanistic interpretability that shifts the success criterion from correlational prediction accuracy to causal functional faithfulness. Prior work — the automated interpretability pipeline of Bills et al. (2023) and Paulo et al. (2024), and the broader SAE explanation literature — evaluates explanations by measuring whether they help a separate model predict feature activations on held-out data. This paper argues, through its experimental design, that these metrics are necessary but insufficient: an explanation can achieve high simulation or detection scores by capturing surface-level statistical patterns without describing the feature's actual computational role in the network. The rewriting experiment operationalizes the microscope AI agenda's core question — "do we understand this component well enough to rewrite it?" — by demanding that the explanation support replacing the original learned encoder with an LLM simulator while preserving downstream behavior.

The magnitude of this shift is methodological rather than results-driven. The paper does not claim to have solved explanation faithfulness; it claims to have built a better test for it. This is analogous to how ImageNet shifted computer vision from algorithm-specific metrics to a standardized benchmark that enabled comparison across approaches. The rewriting protocol is reusable: any future explanation method — whether contrastive, interactive, programmatic, or human-written — can be plugged into the same transcoder/SAE → LLM simulation → quantile normalization → cross-entropy evaluation pipeline, producing a quantitative faithfulness score that directly measures functional fidelity. This creates a feedback loop: improvements to explanation quality can be measured by their effect on rewriting performance, and the zero-ablation baseline provides a clear threshold that must be exceeded for explanations to claim functional usefulness.

The paper also reconciles a tension between the apparent success of automated interpretability (features that "look interpretable" and achieve reasonable detection scores) and the field's ultimate goal of understanding models well enough to rewrite them. Prior work could claim progress because the metrics were improving; this paper demonstrates that the metrics may be measuring something different from what the field needs. The finding that detection scores correlate with simulator sensitivity and specificity (Figure 4, Figure A4) yet fail to enable functional rewriting (Figure 3, green vs. blue bars) reveals that correlational metrics can improve while functional faithfulness remains near zero. This is a specific, quantitative demonstration of Goodhart's law in interpretability evaluation — when a metric becomes a target, it ceases to be a good metric — and it should redirect effort away from optimizing existing simulation/detection scores toward developing fundamentally different explanation formats.

The paper redirects research attention toward specificity as the critical bottleneck. Prior work on explanation quality focused primarily on positive prediction: can the explanation help identify when a feature is active? The rewriting experiment reveals that the limiting factor is false positives — with 32,768 latents and only 32 active per token, even 99% specificity produces 10× too many active predictions. The paper's specificity arithmetic (Section 4.1) is likely robust to experimental variations and establishes a structural challenge: any approach that predicts a sparse subset from a large feature vocabulary must achieve specificity within a factor of ~1000× of the base activation rate. This is not a problem that marginal improvements to existing pipelines can solve; it requires explanations that are qualitatively more precise about when features do not fire. The paper makes this diagnosis explicit and actionable, calling for "contrast pairs of highly similar features" and explanations that specify negative constraints — research directions that were not obvious priorities before this work.

Finally, the paper lowers the barrier to entry for interpretability evaluation. The rewriting protocol requires only that a method produce an encoder-decoder decomposition (transcoder, SAE, or any future sparse feature extractor) and an LLM-based simulator. It does not require human evaluation, manual feature labeling, or task-specific metrics. This standardizes a previously fragmented evaluation landscape where different groups used different metrics (simulation score, detection score, fuzzing score, human agreement) with unclear relationships to the field's ultimate goals. A standardized, automated faithfulness benchmark could accelerate progress in the same way that standardized NLP benchmarks accelerated language model development — by making it easy to compare methods and measure improvement.

Follow-Up Research This Work Enables

Contrastive explanation generation with explicit negative constraints. The paper identifies low specificity as the primary failure mode: explanations describe when features activate but fail to specify when they do not. A direct follow-up would modify the Paulo et al. (2024) pipeline to generate contrastive explanations. For each transcoder latent, in addition to collecting top-activating examples (as current pipelines do), collect highly confusable negative examples — contexts where the latent does NOT fire but surface-level features suggest it should, or where a semantically similar latent fires instead. Present these contrast pairs to the explanation-generating LLM with a prompt like: "This feature activates on [positive examples] but does NOT activate on [negative examples]. What distinguishes them?" The resulting explanations would contain explicit negative constraints (e.g., "activates on mathematical expressions involving integrals, but NOT on expressions involving only arithmetic"). The rewriting protocol then measures whether contrastive explanations improve specificity beyond the ~80% baseline and whether this translates to rewriting performance that exceeds zero ablation. A strong result would be contrastive explanations achieving >99% specificity and rewriting loss significantly below the zero-ablation baseline on the top-scoring latents.

Fine-tuned simulator to disentangle explanation quality from simulator capability. The paper's zero-shot Llama 3 8B simulator confounds two failure modes: poor explanations and poor simulation capability. A critical follow-up experiment would fine-tune the simulator on a subset of latents, training it to predict activation values directly from the explanation text and context. The key measurement: does fine-tuning substantially improve prediction accuracy (specificity, sensitivity) compared to zero-shot, given the same explanations? If yes, the bottleneck is at least partially the simulator's ability to map natural language to geometric activation values, and effort should shift toward better simulator architectures (e.g., training a dedicated regression head on top of the LLM's representations). If no — fine-tuning achieves no better accuracy than zero-shot after quantile normalization — then the explanations genuinely lack the necessary information, strengthening the paper's diagnosis. A particularly informative design: fine-tune on half the latents (selected randomly) and test on the other half. If fine-tuned prediction accuracy generalizes to held-out latents, the simulator has learned a general mapping from explanations to activation patterns, which would be a significant positive result for the rewriting agenda.

Layer-wise and cross-model scaling of the rewriting protocol. The paper tests only layer 6 of Pythia 160M. A systematic follow-up would apply the identical rewriting protocol to every layer of Pythia 160M (or another small model with SAE/transcoder coverage across all layers), measuring whether the specificity bottleneck and zero-ablation equivalence are uniform across layers or concentrated in specific functional regimes. The hypothesis: early layers (which handle local syntactic features) may have more explainable features with higher specificity, while late layers (which handle abstract semantic composition) may have harder-to-explain features. If rewriting performance varies systematically by layer, this maps the "explainability landscape" of a transformer and identifies which layers are ripe for rewriting and which require fundamentally new approaches. Cross-model scaling — testing the same layer (e.g., middle MLP) across Pythia 160M, Pythia 410M, Pythia 1B, and Pythia 2.8B — would test whether larger models produce more monosemantic and explainable features (as Templeton et al., 2024 suggests for Claude 3 Sonnet) or whether the specificity problem scales with feature count. This experiment is computationally expensive (each additional model requires training a transcoder, generating explanations, and running the simulation) but directly addresses the paper's single-model limitation.

Adversarial stress-testing of explanation faithfulness. The paper evaluates only on in-distribution Pile text, which may allow the simulator to exploit spurious correlations rather than genuinely understanding explanations. A stress-test follow-up would construct adversarial evaluation sets designed to expose explanation unfaithfulness. For each high-scoring latent, use the explanation to generate text where the latent should activate according to the explanation but might not (e.g., for a "financial language" feature, generate text about banking that uses unusual vocabulary) and text where it should not activate but might (e.g., text with financial cue words in non-financial contexts). Measure whether the rewritten model's cross-entropy loss degrades more sharply on these adversarial inputs than on random Pile text, compared to the original model's degradation pattern. If explanations are genuinely faithful, the rewritten model should track the original model's behavior on adversarial inputs — both should succeed or fail similarly. If the rewritten model degrades disproportionately, the explanations capture surface correlations rather than robust computational functions. This is a direct test of the microscope AI agenda's central claim: that we can extract actionable insights from models that generalize beyond the training distribution.

Bias-corrected quantile normalization and large-sample scaling. The paper acknowledges that finite-sample bias in the empirical inverse CDF may degrade quantile normalization quality, and speculates that with 100K+ predicted activation samples the simulator might exceed the zero-ablation baseline. A focused follow-up would implement bias-corrected quantile estimators (Hyndman & Fan, 1996) for both the predicted and true activation distributions, then scale the number of evaluation prompts to the maximum feasible given computational budget (ideally 50K–100K prompts for at least the transcoder experiment). The key measurement: does CE loss as a function of sample size show a trend that clearly extrapolates above the zero-ablation line, or does it plateau below it? This is the single most important unresolved question from the paper, because it determines whether the headline result — "increase in loss is statistically similar to zero ablation" — is a fundamental limitation of current explanations or an artifact of insufficient data for calibration. A negative result (performance plateaus below zero ablation even at 100K samples) would strengthen the paper's diagnosis that explanations lack per-token signal. A positive result (performance exceeds zero ablation at scale) would partially rehabilitate current explanation methods while highlighting the practical challenge of calibration cost.

Manual explanation writing for a targeted subset of latents. To establish an upper bound on what the rewriting protocol can achieve with current transcoder features, a follow-up would manually write high-quality explanations for the top-50 or top-100 detection-scored latents. Human-written explanations could include: precise activation conditions, explicit negative examples, graded activation levels with described thresholds, and interactions with other features. The rewriting experiment would then compare human-written vs. automated explanations on this subset, measuring whether human explanations enable rewriting performance that exceeds both the automated-explanation and zero-ablation baselines. If human explanations succeed where automated ones fail, the bottleneck is clearly the explanation generation pipeline — and the human explanations serve as existence proofs and templates for what automated methods should aim to produce. If even human explanations fail, the bottleneck is either in the transcoder features themselves (they are not as interpretable as they appear) or in the simulation interface (mapping natural language to geometric activations is fundamentally too difficult for current LLMs), both of which would redirect research effort away from explanation generation toward feature extraction or simulator design.

Practical Applications and Downstream Use Cases

Standardized faithfulness benchmark for interpretability methods. The rewriting protocol can be packaged as a benchmark — akin to GLUE for NLP or ImageNet for vision — that any interpretability method can submit to. A research group developing a new SAE variant, transcoder training technique, or explanation generation method would: (1) train their decomposition on a standard model (e.g., Pythia 160M layer 6, or a suite of layers/models), (2) generate explanations using their method, (3) run the LLM simulator with quantile normalization, and (4) report cross-entropy loss relative to the zero-ablation and ground-truth transcoder baselines. This would replace the current fragmented evaluation landscape (where every paper uses different metrics, datasets, and baselines) with a single, automated, quantitative measure of functional faithfulness. The computational cost (~327M LLM predictions for 10K prompts on 32,768 latents) is high but feasible for well-resourced groups, and could be reduced by evaluating on a representative subset of latents (e.g., 1,000 randomly sampled) or using smaller simulator models. A public leaderboard tracking rewriting performance over time would create accountability and accelerate progress in the same way that standardized benchmarks have in other ML subfields.

Quality control for interpretability in high-stakes deployments. For organizations deploying LLMs in safety-critical contexts (medical diagnosis, legal reasoning, autonomous systems) where understanding model decisions is required for regulatory compliance or risk management, the rewriting protocol provides a quantitative "explanation audit." Rather than relying on qualitative feature inspections or correlational explanation scores, an auditor could: (1) train transcoders on the deployed model's critical layers, (2) generate automated explanations, (3) run the rewriting experiment, and (4) measure whether the explanations preserve model behavior above the zero-ablation baseline. A model whose explanations fail the rewriting test — meaning the explanations do not capture enough functional information to distinguish the model's actual computation from a disabled layer — would fail the audit, flagging the need for either better interpretability methods or deployment restrictions. The protocol's automation means this audit could run continuously as part of a model monitoring pipeline, detecting regressions in interpretability after fine-tuning or updates.

Data filtering for self-improvement and distillation pipelines. The paper's finding that only high-scoring explanations achieve specificity and sensitivity above chance (Figure 4, Figure A4) suggests a practical filtering strategy: when using automated interpretability to generate training data for self-improvement (e.g., generating reasoning traces, identifying model biases, or distilling knowledge into interpretable form), retain only features whose explanations exceed a detection score threshold empirically linked to adequate specificity. The rewriting protocol itself could be used to set this threshold: measure rewriting performance as a function of detection score cutoff, and identify the score above which simulated latents contribute more information than zero ablation. Features below this threshold would be excluded from downstream use, reducing noise and improving the quality of interpretability-derived data. This is immediately actionable with current tools (the Paulo et al., 2024 pipeline plus a trained transcoder) and does not require solving the full rewriting problem.

Guiding investment in mechanistic interpretability research. The paper's diagnosis — that specificity, not sensitivity, is the bottleneck — has direct resource allocation implications for research groups and funders. Projects aimed at improving positive example collection, generating more detailed descriptions of when features fire, or increasing simulation scores through better prompting are targeting sensitivity — and the paper suggests they will hit diminishing returns because the limiting factor is the overwhelming false positive rate from inadequate negative specification. Instead, resources should shift toward contrastive explanation methods, negative example mining from large text corpora, and specificity-focused evaluation metrics. The paper provides the quantitative evidence (the specificity arithmetic, the zero-ablation baseline) to justify this shift, making it useful for grant proposals, project prioritization, and research roadmapping in interpretability groups. A funder evaluating proposals could ask: "Does this work target the specificity bottleneck identified in Paulo and Belrose (2025)?" as a relevance criterion.