Improving Dictionary Learning with Gated Sparse Autoencoders

ArXiv: 2404.16014

🎯 Pitch

Gated SAEs eliminate the activation shrinkage that plagues standard sparse autoencoders by separating feature selection from magnitude estimation, achieving the same reconstruction fidelity with half the active features on models up to 7B parameters, without sacrificing interpretability.

---

1. Executive Summary

This paper introduces the Gated Sparse Autoencoder (Gated SAE), a modification to the standard SAE architecture that decouples the detection of which features are active from the estimation of their magnitudes — applying the L1 sparsity penalty only to the former task and thereby eliminating shrinkage (the systematic underestimation of feature activations caused by the L1 penalty competing with reconstruction fidelity). Training on language models up to 7B parameters (Gemma-7B, Pythia-2.8B, GELU-1L) across multiple sites (MLP outputs, attention outputs, and residual stream activations), Gated SAEs achieve a Pareto improvement over baseline SAEs on the sparsity–reconstruction fidelity trade-off, requiring approximately half as many firing features to achieve comparable reconstruction fidelity. A double-blind human interpretability study confirms that Gated SAE features are comparably interpretable to baseline SAE features, establishing that the architectural improvements in reconstruction quality do not come at the cost of less meaningful learned directions.

2. Context and Motivation

The Core Problem: Sparse Decomposition of Language Model Activations

This paper addresses a fundamental challenge in mechanistic interpretability: how do we decompose a language model's internal activations into a sparse, linear combination of interpretable feature directions? The key question isn't merely whether such a decomposition exists, but how to learn one that faithfully reconstructs the original activations while using only a small number of active features per input.

The motivation for sparse decomposition traces directly to the superposition hypothesis (Elhage et al., 2022b). In a typical transformer layer, the residual stream has dimension $n$ (e.g., 2,048 or 4,096). Yet language models appear to represent far more concepts than their dimensionality would naively allow — perhaps $M \gg n$ distinct features. The superposition hypothesis proposes a specific mechanism: the model encodes $M$ concepts as nearly-orthogonal linear directions in activation space, with the crucial property that on any given input, only a sparse subset of these concepts are active, minimizing interference between non-orthogonal feature vectors.

This is not merely of theoretical interest. If true, superposition implies that understanding a model's computation requires finding these $M$ feature directions — the model's internal "vocabulary" of concepts. But these directions are not aligned with the standard basis (neurons), which exhibit polysemanticity: individual neurons respond to multiple apparently unrelated concepts (Bolukbasi et al., 2021; Elhage et al., 2022a), making neuron-level analysis an unreliable tool for understanding model internals.

Sparse autoencoders (SAEs) provide a natural tool for this discovery task. Formally, given an activation vector $\mathbf{x} \in \mathbb{R}^n$, we want to find:

$\mathbf{x} \approx \mathbf{x}_0 + \sum_{i=1}^{M} f_i(\mathbf{x}) \mathbf{d}_i$

where $\mathbf{d}_i$ are $M \gg n$ unit-norm feature directions (the dictionary or overcomplete basis), $\mathbf{x}_0$ is a centering term, and $f_i(\mathbf{x}) \geq 0$ are sparse feature activations — only a few are non-zero for any given $\mathbf{x}$. This decomposition has the exact structure of an autoencoder: an encoder maps $\mathbf{x}$ to the sparse coefficients $f_i$, and a decoder linearly reconstructs $\mathbf{x}$ from those coefficients.

Where Existing SAE Methods Fall Short: The L1 Penalty's Unwanted Side Effects

The prevailing approach to SAE training — established by Bricken et al. (2023) and adopted by subsequent work — parameterizes a single-layer autoencoder with a ReLU encoder and linear decoder:

$\mathbf{f}(\mathbf{x}) := \text{ReLU}\left(\mathbf{W}_{\text{enc}}(\mathbf{x} - \mathbf{b}_{\text{dec}}) + \mathbf{b}_{\text{enc}}\right)$

$\hat{\mathbf{x}}(\mathbf{f}) := \mathbf{W}_{\text{dec}} \mathbf{f} + \mathbf{b}_{\text{dec}}$

Training minimizes a loss with two competing terms:

$\mathcal{L}(\mathbf{x}) := \|\mathbf{x} - \hat{\mathbf{x}}(\mathbf{f}(\mathbf{x}))\|_2^2 + \lambda \|\mathbf{f}(\mathbf{x})\|_1$

The L1 penalty on feature activations is necessary — we cannot directly optimize the L0 norm (number of active features) because it's non-differentiable, and L1 provides a convex surrogate that empirically encourages sparsity. However, this same L1 penalty introduces a systematic bias that the paper terms shrinkage (building on Wright and Sharkey, 2024).

To understand shrinkage, consider what the loss function is really doing. For a fixed decoder, the L1 term pushes every feature activation toward zero, while the reconstruction loss pushes feature activations high enough to produce accurate reconstructions. The optimal value sits somewhere between these opposing forces — meaning the encoder systematically underestimates the magnitude of feature activations. The paper illustrates this with a stark toy example in Figure 2: a single-feature SAE with $\lambda = 1$ reconstructs $1/2$ rather than the true value of $1$ when minimizing Equation (4), because shrinking the activation from 1 to 1/2 reduces the total loss ($\mathcal{L}_{\text{shrunken}} = (1/2)^2 + 1 \cdot 1/2 = 0.75$) compared to a perfect reconstruction ($\mathcal{L}_{\text{perfect}} = 0 + 1 = 1.0$).

More subtly, the paper notes that shrinkage cannot be fully cured by post-hoc rescaling. Section 3.1 explicitly argues:

"Conversely, rescaling the shrunk feature activations is not necessarily enough to overcome the bias induced by the L1 penalty: a SAE trained with the L1 penalty could have learnt sub-optimal encoder and decoder directions that are not improved by such a fix."

In other words, shrinkage doesn't merely affect the magnitudes of feature activations at inference time — it biases the entire training process, causing the SAE to learn suboptimal feature directions (both encoder and decoder weights) because the optimizer trades reconstruction accuracy for L1 reduction throughout training, not just at the final step. This is analogous to the difference between biased coefficient estimates and biased model selection in statistics: fixing the coefficient magnitudes after training doesn't undo the fact that training selected different directions in the first place.

The paper provides empirical evidence for this claim in Section 5.2 and Figure 9, which we'll examine later, but the key point is that the L1 penalty's bias is baked into the learned parameters, not merely a rescaling issue at inference time.

The Encoder's Dual Role: A Root Cause Analysis

The paper identifies why the L1 penalty causes such pervasive problems: the encoder output $\mathbf{f}(\mathbf{x})$ serves two distinct roles in the baseline SAE architecture:

1. Feature detection: Determining which features are active. The ReLU naturally maps negative pre-activations to zero (inactive) and positive pre-activations to some positive value (active). For this role, the L1 penalty is genuinely needed — it's what pushes the encoder to produce zeros for most features.

2. Magnitude estimation: Estimating how much each active feature is present. For features that are firing, the exact value matters for reconstruction quality. For this role, the L1 penalty is pure bias — it encourages underestimation without any compensating benefit for sparsity (since the feature is already active).

In the baseline architecture, both roles are performed by the same ReLU encoder. The L1 penalty cannot be applied selectively to just the detection function — it necessarily affects both, introducing bias into magnitude estimation as a side effect of encouraging sparsity. This is the fundamental tension that the paper sets out to resolve.

Practical Consequences of Shrinkage

Why does this matter beyond mathematical elegance? The paper implies several practical consequences:

• Reduced reconstruction fidelity: Shrunken feature activations mean the decoder produces reconstructions that are systematically smaller in norm than the original activations. This degrades the SAE's usefulness as a faithful model of the original network's computation.

• Difficulty interpreting feature magnitude: If feature activations are systematically underestimated, analyses that depend on activation magnitudes — such as identifying which features are most important for a particular input, or comparing the relative importance of different features — become unreliable.

• Degraded performance when spliced into the model: SAEs are often evaluated by splicing their reconstructions into the language model's forward pass and measuring the resulting cross-entropy loss (the "loss recovered" metric, defined in Equation 5). Shrinkage means the spliced-in activations have smaller norm than the originals, potentially degrading the model's downstream computation even when the feature directions are correct.

Prior Approaches and Their Limitations

Bricken et al. (2023) established the baseline. This work provided the first large-scale demonstration that SAEs can discover interpretable features in language model activations, and defined the evaluation metrics (L0 for sparsity, loss recovered for fidelity) and training methodology that subsequent work adopted. However, they did not identify or address the shrinkage problem — their architecture and loss function are exactly the baseline that this paper improves upon.

Wright and Sharkey (2024) identified shrinkage and proposed decoder finetuning. This contemporaneous work recognized that L1-trained SAEs exhibit feature suppression (shrinkage) and proposed addressing it by post-training finetuning of the decoder. The limitation, as this paper notes in Section 6, is that:

"a difficulty with this approach is that it is not possible to fine tune all the SAE's parameters in this way without losing sparsity and/or interpretability of feature directions."

Decoder finetuning can recover some reconstruction fidelity, but it cannot undo the fact that the encoder was trained under L1-induced bias. The encoder directions themselves were optimized under pressure to shrink activations, and finetuning the decoder doesn't change which directions the encoder learned to detect. This paper cites empirical evidence for this claim in Section 5.2 (Figure 9), where a "baseline + rescale & shift" approach — which keeps baseline encoder and decoder directions frozen and only learns magnitude correction parameters — captures only a small fraction of the Gated SAE's performance gain.

Other architectural improvements explored different directions. Batson et al. (2024), Olah et al. (2024), and Templeton et al. (2024) proposed various methodological tweaks to SAE training (the Tanh penalty, different initialization schemes, different decoder norm constraints), but none fundamentally addressed the dual-role problem. Taggart (2024), working in parallel to this paper, explored a similar Jump ReLU architecture but with a different loss function and without this paper's analysis of the connection to shrinkage reduction.

How This Paper Positions Itself

The paper frames its contribution not as an entirely new approach to dictionary learning, but as a targeted architectural modification that addresses a specific, well-characterized failure mode of the prevailing method. The key design principle is separation of concerns: divide the encoder's two roles (detection and magnitude estimation) into distinct computational paths, and apply the L1 penalty only to the detection path.

This is explicitly motivated by analogy to Gated Linear Units (Dauphin et al., 2017; Shazeer, 2020), which use a learned gating mechanism to control information flow. In a GLU, one linear projection produces candidate values while another produces sigmoid gates that modulate those values. The Gated SAE adapts this idea: one path decides which features are active (the "gate"), and another estimates how active they are. The L1 penalty applies only to the gating path's pre-activations, leaving the magnitude estimation path free from shrinkage-inducing sparsity pressure.

The paper explicitly connects this to classical dictionary learning literature. In particular, it notes that k-SVD (Aharon et al., 2006), a well-known dictionary learning algorithm, also uses a two-stage approach — one stage for sparse coding (finding which dictionary elements to use) and another for dictionary update — providing an intuitive parallel to the gating/magnitude split. However, the paper emphasizes that its contribution is independently motivated by the L1 bias problem rather than directly derived from k-SVD:

"We would like to thank Romeo Valentin for conversations that got us thinking about k-SVD in the context of SAEs, which inspired part of our work."

The paper also positions itself within the broader interpretability research agenda. The ultimate goal of SAE research is to enable understanding of large language models through their learned feature representations. The Gated SAE is offered not as a complete solution to interpretability, but as an improvement that reduces the systematic error in SAE reconstructions, making downstream analyses (circuit discovery, feature steering, anomaly detection) more reliable. The explicit hope, stated in the conclusion, is that:

"Gated SAEs can serve to accelerate such work."

What Makes This Problem Worth Solving Now

Several factors converge to make this the right moment for this work:

1. SAEs are gaining adoption. Following Bricken et al. (2023), multiple research groups are training SAEs on increasingly large models and using them for downstream interpretability tasks (circuit analysis in Marks et al., 2024; steering in Nanda et al., 2024; residual stream decomposition in Bloom, 2024). As the user base grows, the practical impact of systematic reconstruction errors becomes more consequential.

2. Scaling SAEs to larger models is challenging. This paper trains on models up to 7B parameters (Gemma-7B) at multiple internal sites. At this scale, inefficiencies in the sparsity–reconstruction trade-off compound — if you need twice as many active features to achieve a given reconstruction quality, the computational cost of analyzing those features (manual interpretation, automated explanation, causal intervention) scales accordingly. The paper's finding that Gated SAEs achieve comparable reconstruction with half the L0 is therefore not just a metric improvement but a practical enabler for scaling interpretability work.

3. The shrinkage problem was identified but not solved. Wright and Sharkey (2024) diagnosed the issue but their proposed solution (decoder finetuning) was limited because it couldn't address the training-time bias in encoder directions. This created a clear opportunity for an approach that eliminated shrinkage during training, not just compensated for it afterward.

4. Gating mechanisms are well-understood and proven. The paper's solution draws on established ideas from the gated architecture literature (GLUs, Shazeer, 2020), giving confidence that the approach is sound and implementable. The innovation is not in inventing gating but in recognizing that gating solves a specific, well-defined problem in SAE training.

Conceptual Foundation: Shrinkage is Not Just a Scaling Problem

Before moving to the technical approach, it's worth emphasizing one more subtle point the paper makes about why the L1 penalty is pernicious. The paper notes that shrinkage manifests in a measurable way: the relative reconstruction bias $\gamma$, defined in Equation (9), is less than 1 for baseline SAEs (indicating systematic underestimation of reconstruction norm). For Gated SAEs, $\gamma \approx 1$ (Figure 6).

But the paper's deeper claim — supported by the "baseline + rescale & shift" experiment in Section 5.2 — is that $\gamma \approx 1$ is a consequence of better training dynamics, not merely a post-hoc correction. Training without L1 pressure on the magnitude path allows the SAE to learn better feature directions (both encoder and decoder) from the beginning, because the optimizer never needs to compromise reconstruction fidelity for L1 reduction on active features. This is why simply rescaling a trained baseline SAE's outputs doesn't close the gap to Gated SAE performance.

This distinction — between fixing the symptom (rescaling activations) and fixing the cause (training without magnitude-path bias) — is central to understanding why Gated SAEs represent a genuine improvement rather than just a post-processing trick. It also explains why the paper invests in a full architectural change rather than a simpler intervention.

3. Technical Approach

3.1 Reader Orientation

We are building a modified sparse autoencoder — a neural network that takes a language model's internal activations and decomposes them into a sparse, linear combination of interpretable feature directions. The system solves the shrinkage problem: the baseline SAE architecture forces the same ReLU encoder to both detect which features are active and estimate how active they are, and applying the L1 sparsity penalty to this shared output systematically underestimates feature magnitudes. The Gated SAE separates these two responsibilities into distinct computational paths — one for detection (which receives the sparsity penalty) and one for magnitude estimation (which does not) — using a gating mechanism inspired by Gated Linear Units.

3.2 Big-Picture Architecture (Diagram in Words)

The Gated SAE has five major components connected in a specific flow:

1. Pre-encoder bias subtraction: The input activation $\mathbf{x} \in \mathbb{R}^n$ (e.g., an MLP layer output or residual stream state from a transformer) is first centered by subtracting the decoder bias $\mathbf{b}_{\text{dec}}$, producing $\mathbf{x}_{\text{centered}} = \mathbf{x} - \mathbf{b}_{\text{dec}}$. This centering is standard in SAE training and means the decoder bias acts as the "default" reconstruction when no features fire.

2. Gating encoder path (feature detection): The centered input passes through a linear layer with weights $\mathbf{W}_{\text{gate}} \in \mathbb{R}^{M \times n}$ and bias $\mathbf{b}_{\text{gate}} \in \mathbb{R}^M$, producing pre-activations $\boldsymbol{\pi}_{\text{gate}}(\mathbf{x}) = \mathbf{W}_{\text{gate}} \mathbf{x}_{\text{centered}} + \mathbf{b}_{\text{gate}}$. These pre-activations are thresholded by a Heaviside step function: features with $\pi_{\text{gate}, i} > 0$ are deemed active (gate = 1), others inactive (gate = 0). This is the pathway that receives the L1 sparsity penalty, applied specifically to the positive parts of these pre-activations before binarization.

3. Magnitude encoder path (feature magnitude estimation): In parallel, the same centered input passes through a second linear layer with weights $\mathbf{W}_{\text{mag}} \in \mathbb{R}^{M \times n}$ and bias $\mathbf{b}_{\text{mag}} \in \mathbb{R}^M$, producing $\mathbf{f}_{\text{mag}}(\mathbf{x}) = \text{ReLU}(\mathbf{W}_{\text{mag}} \mathbf{x}_{\text{centered}} + \mathbf{b}_{\text{mag}})$. This estimates the continuous magnitude of each feature's activation. Critically, this path receives no L1 penalty, so its estimates are unbiased by sparsity pressure.

4. Gating multiplication: The binary gates from the gating path elementwise-multiply the continuous magnitudes from the magnitude path: $\tilde{\mathbf{f}}(\mathbf{x}) = \mathbf{1}[\boldsymbol{\pi}_{\text{gate}}(\mathbf{x}) > 0] \odot \text{ReLU}(\mathbf{W}_{\text{mag}} \mathbf{x}_{\text{centered}} + \mathbf{b}_{\text{mag}})$. This zeros out any inactive features while preserving the unbiased magnitude estimates for active ones.

5. Linear decoder: The gated feature activations $\tilde{\mathbf{f}} \in \mathbb{R}^M$ are decoded by a linear layer with weights $\mathbf{W}_{\text{dec}} \in \mathbb{R}^{n \times M}$ (whose columns are constrained to unit norm during training) and bias $\mathbf{b}_{\text{dec}} \in \mathbb{R}^n$, producing the reconstruction $\hat{\mathbf{x}} = \mathbf{W}_{\text{dec}} \tilde{\mathbf{f}} + \mathbf{b}_{\text{dec}}$.

Parameter sharing between paths: To avoid a 50% increase in parameter count, the two encoder paths share their weight directions. Specifically, $\mathbf{W}_{\text{mag}}$ is defined in terms of $\mathbf{W}_{\text{gate}}$ and a learned rescaling vector $\mathbf{r}_{\text{mag}} \in \mathbb{R}^M$:

$(\mathbf{W}_{\text{mag}})_{ij} := (\exp(\mathbf{r}_{\text{mag}}))_i \cdot (\mathbf{W}_{\text{gate}})_{ij}$

So the magnitude path uses the same projection directions as the gating path, but scales each direction's norm by a learned per-feature factor $\exp(r_{\text{mag}, i})$, while allowing the biases $\mathbf{b}_{\text{gate}}$ and $\mathbf{b}_{\text{mag}}$ to differ independently. With this scheme, a Gated SAE has only $2 \times M$ more parameters than a baseline SAE (the extra bias vector $\mathbf{b}_{\text{mag}}$ and the rescaling vector $\mathbf{r}_{\text{mag}}$).

Reinterpretation as Jump ReLU: With this weight tying, the gated encoder can be expressed as a single linear layer with a parameterized activation function:

$\tilde{\mathbf{f}}(\mathbf{x}) = \sigma_{\boldsymbol{\theta}}(\mathbf{W}_{\text{mag}} \mathbf{x}_{\text{centered}} + \mathbf{b}_{\text{mag}})$

where $\sigma_{\theta}(z) := \mathbf{1}[z > \theta] \cdot \text{ReLU}(z)$ is the Jump ReLU (Erichson et al., 2019), and the per-feature gap parameter is $\theta_i = b_{\text{mag}, i} - \exp(r_{\text{mag}, i}) \cdot b_{\text{gate}, i}$. This means the magnitude path's pre-activation must exceed some positive threshold $\theta$ before the feature activates, and once active, the output is exactly the pre-activation itself (no shrinkage). The standard ReLU is a special case with $\theta = 0$.

3.3 Roadmap for the Deep Dive

• First, the baseline SAE architecture and loss function, establishing the precise starting point that the Gated SAE modifies. Understanding the baseline is essential because the Gated SAE's design is a direct response to its specific failure mode.

• Second, the Gated SAE architecture in full detail — the gating path, the magnitude path, the weight-tying scheme, and the Jump ReLU reinterpretation. This is the core architectural innovation.

• Third, the training loss function, which is where the L1 penalty is carefully restricted to only the gating path. The auxiliary reconstruction task and the frozen decoder are key design choices that prevent the L1 penalty from leaking into the magnitude estimation path.

• Fourth, the training and evaluation methodology — hyperparameters, resampling, decoder norm constraints, and the evaluation metrics (L0, loss recovered, relative reconstruction bias). These details are necessary to understand the experimental results and to replicate the approach.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology and architecture paper. Its core contribution is a specific architectural modification — the gated encoder with tied weights — and an accompanying loss function that together eliminate shrinkage by decoupling feature detection from magnitude estimation. The approach is evaluated empirically across multiple models and sites, but the technical novelty is in the architecture and training procedure design.

---

Baseline Architecture and Loss Function (What We're Improving)

Single-layer ReLU autoencoder. The baseline SAE, as established by Bricken et al. (2023), parameterizes the relationship between an input activation $\mathbf{x} \in \mathbb{R}^n$ and a sparse latent representation $\mathbf{f} \in \mathbb{R}^M$ (with $M \gg n$) through two operations:

$\mathbf{f}(\mathbf{x}) := \text{ReLU}\left(\mathbf{W}_{\text{enc}}(\mathbf{x} - \mathbf{b}_{\text{dec}}) + \mathbf{b}_{\text{enc}}\right)$

$\hat{\mathbf{x}}(\mathbf{f}) := \mathbf{W}_{\text{dec}} \mathbf{f} + \mathbf{b}_{\text{dec}}$

where $\mathbf{W}_{\text{enc}} \in \mathbb{R}^{M \times n}$ is the encoder weight matrix, $\mathbf{b}_{\text{enc}} \in \mathbb{R}^M$ is the encoder bias, $\mathbf{W}_{\text{dec}} \in \mathbb{R}^{n \times M}$ is the decoder weight matrix with unit-norm columns, and $\mathbf{b}_{\text{dec}} \in \mathbb{R}^n$ is the decoder bias.

What it computes: The encoder takes the centered input $\mathbf{x} - \mathbf{b}_{\text{dec}}$, projects it through $\mathbf{W}_{\text{enc}}$, adds bias $\mathbf{b}_{\text{enc}}$, and applies ReLU to produce non-negative feature activations. The decoder linearly combines the decoder weight matrix columns (the learned feature directions $\mathbf{d}_i$) weighted by $\mathbf{f}$, then adds $\mathbf{b}_{\text{dec}}$ to produce the reconstruction. Features with zero activation contribute nothing to the reconstruction; features with positive activation contribute in proportion to their magnitude.

The L1-reconstruction loss. Training minimizes:

$\mathcal{L}(\mathbf{x}) := \|\mathbf{x} - \hat{\mathbf{x}}(\mathbf{f}(\mathbf{x}))\|_2^2 + \lambda \|\mathbf{f}(\mathbf{x})\|_1$

where $\lambda > 0$ is a hyperparameter controlling the sparsity-reconstruction trade-off.

What it computes: The first term $\|\mathbf{x} - \hat{\mathbf{x}}\|_2^2$ is the mean squared error between input and reconstruction — lower is better, driving the SAE to faithfully reproduce the original activation. The second term $\lambda \|\mathbf{f}\|_1$ is the L1 norm of the feature activations — lower is sparser (fewer non-zero entries, or entries closer to zero). The $\lambda$ coefficient balances these competing objectives.

Why this form, and why it causes shrinkage: L1 is used because L0 (the count of non-zero features) is non-differentiable and therefore cannot be optimized by gradient descent. L1 is the tightest convex relaxation of L0 and empirically encourages sparsity. However, the L1 penalty applies to the same quantity that the reconstruction loss depends on — $\mathbf{f}(\mathbf{x})$. This creates a direct tension: the reconstruction loss wants $f_i$ to be exactly the right magnitude for accurate reconstruction, while the L1 penalty wants $f_i$ to be as small as possible, ideally zero. The optimizer compromises by setting $f_i$ somewhere between zero and the true activation magnitude — exactly the shrinkage phenomenon illustrated in Figure 2. Moreover, because this bias is present at every training step, it affects the learned encoder directions $\mathbf{W}_{\text{enc}}$ and decoder directions $\mathbf{W}_{\text{dec}}$, not just the final activation magnitudes. An encoder direction that minimizes the combined loss may be different from one that maximizes reconstruction fidelity alone.

Decoder column norm constraint. To prevent a degenerate solution where the encoder outputs arbitrarily small values and the decoder weights grow arbitrarily large (preserving the reconstruction while reducing the L1 penalty), the columns of $\mathbf{W}_{\text{dec}}$ are constrained to have exactly unit L2 norm during training. The paper follows the approach of Bricken et al. (2023) in using exact unit norm constraints rather than the "at most unit norm" relaxation suggested by Templeton et al. (2024), "for the sake of simplicity." This normalization means that the decoder effectively expresses feature directions $\mathbf{d}_i$ as unit vectors, and the feature activation $f_i$ encodes the scalar magnitude along that direction.

---

Gated SAE Architecture (The Core Innovation)

Motivation via role separation. The baseline encoder output $\mathbf{f}(\mathbf{x})$ serves two functions simultaneously: it determines which features are active (since ReLU maps negative pre-activations to exactly zero), and it estimates the magnitude of active features (since ReLU passes positive pre-activations through unchanged). The L1 penalty is necessary for the first function — it's what pushes most features to zero — but it's harmful for the second, because it pushes active features' magnitudes below their true values. The Gated SAE separates these two functions into distinct computational sub-paths.

The gated encoder. The gated encoder replaces the single ReLU encoder with a two-path structure and an elementwise multiplication:

$\tilde{\mathbf{f}}(\mathbf{x}) := \underbrace{\mathbf{1}[\overbrace{(\mathbf{W}_{\text{gate}}(\mathbf{x} - \mathbf{b}_{\text{dec}}) + \mathbf{b}_{\text{gate}})}^{\boldsymbol{\pi}_{\text{gate}}(\mathbf{x})} > 0]}_{\mathbf{f}_{\text{gate}}(\mathbf{x})} \odot \underbrace{\text{ReLU}(\mathbf{W}_{\text{mag}}(\mathbf{x} - \mathbf{b}_{\text{dec}}) + \mathbf{b}_{\text{mag}})}_{\mathbf{f}_{\text{mag}}(\mathbf{x})}$

where $\odot$ denotes elementwise (Hadamard) multiplication, and $\mathbf{1}[\cdot > 0]$ is the Heaviside step function applied elementwise (outputting 1 if the condition holds, 0 otherwise).

What it computes: The gating path $\mathbf{f}_{\text{gate}}$ produces a binary vector in $\{0, 1\}^M$: feature $i$ gets a 1 if $\pi_{\text{gate}, i} > 0$, and 0 otherwise. This is a hard decision about whether each feature is active. The magnitude path $\mathbf{f}_{\text{mag}}$ produces a non-negative continuous vector in $\mathbb{R}_{\geq 0}^M$ via ReLU: it estimates how strongly each feature is present. The elementwise product combines them: for features deemed inactive (gate = 0), the output is zero regardless of the magnitude estimate; for features deemed active (gate = 1), the output is the magnitude estimate. Crucially, a feature must satisfy both conditions to produce a non-zero output — the gating pre-activation must be positive AND the magnitude pre-activation must be positive.

The gating path pre-activations, not the Heaviside output, link to the loss. The Heaviside function has zero gradient almost everywhere, so no learning signal would flow to $\mathbf{W}_{\text{gate}}$ and $\mathbf{b}_{\text{gate}}$ through the main reconstruction loss. The paper solves this by introducing a separate loss term that operates on the rectified pre-activations $\text{ReLU}(\boldsymbol{\pi}_{\text{gate}}(\mathbf{x}))$ — the continuous-valued, non-negative pre-activations before binarization. This is explained in the training section below.

The weight-tying scheme between paths. Naively, the gate and magnitude paths have separate weight matrices, doubling the encoder parameters from $M \times n + M$ to $2M \times n + 2M$ (a ~100% increase in total SAE parameters). To mitigate this without sacrificing the benefit of the dual-path architecture, the paper constrains the magnitude path weights to share the same directions as the gating path weights, while allowing per-feature rescaling:

$(\mathbf{W}_{\text{mag}})_{ij} := (\exp(\mathbf{r}_{\text{mag}}))_i \cdot (\mathbf{W}_{\text{gate}})_{ij}$

where $\mathbf{r}_{\text{mag}} \in \mathbb{R}^M$ is a learned vector of log-scale factors. Exponentiating ensures the scaling is always positive; a value of $r_{\text{mag}, i} = 0$ means the $i$-th feature has the same weight norm in both paths, while positive or negative values stretch or shrink the norm relative to the gating path.

What this achieves: The two paths share the same set of $M$ projection directions (the rows of $\mathbf{W}_{\text{gate}}$), but each direction can have different effective norm in each path. Additionally, the biases $\mathbf{b}_{\text{gate}}$ and $\mathbf{b}_{\text{mag}}$ are independent per-feature. This means the gating path and magnitude path can have different thresholds for when a feature fires — the gating path fires when the projection exceeds $-b_{\text{gate}, i}$, while the magnitude path fires when the projection exceeds $-b_{\text{mag}, i} / \exp(r_{\text{mag}, i})$. More importantly, once a feature fires, the magnitude path's output is proportional to $\exp(r_{\text{mag}, i})$ times the gating path's equivalent output — allowing the magnitude path to produce a different scale of activation than the gating path would.

Parameter count comparison. With weight tying, the Gated SAE encoder has $M \times n + 2M$ parameters for $\mathbf{W}_{\text{gate}}$ and $\mathbf{b}_{\text{gate}}$, plus $M$ additional parameters for $\mathbf{b}_{\text{mag}}$ and $M$ for $\mathbf{r}_{\text{mag}}$, giving $M \times n + 3M$ total encoder parameters. The baseline SAE encoder has $M \times n + M$ parameters (for $\mathbf{W}_{\text{enc}}$ and $\mathbf{b}_{\text{enc}}$). The difference is $2M$ parameters, which is negligible compared to the $M \times n$ dominant term for typical $n$ (hundreds to thousands). Section 5.1 experimentally verifies that untying the weights (giving each path its own full weight matrix) does not improve performance, confirming that the weight-tying is not bottlenecking the architecture's expressivity.

Reinterpretation as a linear encoder with Jump ReLU. Appendix E of the paper shows that the tied-weight gated encoder is mathematically equivalent to a single linear layer followed by a parameterized Jump ReLU activation. For a single feature (ignoring the decoder bias for simplicity):

$\tilde{f}(\mathbf{x}) = \mathbf{1}[\mathbf{w}_{\text{gate}} \cdot \mathbf{x} + b_{\text{gate}} > 0] \cdot \text{ReLU}(\rho_{\text{mag}} \mathbf{w}_{\text{gate}} \cdot \mathbf{x} + b_{\text{mag}})$

where $\rho_{\text{mag}} = \exp(r_{\text{mag}})$. This can be rewritten as:

$\tilde{f}(\mathbf{x}) = \sigma_{\theta}(\mathbf{w}_{\text{mag}} \cdot \mathbf{x} + b_{\text{mag}})$

where $\mathbf{w}_{\text{mag}} = \rho_{\text{mag}} \mathbf{w}_{\text{gate}}$, and $\sigma_{\theta}(z) := \mathbf{1}[z > \theta] \cdot \text{ReLU}(z)$ with gap parameter $\theta = b_{\text{mag}} - \rho_{\text{mag}} b_{\text{gate}}$.

What this means geometrically. In a standard ReLU with threshold 0, the activation function outputs $z$ when $z > 0$ and 0 otherwise. In a Jump ReLU with gap $\theta > 0$, there is a "dead zone": the function outputs 0 for all $z \leq \theta$, and outputs $z$ for all $z > \theta$. This introduces a discontinuity at $\theta$. The gap parameter $\theta$ is learned per-feature through the interplay of $b_{\text{gate}}$, $b_{\text{mag}}$, and $r_{\text{mag}}$. This allows the SAE to learn features that only activate when there is strong evidence (the pre-activation exceeds a positive threshold), while still outputting the true pre-activation magnitude when active — no shrinkage. The standard ReLU corresponds to $\theta = 0$, which forces the SAE to fire on any positive projection, including noisy near-zero values from superposition interference in the model's activations.

Why Jump ReLU helps beyond resolving shrinkage. Appendix F provides a toy model illustrating an additional advantage. Consider a sparsely activating feature $X$ whose projection onto a unit encoder direction $\hat{\mathbf{v}}$ is $\mathcal{N}(2, 1/4)$ when $X$ is active (on) and $\mathcal{N}(0, 1)$ when inactive (off). A standard ReLU with threshold 0 fires on approximately half of the off-distribution samples (those with positive noise), creating many false positives. Raising the threshold to exclude most noise (e.g., to 1) also shifts the origin from which distance is measured when the feature is active, distorting the magnitude estimate even after rescaling. A Jump ReLU can set the threshold (where it starts firing) at 1 to exclude noise, while keeping the origin (from which magnitude is measured) at 0, giving near-perfect reconstruction. This illustrates the core advantage: the Jump ReLU decouples the firing threshold from the origin point for magnitude estimation.

---

Gated SAE Loss Function (How Training Works)

The naive (incorrect) approach and why it fails. One might think to simply apply the baseline loss to the gated encoder, restricting the L1 penalty to only the gating path outputs:

$\mathcal{L}_{\text{incorrect}}(\mathbf{x}) := \|\mathbf{x} - \hat{\mathbf{x}}(\tilde{\mathbf{f}}(\mathbf{x}))\|_2^2 + \lambda \|\mathbf{f}_{\text{gate}}(\mathbf{x})\|_1$

The problem: $\mathbf{f}_{\text{gate}}(\mathbf{x})$ is produced by a Heaviside step function, which has zero gradient almost everywhere. So no gradient signal would propagate to $\mathbf{W}_{\text{gate}}$, $\mathbf{b}_{\text{gate}}$, or $\mathbf{r}_{\text{mag}}$ through either the reconstruction loss (because the gate values are binary constants with respect to their parameters) or the L1 penalty (because the Heaviside output is either 0 or 1 and has zero gradient with respect to its pre-activation everywhere except the single point of discontinuity). The gating path parameters would never learn.

The actual loss function. To provide gradient signal to the gating parameters while maintaining the separation of concerns, the paper introduces a two-part loss:

$\mathcal{L}_{\text{gated}}(\mathbf{x}) := \underbrace{\|\mathbf{x} - \hat{\mathbf{x}}(\tilde{\mathbf{f}}(\mathbf{x}))\|_2^2}_{\mathcal{L}_{\text{reconstruct}}} + \lambda \underbrace{\|\text{ReLU}(\boldsymbol{\pi}_{\text{gate}}(\mathbf{x}))\|_1}_{\mathcal{L}_{\text{sparsity}}} + \underbrace{\|\mathbf{x} - \hat{\mathbf{x}}_{\text{frozen}}(\text{ReLU}(\boldsymbol{\pi}_{\text{gate}}(\mathbf{x})))\|_2^2}_{\mathcal{L}_{\text{aux}}}$

where $\boldsymbol{\pi}_{\text{gate}}(\mathbf{x}) = \mathbf{W}_{\text{gate}}(\mathbf{x} - \mathbf{b}_{\text{dec}}) + \mathbf{b}_{\text{gate}}$ are the gating pre-activations, $\text{ReLU}(\boldsymbol{\pi}_{\text{gate}})$ is the (elementwise) non-negative version of those pre-activations, and $\hat{\mathbf{x}}_{\text{frozen}}$ is a copy of the decoder with its weights frozen (using a stop-gradient operation, $\hat{\mathbf{x}}_{\text{frozen}}(\mathbf{f}) := \mathbf{W}_{\text{dec}}^{\text{copy}} \mathbf{f} + \mathbf{b}_{\text{dec}}^{\text{copy}}$).

What each term computes:

1. $\mathcal{L}_{\text{reconstruct}}$: The standard MSE between the input and the reconstruction produced by the full gated encoder — that is, using the binary-gated magnitude estimates $\tilde{\mathbf{f}}(\mathbf{x})$. This term trains the magnitude path ($\mathbf{W}_{\text{mag}}$, $\mathbf{b}_{\text{mag}}$) and the decoder ($\mathbf{W}_{\text{dec}}$, $\mathbf{b}_{\text{dec}}$) to produce faithful reconstructions. Since the gate values are binary (0 or 1) and treated as constants (the Heaviside blocks gradient flow), this term provides no gradient to the gating path parameters. This is intentional — it means $\mathcal{L}_{\text{reconstruct}}$ applies no L1-induced bias to the magnitude estimates.

2. $\mathcal{L}_{\text{sparsity}}$: The L1 norm of the rectified gating pre-activations $\text{ReLU}(\boldsymbol{\pi}_{\text{gate}})$. This applies the sparsity pressure to the gating path. Crucially, it operates on the continuous pre-activations before binarization, not on the binary gates themselves. This provides a strong gradient signal: when a gate pre-activation is positive, the L1 penalty pushes it toward zero, encouraging sparsity. When a gate pre-activation is negative, it's already zero after ReLU and receives no penalty — exactly as desired. This term trains $\mathbf{W}_{\text{gate}}$, $\mathbf{b}_{\text{gate}}$, and (indirectly through the pre-activations) $\mathbf{r}_{\text{mag}}$.

3. $\mathcal{L}_{\text{aux}}$: An auxiliary reconstruction task that uses the gating path only — taking $\text{ReLU}(\boldsymbol{\pi}_{\text{gate}})$ as the feature activations and decoding through the frozen decoder. This is necessary because $\mathcal{L}_{\text{sparsity}}$ alone would only push all gate pre-activations to be negative (zero after ReLU) — it provides no incentive for the gating path to actually detect features that matter for reconstruction. The auxiliary loss counterbalances this: it rewards the gating path when its rectified pre-activations, if used as feature activations (instead of the magnitude path's estimates), can reconstruct the input well through the frozen decoder. This ensures the gating path learns to activate for features that are genuinely useful for reconstruction, not just to be sparse.

Why the decoder is frozen in $\mathcal{L}_{\text{aux}}$: This is a critical design choice. If gradients from $\mathcal{L}_{\text{aux}}$ were allowed to update the decoder, the decoder would adapt to the gating path's (potentially shrunken or differently-distributed) activations, potentially degrading its performance on the main reconstruction task where it receives the magnitude path's unbiased estimates. Freezing the decoder forces the gating path to learn pre-activations that are compatible with the decoder's existing feature directions, rather than allowing the decoder to adjust to accommodate the gating path's behavior. Section 5.1 experimentally verifies that unfreezing the decoder in $\mathcal{L}_{\text{aux}}$ reduces performance — evidence supporting the "limit L1's scope" principle.

Computational cost. The loss requires running the decoder twice per training step: once for $\mathcal{L}_{\text{reconstruct}}$ (using $\tilde{\mathbf{f}}$) and once for $\mathcal{L}_{\text{aux}}$ (using $\text{ReLU}(\boldsymbol{\pi}_{\text{gate}})$). This is approximately 50% more computation than computing the baseline SAE loss for a matched-width model. However, the paper notes that in practice, training wall-clock time is often dominated by generating language model activations (if done on-the-fly) or disk I/O (if using cached activations), not by the SAE forward/backward passes, so the overhead is relatively modest. At inference time, the auxiliary loss is not needed — only a single forward pass through the gated encoder and decoder is required, which (with weight tying) has essentially the same FLOP count as the baseline SAE.

What happens to the gradients. During backpropagation:

• Gradients from $\mathcal{L}_{\text{reconstruct}}$ flow through the decoder and the magnitude path but stop at the Heaviside step function, providing no learning signal to the gating path.
• Gradients from $\mathcal{L}_{\text{sparsity}}$ flow through the gating pre-activations, updating $\mathbf{W}_{\text{gate}}$, $\mathbf{b}_{\text{gate}}$, and (via the chain rule through the pre-activations) $\mathbf{r}_{\text{mag}}$.
• Gradients from $\mathcal{L}_{\text{aux}}$ flow through the frozen decoder copy and the gating pre-activations, also updating the gating path parameters. The decoder itself receives no gradient from this term.

This asymmetric gradient flow is precisely what implements the separation of concerns: the magnitude path learns to produce unbiased, high-fidelity reconstructions (driven by $\mathcal{L}_{\text{reconstruct}}$), while the gating path learns to sparsely detect which features are present while remaining compatible with the decoder's feature basis (driven by $\mathcal{L}_{\text{sparsity}}$ and $\mathcal{L}_{\text{aux}}$).

---

Training and Evaluation Details

General training methodology. Beyond the architecture and loss, the paper specifies a number of training hyperparameters and procedures, largely following and extending the practices established by Bricken et al. (2023) and subsequent work:

• Optimizer: Adam with $\beta_1 = 0.0$ and $\beta_2 = 0.999$. The paper follows Templeton et al. (2024) in setting $\beta_1 = 0.0$ rather than the default $0.9$, noting this as a "slight improvement to training." Setting $\beta_1 = 0.0$ disables the momentum term, effectively using RMSProp-style updates where the step direction depends only on the current gradient's variance-normalized direction without historical smoothing. This may help because SAE training involves periodic resampling events that abruptly change the optimization landscape, and momentum could carry the optimizer in stale directions after such resets.

• Learning rate: $3 \times 10^{-4}$ (0.0003) for all Gated SAE experiments and the GELU-1L baseline. For Pythia-2.8B and Gemma-7B baseline SAEs, the paper used higher learning rates ($1 \times 10^{-3}$ and $7.5 \times 10^{-4}$) because the L2 loss was divided by $\mathbb{E}\|\mathbf{x}\|_2^2$ — a normalization motivated by "better hyperparameter transfer" — but the paper states this didn't noticeably affect the Pareto frontier and wasn't swept further.

• Warmup: Learning rate warmup is applied, including after each resampling event where the learning rate is reduced to $0.1 \times$ the ordinary value and increased back with a cosine schedule over 1000 steps.

• Batch size: 4096 for all runs. Training steps: 300,000 for GELU-1L and Gemma-7B, 400,000 for Pythia-2.8B.

• Activation sequence length: 128 tokens for GELU-1L, 2048 for Pythia-2.8B, 1024 for Gemma-7B.

• Training data buffer: A shuffled buffer of activations is used, following Nanda (2023), so that consecutive optimization steps don't use highly correlated activations from the same token positions. This is particularly important for longer sequence lengths (2048 for Pythia-2.8B), where consecutive positions in the same sequence have highly correlated representations.

Decoder weight norm constraints. The columns of $\mathbf{W}_{\text{dec}}$ are constrained to have exactly unit L2 norm during training. After each optimizer step, each column $\mathbf{w}_{\text{dec}, i}$ is projected onto the unit sphere: $\mathbf{w}_{\text{dec}, i} \leftarrow \mathbf{w}_{\text{dec}, i} / \|\mathbf{w}_{\text{dec}, i}\|_2$. The paper acknowledges that Templeton et al. (2024) suggest constraining to at most unit norm (which can help distinguish productive vs. unproductive feature directions, since dead features may shrink below unit norm), but chooses exact unit norm "for the sake of simplicity." This constraint is essential because without it, the encoder could output arbitrarily small values and the decoder could scale up its weights to compensate, making the L1 penalty meaningless.

Resampling (dead feature reinitialization). During training, some features may become "dead" — they never or almost never activate on any input, meaning their encoder directions have been pushed to regions of activation space that contain no signal. The paper uses resampling to revive these features, mostly following Bricken et al. (2023): periodically, features that activate extremely rarely are reinitialized. After each resampling event, learning rate warmup is reapplied (reducing to 0.1×, cosine back to full over 1000 steps). The paper notes that "resampling makes hyperparameter sweeps difficult" because "resampling caused L0 and loss recovered to increase" — newly initialized features can suddenly start firing, changing the sparsity and reconstruction characteristics mid-training.

SAE widths. For most experiments, baseline SAEs have width $2^{17} = 131,072$ features, and Gated SAEs have $3 \times 2^{16} = 196,608$ features — the Gated SAEs have 50% more features to compensate for their 50% higher training compute cost, ensuring fair comparison. For Pythia-2.8B residual stream sites, smaller widths ($2^{15}$ baseline, $3 \times 2^{14}$ Gated) were used "since early runs at these sites had lots of learned feature death." The actual widths vary slightly by experiment and are recorded in detail in Tables 1-8.

Interpreting the L1 coefficient $\lambda$. In the paper's infrastructure, the L2 reconstruction loss is computed and then divided by the input dimensionality $n$ to normalize for varying representation sizes. For the baseline Pythia-2.8B and Gemma-7B experiments, this normalized L2 loss is further divided by $\mathbb{E}\|\mathbf{x}\|_2^2$, which scales the reconstruction loss to be roughly unit-scale regardless of the activation site. This changes the effective $\lambda$ needed to achieve a given sparsity level, which is why the baseline experiments for those models used different learning rates and $\lambda$ values.

Evaluation metrics. The paper uses three quantitative metrics to assess SAE quality:

1. L0: The average number of active features per input, $\mathbb{E}_{\mathbf{x} \sim \mathcal{D}} \|\mathbf{f}(\mathbf{x})\|_0$. This is the direct measure of sparsity — lower L0 means fewer features fire on average. Unlike L1, which is used during training, L0 counts the exact number of non-zero entries (binary threshold at 0). For Gated SAEs, the gate values are binary by construction, so L0 counts the number of gates that are 1.

2. Loss recovered: The fraction of the language model's cross-entropy loss reduction (relative to a zero-ablation baseline) that the SAE preserves when its reconstructions are spliced into the model's forward pass:

$\text{Loss Recovered} := 1 - \frac{\text{CE}(\hat{\mathbf{x}} \circ \mathbf{f}) - \text{CE}(\text{Id})}{\text{CE}(\boldsymbol{\zeta}) - \text{CE}(\text{Id})}$

where $\text{CE}(\phi)$ is the average cross-entropy loss when function $\phi$ is spliced in at the SAE's site during the forward pass, $\hat{\mathbf{x}} \circ \mathbf{f}$ is the SAE (encoder + decoder), $\text{Id}: \mathbf{x} \mapsto \mathbf{x}$ is the identity function (no intervention — the original model's loss), and $\boldsymbol{\zeta}: \mathbf{x} \mapsto \mathbf{0}$ is the zero-ablation (replacing activations with zeros). A loss recovered of 0% means the SAE's reconstruction is equivalent to zeroing out the activations (no useful information preserved); 100% means perfect reconstruction (the model behaves exactly as it would with the original activations). Values above 100% are possible if the SAE reconstruction happens to reduce loss beyond the original model (e.g., by removing noise), though this usually indicates a distribution shift issue.

3. Relative reconstruction bias $\gamma$: A measure of shrinkage defined as the optimal multiplicative rescaling factor for the SAE's reconstructions to minimize L2 error:

$\gamma := \arg\min_{\gamma'} \mathbb{E}_{\mathbf{x} \sim \mathcal{D}}\left[\|\hat{\mathbf{x}}_{\text{SAE}}(\mathbf{x}) / \gamma' - \mathbf{x}\|_2^2\right]$

This is solved analytically (Equation 10):

$\gamma = \frac{2 \mathbb{E}_{\mathbf{x}}\left[\|\hat{\mathbf{x}}_{\text{SAE}}(\mathbf{x})\|_2^2\right]}{\mathbb{E}_{\mathbf{x}}\left[\|\hat{\mathbf{x}}_{\text{SAE}}(\mathbf{x})\|_2^2\right] + \mathbb{E}_{\mathbf{x}}\left[\|\mathbf{x}\|_2^2\right] - \mathbb{E}_{\mathbf{x}}\left[\|\hat{\mathbf{x}}_{\text{SAE}}(\mathbf{x}) - \mathbf{x}\|_2^2\right]}$

When $\gamma = 1$, the reconstruction is unbiased — its expected squared norm equals what it should be given the reconstruction error (an unbiased but imperfect SAE has $\mathbb{E}\|\hat{\mathbf{x}}\|_2^2 = \mathbb{E}\|\mathbf{x}\|_2^2 - \mathbb{E}\|\hat{\mathbf{x}} - \mathbf{x}\|_2^2$). When $\gamma < 1$, the reconstruction is systematically too small — this is the signature of shrinkage. The paper tracks $\gamma$ during training to quantify whether Gated SAEs indeed resolve shrinkage (Figure 6 shows $\gamma \approx 1$ for Gated, $\gamma < 1$ for baseline).

Evaluation dataset. All SAEs are evaluated on held-out activation data — over one million tokens that were not used during training.

---

Design Choices and Their Justifications

Choice 1: Separate gating and magnitude paths rather than post-hoc rescaling. The paper could have addressed shrinkage by training a baseline SAE and then learning rescaling factors for its feature activations (as in Wright and Sharkey, 2024). The decision to instead redesign the architecture reflects the paper's central claim that shrinkage biases the entire training process, not just the final outputs. Learning separate paths from scratch allows the magnitude path to learn different encoder directions than the gating path would have learned under L1 pressure, and allows the decoder to adapt to unbiased magnitude estimates. Section 5.2 empirically demonstrates this: a baseline SAE with learned rescaling and bias shifting ("baseline + rescale & shift") captures only a fraction of the Gated SAE's performance gain, confirming that the benefits go beyond magnitude correction.

Choice 2: Weight tying rather than independent encoder paths. The paper could have trained two completely independent encoders (doubling parameters). The ablation in Section 5.1 ("Untied encoders") shows that untying the weights does not improve performance over the tied scheme — in fact, it slightly deteriorates performance despite the greater expressive capacity. This suggests that sharing projection directions acts as a useful regularization, preventing the two paths from learning incompatible feature bases. It also keeps the parameter count and inference cost nearly identical to the baseline, which is a practical advantage for large-scale deployment.

Choice 3: L1 applied to rectified gating pre-activations rather than to binary gates. This is a necessity rather than a choice — binary gates have no gradient and cannot be trained directly with L1. But it also has a subtle benefit: the L1 penalty applied to $\text{ReLU}(\boldsymbol{\pi}_{\text{gate}})$ provides a continuous gradient that pushes gate pre-activations toward zero (and below), rather than the binary penalty $\|\mathbf{1}[\boldsymbol{\pi}_{\text{gate}} > 0]\|_1$ which would give zero signal for any pre-activation already above zero. This means the optimizer can continuously trade off the benefit of keeping a gate barely positive against the cost of the L1 penalty on its magnitude, providing a richer optimization landscape.

Choice 4: Frozen decoder in the auxiliary loss. If the decoder were trainable through $\mathcal{L}_{\text{aux}}$, the gating path could learn to produce activations that are convenient for the decoder under L1 pressure — likely shrunken, since the L1 penalty encourages small values — and the decoder would adapt to expect those shrunken values. This would defeat the purpose of having a separate, unbiased magnitude path. Freezing the decoder forces the gating path's rectified pre-activations to be compatible with the decoder's actual feature basis, as trained by the magnitude path through $\mathcal{L}_{\text{reconstruct}}$. Section 5.1 confirms unfreezing the decoder hurts performance.

Choice 5: The $\mathbf{r}_{\text{mag}}$ rescaling parameter. Why allow the magnitude path to scale each feature direction's norm independently? Without $\mathbf{r}_{\text{mag}}$, the two paths would have identical weight norms (only the biases could differ). This would mean the only way the magnitude path could produce different outputs than the gating path would be through its bias shift — limiting its ability to compensate for different signal-to-noise ratios across features. The ablation "No r_mag" in Section 5.1 shows a slight drop in performance when $\mathbf{r}_{\text{mag}}$ is removed, confirming it contributes meaningful flexibility.

Choice 6: Training with matched compute rather than matched width. To ensure fair comparison given that Gated SAE training requires 50% more compute per step (due to the auxiliary loss's decoder forward pass), the paper compares Gated SAEs to baseline SAEs with 50% more learned features (widths of ~200K vs. ~130K). This means the baseline SAEs have more capacity (more feature directions to use), compensating for the Gated SAEs' higher training cost. The Pareto improvement of Gated SAEs despite this handicap strengthens the conclusion that the architectural improvement is genuine, not just a compute efficiency artifact.

4. Key Insights and Innovations

Innovation 1: Diagnosing the "Dual-Role" Problem as the Root Cause of SAE Degradation

The Gated SAE's architectural novelty is grounded in a specific, falsifiable diagnosis: the baseline SAE's performance ceiling is not a generic consequence of the sparsity–fidelity trade-off, but specifically a consequence of encoding two distinct functions — detection and magnitude estimation — into a single quantity that receives the sparsity penalty. Prior work (Bricken et al., 2023; Templeton et al., 2024; Wright and Sharkey, 2024) implicitly treated the encoder output f(x) as a monolithic object: it's the "feature activations," and making it sparse via L1 is simply how dictionary learning works. Any side effects (like shrinkage) were seen as inevitable costs of doing business with L1 regularization, to be mitigated post-hoc rather than designed out of the architecture.

The key conceptual move this paper makes is reframing the encoder output as serving two roles that happen to share a carrier signal, and recognizing that the L1 penalty is necessary for only one of those roles. This is not an obvious decomposition. One could reasonably argue that both roles benefit from sparsity — after all, you want both detection and magnitude estimation to be zero for irrelevant features. But the paper's insight is subtler: for features that are active, the L1 penalty on magnitude is pure bias with no compensating benefit. It doesn't make the decomposition sparser — the feature is already firing — it just makes the reconstruction worse. This is fundamentally different from the Lasso setting (Tibshirani, 1996) where shrinkage is a deliberate feature of the estimator for variable selection, because in SAEs the sparsity goal has already been achieved once the detection decision is made.

The evidence that this diagnosis is not merely a post-hoc rationalization comes from Section 5.2 and Figure 9. If the problem were just that the final activation magnitudes are shrunken, then learning rescale and shift parameters on a frozen baseline SAE — which exactly addresses magnitude bias — should close most of the performance gap. Instead, Figure 9 shows that this captures only a small fraction of the Gated SAE's advantage. The bulk of the gain comes from training from scratch with the dual paths, implying that the L1 penalty's bias during training caused the baseline SAE to learn different, suboptimal encoder and decoder directions — not just wrong magnitudes, but wrong geometry. This is the smoking gun: the dual-role problem is a training-time pathology, not an inference-time scaling issue.

This reframing connects SAE training to a broader principle in machine learning: when a single learned quantity must satisfy two objectives, and a regularizer is appropriate for only one, applying it uniformly degrades both. The Gated SAE's solution — separate the objectives into distinct computational paths, apply the regularizer selectively — is a specific instance of this principle, and the paper's contribution is in identifying that this principle applies here and demonstrating that applying it yields a Pareto improvement.

Innovation 2: The Gating Mechanism as a Training-Time Separation of Concerns, Not Just an Architectural Gimmick

The Gated SAE uses a gating mechanism — a Heaviside step function multiplying a ReLU output — which superficially resembles Gated Linear Units (Dauphin et al., 2017; Shazeer, 2020) and might appear to be a straightforward architectural transplant. This would miss what's distinctive. The innovation is not gating per se, but the specific way gating is deployed to create an asymmetric gradient flow that limits the L1 penalty's scope to detection parameters only.

In a standard GLU, both the gate and the value pathway receive the same training signal — the gate learns to modulate information flow to improve the final objective, and both pathways are updated by the same loss. In the Gated SAE, the gradient flow is deliberately asymmetric: the Heaviside function in the gating path blocks gradients from the main reconstruction loss L_reconstruct from reaching the gating parameters. This means the gating path's parameters (W_gate, b_gate) receive gradient signal only through the sparsity penalty L_sparsity and the auxiliary reconstruction loss L_aux — never from the main reconstruction objective. Conversely, the magnitude path's parameters receive gradient signal only from the main reconstruction objective — never from the sparsity penalty.

This is a training-time mechanism, not an inference-time one. At inference, the gated encoder is equivalent to a Jump ReLU (Appendix E), and the gating structure is just one way to achieve that activation function. But during training, the Heaviside's gradient-blocking property is essential: it's what enforces the separation of concerns by physically preventing the L1 penalty's bias from propagating into the magnitude estimation pathway through gradient descent. The Jump ReLU interpretation explains what the trained model computes; the gating structure explains how training converges to that computation without L1-induced bias.

The ablation in Section 5.1 (Figure 8) confirms that this asymmetry matters. When the decoder is unfrozen in L_aux ("Ablation: unfreeze decoder"), performance degrades because the decoder can now adapt to the gating path's (L1-influenced) activations, partially re-coupling the two pathways through the shared decoder weights. The auxiliary loss is carefully designed so that the gating path must learn to produce activations that work well with the same decoder that the magnitude path uses — and freezing the decoder is what enforces this compatibility constraint.

This connects to a broader idea in multi-task and multi-objective learning: when different objectives should affect different subsets of parameters, architectural choices (like stop-gradient operations and frozen weight copies) can enforce this separation more reliably than carefully weighted loss terms alone. The Gated SAE's loss function (Equation 8) is actually three separate objectives affecting three overlapping but non-identical parameter subsets — and the architecture's gradient flow (not just the loss weights) determines which parameters respond to which objectives.

Innovation 3: Relative Reconstruction Bias as a Diagnostic Metric for Shrinkage

Beyond the architectural contribution, the paper introduces a clean, principled metric for quantifying shrinkage that goes beyond simply observing that reconstructions have smaller norm than inputs. The relative reconstruction bias γ (Equation 9-10) measures the optimal multiplicative rescaling factor for an SAE's reconstructions: γ = 1 for unbiased reconstructions, γ < 1 when shrinkage is present.

What makes this metric intellectually distinctive is that it distinguishes shrinkage from mere reconstruction error. An SAE with non-zero reconstruction error will naturally have E[‖x̂‖²] < E[‖x‖²] even without shrinkage, because imperfect reconstructions lose information that contributes to the input's variance. The γ metric accounts for this: it asks whether the reconstruction norm is even smaller than would be expected given the reconstruction error alone. This is formalized in the analytical expression (Equation 10), which relates γ to three observable expectations — the mean squared reconstruction norm, the mean squared input norm, and the mean squared reconstruction error.

Prior work on shrinkage (Wright and Sharkey, 2024) characterized the phenomenon qualitatively or through the lens of feature-level activation suppression, but did not provide a single scalar metric that tracks the overall severity of shrinkage across an SAE's entire output distribution. By defining γ and showing its behavior across training runs (Figure 6, Appendix C Figure 14), the paper makes shrinkage quantitatively trackable in a way that enables systematic comparison between architectures and hyperparameter settings.

The metric also serves a diagnostic purpose beyond this paper. The observation that baseline SAEs' γ decreases (shrinkage worsens) as the L1 coefficient λ increases and L0 decreases (Figure 6) quantitatively confirms the intuitive trade-off: stronger sparsity pressure → more shrinkage. For Gated SAEs, γ ≈ 1 across all λ values tested, providing a clean quantitative demonstration that the architecture achieves its design goal. Future work on SAE architectures or training methods can adopt γ as a standard diagnostic alongside L0 and loss recovered, creating a three-dimensional evaluation space (sparsity, fidelity, bias) rather than the two-dimensional one that dominated prior work.

Innovation 4: The Finding that Architectural Innovation Outperforms Post-Hoc Bias Correction

Section 5.2 and Figure 9 deliver a result that is conceptually significant beyond the specific method: training from scratch with the Gated SAE architecture substantially outperforms taking a converged baseline SAE and correcting its shrinkage, even when the correction mechanism is architecturally equivalent to the Gated SAE's inference-time computation. This is the "baseline + rescale & shift" experiment: freeze a trained baseline SAE's encoder and decoder weights, then learn per-feature rescaling and bias-shift parameters that should, in principle, be able to reproduce the Jump ReLU behavior that the Gated SAE exhibits at inference.

The result is that this post-hoc correction captures only a small fraction of the Gated SAE's performance gain. The interpretation — that the baseline SAE learned worse feature directions because the L1 penalty biased the entire training trajectory — is not obvious a priori. One might have expected that shrinkage is primarily a magnitude issue, and that a baseline SAE's encoder and decoder directions would be roughly correct (just with shrunken activations), making post-hoc rescaling nearly sufficient. The data reject this hypothesis.

This finding has implications for how the field thinks about regularization-induced bias in representation learning more broadly. It suggests that when a regularizer is applied to a quantity that serves multiple purposes, the resulting bias can be path-dependent — the optimizer finds a different local minimum in parameter space, not just a rescaled version of the unbiased minimum — and that post-hoc corrections cannot recover what was never learned. This is analogous to the difference between biased coefficient estimates (which can be debiased post-hoc) and biased model selection (which affects which variables enter the model at all).

For practitioners, this finding provides the rationale for adopting Gated SAEs as a replacement for baseline SAEs rather than as an optional post-processing step. It also suggests a general principle for designing architectures that will be trained with competing objectives: when a regularizer should apply to only one aspect of a learned representation, architectural separation (not just loss weighting) is the reliable way to prevent the regularizer from distorting the representation in ways that are difficult or impossible to correct after training.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses language model activations rather than a standard supervised dataset. These activations are collected by running forward passes of the target language model on its pretraining data (or similar distribution), extracting the representation at a specific layer and site (e.g., "MLP output at layer 20," "residual stream post-MLP at layer 12," "attention output pre-linear at layer 6"). The number of training tokens varies: "hundreds of millions to billions of activations from LM forward passes" (Appendix D.1.1). For evaluation, all SAEs are tested on over one million held-out tokens that were not used during training. The sequence lengths used for generating activations are 128 for GELU-1L, 2048 for Pythia-2.8B, and 1024 for Gemma-7B.

• Base model(s). Experiments are conducted on three model families at different scales:

  • GELU-1L (Nanda, 2022): A 1-layer transformer with GELU activations, chosen as the closest direct comparison to Bricken et al. (2023) and for rapid iteration.
  • Pythia-2.8B (Biderman et al., 2023): A 2.8B-parameter decoder-only transformer. The paper tests 5 different layers (4, 12, 16, 20, 28) and 3 different sites per layer (MLP output, attention output pre-linear, residual stream post-MLP).
  • Gemma-7B (Gemma Team et al., 2024): A 7B-parameter base model. The paper tests 4 different layers (6, 13, 20, 27) and the same 3 site types.

  These models span a meaningful scaling range (1 layer → 2.8B → 7B), allowing the paper to assess whether the Gated SAE's benefits generalize beyond small-scale settings.

• Metrics. Three metrics are used, assessed on held-out activation data:

  1. L0: The average number of active (non-zero) features per input, 𝔼_{x~D} ‖f(x)‖₀, computed from the binary gate values for Gated SAEs and from the ReLU output for baseline SAEs. Lower is sparser.
  2. Loss Recovered (as a percentage): The fraction of the language model's cross-entropy loss that the SAE preserves when its reconstructions are spliced into the forward pass. Formally: 1 − (CE(SAE) − CE(Identity)) / (CE(Zero Ablation) − CE(Identity)). Loss recovered of 100% means perfect reconstruction (model behaves identically to the original); 0% means the reconstruction is equivalent to zeroing out the activations.
  3. Relative Reconstruction Bias γ (Equation 9-10): The optimal multiplicative rescaling factor for an SAE's reconstructions to minimize L2 error. γ < 1 indicates shrinkage (systematic underestimation of reconstruction norm); γ ≈ 1 indicates unbiased reconstruction. Computed analytically from the mean squared reconstruction norm, mean squared input norm, and mean squared reconstruction error.

• Baselines. The primary comparison is against the baseline SAE (Baseline), defined by the architecture and loss in Equations (2)-(4) of Section 2.1-2.2, following Bricken et al. (2023). In Sections 4.3 and 5.2, additional comparison points include:

  • Baseline + rescale & shift: A frozen baseline SAE with additional learned rescaling and bias-shift parameters on top, inspired by Wright and Sharkey (2024). This tests whether post-hoc magnitude correction can match Gated SAE performance without retraining the encoder and decoder directions.
  • Inference-Time Optimization (ITO): Replaces the SAE encoder at test time with a gradient pursuit algorithm (Blumensath and Davies, 2008) that performs sparse approximation against the learned decoder. This isolates decoder quality from encoder quality. Used in Appendix A.

• Generation budget / compute accounting. Compute is measured per training step rather than per inference step. Gated SAE training requires running the decoder twice (once for L_reconstruct, once for L_aux), producing approximately 1.5× the compute cost of an equally wide baseline SAE training step. To ensure fair comparison, the paper compares Gated SAEs to baseline SAEs with 50% more learned features (dictionary size). For example, at most sites the baseline SAE has width 2^17 = 131,072 while the Gated SAE has width 3 × 2^16 = 196,608. This equalizes total training compute between the two methods. At inference time, Gated SAEs with weight tying have essentially identical compute to matched-width baseline SAEs.

• Cross-validation / statistical protocol. The paper does not use cross-validation in the traditional sense — there is no model selection between different hyperparameters on validation folds. Instead:

  • Held-out data: All evaluations use activations from tokens not seen during training.
  • Pareto frontier comparison: For each (model, site, layer) combination, multiple SAEs are trained with different L1 coefficients λ (typically 6-12 values), producing points in the (L0, Loss Recovered) plane. The Pareto frontier — the set of points not dominated by any other point in both metrics — is compared between Gated and baseline SAEs. This is a non-parametric comparison across the full range of achievable sparsity–fidelity trade-offs.
  • Interpretability study: Section 4.3 uses a double-blind design with expert raters seeing features in random order without knowing the SAE type, site, or layer. Statistical testing uses a one-sided paired Wilcoxon-Pratt signed-rank test and 90% BCa bootstrap confidence intervals for the mean difference in interpretability labels (coded as No=0, Maybe=1, Yes=2). The study includes 150 features from Pythia-2.8B (5 raters) and 192 features from Gemma-7B (7 raters), paired on (model, layer, site, rater) to control for confounding variables.

---

Main Quantitative Results

Pareto Improvement in the Sparsity–Fidelity Trade-off (Section 4.1)

The headline result is that Gated SAEs achieve a Pareto improvement in the (L0, Loss Recovered) trade-off over baseline SAEs across all tested models, layers, and sites. Specifically:

• Figure 1 and Figure 5 (GELU-1L MLP neuron activations): The Gated SAE curve lies everywhere above and to the left of the baseline SAE curve. For a given loss recovered value, the Gated SAE requires fewer active features (lower L0). For example, at a loss recovered of approximately 0.95, the Gated SAEs achieve this with an L0 of roughly 20-30, while baseline SAEs with 1.5× width require L0 ≈ 60 (Figure 5). The improvement is consistent across the full λ sweep, not just at specific operating points.

• Figure 12 (Pythia-2.8B, all sites and layers): Across 5 layers and 3 sites (MLP output, attention output pre-linear, residual stream post-MLP), Gated SAE Pareto frontiers dominate baseline SAE frontiers. The paper states: "At all sites we tested, Gated SAEs are a Pareto improvement" (Figure 12 caption). The improvement is particularly pronounced at the residual stream post-MLP site, where baseline SAEs struggle to achieve high loss recovered without high L0, while Gated SAEs maintain high fidelity at moderate sparsity. At the attention output sites, the gap is smaller but still consistent — Gated SAEs achieve the highest loss recovered at most L0 values.

• Figure 13 (Gemma-7B, all sites and layers): Similar pattern, with one anomalous site: attention output pre-linear at layer 27 shows loss recovered values above 100% (exceeding 120%). The paper explains this in Appendix B: the dataset used to train the SAE did not match Gemma's pretraining distribution, and at this site it was possible to mean-ablate the activations and actually decrease the model's loss — meaning the SAE reconstructions happened to be "better" than the original activations for the evaluation distribution. This is noted as a distribution shift artifact rather than a genuine quality signal.

• Quantifying the improvement: The paper claims in the conclusion that "at many sites, Gated SAEs require half the L0 to achieve the same loss recovered (Figure 12)." The exact ratio varies by site and operating point, but Figure 12 shows that for residual stream post-MLP and MLP output sites in Pythia-2.8B, achieving ~95% loss recovered requires baseline SAEs to have L0 ≈ 100-150 while Gated SAEs achieve this with L0 ≈ 40-60. This is approximately a 2-3× reduction in required active features.

• Pareto frontier non-monotonicity (Appendix B, Figures 12-13): Some Pareto frontiers show small regions of non-monotonic behavior — where increasing λ does not monotonically decrease L0 or increase loss recovered. The paper attributes this to "difficulties training SAEs" (Appendix D.1.3), particularly the interaction between resampling and the L1 coefficient. These artifacts do not affect the overall conclusion that Gated SAEs dominate baseline SAEs.

Computational parity caveat: All comparisons use baseline SAEs with 50% more features (Tables 1-8 confirm widths of ~131K for Gated vs. ~197K for baseline at most Gemma-7B sites; 3×2^14 = 49,152 for Gated vs. 2^15 = 32,768 for baseline at Pythia residual stream sites). The baseline SAEs have higher capacity to compensate for their lower training compute per step. The fact that Gated SAEs still Pareto-dominate despite this handicap strengthens the conclusion that the architectural improvement is genuine and not merely a compute efficiency artifact.

---

Resolution of Shrinkage (Section 4.2)

The paper demonstrates that Gated SAEs eliminate the shrinkage problem, as measured by the relative reconstruction bias γ:

• Figure 6 (GELU-1L MLP neuron activations): For baseline SAEs with 1.5× width, γ decreases systematically as L0 decreases (as λ increases and the sparsity penalty strengthens). At the sparsest baseline SAEs (L0 ≈ 5-10), γ ≈ 0.85, indicating that reconstructions are systematically ~15% too small in norm relative to optimal rescaling. For Gated SAEs, γ ≈ 1 across the entire range of L0 values, including the sparsest runs (L0 ≈ 2-5). The Gated SAE γ values show slightly more variance than 1.0 (some points at γ ≈ 1.05), but with no systematic trend with L0.

• Figure 14 (Pythia-2.8B, all sites and layers): The pattern generalizes across model scale and multiple sites. At residual stream post-MLP, MLP output, and attention output pre-linear sites across layers 6, 13, 20, and 27, Gated SAEs maintain γ ≈ 1 while baseline SAEs show γ values consistently below 1.0, with the severity of shrinkage varying by site. The worst shrinkage in Pythia-2.8B baseline SAEs appears at attention output sites (where γ drops to ~0.75 at low L0 values), while residual stream and MLP output sites show more moderate shrinkage (γ ≈ 0.85-0.95 at comparable L0 levels).

• Connection to reconstruction quality: The paper does not directly regress loss recovered against γ, but the per-figure combination of Figure 6 with Figure 5 (GELU-1L) and Figure 14 with Figure 12 (Pythia-2.8B) shows that the sites and λ values where baseline SAEs exhibit strong shrinkage are also where they fall below the Gated SAE Pareto frontier. This is consistent with — though does not prove — a causal relationship: the L1-induced bias in magnitude estimation limits reconstruction fidelity.

Interpretation: The γ ≈ 1 result is the quantitative confirmation that the Gated SAE's core design principle — separating detection from magnitude estimation — works as intended. The baseline SAEs' γ < 1 at low L0 confirms that the shrinkage problem is not hypothetical — it is a real, measurable consequence of the L1 penalty that worsens as sparsity pressure increases. The Gated SAE's immunity to this trend is direct evidence that the dual-path architecture successfully insulates magnitude estimation from sparsity pressure.

---

Human Interpretability Study (Section 4.3)

Beyond quantitative metrics, the paper conducts a double-blind human rater experiment to assess whether Gated SAE features are as interpretable as baseline SAE features (or more so). This matters because improving reconstruction fidelity and sparsity could, in principle, come at the cost of learning features that are "cheating" — exploiting discontinuities or learning directions that reconstruct well but don't correspond to coherent concepts.

• Experimental design (Section 4.3.1):

  • Pythia-2.8B: 5 raters each rated 1 feature from baseline and Gated SAEs at each (Site, Layer) pair from Figure 12, yielding 150 total features. All features were shown in random order, with raters blind to SAE type, site, and layer.
  • Gemma-7B: 7 raters (one rated 2 features, the rest 1 feature) from each (Site, Layer) pair from Figure 13, yielding 192 total features. A debugging issue meant Gemma attention SAEs were rated separately (raters were not blind to site for those), but were still blind to SAE type.
  • Rating procedure: For each feature, raters examined its highest activating examples, decided whether there is a coherent explanation of the feature's behavior, entered the explanation (if applicable), and selected: 'Yes' (interpretable), 'No' (uninterpretable), or 'Maybe' (maybe interpretable). The interface used the open-source SAE visualizer library (McDougall, 2024).

• Statistical analysis (Section 4.3.2, Figure 7 contingency tables):

  • Overall result: The one-sided paired Wilcoxon-Pratt signed-rank test of the null hypothesis that Gated SAE features are at most as interpretable as baseline SAE features gives p = .060 (estimate 0.13, mean difference 90% CI [0, 0.26]). This is marginal evidence against the null — not statistically significant at α = 0.05 — but trending in the direction of Gated features being more interpretable.
  • Per-model breakdown: Pythia-2.8B alone gives p = .15 (mean difference CI [-0.07, 0.33]); Gemma-7B alone gives p = .13 (mean difference CI [-0.04, 0.29]). Neither individually reaches significance.
  • Model comparison: A Mann-Whitney U test on label differences between the two models fails to reject the null that they come from the same distribution (p = .95); the same test on raw labels also fails to reject (p = .84).
  • Per-site analysis (Appendix H.1): A Friedman test across sites shows significant differences between the Gated-vs-Baseline differences (p = .047), though not between raw labels (p = .92). Breaking down by site: MLP outputs show the strongest result, with mean difference 0.40, significance p = .003, and CI [0.18, 0.63]. Attention outputs show p = .47 (mean 0.05, CI [-0.16, 0.26]), and residual stream shows p = .59 (mean -0.07, CI [-0.28, 0.12]). This suggests Gated SAE features may be more interpretable specifically at MLP output sites, but not conclusively at other sites.
  • Contingency tables (Figure 7): Descriptively, for the pooled data, there are many (Gated=Yes, Baseline=Maybe) and (Gated=Yes, Baseline=No) pairs — instances where Gated features were clearly interpretable but the corresponding baseline features were not. The exact counts are shown in the figure and are broadly balanced in the other direction (some baseline features more interpretable than their paired Gated counterparts).

• Conclusion stated by the paper: "while we can't definitively say the Gated SAE features are more interpretable than those from the Baseline SAEs, they are at least comparable" (Section 4.3.2). The stronger result at MLP output sites is noted but not overclaimed. The paper transparently acknowledges that the statistical evidence is marginal and does not reach conventional significance thresholds.

Interpretation: This result is important as a sanity check rather than a positive claim of superiority. The concern — that Gated SAEs might "cheat" by learning uninterpretable Jump ReLU features that reconstruct well without corresponding to coherent concepts — is not borne out by the data. Gated SAE features are at minimum as interpretable as baseline features, and may be somewhat more interpretable at MLP outputs, though the evidence for the latter is not statistically robust. The paper's handling of this is appropriately cautious.

---

Why Gated SAEs Improve Performance: Ablation and Mechanism Studies (Section 5.1)

Ablation study design (Figure 8): On GELU-1L MLP neuron activations, the paper trains variants of the Gated SAE that remove or modify specific components, and compares their Pareto frontiers to the full Gated SAE and the baseline SAE (1.5× width).

• Ablation: Unfreeze decoder in L_aux. When the decoder weights in the auxiliary loss are unfrozen — allowing gradients from L_aux to update them in addition to training the gating path — the Pareto frontier drops below the full Gated SAE. The reduction is visible but moderate: the unfrozen-decoder curve sits between the baseline and full Gated SAE curves at most L0 values. At low L0 (~10-20), the unfrozen-decoder variant achieves loss recovered of ~0.94 vs. ~0.96 for full Gated and ~0.92 for baseline. The paper interprets this as supporting the "limit the scope of L1" principle: allowing the decoder to adapt to the gating path's L1-influenced activations partially re-couples the two pathways, reintroducing some of the bias the architecture was designed to prevent.

• Ablation: No r_mag (remove rescaling parameter). Setting r_mag = 0 in Equation (7) — so the two encoder paths share identical weight norms and differ only in biases — causes a slight drop in performance. The no-r_mag curve is almost identical to the full Gated SAE at higher L0 values (>40) but diverges slightly at lower L0 (~10-20), where the full Gated SAE achieves ~0.96-0.97 loss recovered vs. ~0.95-0.96 for no-r_mag. The paper interprets this as r_mag contributing "somewhat (but not critically) to the improved performance." The small effect size is consistent with the interpretation that the critical components are the dual-path architecture itself and the frozen-decoder auxiliary loss, with r_mag providing modest additional flexibility.

• Ablation: Untie encoder layers. Training Gated SAEs with completely independent weight matrices for the gating and magnitude paths — no weight tying at all, doubling the encoder parameter count — produces a Pareto frontier that is essentially identical to (perhaps very slightly below) the tied-weight Gated SAE. The paper states "we see no improvement in performance — in fact a slight deterioration. This suggests our tying scheme (Eq. (7)) — where encoder directions are shared, but magnitudes and biases aren't — is effective at capturing the advantages of using a gated SAE while avoiding the 50% increase in parameter count and inference-time compute." The slight deterioration suggests weight sharing may act as a regularizer, preventing the two paths from learning incompatible feature bases.

Interpretation of the ablation results: The performance ordering is: Gated ≈ Gated (untied) > Gated (no r_mag) > Gated (unfrozen decoder) > Baseline. The gaps between Gated and the ablations are relatively small compared to the gap between Gated and Baseline, but consistent. This suggests that (a) the core benefit comes from the dual-path architecture with separated gradient flow, not from any one hyperparameter choice; (b) the weight-tying scheme is not bottlenecking performance (untying doesn't help); (c) the frozen decoder in L_aux matters more than r_mag, though both contribute.

---

Post-Hoc Shrinkage Correction vs. Full Gated Training (Section 5.2)

This experiment addresses the question: Is the Gated SAE's advantage solely attributable to addressing shrinkage (magnitude bias), or does it also learn better feature directions?

Experimental design (Figure 9): On GELU-1L MLP neuron activations, the paper compares:

1. Baseline SAE (equal width to Gated, not 1.5× width as in the main comparisons).
2. Gated SAE (standard training).
3. Baseline + rescale & shift: Take a converged baseline SAE, freeze its encoder and decoder weights, and learn per-feature rescaling parameters and bias shifts — changes that should, in principle, enable the baseline SAE to reproduce Jump ReLU behavior (matching the Gated SAE's inference-time computation) without retraining the feature directions. The architectural equivalence means any remaining performance gap must be due to the feature directions themselves.

Results:

• Baseline + rescale & shift improves over the baseline SAE across most of the L0 range, confirming that addressing shrinkage alone provides some benefit. For example, at L0 ≈ 60, baseline + rescale & shift achieves loss recovered of ~0.93, compared to ~0.92 for baseline and ~0.96 for Gated (values approximated from Figure 9).
• However, a substantial gap remains between baseline + rescale & shift and the Gated SAE, particularly at low to moderate L0 values (20-80). At L0 ≈ 20, baseline + rescale & shift achieves loss recovered of ~0.91 vs. ~0.96 for Gated.
• The gap narrows at very high L0 (>100), but this is less practically relevant (these operating points are very dense).

Interpretation (as stated by the paper): "Although resolving shrinkage only ('baseline + rescale & shift') does improve baseline SAEs' performance a little, a significant gap remains with respect to the performance of gated SAEs. This suggests that the benefit of the gated architecture and loss comes from learning better encoder and decoder directions, not just from overcoming shrinkage" (Section 5.2). In other words, the L1 penalty in baseline training doesn't just shrink the final outputs — it causes the optimizer to find a different local minimum in parameter space, with different (worse) feature geometry. The Gated SAE finds better directions because the magnitude path is never exposed to sparsity pressure, so reconstruction quality alone drives its learning.

Caveat: The equal-width comparison (rather than 1.5× width for baseline) means the baseline SAE here has ~33% fewer features than in the main Pareto comparisons, making the gap appear slightly larger than it would in the 1.5× width setting. However, the qualitative conclusion — that post-hoc correction explains only a fraction of the gain — would not change with increased width, as the Gated SAE outperforms even the 1.5× width baseline in Figures 5 and 12.

---

Inference-Time Optimization (Appendix A)

This analysis compares the quality of learned dictionaries (decoder weights) independently of encoder quality, by replacing the trained encoder at inference time with a gradient pursuit algorithm (Blumensath and Davies, 2008).

• Figure 10 (ITO at three test-time L0 thresholds): At target sparsities of 10, 30, and 100, the paper sweeps which trained decoder (from either Gated or baseline SAEs, at various training L0 values) achieves the best loss recovered under ITO. A surprising finding: the best-performing decoder at a given test-time L0 often came from an SAE trained with a much higher L0. For instance, at a test-time target L0 of 30, the best baseline SAE dictionary had a training L0 of around 100. Gated SAE dictionaries trained at lower target sparsities consistently outperform baseline dictionaries — at each test-time L0, the best Gated decoder achieves higher loss recovered than the best baseline decoder.

• Figure 11 (encoder vs. ITO Pareto frontiers): Comparing encoder-based reconstructions to ITO-based reconstructions for the best dictionary of each type, the gap between encoder and ITO performance is smaller for Gated SAEs than for baseline SAEs. The Gated encoder is closer to optimal sparse approximation (as approximated by ITO) than the baseline encoder is, suggesting that the Gated architecture enables the encoder to learn a better mapping from activations to feature coefficients — not just a better dictionary.

Interpretation: The Gated SAE's benefit is not purely in learning better feature directions (though it does that), but also in learning a better encoder that more closely approximates optimal sparse coding for its dictionary. This makes sense given the architecture: the gating path is specifically optimized for detection (driven by L_sparsity and L_aux), while the magnitude path is optimized for accurate coefficient estimation (driven by L_reconstruct). The baseline encoder must compromise between these objectives, leading to an encoder that is suboptimal at both.

---

Ablation Studies and Robustness Checks

Unfreeze decoder in L_aux (Section 5.1, Figure 8): Allowing the auxiliary reconstruction loss to update the decoder weights — so the decoder can adapt to the gating path's L1-influenced activations — reduces performance, confirming the principle that limiting the L1 penalty's scope to only the gating path parameters matters. The ablation's Pareto frontier sits between the baseline and the full Gated SAE, suggesting that while the dual-path architecture with unfrozen decoder still provides some benefit over the baseline (the gating and magnitude paths are still separated), the full separation with frozen decoder is important for achieving the best trade-off.

No r_mag (Section 5.1, Figure 8): Removing the per-feature rescaling parameter — so W_mag and W_gate have identical norms — causes a slight but consistent degradation, confirming that allowing the magnitude path to rescale each feature direction independently provides meaningful (though not critical) flexibility. The paper frames this as r_mag contributing modestly to performance.

Untied encoders (Section 5.1, Figure 8): Allowing the gating and magnitude paths to have completely independent weight matrices — doubling the encoder parameters — does not improve performance over the weight-tied scheme (in fact, a slight deterioration). This is an important negative result because it validates the paper's choice to share weight directions: the additional expressivity of independent encoders is either unnecessary or actively harmful (perhaps through overfitting or the two paths learning incompatible feature bases). This justifies the weight-tying as both parameter-efficient and performance-preserving.

Baseline + rescale & shift (Section 5.2, Figure 9): [Covered in detail above — shows that post-hoc correction captures only a fraction of the Gated SAE's gain, demonstrating that the benefit is in learned feature geometry, not just magnitude correction.]

Inference-Time Optimization (Appendix A, Figures 10-11): [Covered in detail above — shows that Gated SAEs learn better dictionaries and better encoders, as assessed by an external sparse approximation algorithm.]

Resampling behavior (Appendix D.1.3): The paper notes that resampling introduces complexity into hyperparameter sweeps: "resampling caused L0 and loss recovered to increase." This is an expected consequence of reviving dead features mid-training (they start firing, increasing L0; if they're useful, loss recovered may also increase). The resampling schedule and warmup procedure appear consistent across all experiments, so this should affect both Gated and baseline SAEs similarly — but it adds noise to the Pareto frontiers, particularly at the low-L0 end where resampling events can cause discontinuous jumps in sparsity.

Learning rate sensitivity (Appendix D.1.2): For the Pythia-2.8B and Gemma-7B baseline SAEs, the paper used a different learning rate than for Gated SAEs (0.001 and 0.00075 vs. 0.0003), due to a different L2 loss normalization scheme. The paper states "We didn't see noticeable difference in the Pareto frontier and so did not sweep this hyperparameter further." This is a minor robustness check — the normalization scheme and learning rate appear not to be pivotal for the relative performance ordering.

Dead feature rates (Tables 1-8): The full per-run statistics in Tables 1-8 include the percentage of "alive features" (features that activate at least occasionally). There is no systematic difference in dead feature rates between Gated and baseline SAEs — both exhibit a wide range (from <1% to >99% alive), varying primarily with the L1 coefficient λ rather than SAE type. This is a robustness check that the Gated architecture does not exacerbate or resolve dead feature problems; those are driven by the sparsity penalty and resampling schedule.

Distribution shift at Gemma-7B Layer 27 attention output (Appendix B): The anomalous >100% loss recovered values at this site are explained as a dataset mismatch (the SAE training data did not match Gemma's pretraining distribution), and the paper notes this explicitly. This is a known issue rather than an artifact of the architecture, and it affects both Gated and baseline SAEs similarly.

---

Critical Assessment

Does the Pareto Improvement Claim Hold Universally?

The paper's central quantitative claim — that Gated SAEs Pareto-improve the (L0, Loss Recovered) trade-off — is strongly supported by the experimental evidence in Figures 5, 12, and 13. The improvement is consistent across three model families spanning from a 1-layer model to a 7B-parameter model, across three different internal sites (MLP output, attention output, residual stream), and across multiple layers within each model. At no (model, site, layer, L0) combination tested does the baseline SAE Pareto-dominate the Gated SAE.

However, the claim should be qualified in two ways:

1. The Pareto improvement is relative to a baseline with 50% more features. This is the fair training-compute-matched comparison, but it means the baseline SAEs have higher representational capacity (more latent dimensions to work with). A matched-width comparison (same dictionary size) would show an even larger gap favoring Gated SAEs, as the Section 5.2 equal-width experiment suggests. Conversely, a comparison where baseline SAEs are given 50% more training steps (rather than 50% more features) is not performed — training compute and representational capacity are not perfectly substitutable, and the current experimental design conflates them to some degree.

2. The Pareto improvement is assessed only on the metrics L0 and Loss Recovered. While these capture reconstruction quality and sparsity — the two primary desiderata for SAEs — they do not capture all aspects of SAE quality. The human interpretability study (Section 4.3) partially addresses this gap (showing at minimum no degradation), but other potential dimensions — such as whether the learned features are causally meaningful in the model's computation, whether they enable circuit discovery, or whether the Jump ReLU discontinuity causes issues for gradient-based attribution methods — are not assessed. The paper acknowledges these limitations explicitly in the Limitations section.

Does the Claim that Gated SAEs "Solve Shrinkage" Hold?

Strongly supported. The relative reconstruction bias γ metric (Figures 6 and 14) provides clean, quantitative evidence: Gated SAEs maintain γ ≈ 1 across the full range of L0 values, while baseline SAEs exhibit γ < 1 that worsens with increasing sparsity pressure. This is replicated across GELU-1L and Pythia-2.8B, and across multiple sites. The γ metric is well-motivated (it distinguishes shrinkage from mere reconstruction error) and analytically tractable. The paper could strengthen this claim by:

• Reporting γ for Gemma-7B as well (not currently included), to confirm the pattern at the largest scale tested.
• Showing that γ correlates with downstream effects on the model's behavior when SAE reconstructions are spliced in (e.g., does γ < 1 systematically change the model's next-token predictions in ways that γ ≈ 1 does not?). Currently, loss recovered is the only splicing-based metric, and it conflates shrinkage effects with other reconstruction errors.
• Characterizing whether γ is uniform across features or varies systematically — a global γ ≈ 1 could mask some features being overestimated while others are underestimated (though the L2-based metric would catch systematic bias in either direction).

Does the Claim that Gated SAE Features are "Comparably Interpretable" Hold?

Supported, but with important caveats. The study design (double-blind, paired, with statistical testing) is rigorous for an interpretability study, but the sample size (342 features total across two models) and the p = .060 result leave room for doubt. The paper is appropriately cautious in its conclusions ("while we can't definitively say... they are at least comparable").

Specific concerns:

• The definition of "interpretable" is inherently subjective. The three-level scale (No/Maybe/Yes) is reasonable but coarse, and the raters likely had different internal thresholds. Inter-rater reliability is not reported (though rater-level tests in Appendix H.3 suggest some differences, which the paper acknowledges).
• The strongest result is at MLP output sites only (p = .003 in Appendix H.1), while attention outputs and residual stream show no significant difference. The paper does not overclaim this — it's reported in the appendix — but readers may incorrectly generalize the per-site result to all sites.
• The study can only show comparability, not superiority, with the given sample size. A larger study (more raters, more features, more models) would be needed to resolve the question of whether Gated SAE features are genuinely more interpretable. The paper's claim of "comparable" is conservative and supported.

Does the Claim that the Gated Architecture Learns "Better Directions" (Not Just Unbiased Magnitudes) Hold?

Supported, with one caveat. The baseline + rescale & shift experiment in Section 5.2 and Figure 9 is the key evidence. The substantial gap between post-hoc-corrected baseline and full Gated SAE training strongly suggests that the L1 penalty affects the learned geometry, not just the output magnitudes. However, there is a design subtlety: the baseline + rescale & shift approach learns rescaling and shift parameters on frozen encoder and decoder weights. This addresses magnitude bias but does not adjust the learned thresholds (the θ parameters in the Jump ReLU interpretation). The Gated SAE, by training from scratch, can learn per-feature thresholds that optimally trade off false positives and false negatives, while the rescale & shift approach is stuck with the thresholds that emerged from the baseline training (where the ReLU threshold is always at 0 in pre-activation space, determined by the encoder bias). This means the comparison is not purely about feature directions — it's also about learned thresholds. The paper's interpretation ("learning better encoder and decoder directions") is correct but incomplete; "learning better directions AND better detection thresholds" would be more precise.

A stronger test of the "better directions" claim would be: train a Gated SAE normally, then take its magnitude path and decoder weights, and train a new gating path from scratch on top (with the decoder frozen). If the Gated SAE's advantage over baseline + rescale & shift is entirely due to better directions, this combination should perform similarly to the full Gated SAE. If not, the learned thresholds (or other aspects of joint training) are also contributing. This experiment is not performed.

What Experiments Are Missing?

1. Causal analysis of features. The paper does not assess whether Gated SAE features are causally meaningful — i.e., whether intervening on a feature's activation changes the model's output in the way the feature's interpretation would predict. This is a standard desideratum in mechanistic interpretability (Marks et al., 2024; Nanda et al., 2024) and would provide stronger evidence that the Gated SAE's fidelity improvement translates to practical interpretability benefits. The paper acknowledges this limitation.

2. Larger-scale models (>7B parameters). The paper states in the conclusion that "future work could verify that Gated SAEs continue to improve dictionary learning beyond 7B base LLMs." The current results are consistent with scaling but do not demonstrate it. The Gemma-7B results (Figure 13) are qualitatively similar to the GELU-1L results (Figure 5), which is encouraging but not proof of scalability.

3. Different domains beyond language models. All experiments are on transformer language models. Whether Gated SAEs benefit dictionary learning in other architectures (CNNs, vision transformers) or modalities (vision, audio) is untested.

4. Reconstruction quality at the level of individual features. The evaluation focuses on aggregate metrics (L0, loss recovered, γ). It would be informative to see per-feature or per-example analyses: do Gated SAEs better reconstruct rare features? Do they better handle superposition interference (where many features are partially active)? The ITO analysis in Appendix A partially addresses this by comparing decoder quality, but doesn't provide feature-level decomposition of the improvement.

5. Impact on downstream interpretability tasks. The paper's conclusion expresses hope that Gated SAEs will "accelerate" circuit finding and steering work, but no experiments demonstrate this. A direct comparison of circuit recovery quality (e.g., using Marks et al., 2024's methodology) or steering effectiveness (e.g., using Turner et al., 2023's approach) between Gated and baseline SAE features would substantially strengthen the practical case for adoption.

6. Sensitivity to the L1 coefficient schedule. The paper sweeps λ across fixed values but does not explore whether annealing λ during training (starting with low sparsity pressure, increasing it) or using adaptive sparsity schedules (target a specific L0) would change the relative performance of Gated vs. baseline SAEs. Given that both architectures are sensitive to the choice of λ, an adaptive schedule might reduce the gap or change the optimal allocation of the sparsity penalty.

Where the Paper's Claims Should Be Considered Conditional

• "Half the L0 to achieve the same loss recovered" (Conclusion): This is a ballpark figure that applies "at many sites" but not uniformly. The ratio varies across sites and operating points. At attention output sites in Pythia-2.8B (Figure 12, right column), the gap is notably smaller than at residual stream or MLP output sites. Practitioners should not assume a universal 2× factor.

• Pareto improvement: Holds for all tested (model, site, layer, λ) combinations, but should be understood as "in the training-compute-matched setting with 1.5× width for baseline SAEs." In other compute regimes or width ratios, the ordering could potentially shift.

• Interpretability comparability: Holds in the sense that Gated SAE features are not worse, but the evidence for "better" is marginal and site-dependent (stronger at MLP outputs). The safe interpretation is "no evidence of interpretability degradation despite improved reconstruction."

• Shrinkage resolution: Holds robustly across GELU-1L and Pythia-2.8B. Untested at Gemma-7B (the paper evaluates γ only on the first two models in Figures 6 and 14).

• The Jump ReLU interpretation as an advantage: The toy model in Appendix F provides intuition for why Jump ReLUs might be better than standard ReLUs, but this is a pedagogical example, not empirical evidence from real models. The actual mechanism of improvement could involve additional factors beyond the thresholding behavior illustrated in the toy model.

6. Limitations and Trade-offs

The Gated SAE Assumes Linear, Sparse Feature Representations

This work, like all SAE-based dictionary learning, is motivated by the superposition hypothesis — the assumption that language models represent concepts as sparse linear combinations of feature directions in activation space. The paper explicitly acknowledges this in the Limitations section:

"Our work, like all sparse autoencoder research, is motivated by several assumptions about the sparsity and linearity of computation in Large Language Models (Section 1). If these assumptions are false, our work may still be useful (see footnote 1), but we may be making incorrect conclusions from work using SAEs, since they bake in the sparsity and linearity assumptions."

The consequence is that even a perfect Gated SAE — one that achieves 100% loss recovered with minimal L0 and no shrinkage — could still misrepresent the model's computation if the model itself does not operate primarily through linear feature composition. If a fraction of the model's computation involves non-linear interactions that cannot be decomposed into a sparse linear basis, the SAE's reconstruction — however faithful in L2 norm — would attribute that computation to spurious linear feature combinations. The paper provides no way to detect such non-linear contributions or to quantify what fraction of the model's behavior they explain. The assumption is baked into the architecture: the decoder is purely linear, and any non-linear computation in the original model must either be projected onto the learned linear basis or appear as reconstruction error. Unlike some other limitations the paper explicitly measures (e.g., shrinkage via γ), this assumption is untested — no experiment attempts to determine whether the linearity assumption is violated at any of the tested model sites. The paper acknowledges this as a limitation but offers no mitigation beyond the observation in footnote 1 that a faithful, sparse decomposition "would be a useful basis in its own right for downstream interpretability tasks... even if some fraction of the model's computation is represented non-linearly."

---

The Jump ReLU Discontinuity Creates Challenges for Gradient-Based Attribution Methods

The Gated SAE is equivalent at inference time to an SAE with a Jump ReLU activation function — a discontinuous step from zero to the pre-activation value at a learned threshold θ (Appendix E). While the paper does not present this discontinuity as a problem during training (the Heaviside's gradient-blocking property is actively used to separate gradient flows), it raises issues for downstream interpretability methods that rely on gradients through the SAE. The paper states:

"The discontinuity in particular introduces issues with methods like integrated gradients (Sundararajan et al., 2017) that discretely approximate a path integral, as applied to SAEs by Marks et al. (2024)."

Integrated gradients computes feature attributions by integrating gradients along a path from a baseline input to the target input. A discontinuous activation function has zero gradient almost everywhere (except at the single point of the jump), making the path integral ill-defined or degenerate. This directly affects the methodology of Marks et al. (2024), who use SAEs for circuit discovery and rely on gradient-based attribution through the SAE encoder to identify which features causally influence the model's output. If that methodology fails or degrades when applied to Gated SAEs, then improved reconstruction fidelity comes at the cost of reduced compatibility with an important downstream tool. The paper provides no experimental quantification of this effect — no comparison of integrated gradients faithfulness between Gated and baseline SAEs, no exploration of alternative attribution methods that might handle discontinuities better, and no demonstration that circuit discovery results are preserved or improved. This limitation is acknowledged but left entirely as future work, making it a significant practical uncertainty for the primary use case the paper envisions.

---

Computational Overhead from the Auxiliary Loss is Partially Unaccounted For in Deployment Scenarios

The paper is transparent that Gated SAE training requires approximately 50% more compute per step than training a matched-width baseline SAE, due to running the decoder twice (once for L_reconstruct, once for L_aux). To ensure fair comparison, the paper gives baseline SAEs 50% more features (Section 3.2.2, Section 4.1). The paper argues that "the increase in overall training time is typically much less, as in our experience much of the training wall clock time goes to generating language model activations... or disk I/O" (Section 3.2.2). However, this accounting has several gaps.

First, the relative cost of the SAE forward/backward pass versus activation generation depends heavily on the deployment context. For researchers training SAEs on pre-collected activation datasets (which is increasingly common as SAE research scales), the SAE training cost may dominate. Second, the auxiliary loss requires storing or recomputing the gating pre-activations and running a second decoder forward pass, which increases memory consumption during training — a practical constraint for very large SAEs where GPU memory is already tight. Third, the paper does not measure the actual wall-clock time difference in any experiment, relying instead on the claim that activation generation dominates. Finally, at inference time the auxiliary loss is not needed, so Gated SAEs with weight tying have essentially identical compute to baseline SAEs — but during hyperparameter sweeps (which are essential to find good λ values and produce the Pareto frontiers in Figures 5, 12, 13), the 50% per-step overhead compounds across dozens of runs. The paper does not discuss how this affects the practical cost of tuning Gated SAEs relative to baseline SAEs or whether the overhead can be reduced (e.g., by only enabling the auxiliary loss periodically rather than every step). No ablation studies investigate whether the auxiliary loss can be made cheaper without sacrificing performance.

---

Scaling Beyond 7B Parameters and to Non-Transformer Architectures is Untested

All experiments are conducted on three transformer language model families: GELU-1L, Pythia-2.8B, and Gemma-7B. The paper acknowledges this scope limitation:

"Future work could verify that Gated SAEs continue to improve dictionary learning beyond 7B base LLMs, such as by extending to larger chat models, or even to multimodal or Mixture-of-Experts models."

The consequence is uncertainty about whether the Gated SAE's benefits persist at larger scales or in different architectures. Several scaling-related failure modes are plausible but unexamined. First, at larger model sizes, activations may exhibit different noise characteristics or feature density patterns — the Jump ReLU's gating threshold, which the toy model in Appendix F suggests is useful for filtering superposition interference, might need to adapt differently at scale. Second, Mixture-of-Experts models route different inputs through different expert sub-networks, producing activations with potentially different statistical properties than the dense transformer layers tested here — the paper provides no evidence that Gated SAEs would handle this heterogeneity. Third, multimodal models produce activations that interleave representations from different modalities (text, image, audio tokens), which may violate the implicit assumption that activations are drawn from a relatively homogeneous distribution. The paper provides no extrapolation analysis — no trend showing how the Gated SAE's advantage over baseline SAEs changes as model size increases from GELU-1L → Pythia-2.8B → Gemma-7B, which would provide some evidence about scaling behavior. The results at Gemma-7B (Figure 13) are qualitatively similar to those at GELU-1L (Figure 5), which is encouraging but does not constitute a scaling trend analysis. No experiments use models with non-transformer architectures (CNNs, state-space models) or non-language modalities.

---

There is No Evidence That Improved Reconstruction Translates to Better Downstream Interpretability

The paper's stated motivation is to improve dictionary learning for mechanistic interpretability, with explicit hopes that Gated SAEs will "accelerate" circuit finding and steering work (Conclusion). The paper demonstrates that Gated SAEs improve reconstruction fidelity and reduce shrinkage, and that their features are comparably interpretable to baseline SAE features in a human labeling study. However, it provides no evidence that these quantitative improvements translate to better performance on actual interpretability tasks.

Several gaps exist. First, the human interpretability study (Section 4.3) assesses whether individual features have coherent explanations, but does not test whether Gated SAE features are more useful for specific interpretability workflows — identifying circuits, localizing factual knowledge, or enabling effective steering interventions. A feature can be "interpretable" (a human can describe what it responds to) without being causally important or useful for understanding the model's algorithms. Second, the paper does not evaluate whether the reduced L0 (half as many active features for comparable reconstruction) makes manual analysis substantially easier — less labor is implied but not demonstrated. Third, the paper cannot rule out that the increased architectural complexity of Gated SAEs introduces subtle failure modes that harm interpretability in ways not captured by the metrics. For example, the Jump ReLU's gating mechanism means a feature could be "marginally active" in the gating path (pre-activation just above zero) but produce a large magnitude estimate — a behavior with no analog in baseline SAEs that might confuse downstream analyses that assume feature activations are monotonically related to detection confidence. The paper's own limited evidence — the interpretability study's marginal p = .060 result — does not establish superiority and should temper expectations that Gated SAEs will dramatically improve interpretability outcomes. The authors are transparent about this gap: the conclusion frames Gated SAEs as a tool that "can serve to accelerate such work" rather than claiming demonstrated acceleration.

---

The Rescaling and Shift Ablation (Section 5.2) Does Not Isolate the Effect of Learned Thresholds from the Effect of Learned Directions

The paper's claim that Gated SAEs learn "better encoder and decoder directions" (Section 5.2) relies on the "baseline + rescale & shift" experiment (Figure 9), which takes a converged baseline SAE and learns per-feature rescaling and bias shift parameters on top. The interpretation is that any remaining gap between this corrected baseline and a full Gated SAE must be due to differences in the learned feature directions. However, this experiment does not control for a potentially important confound: the learned detection thresholds.

In the Jump ReLU interpretation (Appendix E), a Gated SAE has per-feature gap parameters θ_i = b_mag,i − exp(r_mag,i) · b_gate,i. These thresholds control where the activation function jumps from zero to the pre-activation value. In a baseline SAE, the ReLU threshold is always at zero in pre-activation space — the encoder bias b_enc,i determines where the pre-activation crosses zero, but there is no separate mechanism to set a positive firing threshold. The baseline + rescale & shift correction can adjust magnitudes (through rescaling) and shift the effective origin (through bias shifts), but cannot retroactively introduce a non-zero firing threshold — the baseline encoder was trained with ReLU, so its pre-activations near zero contain a mix of genuine weak feature detections and superposition noise, and the learned rescaling and shift cannot separate these after the fact.

The Gated SAE, by training from scratch, can learn per-feature θ values that optimally balance false positives (firing on noise) and false negatives (missing weak but genuine feature instances). The performance gap in Figure 9 could therefore be partly or largely attributable to better thresholds rather than better directions — the Gated SAE fires on cleaner signals because it can set a positive threshold, not because its projection directions are fundamentally different. The paper does not disentangle these two contributions. A more diagnostic experiment would be: take a Gated SAE's trained magnitude path and decoder weights, then train only the gating path from scratch (with the decoder frozen). If this combination matches the full Gated SAE, the advantage is in the directions; if it falls short, the advantage is in the joint optimization of thresholds and directions. This experiment is not performed, so the claim that "the benefit of the gated architecture and loss comes from learning better encoder and decoder directions, not just from overcoming shrinkage" is partially supported but not fully isolated — the benefit could come significantly from learning better detection thresholds, which is a distinct mechanism from learning better directions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around SAE training from treating the L1 penalty as an unavoidable source of bias to designing architectures that compartmentalize that bias. Prior to this work, the field's approach to the L1-induced shrinkage problem was largely reactive: Wright and Sharkey (2024) diagnosed the phenomenon and proposed post-hoc decoder finetuning as mitigation, while other methodological improvements (Templeton et al., 2024; Batson et al., 2024) tweaked training hyperparameters but left the fundamental architecture — a single ReLU encoder with a uniform L1 penalty — unchanged. The implicit assumption was that shrinkage is an inevitable side effect of using L1 as a sparsity surrogate, to be compensated for after training rather than eliminated during training.

The Gated SAE overturns this assumption. By separating the encoder's two functions — detection (which needs the L1 penalty) and magnitude estimation (which does not) — into distinct computational paths with asymmetric gradient flow, the paper demonstrates that shrinkage can be designed out of the architecture entirely, not just patched after the fact. The γ ≈ 1 result across all tested models and sites (Figures 6, 14) is a clean existence proof: an SAE trained with L1 sparsity pressure need not exhibit systematic underestimation of feature activations. This reframes the design space for future SAE research: rather than asking "how can we mitigate the side effects of L1?", researchers can ask "which parameters should the sparsity penalty influence, and how do we architecturally restrict it to only those parameters?"

The paper also reconciles a tension in the SAE scaling literature. Bricken et al. (2023) demonstrated that SAEs can discover interpretable features at small scale (1-layer models), but subsequent work scaling to larger models (Templeton et al., 2024; this paper's Pythia-2.8B and Gemma-7B experiments) encountered diminishing returns — higher L0 was needed to maintain reconstruction fidelity, and dead features proliferated. The Gated SAE's 2× reduction in required L0 at comparable loss recovered (Figure 12) suggests that part of this "scaling problem" was actually a methodological problem — baseline SAEs were wasting sparsity budget on many weakly-active, shrunken features that contributed little to reconstruction. The Gated architecture eliminates these low-magnitude features (because the gating mechanism enforces a hard threshold via the Jump ReLU gap, rather than the soft threshold of a ReLU), concentrating the sparsity budget on genuinely active features. This implies that improved architectures, not just more compute or wider SAEs, are the key to scaling dictionary learning to frontier models.

A more subtle shift is in the evaluation culture for SAE research. The paper introduces relative reconstruction bias γ as a third evaluation axis alongside L0 and loss recovered, creating a three-dimensional evaluation space (sparsity, fidelity, bias). Prior work implicitly used a two-dimensional Pareto frontier; this paper demonstrates that two SAEs can have identical L0 and loss recovered but different γ (indicating different reconstruction bias), and that γ captures a practically important property — systematic underestimation — that the other metrics miss. The γ metric is clean, analytically tractable (Equation 10), and cheap to compute during training, making it straightforward for other researchers to adopt. This is a methodological contribution that improves the field's ability to diagnose and compare SAE training methods, independent of the specific Gated architecture.

In terms of research prioritization, the paper's findings redirect effort in several ways:

• More attractive: Architectural innovations that separate detection from magnitude estimation — alternative gating mechanisms, different activation functions with explicit gap parameters, or entirely different decompositions of the encoder's role — are now a clearly promising direction. The Jump ReLU interpretation (Appendix E) suggests a rich design space of parameterized activation functions beyond the binary choice of ReLU vs. Jump ReLU.

• More attractive: Improving decoder quality independently of encoder quality, now that the ITO analysis (Appendix A) shows Gated SAEs learn better dictionaries (decoder weights) as well as better encoders. The finding that the best baseline dictionary at a given test-time L0 came from an SAE trained with much higher L0 (Figure 10) raises questions about whether existing SAE training procedures are optimal for dictionary learning or merely for encoder warm-starting.

• Less attractive: Post-hoc shrinkage correction methods (decoder finetuning, output rescaling) are now demonstrated to be fundamentally limited — they cannot recover the feature direction quality that is lost by training under L1 bias (Figure 9). This finding should shift effort toward architectural solutions that prevent bias during training rather than compensating for it afterward.

• Less attractive: Improvements to the L1 penalty itself (e.g., alternative sparsity regularizers like the Tanh penalty from Templeton et al., 2024) that still apply uniformly to a single encoder output, because the paper's core insight is that the problem is not which sparsity regularizer is used but which parameters it affects. A "better" regularizer applied to a combined detection-and-magnitude pathway will still introduce some form of bias, even if less severe than L1.

The paper does not represent a paradigm shift — it does not challenge the superposition hypothesis, the linear representation assumption, or the fundamental autoencoder approach to dictionary learning. It is a targeted architectural refinement that addresses a specific, well-characterized failure mode. However, the magnitude of the practical improvement (2× reduction in L0 at comparable fidelity) and the conceptual clarity of the diagnosis (the dual-role problem) make it a refinement that substantially changes what practitioners should expect from SAEs and how they should design them.

---

Follow-Up Research This Work Enables

1. Causal validation: do Gated SAE features support circuit discovery and steering better than baseline features?

The paper demonstrates reconstruction fidelity and interpretability improvements, but does not test whether Gated SAE features are causally meaningful in the model's computation — whether intervening on a Gated feature's activation produces the expected downstream effect. This is the central promise of SAEs for mechanistic interpretability (Marks et al., 2024; Nanda et al., 2024), and the paper's explicit hope is that Gated SAEs will "accelerate such work" (Conclusion). A strong follow-up would replicate the circuit-finding methodology of Marks et al. (2024) — which uses sparse feature circuits to identify causally important feature interactions — on both baseline and Gated SAEs trained at the same model site and layer, and compare: (a) the fraction of model behavior explained by circuits of a given size, (b) the faithfulness of the circuits (do they reproduce the model's output distribution when ablated?), and (c) whether the Jump ReLU discontinuity causes practical problems for the integrated gradients attribution method that Marks et al. (2024) uses. A negative result — that Gated SAE features are less causally tractable despite being more faithful and equally interpretable — would substantially change the field's assessment of the architecture's value.

2. Adaptive or learned sparsity schedules instead of fixed λ.

The paper sweeps a fixed L1 coefficient λ for each training run and selects points on the Pareto frontier post-hoc. In practice, practitioners must commit to a specific λ (and thus a specific L0 regime) before training, which is expensive and often requires multiple runs. A natural extension is to train a Gated SAE with an adaptive sparsity schedule that targets a specific L0 value dynamically — for example, adjusting λ online to maintain a target average L0, similar to how some SAE training pipelines use a target sparsity rather than a fixed λ. The Gated SAE's separation of the sparsity penalty onto the gating path only makes this cleaner: the L_sparsity term can be rescaled dynamically based on the current L0 without destabilizing the magnitude path's training (since the magnitude path doesn't receive the sparsity gradient). A strong experiment would compare Gated vs. baseline SAEs trained with adaptive λ schedules targeting the same L0 trajectory, measuring whether the Gated SAE's advantage persists or even widens when λ is not fixed. The paper's ablation showing unfreezing the decoder in L_aux degrades performance (Figure 8) provides suggestive evidence that gradient flow separation matters most when training objectives compete, and adaptive schedules may intensify this competition.

3. Direct measurement of non-linear computation in language model representations.

The SAE approach — both baseline and Gated — assumes that model activations can be faithfully reconstructed as sparse linear combinations of feature directions (Equation 1). The paper acknowledges that "if these assumptions are false, our work may still be useful... but we may be making incorrect conclusions from work using SAEs" (Section 7, Limitations). The Gated SAE's improved reconstruction fidelity creates an opportunity to quantify the non-linear residual — the component of the model's activation that cannot be explained by any sparse linear decomposition, even an optimized one. Specifically: train a Gated SAE to convergence, then compute the residual x − x̂(x) averaged over the data distribution, and analyze whether this residual exhibits structure (e.g., it correlates with specific linguistic features, it is predictable from the input, it varies systematically across layers) or whether it appears to be isotropic noise. If the residual is structured, this would be evidence against the linearity assumption and would motivate hybrid architectures (e.g., SAE + a small non-linear corrector network). If it is unstructured, this would strengthen confidence in the linear representation hypothesis. The Gated SAE's γ ≈ 1 property is important here because shrinkage in baseline SAEs would create a spurious correlation between reconstruction error and activation magnitude, making the residual analysis misleading.

4. Scaling trend analysis from 1-layer to 7B+ models.

The paper tests three model scales (GELU-1L, Pythia-2.8B, Gemma-7B) but does not analyze how the Gated SAE's advantage over baseline SAEs changes with model size, layer depth, or site type. A scaling analysis would answer: does the 2× L0 reduction hold at larger scales? Does the advantage grow, shrink, or stay constant? The existing data in Figures 5, 12, and 13 could be mined for this — plotting the L0 ratio (baseline / Gated) at a fixed loss recovered threshold against model parameters, layer index, and site type (MLP vs. attention vs. residual stream). If the advantage diminishes at larger scales (e.g., because larger models have cleaner, more linearly separable representations that baseline SAEs handle fine), that would bound the practical importance of the Gated architecture. If it grows (e.g., because larger models have more severe superposition, making the Jump ReLU's noise-filtering threshold more valuable), that would strengthen the case for scaling Gated SAEs to frontier models. The paper's current result that Gated SAEs are a Pareto improvement "at all sites tested" (Figure 12 caption) treats scale as a robustness check rather than an independent variable, but the data for a scaling analysis is largely present.

5. Gradient-attribution-compatible Gated SAE variants.

The Jump ReLU discontinuity is explicitly flagged as problematic for integrated gradients (Section 7, Limitations). A practically important follow-up would be to develop and test soft-gating variants that preserve the gradient flow separation during training but provide smooth activations at inference time that are compatible with gradient-based attribution. One approach: during training, use the hard Heaviside gate (to enforce the separation of concerns), but at inference time, replace it with a sigmoid gate with temperature annealing — σ(π_gate / τ) with small τ — which is differentiable everywhere. The trade-off is that a soft gate may reintroduce some shrinkage (since the gate values are less than 1 for marginally active features), and the paper would need to measure how much the γ ≈ 1 property degrades as a function of τ. A strong experiment would train Gated SAEs normally, then at test time sweep the soft-gate temperature and measure: (a) the change in loss recovered and L0 (how much does reconstruction fidelity degrade?), (b) the faithfulness of integrated gradients attributions compared to baseline SAE attributions, and (c) whether the soft gate's gradient provides useful signal for circuit discovery (e.g., using Marks et al., 2024's methodology).

6. Cross-architecture dictionary learning: do Gated SAEs benefit vision or multimodal models?

All experiments are on transformer language models. The superposition hypothesis — that models represent more features than dimensions by exploiting sparsity — is not language-specific; it should apply to any high-capacity neural network trained on high-dimensional data with compositional structure. A natural extension is to train Gated SAEs on intermediate activations from vision transformers (ViTs), multimodal models (e.g., a CLIP-style vision encoder), or even convolutional networks, and compare the L0 / loss recovered Pareto frontier against baseline SAEs. The primary question is whether the Jump ReLU's noise-filtering threshold — which the toy model in Appendix F suggests is useful when there is superposition interference (many non-orthogonal features partially active, creating noisy near-zero activations) — provides similar benefits in vision models, which may have different sparsity patterns than language models. A secondary question is whether the improved reconstruction fidelity translates to more interpretable visual features (e.g., features that correspond to coherent visual concepts like textures, shapes, or object parts). The Gemma-7B experiments already test a 7B-parameter model, demonstrating feasibility at moderate scale, so a 7B-class vision model (e.g., ViT-G) would be a natural next target.

---

Practical Applications and Downstream Use Cases

1. Sparse feature extraction for interpretability at scale. The paper's headline result — Gated SAEs require approximately half the L0 to achieve comparable reconstruction fidelity — has a direct practical implication for any workflow that involves human inspection or automated interpretation of SAE features. If an interpretability team is analyzing a 7B-parameter model's MLP outputs and wants to identify all features with loss recovered >95%, they would need to examine roughly half as many active features per input with a Gated SAE compared to a baseline SAE. For a typical analysis that examines hundreds to thousands of inputs per feature, this reduction in active features translates to substantially less manual labor or automated interpretation compute. The interpretability study (Section 4.3) provides evidence that this efficiency gain does not come at the cost of feature meaningfulness — Gated SAE features are at least as interpretable. Purely from a cost perspective, the training overhead (1.5× per step, partially offset by the fact that "much of the training wall clock time goes to generating language model activations or disk I/O," Section 3.2.2) is modest compared to the downstream savings from analyzing half as many features.

2. Improved SAE-based model steering. Nanda et al. (2024) and Turner et al. (2023) propose using SAE features as "steering vectors" — intervening on specific feature activations to modify the model's behavior in predictable ways (e.g., adjusting a "sentiment" feature to make generations more positive). The effectiveness of such steering depends critically on the accuracy of feature activation magnitudes: if the SAE systematically underestimates how strongly a feature is active, the steering intervention will be miscalibrated (systematically too weak for the desired effect). The paper shows that baseline SAEs exhibit shrinkage (γ < 1, Figure 6), with γ dropping to ~0.85 at low L0 values — meaning reconstructions are systematically ~15% too small in norm. For a steering application that multiplies the current feature activation by a target factor, this 15% underestimation would propagate to a 15% error in the intervention strength. The Gated SAE's γ ≈ 1 property eliminates this systematic calibration error, making steering interventions more predictable. This benefit is independent of whether the Gated SAE's features are more interpretable — it's purely a consequence of unbiased magnitude estimation.

3. Higher-quality training data for automated interpretability pipelines. Bills et al. (2023) introduced automated interpretability (using a separate language model to generate natural-language explanations of SAE features based on their activating examples) as a way to scale feature interpretation beyond what manual inspection can achieve. The quality of these automated explanations depends on the clarity of the feature's activation pattern — features that fire inconsistently, with many borderline near-zero activations, produce noisy example sets that lead to worse auto-generated explanations. The paper notes in its Limitations that baseline SAEs "don't have a thresholding mechanism like Gated SAEs do," and that "low activating features increase baseline SAEs' L0 at a given loss recovered without contributing much to reconstruction... with unclear impact on interpretability." The Gated SAE's hard gating threshold (via the Jump ReLU gap) eliminates these marginal activations — a feature either fires cleanly (when the gating pre-activation is positive) or doesn't fire at all (when it's below threshold). This produces cleaner, more binary feature activation patterns that should improve the signal-to-noise ratio for automated interpretability methods, though this prediction is not empirically tested in the paper.

4. Calibrated feature importance for anomaly detection and model auditing. SAEs are being explored as tools for detecting anomalous model behavior — if a feature that usually activates for benign inputs suddenly fires strongly on an adversarial or out-of-distribution input, that's a signal worth investigating. However, such anomaly detection requires that feature activation magnitudes be well-calibrated: a shrunken activation on a normal input (due to L1 bias) might be indistinguishable from a genuinely low-but-normal activation on an unusual input. The Gated SAE's γ ≈ 1 property means that activation magnitudes are not systematically compressed, making it easier to set meaningful thresholds for "unusual" activation levels. For example, an auditor monitoring a deployed model for signs of jailbreak attempts could track the activations of "harmfulness" features; with baseline SAEs, the systematic shrinkage means the auditor would need to estimate and compensate for the shrinkage factor per feature, while Gated SAEs provide feature magnitudes that are directly interpretable as the model's estimate of the feature's strength.

---

When to Prefer This Method

The paper does not articulate an explicit trade-off between Gated SAEs and named alternative methods beyond the baseline SAE — the comparison is uniformly "Gated SAEs are a Pareto improvement," with no operating point or setting where baseline SAEs are preferable. The ablation study (Section 5.1) examines variants of the Gated architecture but does not position Gated SAEs against, for example, the Tanh penalty (Templeton et al., 2024), decoder finetuning (Wright and Sharkey, 2024), or top-k SAEs (Gao et al., 2024) as competing approaches with different strengths. The paper's own Limitations section notes potential failure modes — the Jump ReLU discontinuity is incompatible with integrated gradients, the auxiliary loss adds training overhead, the linearity assumption is untested — but these are presented as open questions rather than conditions where baseline SAEs would be the stronger choice. The comparison with "baseline + rescale & shift" (Section 5.2, Figure 9) shows that post-hoc shrinkage correction cannot match full Gated training, but this is not framed as a trade-off — it's offered as evidence that the Gated architecture's benefit goes beyond magnitude correction. Since the paper does not identify a specific regime (e.g., a particular model scale, site type, L0 range, or downstream task) where the baseline SAE outperforms the Gated SAE on the metrics it measures, a formal "Prefer X when Y" decision rule would impose a structure the paper does not provide. The most the paper offers is the acknowledgment that "it could be argued that some of the performance gap between Gated and baseline SAEs could be closed by inexpensive inference-time interventions that prune the many low activating features" (Section 7, Limitations) — but this is a speculation about narrowing the gap, not a condition where the baseline is superior.