Sparse Autoencoders Trained on the Same Data Learn Different Features

🎯 Pitch

When trained on identical data and model activations but with different random seeds, sparse autoencoders converge to substantially different feature sets—only 30% of features are shared in Llama 3 8B. This seed-dependence grows with model width and sparsity, revealing that SAE features are not a unique, ground-truth decomposition of a network but a pragmatic, initialization-sensitive view into activation space.

1. Executive Summary

This paper analyzes whether sparse autoencoders (SAEs) converge to a unique, objective set of features when trained on the same model and data, by training multiple SAEs differing only in random initialization seed and measuring feature overlap via optimal matching with the Hungarian algorithm. Studying SAEs trained on feedforward network activations from Llama 3 8B, Pythia 160M, GPT-2, and SmolLM across two datasets and multiple architectures, the authors find that only ~30% of features are shared across seeds in the largest setting (a 131K-latent TopK SAE on Llama 3 8B), with feature overlap decreasing as SAE width and the number of active latents increase. The paper introduces a formal shared latent definition — a latent matched to the same counterpart in both encoder and decoder matchings with cosine similarity ≥ 0.7 — and observes that while frequently-firing latents tend to be shared, a significant fraction of high-scoring interpretable features appear as "orphan" latents found in only a single seed, establishing that the feature set discovered by an SAE is a pragmatically useful but non-universal decomposition of activation space rather than an exhaustive enumeration of features the model "truly uses."

2. Context and Motivation

The Core Problem: Do SAEs Uncover the Model's "True" Features?

The fundamental question this paper tackles is whether sparse autoencoders converge to a unique, objective decomposition of a neural network's activation space into interpretable features. This is not merely an academic curiosity — it has direct implications for how the interpretability community should think about what SAEs are actually doing and what claims can be made based on SAE-discovered features.

The paper frames this around a specific presupposition that has emerged in the mechanistic interpretability literature: the hope that SAEs can "identify and enumerate over all features in a model" (Elhage et al., 2022), which might enable verifying safety properties such as "a model will never lie" (Olah, 2023). This hope — which the paper attributes to what Smith (2024) calls the "Strong Feature Hypothesis" — implicitly assumes that there exists some ground-truth set of features inside the model, and that SAEs are a reliable instrument for recovering them.

The authors test this assumption directly by asking: if you train two SAEs identically except for the random seed used to initialize their weights, do they find the same features? If SAEs were converging to a unique, objective decomposition, we would expect high overlap regardless of initialization. The paper's central finding — that overlap can be as low as ~30% in large-scale settings — fundamentally challenges this presupposition.

Why This Matters: Theoretical and Practical Stakes

The paper's motivation sits at the intersection of several important concerns in the interpretability landscape:

The safety verification agenda. A significant thread of interpretability research is motivated by the goal of formally verifying safety properties of AI systems. If we could enumerate all features in a model, we might, for example, check whether any feature corresponds to "deception" or "scheming," or verify that the model's reasoning process never passes through dangerous intermediate states. This verification agenda depends on the features being objective and exhaustive — if different SAE training runs produce different feature sets, then a safety audit based on one SAE might miss features that another run would have found. The fact that some interpretable features appear as "orphans" in only a single seed (Section 5, Table 1) means that any single SAE is likely "missing out" on a number of meaningful features, which is a direct challenge to the exhaustiveness assumption.

The nature of neural network representations. Beyond safety applications, the question of whether SAEs converge to unique features bears on a deeper theoretical issue: do neural networks have a preferred basis for their internal representations, or are representations fundamentally ambiguous up to some equivalence class? The toy models of superposition literature (Elhage et al., 2022) demonstrated that models can represent more features than they have dimensions by encoding them in overlapping, quasi-orthogonal directions. But whether these directions are uniquely determined by the training process — or whether multiple equally valid decompositions exist — has been an open question. This paper provides evidence for the latter: the nonconvexity of the SAE loss function gives rise to many local optima (Section 7), and different random initializations find qualitatively different feature sets.

Methodological implications for interpretability research. If SAEs are not converging to a unique feature set, then interpreting a single trained SAE provides only a partial view of the model's internal structure. This has concrete methodological implications: papers that claim to have "found the feature for X" based on a single SAE run may be overclaiming, since another SAE trained on the same data might not contain that feature, or might represent it differently. The paper explicitly advocates for viewing feature discovery as "a compositional problem, wherein we look for useful ways of cutting up the activation space into categories, and these categories can themselves be cut up into further categories, hierarchically" (Section 7), shifting the framing from discovery to construction.

Prior Approaches and Where They Fall Short

The "SAEs converge" narrative. Prior work had produced evidence suggesting that SAEs are relatively stable across random seeds. Anonymous (2024) and Braun et al. (2024) found that ReLU SAEs trained with an L1 sparsity penalty showed high feature overlap across seeds, with Braun et al. reporting that >90% of features were shared between GPT-2 seeds. These findings contributed to an emerging narrative that SAEs were indeed converging to something like the "true" features of the model.

The present paper identifies several reasons why these prior results may be misleading:

Scale mismatch. The prior work used SAEs with relatively small numbers of latents. The paper demonstrates that feature overlap decreases as the number of SAE latents increases (Figure 4, left), meaning that the apparent stability in small SAEs does not generalize to the large-scale regime where SAEs are most practically useful (e.g., 131K latents on Llama 3 8B).
Architecture dependence. The high-overlap results were obtained with ReLU SAEs trained with L1 sparsity penalties. The paper shows that the state-of-the-art TopK activation function — which has become the standard in large-scale SAE training (Gao et al., 2024; Templeton et al., 2024) — exhibits significantly more seed dependence than ReLU SAEs, even when controlling for the level of sparsity (Section 6, Figure 5, right). Since the field has largely migrated to TopK SAEs, the prior stability results on ReLU SAEs may not apply to the architectures currently in use.
Methodological differences. Prior work often used maximum cosine similarity as the metric for comparing features across SAEs. The paper notes that this metric does not enforce a bijective (one-to-one) matching — many features in SAE A could be mapped to the same feature in SAE B, potentially inflating overlap estimates by ignoring cases where features are "ganging up" on a single counterpart. The Hungarian algorithm approach enforces a one-to-one matching, providing a more stringent comparison (Section 3).

Concurrent work pointing in different directions. Marks et al. (2024) found that TopK SAEs could be improved by training two different seeds and forcing them to be "aligned," which implicitly suggests that they are not sufficiently aligned by default — consistent with this paper's findings. Karvonen et al. (2024), in a benchmark of feature splitting, found that JumpReLU and TopK latents had higher feature splitting rates than ReLU SAEs, a convergent result that suggests different activation functions lead to qualitatively different feature decompositions. The present paper synthesizes and extends these observations into a systematic study of seed dependence across architectures, scales, and models.

The GemmaScope collection and a revealing methodological detail. Balagansky et al. (2024) used the Hungarian algorithm to align features from SAEs trained on adjacent layers of Gemma 2 models, reporting positive results. However, the paper reveals (Section 2) that "the same random seed was used to initialize every SAE in the Gemmascope collection" — meaning that what appeared to be convergence across independently trained SAEs was actually a consequence of identical initialization. The paper notes, through personal communication with the DeepMind interpretability team, that Balagansky et al.'s results are "likely dependent on this hyperparameter choice." This is a striking example of how the field's assumptions about SAE convergence have been shaped by an unnoticed experimental artifact.

Theoretical motivations for non-uniqueness. The paper situates its findings within a broader set of observations that challenge the idea of SAEs finding atomic, universal features:

Feature splitting and absorption. Chanin et al. (2024) observed that a feature in a small SAE may be replaced by multiple, more specific features in a larger SAE, and that general features like "starts with the letter L" can appear alongside specific features like "the token 'lion'," preventing the general feature from activating in contexts where the specific feature applies. This suggests that the "flat" design of standard SAEs may not accommodate the hierarchical structure of human concepts (Ayonrinde et al., 2024).
Meta-SAEs and non-atomicity. Anonymous (2024) showed that a "meta SAE" can decompose individual SAE features into more specific features, suggesting that what appears as a single feature in one SAE is actually a mixture that another decomposition can further separate.
Nonlinear features. Engels et al. (2024) provided evidence that irreducibly nonlinear features exist in language models. If true, this implies that SAEs — which presuppose linear representations — might converge to different ways of "linearizing" the nonlinear features in activation space depending on random initialization. The paper explicitly raises this as a possible mechanism for the seed dependence they observe (Section 2).

How This Paper Positions Itself

The paper positions itself not as proposing a new SAE architecture or training method, but rather as interrogating a foundational assumption that has been implicit in much of the SAE literature: that SAEs converge to something objective. The authors are careful not to claim that SAEs are useless or that interpretability is impossible — they explicitly describe feature sets as "a pragmatically useful decomposition of activation space" (Abstract). Instead, they are advocating for a shift in framing: from viewing SAEs as discovering pre-existing features to viewing them as constructing useful decompositions that are one of many possible ways to carve up the activation space.

This framing has direct implications for several active research directions:

Safety verification. If features are construction rather than discovery, then safety audits based on SAEs need to account for the possibility that dangerous features might be represented differently — or not at all — in any given SAE run. Ensemble approaches (training multiple SAEs and combining their feature sets) or hierarchical approaches might be necessary.
Feature universality across models. If features aren't even universal across different seeds of the same SAE on the same model, the prospects for finding universal features across different models are even more challenging. This connects to ongoing debates about whether different language models learn similar internal representations.
Benchmarking and evaluation. If different SAE runs produce different feature sets, then evaluating SAEs based on the interpretability or usefulness of a single run's features may be misleading. The paper's finding that orphan features can be highly interpretable (Table 1) suggests that any single run is missing interpretable features, which has implications for how SAE quality should be evaluated.

The paper also explicitly connects to the broader deep learning phenomenon of nonconvex loss landscapes with many local optima. The lack of feature overlap is attributed mathematically to the nonconvexity of the SAE loss function (Section 7). But the authors note an important subtlety: "One might have expected a priori, however, that different local optima would have more feature overlap than we found in this study" — suggesting that the degree of seed dependence is surprisingly high even given the known nonconvexity, and that the SAE loss landscape has more distinct basins than one might have anticipated from prior work on network merging (Ainsworth et al., 2023) or linear mode connectivity.

In summary, the paper addresses a gap that is at once empirical (we didn't know how much seed dependence existed in modern SAE training), methodological (we didn't have rigorous tools for comparing feature sets across SAEs), and philosophical (we didn't have clarity on whether SAE features should be interpreted as discovered or constructed). By providing systematic evidence for substantial seed dependence — and showing that it grows with scale rather than diminishing — the paper challenges the field to reconsider what claims can be made based on SAE-discovered features and motivates new approaches that embrace the non-uniqueness of feature decompositions.

3. Technical Approach

3.1 Reader Orientation

This paper constructs a measurement framework — not a new model or training method — for quantifying how much feature overlap exists between independently trained sparse autoencoders. It solves the problem that SAE latents have no inherent ordering across training runs: two SAEs that have learned identical features would look completely different if you compared their weight matrices directly because the features would be arranged in different, arbitrary orders. The solution's shape is a two-stage pipeline: first, compute an optimal one-to-one matching between the features of two SAEs using the Hungarian algorithm on cosine similarity, then classify each feature as "shared" or "orphan" based on whether it has a close counterpart in the other SAE that both the encoder and decoder directions agree on.

3.2 Big-Picture Architecture (Diagram in Words)

The measurement system has four major components:

SAE Training Pipeline — trains multiple sparse autoencoders on identical data in identical order, differing only in the random seed used for weight initialization. This produces a set of independently trained SAEs that should be identical if the optimization were converging to a unique solution.
Feature Extraction — extracts encoder vectors (rows of the encoder weight matrix, mapping activation space to latent space) and decoder vectors (columns of the decoder weight matrix, mapping latent space back to activation space) from each trained SAE. Each such vector pair constitutes one "feature" or "latent."
Hungarian Matching Engine — given the encoder vectors from SAE A and SAE B, computes a bijective (one-to-one) assignment that maximizes the total cosine similarity between matched pairs. This is done separately for encoder vectors and decoder vectors, producing two possibly different matchings. The algorithm ensures that no two latents from SAE A are assigned to the same latent in SAE B, and vice versa.
Shared/Orphan Classifier — for each latent in SAE A, checks two conditions: (a) whether the encoder matching and decoder matching assign it to the same counterpart latent in SAE B, and (b) whether both matched cosine similarities exceed a threshold of 0.7. If both conditions hold, the latent is classified as "shared"; otherwise it is "orphan." The fraction of shared latents across all latents in the SAE is the primary quantitative metric.

Information flows as follows: identical activation data → multiple SAE training runs (different seeds) → extraction of encoder and decoder vectors from each trained SAE → pairwise Hungarian matching between all SAE pairs → shared/orphan classification per latent per pair → aggregation across seeds to measure asymptotic shared fraction.

3.3 Roadmap for the Deep Dive

First, the SAE training setup — exactly what is being trained, on what data, with what hyperparameters, and what variants are explored — because the seed-dependence phenomenon is measured on these trained SAEs and all subsequent analysis depends on understanding what's being compared.
Second, the formal definition of a "shared latent" and the two-part matching criterion (encoder agreement + decoder agreement + threshold) — because this is the paper's core operational definition and every quantitative result traces back to it.
Third, the Hungarian algorithm and why a bijective matching is necessary — because without enforcing one-to-one assignment, similarity metrics can inflate apparent overlap by letting many features crowd onto a single counterpart.
Fourth, the distinction between encoder matching and decoder matching, why they can disagree, and why agreement between them is used as the sharing criterion — because this is the paper's key methodological insight for distinguishing genuine feature similarity from spurious matches.
Fifth, the asymptotic trend analysis — how the paper moves from pairwise SAE comparisons to quantifying how many features are unique to a single seed when multiple SAEs are available, and why this reveals persistent orphan features that don't disappear with more training runs.
Sixth, the ablation dimensions (SAE width, number of active latents, training duration, architecture choice, model layer) — because these establish the generality of the phenomenon and identify which factors amplify or suppress seed dependence.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a measurement and analysis paper whose core idea is that the degree of feature overlap between independently trained SAEs can be rigorously quantified using optimal matching, and that this measurement reveals substantial non-overlap that grows with SAE scale and depends strongly on architectural choices (TopK vs. ReLU activation).

SAE Training Setup: What Is Being Trained and Compared

The paper trains multiple sparse autoencoders on the same underlying data, varying only the random seed used for weight initialization, and then compares the learned features across seeds. An SAE is a two-layer network consisting of an encoder that maps activation vectors from the target model into a higher-dimensional sparse latent space, and a decoder that reconstructs the original activation vector from the sparse latent representation. Both components are trained jointly to minimize reconstruction error while enforcing sparsity in the latent activations.

Target models and layers. The paper trains SAEs on feedforward network (MLP) outputs from several language models to establish generality. Specifically, the primary experiments use the sixth layer MLP output of Pythia 160M (Biderman et al., 2023), a 160-million-parameter autoregressive transformer. Additional experiments extend to MLP outputs from GPT-2, SmolLM, and Llama 3 8B (Dubey et al., 2024), as well as residual stream activations for some models (Section 6, Figure 5). The choice of MLP outputs rather than residual stream is a design decision that the paper does not extensively motivate, but it enables comparison with prior work that has studied MLP SAEs and provides a consistent target across models of different scales.

Training data. All SAEs are trained on activations collected from the target model's forward pass over text from the Pile (Gao et al., 2020), the same dataset used to pretrain Pythia 160M. For the primary experiments, SAEs are trained on the first 8 billion tokens of the Pile. This is a deliberate choice: using the model's actual pretraining data ensures that the activation distribution the SAE sees matches the distribution the model was trained on, avoiding distribution-shift confounds that could artificially reduce feature overlap.

Critical experimental control: identical data order. Each SAE sees exactly the same data in the same order — the paper states explicitly that "both SAEs see exactly the same data in the same order" (Section 3). This is essential for the experimental design because it isolates the random seed as the only source of variation between training runs. If data order were also randomized, any observed differences in learned features could be attributed to seeing different data rather than to the nonconvexity of the loss landscape. By fixing the data stream, the paper ensures that any two SAEs with different seeds are solving the same optimization problem (same loss function on the same data sequence) from different starting points.

SAE architectures studied. The paper examines three SAE architectures to test whether seed dependence is architecture-specific:

TopK SAEs: The primary architecture used throughout most experiments. In a TopK SAE, the activation function in the latent space retains only the k largest activations and sets all others to zero. This enforces exact sparsity — exactly k latents are active for any input. The paper sweeps different values of k (corresponding to different numbers of active latents) in the ablation studies (Figure 4, middle; Figure 5).
ReLU SAEs with L1 sparsity penalty: The standard architecture from Bricken et al. (2023), where sparsity is encouraged by adding an L1 penalty on the latent activations to the reconstruction loss. Sparsity is soft — the number of active latents varies per input. The paper compares these against TopK SAEs and finds that ReLU SAEs show "greater stability across seeds" (Section 1, abstract) and "have a larger overlap between latents" (Section 6).
Gated SAEs: The architecture from Rajamanoharan et al. (2024), which introduces a gating mechanism that separates the selection of which latents to activate from the estimation of their magnitudes. Like ReLU SAEs, these are trained with an L1 sparsity loss. The paper includes them in the architecture comparison (Section 6, Figure 5, right) and finds they behave similarly to ReLU SAEs in showing higher overlap than TopK SAEs.

Training library. All SAEs are trained using the sae library (Belrose, 2024), an open-source implementation from EleutherAI. The paper does not enumerate full training hyperparameters (learning rate, batch size, optimizer settings) in the main text, but the library uses standard practices from the SAE literature: the decoder vectors are constrained to have unit norm (eliminating the scaling symmetry described below), the encoder and decoder are trained jointly with reconstruction loss (typically MSE) plus the architecture-specific sparsity term, and the TopK activation is applied as a hard constraint during the forward pass.

The scaling symmetry and why it's eliminated. The paper notes an important technical detail about the SAE parameterization (Section 3, footnote): for ReLU networks, there exists a continuous symmetry where "the function represented by the model is unchanged when the pre-activation is scaled by $s$ and the post-activation is scaled by $s^{-1}$ ." In the context of an SAE, this would mean that multiplying an encoder row by a scalar $s$ and dividing the corresponding decoder column by $s$ leaves the reconstructed output unchanged. If this symmetry were present, cosine similarity between encoder or decoder vectors would be an unreliable measure of feature similarity because the directions could be arbitrarily scaled. The standard SAE training recipe eliminates this symmetry by constraining decoder vectors to be unit norm (Bricken et al., 2023). This means the decoder vectors are fixed to lie on the unit sphere, and any scaling in the encoder must be compensated only through the latent activation magnitude, not the decoder direction. This normalization is what makes cosine similarity a meaningful metric: it measures directional agreement independent of magnitude, and since the decoder vectors are already normalized, comparing their directions is comparing the actual learned feature directions.

The Formal Definition of a Shared Latent

The paper introduces a specific operational definition for when a latent in one SAE is considered "shared" with a latent in another SAE. This is the central measurement construct, and every quantitative result — the 30% figure for Llama 3 8B, the asymptotic analysis, the ablation curves — depends on how "shared" is defined.

The definition has two components that must both be satisfied:

Component 1: Matching agreement. Let SAE A and SAE B be two independently trained SAEs. The Hungarian algorithm is run twice for the pair:

Once on the encoder vectors: for each latent $i$ in SAE A, find a unique latent $j$ in SAE B such that the total cosine similarity $\sum_i \text{cosine}(e_i^A, e_{j}^B)$ is maximized across all bijective assignments.
Once on the decoder vectors: for each latent $i$ in SAE A, find a unique latent $j'$ in SAE B such that the total cosine similarity $\sum_i \text{cosine}(d_i^A, d_{j'}^B)$ is maximized across all bijective assignments.

A latent $i$ in SAE A is said to have matching agreement if the encoder matching and decoder matching assign it to the same counterpart in SAE B — that is, if $j = j'$ . When the encoder and decoder matchings disagree (assign latent $i$ to different counterparts), this is treated as evidence that the similarity is spurious or partial: one direction matches well but the other doesn't, suggesting the features are not truly the same.

Component 2: Similarity threshold. The matched counterpart must have cosine similarity ≥ 0.7 in both the encoder and decoder matchings. This threshold was chosen based on the empirical observation that the distribution of matched cosine similarities is bimodal (Figure 1): there is a high-similarity mode (features that have genuine close counterparts) and a low-similarity mode (features matched to unrelated latents because the Hungarian algorithm must assign every latent to something). The 0.7 threshold sits in the valley between these modes, separating the two populations.

Formal statement. Let $\text{enc\_match}(i)$ be the index in SAE B that latent $i$ of SAE A is matched to by the encoder matching, and $\text{dec\_match}(i)$ be the index from the decoder matching. Let $\text{enc\_sim}(i) = \text{cosine}(e_i^A, e_{\text{enc\_match}(i)}^B)$ and $\text{dec\_sim}(i) = \text{cosine}(d_i^A, d_{\text{dec\_match}(i)}^B)$ . Then:

A latent $i$ is shared if and only if $\text{enc\_match}(i) = \text{dec\_match}(i)$ AND $\text{enc\_sim}(i) \geq 0.7$ AND $\text{dec\_sim}(i) \geq 0.7$ .

Why this definition? The paper motivates each component:

Requiring encoder and decoder agreement (component 1): The encoder and decoder represent the same feature from two perspectives — the encoder weights encode which directions in activation space cause this latent to fire, while the decoder weights encode which direction in activation space this latent adds back during reconstruction. If two SAEs have genuinely learned the same feature, the latent should match to the same counterpart whether you look at the encoding direction or the decoding direction. When the matchings disagree, it suggests that the encoder direction in SAE A is similar to the encoder direction of latent X in SAE B, but the decoder direction in SAE A is similar to the decoder direction of a different latent Y in SAE B — this is inconsistent with the two latents being the same feature, and is more likely a coincidental partial similarity.
Requiring a threshold (component 2): The Hungarian algorithm always produces a matching — every latent in SAE A gets assigned to exactly one latent in SAE B. For latents that have no genuine counterpart, the algorithm will pair them with the "least dissimilar" available latent in SAE B, but the cosine similarity will be low. The threshold of 0.7 identifies the point where the similarity is high enough to be in the genuine-match mode rather than the forced-match mode. The threshold choice is validated empirically by the bimodal distribution in Figure 1 and by the robustness check in Figure A1 (right panel), which shows that the fraction classified as shared is stable across a range of threshold values.

Robustness to matching method. The paper compares the Hungarian matching criterion with a simpler alternative — maximum cosine similarity, where each latent in SAE A is simply paired with its single nearest neighbor in SAE B by cosine similarity, without requiring a bijective matching — and finds that empirically they produce very similar results. Figure A2 shows that "the vast majority [of latents] have the same value for both metrics, suggesting that the Hungarian algorithm has chosen to match most latents with their nearest neighbors." Figure A1 (right panel) shows that the fraction of latents classified as shared is essentially unchanged whether using matched cosine or maximum cosine similarity with the 0.7 threshold. This is an important robustness check: it means that the results are not an artifact of the Hungarian algorithm's bijectivity constraint. The paper notes that the Hungarian approach is "more principled" because it guarantees a one-to-one mapping and prevents the inflation of overlap estimates that could occur if many features crowd onto the same counterpart in the maximum cosine approach, but the empirical convergence of the two methods suggests that this theoretical concern does not materially affect the measurements in practice.

The Hungarian Algorithm and Why Bijective Matching Is Necessary

The Hungarian algorithm (also known as the Kuhn-Munkres algorithm) solves the assignment problem: given an $n \times n$ cost matrix where entry $(i, j)$ is the cost of assigning item $i$ from set A to item $j$ from set B, find a bijective assignment (each item in A assigned to exactly one item in B, and vice versa) that minimizes the total cost.

In the paper's usage, the "items" are the latents of two SAEs, and the "cost" is the negative cosine similarity (turning the maximization problem into a minimization problem). For an SAE with $L$ latents, the cost matrix is $L \times L$ . The algorithm runs in $O(L^3)$ time, which is computationally tractable even for the largest SAEs studied (131K latents on Llama 3 8B).

Why a bijective matching matters. The fundamental problem with simpler metrics like "for each feature in SAE A, find its maximum cosine similarity in SAE B and average" is that they allow many-to-one mappings. If SAE B has one feature that is a good match for ten different features in SAE A, the maximum cosine metric would report high similarity for all ten, even though they're all converging on the same counterpart. A bijective matching forces an honest accounting: each feature in SAE B can be paired with at most one feature in SAE A, so if ten features in A are all most similar to the same feature in B, only one gets that match and the other nine must find different (less similar) counterparts. This makes the bijective matching a more conservative and accurate measure of how many features genuinely have unique counterparts in the other SAE.

The paper explicitly addresses this in Section 3, noting that the maximum cosine approach "has the downside that it does not yield a bijective matching: many features in the first SAE may be mapped to one feature in the second SAE." The Hungarian algorithm avoids this issue by construction.

The cost function. For encoder matching, the cost matrix entry $C_{ij}^{\text{enc}}$ for matching latent $i$ in SAE A to latent $j$ in SAE B is:

$C_{ij}^{\text{enc}} = 1 - \text{cosine}(e_i^A, e_j^B)$

where $\text{cosine}(e_i^A, e_j^B) = \frac{e_i^A \cdot e_j^B}{\|e_i^A\| \|e_j^B\|}$ is the cosine similarity between the two encoder vectors.

What it computes: the cosine similarity measures the angular distance between the two vectors on the unit sphere, taking values in $[-1, 1]$ (with 1 indicating identical direction, 0 indicating orthogonal directions, and -1 indicating opposite directions). Subtracting from 1 converts this to a non-negative cost where 0 means perfect alignment and larger values mean greater dissimilarity. The Hungarian algorithm then finds the permutation $\pi$ of $\{1, \ldots, L\}$ that minimizes $\sum_{i=1}^L C_{i, \pi(i)}^{\text{enc}}$ , which is equivalent to maximizing the sum of cosine similarities.

Why this form: Cosine similarity is invariant to the magnitude of the vectors, which is essential because the decoder vectors are constrained to unit norm while the encoder vectors are not. Cosine similarity measures only whether the two vectors point in the same direction, which captures the geometric meaning of a "feature direction" in activation space — if two SAEs have learned the same feature, their decoder vectors should point in roughly the same direction regardless of the encoder's scaling. An alternative like Euclidean distance would be dominated by magnitude differences and would not isolate directional agreement.

The decoder matching uses an analogous cost matrix $C_{ij}^{\text{dec}} = 1 - \text{cosine}(d_i^A, d_j^B)$ . The two matchings are computed independently — the encoder matching does not constrain the decoder matching and vice versa. It is precisely the comparison of these two independent matchings that provides the signal for the "matching agreement" component of the shared latent definition.

Computational scalability. For an SAE with 131K latents (the largest studied, on Llama 3 8B), the cost matrix has $131000^2 \approx 1.7 \times 10^{10}$ entries, requiring about 17 billion cosine similarity computations for each matching. The $O(L^3)$ Hungarian algorithm on a matrix of this size is computationally intensive but feasible with optimized implementations. The paper does not report computational cost explicitly, but the fact that the analysis is run for " $\binom{9}{2} = 36$ " pairwise matchings for the asymptotic analysis (Section 4) indicates that the computation is practical at scale — it is a one-time cost per SAE pair, not an inner-loop training cost.

Encoder vs. Decoder Matching and Why Agreement Matters

A sparse autoencoder has a natural symmetry: the encoder and decoder represent the same feature from complementary perspectives. For a latent $i$ , the encoder vector $e_i$ (a row of the encoder weight matrix) determines what pattern in the input activation space causes this latent to activate, while the decoder vector $d_i$ (a column of the decoder weight matrix) determines what pattern gets added back to the reconstruction when this latent fires. In a well-trained SAE, $e_i$ and $d_i$ for the same latent $i$ tend to be roughly aligned — the feature that causes firing is similar to the feature that gets reconstructed — but they are not identical because the encoder must also contend with interference from other latents and the sparsity constraint.

When comparing two SAEs, it is possible that a latent $i$ in SAE A has an encoder vector similar to latent $j$ in SAE B, but a decoder vector more similar to latent $k \neq j$ in SAE B. This would mean that SAE A's encoding of when to fire matches SAE B's encoding for one feature, but SAE A's reconstruction direction matches SAE B's reconstruction for a different feature. The paper interprets this situation as evidence that the latents are not truly the same feature — a genuine shared feature should match to the same counterpart from both perspectives.

Empirical observation of encoder-decoder agreement. Figure 1 visualizes this phenomenon. For each matched pair of latents between two SAEs, the plot shows the encoder cosine similarity on one axis and the decoder cosine similarity on the other. The points form two distinct clusters:

Blue points (matching agreement): latents where the encoder and decoder matchings point to the same counterpart. These cluster in the high-similarity region (top right), with both encoder and decoder cosine similarities typically above 0.7.
Orange points (matching disagreement): latents where the encoder and decoder matchings point to different counterparts. These cluster in the low-similarity region (bottom left), with both cosine similarities typically below 0.5.

The paper reports that "the average alignment of points with equal decoder and encoder indices is 0.72 and of the ones that have different indices is 0.33" (Figure A1 caption). This 2:1 ratio in average similarity provides quantitative support for the bimodal interpretation: latents whose encoder and decoder matchings agree are substantially more similar to their counterparts than latents whose matchings disagree.

Why this distinction matters. If the paper only used encoder similarity (or only decoder similarity) to define shared features, it would either miss cases where one direction matches but the other doesn't (potentially inflating overlap estimates) or would need a higher threshold to compensate (potentially deflating them). By requiring agreement between both perspectives, the definition captures the intuition that a shared feature should be consistently matched across both the "when to fire" (encoder) and "what to reconstruct" (decoder) aspects of the feature. This is a more stringent criterion than either single-perspective approach alone.

The correlation between encoder and decoder similarities. Figure 1 also shows that even for latents with matching agreement (blue points), encoder and decoder similarities are strongly but imperfectly correlated — there is scatter around the diagonal. This means that even for genuinely shared features, the encoder direction in SAE A can be more or less similar to the encoder direction in SAE B than the decoder directions are to each other. This is expected because the encoder and decoder serve different functions and are subject to different optimization pressures during training.

The Asymptotic Analysis: How Many Features Are Unique to a Single Seed?

The pairwise shared latent definition (§3) tells us how many features two SAE seeds have in common. But a natural follow-up question is: if we train many SAEs with different seeds, do the orphan features from one seed eventually appear in some other seed, or are there features that are persistently unique to a single run? The asymptotic analysis in Section 4 addresses this by measuring how the number of "seed-unique" latents changes as more seeds are added.

Experiment design. The paper trains nine independently seeded SAEs on the same data (Section 4: "seven more SAEs with the same data order, but with seeds different from the first two, yielding a total of nine independently trained SAEs"). For each possible number of seeds $k$ (from $k=2$ to $k=9$ ), the paper considers all $\binom{9}{k}$ combinations of $k$ SAEs drawn from the nine. For each combination:

Select one SAE from the combination as the "base SAE."
Run the Hungarian matching between the base SAE and each of the other $k-1$ SAEs in the combination.
For each latent in the base SAE, compute a binary mask for each of the $k-1$ pairings: is this latent shared (by the §3 definition) with its counterpart in the other SAE?
A latent is classified as "only in the base SAE" if it is an orphan according to all $k-1$ binary masks — that is, it fails to be shared with every other SAE in the combination.
Compute the proportion of latents in the base SAE that are "only in the base."

This experiment is run $k \times \binom{9}{k}$ times for each $k$ — once for each combination of $k$ SAEs and each possible choice of base SAE within the combination. The results are averaged to produce the curve in Figure 2.

Interpretation of "only in the base SAE." A latent that is classified as "only in the base SAE" with $k=9$ is one that failed to find a shared counterpart in any of the other eight independently trained SAEs. This does not necessarily mean the latent is meaningless or uninterpretable — Section 5 shows that many orphan features are highly interpretable. Rather, it means that this particular feature decomposition is not consistently rediscovered across random initializations. The other eight SAEs either represent this activation pattern differently (e.g., split across multiple latents, or combined with other patterns into a polysemantic latent) or don't represent it at all in a way that meets the similarity threshold.

Key result and curve fitting. Figure 2 (left panel) shows that when $k=9$ , about 35% of latents are classified as "only in the base SAE" — meaning these latents have no shared counterpart in any of the eight other independently trained SAEs. The right panel shows that the count decreases slowly as $k$ increases, but appears to approach an asymptote greater than zero.

The paper explicitly tests whether this curve goes to zero or to a positive asymptote by fitting two functional forms: a pure power law and a power law with an offset term. The finding: "a power law with an offset term fits the data significantly better than one without the offset" (Section 4). This is a crucial result — the offset term being statistically preferred implies that some features are persistently unique to individual seeds even as we add more and more SAE training runs. If the pure power law (without offset) had fit better, it would suggest that orphan features eventually all find counterparts if we train enough seeds, just at a slow (power-law) rate. The offset term fit implies the opposite: a nonzero fraction of features will remain unique to a single seed no matter how many SAEs we train.

Why this matters. If orphan features were simply random noise that would disappear with more training runs, the asymptotic analysis would have found a pure power law approaching zero. The offset result instead suggests that the SAE loss landscape has many distinct local minima, and different seeds fall into different basins that produce genuinely different feature decompositions — not just the same features in slightly different forms, but qualitatively different ways of partitioning the activation space. This is the strongest evidence in the paper against the "SAEs converge to unique features" hypothesis.

Relationship to firing frequency. Figure 3 shows how the number of seeds that share a latent relates to how frequently that latent fires. The finding: "the latents that most frequently fire in the first SAE are the ones that are shared with all eight SAEs, and that the ones that most infrequently fire are the ones that are not shared with any other SAE." However, the paper notes an important nuance: "a significant number of orphan latents have a higher firing rate on average than latents that shared with all SAEs." This means that firing frequency is correlated with shareability but is not a sufficient explanation — there are frequently-firing features that are nevertheless unique to a single seed, which further supports the idea that seed dependence reflects genuine differences in how the activation space is decomposed, not just the presence or absence of rare features.

Ablation Dimensions: What Affects Feature Overlap?

The paper systematically varies several dimensions of the SAE training setup to understand what factors increase or decrease the seed dependence of learned features.

Number of SAE Latents (SAE Width)

Figure 4 (left) and Figure 5 (left) show that the fraction of shared latents decreases as the total number of latents in the SAE increases. For a Pythia 160M MLP SAE trained on 8B tokens, the shared fraction drops from approximately 65–70% for small SAEs (a few thousand latents) to approximately 42% for 32K latents (Figure 4, left), and in Figure 5 (left, the points labeled "Pythia 160M") the trend continues downward with increasing latent count. For Llama 3 8B SAEs with 131K latents (the largest studied), the overlap is only about 30% (reported in the abstract and Section 6).

Why width might reduce overlap. The paper does not provide a definitive mechanistic explanation, but several factors likely contribute. First, with more latents, the SAE has more capacity to represent fine-grained, specialized features that may be more sensitive to initialization. Second, the optimization problem becomes higher-dimensional and the loss landscape likely has more local minima — with more parameters, there are more ways to partition the activation space. Third, feature splitting effects (Karvonen et al., 2024) mean that the same underlying pattern might be represented as one feature in one SAE and split across multiple features in another, reducing pairwise overlap even when the same information is captured.

Figure A3 (left) shows the same trend using mean matched cosine similarity (rather than the fraction below a threshold): average cosine similarity of matched latents also decreases with increasing latent count. This confirms that the effect is not an artifact of the threshold choice — the matched pairs are genuinely less similar on average in wider SAEs.

Number of Active Latents (Sparsity Level, `k` for TopK)

For TopK SAEs, the sparsity is controlled by the parameter $k$ — the number of latents that are kept active per input. Figure 4 (middle) shows that increasing $k$ (making the SAE less sparse, with more latents active per input) decreases the fraction of shared latents. The paper sweeps $k$ values and finds a monotonically decreasing relationship.

Why lower sparsity reduces overlap. With more active latents per input, each latent has a "broader" role — it needs to be active in more diverse contexts. This may create more ambiguity in how to partition the activation space, since different initializations could settle on different ways of assigning broadly-active features. Alternatively, lower sparsity may increase the degree of feature superposition (more features represented in overlapping directions), which creates more degrees of freedom in how the decomposition is performed. Figure A3 (middle) confirms that mean matched cosine similarity also decreases with increasing $k$ .

Interpretation: not just feature absorption. The paper explicitly addresses whether seed dependence might be driven by feature absorption (a phenomenon where one feature "absorbs" or co-opts the activation pattern of another). Karvonen et al. (2024) found that absorption increases when sparsity is decreased. If absorption were the primary driver, we would expect more seed dependence when sparsity is lower ( $k$ is higher), which is what Figure 4 (middle) shows. However, the paper also finds that overlap increases with training time (Figure 4, right) — if absorption were the primary mechanism, we might expect training longer to increase absorption and thus decrease overlap, but the opposite occurs. The paper concludes that "seed dependence is not mainly due to feature absorption," or at least that absorption alone cannot explain the pattern of results.

Training Duration (Number of Tokens)

Figure 4 (right) shows that the fraction of shared latents increases with training duration. For a 32K-latent TopK SAE on Pythia 160M, the overlap increases from approximately 35% at 1B training tokens to approximately 42% at 8B tokens. Figure A3 (right) shows the same monotonic increase in mean matched cosine similarity.

Why training longer increases overlap. Longer training allows the optimization to settle more fully into its local minimum and may reduce the variance due to initialization transients. Early in training, different seeds may have wandered into different regions of the loss landscape due to stochastic gradient noise interacting with their different initializations. With more training, the optima they find may become more similar — but the paper's asymptotic analysis shows they never fully converge to the same features, just that the overlap increases modestly with additional training tokens.

Note on interpretation. Even at 8B tokens, the overlap for a 32K-latent SAE is only about 42% — the increase from training is real but does not eliminate seed dependence. The paper has not established whether this trend would continue with substantially more training (e.g., 100B tokens) or whether it plateaus. Given that the 131K-latent Llama 3 8B SAEs were presumably trained on substantial data and still show only ~30% overlap, the evidence suggests that training time alone cannot close the gap.

SAE Architecture (TopK vs. ReLU vs. Gated)

Figure 5 (right panel) compares the fraction of shared latents across architectures for GPT-2 SAEs. The finding is striking: ReLU SAEs trained with L1 sparsity loss and Gated SAEs both show substantially higher overlap than TopK SAEs. The paper states that "standard and Gated SAEs trained with L1 loss have a larger overlap between latents" compared to TopK (Section 6).

Why architectures differ. The paper does not provide a detailed mechanistic explanation, but the architectural difference is fundamental:

In TopK SAEs, exactly $k$ latents are activated per input, selected as the $k$ largest pre-activation values. This is a hard, non-differentiable selection that creates a combinatorial optimization problem — which $k$ latents to activate for each input — interacting with which directions those latents encode. The discrete selection may create a more rugged loss landscape with many distinct local minima corresponding to different assignments of features to latents.
In ReLU SAEs with L1 penalty, all latents with positive pre-activation fire, but the L1 penalty pushes most activations toward zero. The sparsity is soft — the number of active latents varies per input, and the optimization is continuous (no hard selection). This smoother loss landscape may have fewer distinct basins, leading different initializations to converge to more similar solutions.
Gated SAEs separate the "should this latent fire?" decision (gating) from the "how strongly should it fire?" decision (magnitude estimation), but they use an L1 penalty on the gating pathway, making the optimization more similar to ReLU SAEs than to TopK SAEs in terms of smoothness.

The paper notes that TopK SAEs are "state-of-the-art" (used in Gao et al., 2024; Templeton et al., 2024) and that their greater seed dependence is particularly concerning because "the field has largely migrated to TopK SAEs" (Section 2), meaning the architecture with the most seed dependence is also the one most commonly deployed at scale. The earlier stability results on ReLU SAEs (Anonymous, 2024; Braun et al., 2024) may have given a misleading impression of convergence that does not apply to the architectures now in use.

In Figure 5 (right panel), the exact numbers depend on SAE width and other hyperparameters, but the qualitative ranking (ReLU ≈ Gated > TopK) is consistent.

Model Layer and Model Scale

Figure 5 (left) shows how the fraction of shared latents varies across layers for Pythia 160M SAEs with 32K latents. The overlap is "almost constant across the middle layers of the model, being lower for the earlier layers and the last layer." This U-shaped pattern (lower overlap at the extremes, higher in the middle) suggests that features in early and late layers may be more sensitive to initialization or more ambiguous in how they can be decomposed. Early layers process more local, syntactic patterns where multiple valid decompositions may exist; late layers may have more task-specific features that interact in complex ways.

Figure 5 (middle) compares SAEs with the same expansion factor (ratio of SAE latents to model dimension) across different models. The finding is that "the overlap is much lower for the Llama 8B SAEs" — even when controlling for expansion factor, larger models produce SAEs with less feature overlap. For instance, with an expansion factor of 36, Pythia 160M and GPT-2 SAEs show much higher overlap than Llama 3 8B SAEs. This suggests that the seed-dependence phenomenon scales with model capability or activation space complexity, not just with SAE width alone.

Activation Source (MLP vs. Residual Stream)

Figure 5 (middle) also compares SAEs trained on MLP outputs versus residual stream activations. The paper states that "on SmolLM and GPT2 the MLP latents have more overlap between seeds than the residual stream ones, but the same is not true for Pythia." This is a model-dependent result — the relative overlap of MLP vs. residual stream SAEs varies across model families, suggesting that the seed dependence is influenced by the specific geometry of the activation space being decomposed, not just by the SAE architecture or training procedure.

The paper analyzes which properties of individual features predict whether they will be shared across seeds.

Firing frequency (Figure 3). The most striking pattern relates to how often each latent fires. Figure 3 plots matched cosine similarity against firing frequency for all latents in a base SAE, colored by the number of other seeds (out of 8) in which that latent is classified as shared. The result: frequently-firing latents are overwhelmingly shared across many seeds, while rarely-firing latents are predominantly orphan. The paper states: "the latents that most frequently fire in the first SAE are the ones that are shared with all eight SAEs, and that the ones that most infrequently fire are the ones that are not shared with any other SAE."

The counterintuitive finding. However, the paper notes that "as the average alignment between latents increases, the firing frequency seems to decrease" — meaning that the most strongly similar shared latents (highest cosine similarity to their counterparts) are not the most frequently firing ones, but rather are latents with intermediate firing rates. The very highest-frequency latents (right side of the histogram in Figure 3) show good but not perfect alignment across seeds. This is a non-obvious pattern that suggests a nuanced relationship: high-frequency features are reliably present across seeds (they are almost always shared rather than orphan), but their exact learned directions may vary somewhat (they don't achieve the very highest cosine similarities). Meanwhile, some features with intermediate firing rates achieve near-perfect alignment across seeds — these may correspond to particularly "clean" or well-separated features that the optimization reliably finds in essentially the same form.

Interpretability of orphan features (Section 5, Figure 6, Table 1). The paper uses the automated interpretability pipeline from Paulo et al. (2024) to test whether orphan features are less interpretable than shared features. For each latent, the pipeline generates a natural-language explanation of what makes the latent fire and scores that explanation using detection scoring — a metric in $[0, 1]$ that measures how well the explanation predicts whether the latent fires on held-out text.

The results:

Latents shared across many seeds have higher average interpretability scores than orphans, meaning they are on average easier to describe with simple explanations.
However, "a significant fraction of latents found only on one SAE have high scoring latents" — some orphan features achieve scores comparable to the best shared features.
Table 1 provides concrete examples: a latent with alignment score 0.10 (strongly orphan) in Seed 1 has explanation "Abbreviated country name in United States Supreme Court case citations" with a detection score of 0.865, while its matched counterpart in Seed 2 has an explanation about non-English characters with a score of only 0.56. Another orphan latent (alignment 0.44) has explanation "Abbreviated geographical or institutional references, usually in the context of legal citations" with score 0.85, matched to a counterpart that is simply "Punctuation marks" with score 0.49.

This demonstrates that orphan features are not simply "failed" or "degenerate" features — many are highly interpretable and capture meaningful patterns that happen not to appear in the other SAE's decomposition. An SAE training run is "likely 'missing out' on a number of interpretable features" (Section 5) that another seed would have discovered.

Summary of Key Design Choices and Their Justifications

Identical data order across seeds (Section 3): isolates random initialization as the sole source of variation. If data order were randomized, differences could be attributed to seeing different data sequences, making it impossible to determine whether seed dependence reflects loss landscape nonconvexity or data sensitivity.
Hungarian algorithm for bijective matching (Section 3): prevents many-to-one matching that would inflate apparent overlap. While the paper finds that maximum cosine similarity yields similar results empirically (Figure A2), the bijective approach is more principled and ensures each latent in SAE B is used at most once.
Dual encoder and decoder matching with agreement requirement (Section 3): leverages the inherent symmetry of SAEs to increase the stringency of the shared feature criterion. A feature that matches to one counterpart by encoder direction but a different counterpart by decoder direction is treated as not genuinely shared, which captures the intuition that a true feature should be consistent across both perspectives.
Cosine similarity threshold of 0.7 (Section 3): chosen based on the empirically observed bimodal distribution of matched similarities (Figure 1). The robustness check in Figure A1 (right panel) shows that the fraction classified as shared is stable across a range of thresholds, so the exact value is not critical — the bimodality means any threshold in the valley between modes produces similar classification.
Nine independently trained SAEs for asymptotic analysis (Section 4): provides a combinatorial sample of $\binom{9}{k}$ subsets for each $k$ , enabling statistically meaningful estimates of how seed-unique features scale with ensemble size. The power-law-plus-offset fit provides statistical evidence for a nonzero asymptotic orphan fraction.
Decoder vector normalization to unit norm (Section 3, footnote): eliminates the continuous scaling symmetry where encoder scaling can be compensated by decoder inverse-scaling. Without this constraint, cosine similarity would be sensitive to arbitrary scaling factors rather than measuring genuine directional agreement.
Ablation across architectures, widths, sparsity levels, training durations, layers, and models (Section 6): establishes that seed dependence is not idiosyncratic to a particular setup but is a robust phenomenon that varies systematically with controllable parameters. The consistent direction of effects (wider SAEs → less overlap, TopK → less overlap than ReLU, more training → more overlap) provides a coherent picture of when and why seed dependence matters most.

4. Key Insights and Innovations

Innovation 1: Seed Dependence as a First-Class Phenomenon That Inverts the Expected Scaling Relationship

The paper's most fundamental contribution is not the observation that SAE features vary with random seed — prior work had noted some instability — but rather the finding that this variation grows with SAE scale rather than diminishing, and is most pronounced in the architectures the field has converged on as state-of-the-art. This inverts the natural expectation: one might assume that as SAEs become larger and better-optimized, they would converge more reliably to a unique feature decomposition. The paper shows the opposite happens.

What makes this intellectually distinctive is that it transforms seed dependence from an implementation detail into a diagnostic phenomenon that reveals something about the nature of the SAE optimization landscape and about what "features" even are. The dominant narrative in mechanistic interpretability — articulated by Elhage et al. (2022) in their hope to "identify and enumerate over all features in a model" and by Olah (2023)'s vision of verifying that "a model will never lie" — implicitly assumed that there exists a ground-truth set of features and that SAEs are converging toward it. The evidence for this narrative came primarily from small-scale ReLU SAEs, where Braun et al. (2024) found >90% feature overlap.

The paper systematically dismantles the basis for that assumption by showing that the prior evidence came from the regime (small SAEs, ReLU architecture) where overlap is highest. When measured in the regime the field has moved toward — large TopK SAEs on models like Llama 3 8B — overlap drops to ~30% (Abstract, Section 6). The scaling trend (Figure 4, left; Figure 5, left) shows that larger SAEs have monotonically less feature overlap, and the asymptotic analysis (Figure 2) provides statistical evidence that a nonzero fraction of features remain unique to individual seeds no matter how many SAEs are trained.

This is a fundamental rather than incremental finding because it challenges the epistemological status of SAE-discovered features. If the field had stayed with small ReLU SAEs, the convergence narrative might have persisted. The paper shows that the architectures preferred for their superior reconstruction-sparsity tradeoffs (TopK, used in Gao et al., 2024 and Templeton et al., 2024) are precisely the ones where features are least universal. This means there is a genuine tension between SAE performance (reconstruction quality, sparsity) and feature universality — improving one may come at the cost of the other.

Innovation 2: The Dual-Matching Criterion as a Principled Framework for Comparing Learned Representations

Prior work comparing features across independently trained models typically used maximum cosine similarity: for each feature in model A, find the most similar feature in model B and average those similarities. This paper identifies a subtle but important flaw in that approach — it permits many-to-one matching, where multiple features in A all map to the same feature in B — and replaces it with a bijective matching via the Hungarian algorithm combined with a novel consistency check: the matched counterpart must be the same whether you look at encoder directions or decoder directions.

The innovation here is not the Hungarian algorithm itself (which Ainsworth et al., 2023 used for network merging, and Balagansky et al., 2024 applied to SAE layers), but the encoder-decoder agreement criterion as a principled way to distinguish genuine feature similarity from spurious partial matches. The intuition is elegant: an SAE feature has two complementary representations — the encoder vector encodes when the feature fires, and the decoder vector encodes what the feature contributes to the reconstruction. If two SAEs have genuinely learned the same feature, these two perspectives should point to the same counterpart latent. When they don't, it suggests that the encoder of feature A resembles feature X in SAE B but the decoder of feature A resembles feature Y — an inconsistency that signals the features are not truly the same.

This is a conceptual contribution rather than a performance improvement. It provides the field with a more rigorous operational definition of what it means for two features to be "the same" across different training runs, which is essential for any research program that aims to establish feature universality. The empirical validation that the criterion produces a clean bimodal separation (Figure 1: blue cluster at high similarity with matching agreement, orange cluster at low similarity with disagreement) demonstrates that it captures something real about the structure of the SAE loss landscape, not just an arbitrary threshold.

The finding that maximum cosine similarity and the Hungarian matching produce empirically similar results (Figure A2) might seem to diminish the innovation, but the paper correctly argues that the bijective approach is more principled — and the fact that it doesn't change the results in this case is a useful robustness check, not a weakness. In settings with more severe many-to-one mapping, the approaches could diverge, and the Hungarian matching would be the more trustworthy metric.

Innovation 3: The "Orphan Feature" as a New Diagnostic Category with Practical Implications

The paper introduces and operationalizes the concept of an orphan feature: a latent in one SAE that has no close counterpart in another independently trained SAE on the same data. This is not merely a relabeling of "features that differ across seeds" — it is a specific diagnostic category defined by the dual-matching criterion and the bimodal similarity distribution, which allows the paper to ask: what properties distinguish shared from orphan features, and are orphans interpretable or degenerate?

The answer turns out to be nuanced and counterintuitive. Orphan features are not simply failed or degenerate features. Section 5 shows that many orphans achieve high interpretability scores under automated explanation pipelines (Table 1), with concrete examples where a highly interpretable feature in one seed (e.g., "Abbreviated country name in United States Supreme Court case citations," detection score 0.865) is matched to a completely unrelated and less interpretable feature in another seed. The paper quantifies this: orphans have lower average interpretability than shared features, but the distributions overlap substantially, and a "significant fraction" of orphans are as interpretable as the best shared features.

This is a fundamental shift in how to think about what SAEs are doing. The orphan category reveals that any single SAE training run is a sample from a distribution of possible feature decompositions, and that sample can miss interpretable features that other runs would have found. This has direct practical implications: if you're using SAEs for safety auditing and you only train one, you might miss a "deception" feature that another seed would have cleanly isolated. The paper doesn't solve this problem, but by naming and characterizing the orphan phenomenon, it creates a new axis for SAE evaluation — feature coverage across seeds — that was not previously part of the conversation.

The orphan concept also provides a bridge to the feature splitting/absorption literature (Chanin et al., 2024; Karvonen et al., 2024). A feature that appears as an orphan in one SAE might be represented as a split feature (distributed across multiple latents) in another, or might be absorbed into a more general feature. The orphan diagnostic doesn't distinguish these mechanisms, but it provides the measurement framework within which such distinctions could be made in future work.

Innovation 4: Architecture-Dependent Universality as a Challenge to the "Better SAEs Converge" Assumption

The finding that TopK SAEs show substantially lower feature overlap than ReLU or Gated SAEs (Section 6, Figure 5 right) is more than an architectural comparison — it challenges a plausible and widely-held assumption that as SAE training methods improve (better reconstruction, better sparsity, better scalability), the features they discover should become more canonical. TopK SAEs are "state-of-the-art" by the standard metrics (Gao et al., 2024; Templeton et al., 2024), yet they are the least universal in their feature decompositions.

This creates a genuine dilemma. The hard selection mechanism in TopK (exactly k latents activated, chosen as the maximum pre-activations) is what gives it advantages in sparsity control and reconstruction quality. But the paper's results suggest this same mechanism may create a more rugged optimization landscape with many distinct local minima corresponding to different ways of assigning the k-selection pattern. The ReLU + L1 approach, while producing softer (and sometimes less desirable) sparsity patterns, appears to have a smoother loss landscape that different initializations navigate toward more similar endpoints.

The intellectual contribution here is reframing the SAE design problem: feature universality is a property that architectures can have more or less of, and it may trade off against other desirable properties. This is analogous to the bias-variance tradeoff in statistical learning — it's not that one architecture is strictly better, but that different architectures occupy different points on a Pareto frontier. The field had been optimizing primarily for reconstruction fidelity and sparsity; this paper adds universality as a dimension that should be considered, and provides the measurement tools to do so.

The deeper implication, which the paper gestures toward but doesn't fully develop, is that the very notion of what constitutes a "feature" may depend on the architecture used to extract it. A TopK SAE and a ReLU SAE trained on the same activations may both produce useful decompositions, but they may be carving up the activation space along different principles — the TopK favoring sharp, mutually-exclusive boundaries (since exactly k features compete for activation at each input) and the ReLU allowing more graded, overlapping assignments. Neither decomposition is "wrong," but they reflect different answers to the question "what is a feature?" — answers that are partly determined by the architecture rather than purely discovered from the data.

Innovation 5: The Conceptual Shift from Feature Discovery to Feature Construction

The paper's most philosophically significant move is its explicit reframing of what SAEs are doing: not discovering pre-existing features, but constructing useful decompositions of activation space. This is articulated most clearly in the conclusion: "We think feature discovery is best viewed as a compositional problem, wherein we look for useful ways of cutting up the activation space into categories, and these categories can themselves be cut up into further categories, hierarchically."

This reframing is backed by the paper's empirical results but goes beyond them to make a broader methodological argument. If features were being discovered, we would expect convergence — different scientists using the same instrument (SAE) on the same object (the model) should find the same underlying structure. The fact that they don't — that overlap is ~30% in large-scale settings and the asymptotic analysis suggests a persistent fraction of seed-unique features — implies that the SAE is not functioning as a measurement instrument that passively reads out pre-existing structure. It is functioning as a modeling tool that imposes a particular kind of structure (sparse linear decomposition) on activation space, and different random initializations lead to different valid impositions.

This shift has practical consequences for several research programs:

Safety verification: If features are constructed rather than discovered, then auditing a model by enumerating its "features" is category error — you're auditing a particular decomposition, not the model itself. Ensemble approaches (training multiple SAEs and checking whether dangerous patterns appear in any of them) become necessary rather than optional.
Feature interpretability benchmarks: If different SAE runs produce different feature sets, then evaluating SAE quality by the average interpretability of a single run's features (as is standard practice) may be misleading. A run might achieve high interpretability scores by "cherry-picking" easy-to-explain features while missing harder-to-interpret but important ones that another seed would have found.
Cross-model feature comparison: If features aren't even universal across seeds of the same SAE on the same model, the prospects for finding universal features across different models become much more challenging. Research programs that aim to identify "the same feature" in different language models may need to account for the fact that the feature decomposition itself is non-unique even within a single model.

The paper attributes the non-uniqueness mathematically to "the nonconvexity of the SAE loss function, which gives rise to many local optima" (Section 7). But it notes that this alone doesn't predict the degree of non-overlap observed: "One might have expected a priori, however, that different local optima would have more feature overlap than we found in this study." This suggests that the SAE loss landscape has more distinct basins — corresponding to more qualitatively different ways of decomposing activation space — than standard intuitions about local minima in neural networks would predict. The finding connects to broader questions in deep learning theory about why nonconvex optimization finds generalizable solutions, but casts it in a new light: here, the multiplicity of solutions is not a problem to be solved (as in, "how do we ensure convergence to the right minimum?") but a feature of the representational space that should be embraced and studied.

5. Experimental Analysis

Evaluation Methodology

Dataset. The text corpus used for training all SAEs is the Pile (Gao et al., 2020), an 800GB dataset of diverse text that was also used to pretrain Pythia 160M. The SAEs are trained on activations collected from the target language model's forward pass over this text, not on the raw text itself. For the primary experiments, SAEs are trained on the first 8 billion tokens of the Pile (Section 3). When analyzing latent firing frequencies (Figure 3), the paper collects latent occurrences over 10M tokens of the Pile, the same dataset the SAEs were trained on.
Base model(s). The paper studies SAEs trained on activations from multiple language model families to establish generality. The primary experiments use Pythia 160M (Biderman et al., 2023), a 160-million-parameter autoregressive transformer, focusing on the sixth layer MLP output (Section 3). Additional experiments extend to GPT-2, SmolLM, and Llama 3 8B (Dubey et al., 2024), with SAEs trained on both MLP outputs and residual stream activations for some models (Section 6). The choice of MLP outputs for the primary experiments provides a consistent target across model scales and connects to prior work that has studied MLP SAEs. The Llama 3 8B results represent the largest-scale experiments, with 131K-latent SAEs where only ~30% of features are shared across seeds (abstract, Section 6).
Metrics. The paper defines several quantitative metrics for measuring feature overlap between SAEs:
- Fraction of shared latents: the proportion of latents in SAE A that satisfy the shared latent criterion (matched to the same counterpart in both encoder and decoder Hungarian matchings, with cosine similarity ≥ 0.7 for both) when compared to SAE B. This is the primary metric used throughout the paper and defines "overlap" in most figures.
- Mean matched cosine similarity: the average cosine similarity across all matched pairs from the Hungarian matching. This is a continuous measure of feature similarity that avoids thresholding (Figure A3).
- Fraction of latents "only in the base SAE": for the asymptotic analysis, a latent in a base SAE is classified as unique to that SAE if it fails to be shared with any of the other k-1 SAEs in a combination of size k (Section 4). This is plotted against k in Figure 2.
- Detection score (Paulo et al., 2024): a number in [0, 1] measuring how well an automatically generated natural-language explanation predicts whether a latent fires on held-out text. Used in Section 5 to assess whether orphan features are interpretable.
Baselines. The paper does not compare against alternative feature-extraction methods. Its primary comparison is between independently trained SAEs (different random seeds) that are otherwise identical in architecture, data, and training procedure. The baseline is the pairwise shared fraction between two seeds, with the measurement framework then extended to analyze how shared fractions scale with the number of seeds (Section 4). The paper also compares results using the Hungarian matching against results using maximum cosine similarity as an alternative matching method, finding that "empirically the difference between these two approaches is small" (Section 3, Figure A2). Additionally, the paper compares ReLU SAEs with L1 sparsity penalty against TopK and Gated architectures (Section 6, Figure 5).
Generation budget / compute accounting. The paper does not use a generation budget in the sense of LLM inference. The relevant "compute" metrics are:
- SAE training tokens: SAEs are trained on 8B tokens of Pile activations for the primary experiments, with ablation sweeps from 1B to 8B tokens to study the effect of training duration (Figure 4, right).
- Number of SAE latents: the SAE width is swept across a range (e.g., 2^15 = 32,768 for primary Pythia 160M experiments; up to 131K for Llama 3 8B) to study how scale affects seed dependence (Figure 4, left; Figure 5, left).
- Number of active latents (k in TopK SAEs): swept to study how sparsity affects overlap (Figure 4, middle).
- Number of independently trained SAEs: nine seeds are trained for the asymptotic analysis, enabling C(9,2) = 36 pairwise comparisons and combinatorial subset analysis (Section 4).
- Hungarian matching computational cost: O(L^3) for SAEs with L latents, run for all SAE pairs. Not reported in terms of GPU hours or wall-clock time.
Cross-validation / statistical protocol. The paper uses a combinatorial approach for the asymptotic analysis rather than train-test splits. For each k from 2 to 9, all C(9,k) subsets of SAEs are considered. Within each subset, each SAE serves as the base SAE, and the "only in the base" proportion is computed by checking whether each latent is orphaned against all k-1 other SAEs. Results are averaged across all k × C(9,k) trials (Section 4). This provides a statistically robust estimate of how the fraction of seed-unique latents scales with ensemble size without requiring held-out data. The paper fits two functional forms — a pure power law and a power law with an offset term — and compares their fit quality to determine whether the orphan fraction asymptotically approaches zero or approaches a nonzero constant (Section 4, Figure 2). The paper does not report confidence intervals or standard errors for the overlap measurements, and the test set of 500 questions mentioned in the reference example is from a different paper entirely — this paper uses the full 8B-token training set for SAE training and does not employ a train-test split for the SAE comparison itself.

Main Quantitative Results

Pairwise Feature Overlap Between Two Seeds

The paper's foundational measurement establishes the bimodal distribution of feature similarities that motivates the shared/orphan distinction. Two SAEs with 32K latents each, trained on the sixth MLP of Pythia 160M over 8B tokens of the Pile with different random seeds, are compared via Hungarian matching on both encoder and decoder vectors.

Figure 1 visualizes the joint distribution of encoder cosine similarity and decoder cosine similarity for all 32K matched latents. The distribution is bimodal: a high-similarity cluster in the upper-right region (both encoder and decoder similarities above approximately 0.7) and a low-similarity cluster in the lower-left region (both similarities below approximately 0.5). The paper reports:

"the average alignment of points with equal decoder and encoder indices is 0.72 and of the ones that have different indices is 0.33" (Figure A1 caption, Section 3).

The latents where the encoder and decoder matchings agree on the counterpart (colored blue in Figure 1) are concentrated in the high-similarity cluster, while latents where the matchings disagree (colored orange) are concentrated in the low-similarity cluster. This validates the paper's decision to require matching agreement as part of the shared latent criterion: disagreement between the two matching perspectives is strongly associated with low similarity.

Using the formal shared latent definition — same counterpart in both encoder and decoder matchings, both cosine similarities ≥ 0.7 — the paper finds that only 42% of latents are shared across the two independently trained SAEs (Section 3). This is the core quantitative result for this experimental configuration. The paper notes: "If we consider that each latent of an SAE corresponds to a learned feature this result could mean that close to 60% of learned features can be seed dependent."

The robustness of the threshold choice is validated in Figure A1 (right panel): the fraction of latents classified as shared is relatively stable for thresholds between approximately 0.6 and 0.8, and the classification results are "essentially unchanged" whether using matched cosine similarity (from the Hungarian algorithm) or maximum cosine similarity.

Asymptotic Analysis: Scaling With Number of Seeds

Section 4 addresses whether orphan features eventually find counterparts if more SAEs are trained, or whether a persistent fraction remains unique to individual seeds. The experiment uses nine independently trained SAEs (all 32K-latent TopK on Pythia 160M sixth MLP, 8B tokens) and measures how the fraction of latents "only in the base SAE" changes as more SAEs are included in the comparison.

The key result appears in Figure 2 (left panel): when k=9 (all SAEs considered), approximately 35% of latents in the base SAE are "only in the base" — meaning they fail to be shared with any of the other eight SAEs. This represents a substantial fraction of the feature set that is unique to a single random initialization even when eight other independently trained SAEs are available for comparison.

Figure 2 (right panel) shows that the number of "only in the base" latents decreases slowly as k increases, but the curve appears to flatten out. The paper explicitly tests whether this approaches zero:

"We found that a power law with an offset term fits the data significantly better than one without the offset" (Section 4).

The statistical preference for the offset model implies that a nonzero fraction of features will remain unique to individual seeds no matter how many SAEs are trained. The decline is real but approaches an asymptote above zero. This is the paper's strongest evidence against the hypothesis that orphan features are simply rare features that would eventually appear in some seed given enough training runs.

Firing Frequency and Shareability

Figure 3 examines the relationship between how often a latent fires and how many other SAE seeds it is shared with, using Seed 1 as the base SAE and all eight other seeds for comparison. The latents are colored by the number of matchings (out of 8) in which they are classified as shared.

The dominant pattern: frequently-firing latents are overwhelmingly shared across all eight seeds, while rarely-firing latents are predominantly orphan (shared with zero seeds). The histogram at the bottom of Figure 3 (with log-scale from 0 to 500 and linear-scale above) shows that the mass of low-frequency latents is concentrated in the "shared with 0 seeds" category.

However, the paper identifies an important countervailing pattern:

"Interestingly, a significant number of orphan latents have a higher firing rate on average than latents that shared with all SAEs. In fact, as the average alignment between latents increases, the firing frequency seems to decrease" (Section 4, Figure 3 discussion).

This means that while frequency strongly predicts shareability in the extremes (very rare → orphan; very frequent → shared), the relationship is not monotonic in the middle range. Some latents with moderate-to-high firing rates remain orphans, and the latents with the very highest cosine similarities to their counterparts are not the most frequently firing ones. This suggests that firing frequency and the "cleanliness" of feature recovery are distinct dimensions of variation.

Interpretability of Shared vs. Orphan Features

Section 5 tests whether orphan features are simply degenerate or uninterpretable, using the automated interpretability pipeline from Paulo et al. (2024). For two seeds of a 32K-latent SAE, the pipeline generates natural-language explanations for every latent and scores them using detection scoring (0 to 1) over 100 active and 100 non-active sequences.

The overall results: the average explanation score is 0.72, with 25% of explanations scoring below 0.62 and 25% scoring above 0.8 (Section 5). This establishes that the SAE's features are generally interpretable by this metric, providing a baseline for comparison.

When conditioned on the number of SAEs that share each latent (Figure 6, left panel), a clear gradient emerges: latents shared across more SAE seeds have higher average interpretability scores. The distribution of scores shifts rightward as the shared count increases. This means that features that reliably appear across random initializations are easier to explain on average.

However, the critical finding for the paper's argument is what happens at the low end:

"In spite of this, a significant fraction of latents found only on one SAE have high scoring latents" (Section 5).

Figure 6 (right panel) plots the scores of matched latent pairs across two seeds, showing that most latents with low alignment have either low or average scores, but some orphan latents achieve high scores in one SAE that their matched counterpart in the other SAE does not achieve. Table 1 makes this concrete with five examples spanning the alignment spectrum:

A latent with alignment 0.10 (strongly orphan) in Seed 1 has the explanation "Abbreviated country name in United States Supreme Court case citations" with a detection score of 0.865, while its matched counterpart in Seed 2 has an explanation about non-English characters scoring only 0.56.
A latent with alignment 0.44 has the explanation "Abbreviated geographical or institutional references, usually in the context of legal citations" (score 0.85 in Seed 1) matched to one described as "Punctuation marks" (score 0.49 in Seed 2).
At alignment 0.75, both latents have high-scoring, semantically similar explanations about percent symbols (scores 0.95 and 0.97).
At alignment 0.97, both latents have near-identical, high-scoring explanations about frequency adverbs (scores 0.94 and 0.99).

This demonstrates that orphan features are not failed features — many are highly interpretable and capture meaningful linguistic patterns, but those patterns happen not to appear as cleanly separated features in the other SAE's decomposition. The paper concludes that "an individual SAE training run is likely 'missing out' on a number of interpretable features" (Section 5).

Ablation Studies and Robustness Checks

Number of SAE latents (SAE width): Figure 4 (left) shows that the fraction of shared latents decreases monotonically as the number of SAE latents increases. For Pythia 160M MLP SAEs trained on 8B tokens, the overlap drops from approximately 65-70% at small latent counts to approximately 42% at 32K latents. Figure 5 (left) extends this pattern: for Pythia 160M, the downward trend continues with larger latent counts across different layers, and the overlap is even lower for Llama 3 8B SAEs (which have 131K latents, with only ~30% shared as reported in the abstract). Figure A3 (left) confirms that mean matched cosine similarity also decreases with increasing latent count, establishing that the effect is not an artifact of the 0.7 threshold.

Number of active latents (sparsity level, k in TopK): Figure 4 (middle) demonstrates that increasing k (making the SAE less sparse) decreases the fraction of shared latents. The paper sweeps k values for a 32K-latent TopK SAE on Pythia 160M and finds a monotonically decreasing relationship between k and overlap. Figure A3 (middle) confirms the same trend for mean matched cosine similarity. The paper notes that this is consistent with feature absorption increasing at lower sparsity levels (Karvonen et al., 2024), but argues that absorption alone cannot explain the full pattern since training time has the opposite effect (see below).

Training duration (number of tokens): Figure 4 (right) shows that the fraction of shared latents increases with training duration, from approximately 35% at 1B tokens to approximately 42% at 8B tokens for a 32K-latent SAE on Pythia 160M. Figure A3 (right) shows that mean matched cosine similarity also increases. The paper explicitly contrasts this with the sparsity result: "seed dependence is not mainly due to feature absorption, as absorption increases when sparsity is decreased [and] the model is trained for longer, while [overlap] does increase when the number of latents increases" (Section 6). In other words, if absorption were the primary driver, longer training (which might increase absorption) would decrease overlap, but the opposite occurs.

SAE architecture (TopK vs. ReLU vs. Gated): Figure 5 (right panel) compares the fraction of shared latents across architectures for GPT-2 SAEs. The paper reports that "standard and Gated SAEs trained with L1 loss have a larger overlap between latents" than TopK SAEs (Section 6). ReLU SAEs with L1 sparsity penalty and Gated SAEs both show substantially higher feature overlap than TopK SAEs, even when controlling for sparsity level. This is one of the paper's most practically significant findings, since the field has migrated toward TopK as the state-of-the-art architecture (Gao et al., 2024; Templeton et al., 2024).

Model layer: Figure 5 (left) shows how overlap varies across layers for Pythia 160M SAEs with 32K latents. The paper reports that overlap is "almost constant across the middle layers of the model, being lower for the earlier layers and the last layer" (Section 6). This U-shaped pattern suggests that features in early and late layers may be more sensitive to initialization or admit more valid alternative decompositions.

Activation source (MLP vs. residual stream): Figure 5 (middle) compares SAEs trained on MLP outputs versus residual stream activations at the same expansion factor across different models. The result is model-dependent: "on SmolLM and GPT2 the MLP latents have more overlap between seeds than the residual stream ones, but the same is not true for Pythia" (Section 6). This indicates that the activation geometry differs across model families in ways that affect seed dependence.

Model scale: Figure 5 (middle) compares SAEs with the same expansion factor of 36 across Pythia 160M, GPT-2, SmolLM, and Llama 3 8B. The paper reports that "the overlap is much lower for the Llama 8B SAEs" — controlling for expansion factor, the largest model produces SAEs with the least feature overlap. This suggests that seed dependence scales with model capability or activation space complexity, not just SAE width alone.

Hungarian matching vs. maximum cosine similarity: This robustness check (Section 3, Figure A2, Figure A3) compares the two matching methods. The paper finds that "while for some latents the max cosine similarity is higher than the matched cosine similarity, the vast majority have the same value for both metrics." Figure A3 shows that mean matched and mean max cosine similarity track each other closely across all ablation dimensions (latent count, active latents, training time), with max cosine similarity being "just slightly higher." Figure A1 (right panel) confirms that the fraction classified as shared is essentially unchanged between the two methods. The paper acknowledges that this reduces the practical importance of the bijective matching requirement but argues the Hungarian approach is "more principled."

Shared latent threshold sensitivity: Figure A1 (right panel) plots the fraction of latents classified as shared as a function of the cosine similarity threshold. The curve is relatively flat in the range 0.6 to 0.8, with the fraction changing gradually rather than showing a sharp cliff. This validates that the 0.7 threshold sits in a stable region between the two modes of the bimodal distribution and that the exact numerical choice does not drive the results.

Negative result — no evidence of feature absorption in MLP SAEs: Despite exploring parameter ranges where absorption is expected to occur (Chanin et al., 2024; Karvonen et al., 2024), the paper reports: "We have found no evidence of feature absorption on the MLP SAEs we trained, but that may be due to the fact that the current metric is not tuned to find absorption on MLP SAEs, as it was mostly used on residual stream ones" (Section 6). This negative finding suggests that absorption (where one feature takes over the activation pattern of another) may be architecture-dependent or require different detection methods for MLP versus residual stream SAEs, and that seed dependence is not primarily explained by absorption dynamics.

Critical Assessment

Claim 1: "SAEs trained on the same model and data, differing only in the random seed used to initialize their weights, identify different sets of features."

This central claim is strongly and directly supported by the experiments. The paper demonstrates this across multiple architectures, model families, SAE widths, and sparsity levels. For the specific configuration highlighted in the abstract (TopK SAE with 131K latents on Llama 3 8B), the 30% overlap figure directly operationalizes the degree of difference. The bimodal similarity distribution (Figure 1) provides visual evidence that the difference is qualitative (two distinct populations of features) rather than continuous variation around a single mode. The asymptotic analysis (Figure 2) strengthens this by showing that a nonzero fraction of features remain seed-unique even with nine independently trained SAEs.

One important caveat: the claim is established for the features as operationalized by the SAE, not directly for features in the underlying model. The paper's measurement framework compares SAE latents to SAE latents — it does not independently verify whether the underlying model "uses" those features. If SAEs are imposing a non-unique decomposition on a model whose true features are unique, the paper's measurement would detect non-uniqueness correctly, but for a different reason than if the model's features themselves are non-unique. The experiments cannot distinguish between "the model has unique features but SAEs don't find them consistently" and "the model itself doesn't have unique features," but this distinction does not affect the practical conclusion that SAE-discovered feature sets are seed-dependent.

Claim 2: "Only 30% of the features were shared across different seeds" for Llama 3 8B SAEs with 131K latents.

This specific number is supported but with limited detail in the paper. The 30% figure appears in the abstract and is attributed to "an SAE with 131K latents trained on a feedforward network in Llama 3 8B," but Section 6 only discusses Llama 3 8B results at a higher level (Figure 5, middle panel, showing that overlap is "much lower for the Llama 8B SAEs" at the same expansion factor). The paper does not present a dedicated figure breaking down the Llama 3 8B comparison in the same detail as the Pythia 160M experiments (no scatter plot analogous to Figure 1, no asymptotic analysis analogous to Figure 2, no interpretability analysis analogous to Section 5, no firing frequency analysis analogous to Figure 3 for Llama 3 8B). The 30% figure is a single number without error bars, without a report of how many seeds were compared, and without ablation across k values or training durations for that specific configuration. Given that this is the paper's headline number, the experimental support for it is surprisingly sparse — most of the detailed analysis is on the smaller Pythia 160M 32K-latent configuration (42% overlap), and the reader must extrapolate the scaling trend to trust the 30% figure.

What would strengthen this claim: a figure comparable to the Pythia 160M analyses but for Llama 3 8B — bimodal similarity distribution, shared vs. orphan classification threshold validation, and ideally a multi-seed analysis to confirm that the lower overlap is robust across comparisons rather than a single pair of seeds.

Claim 3: "ReLU SAEs trained with the L1 sparsity loss showed greater stability across seeds, [while] SAEs using the state-of-the-art TopK activation function were more seed-dependent, even when controlling for the level of sparsity."

This claim is supported by Figure 5 (right panel), which shows substantially higher overlap for ReLU and Gated SAEs compared to TopK SAEs on GPT-2. The paper reports that TopK has greater seed dependence "even when controlling for the level of sparsity," which is an important qualification — it means the difference is not simply because TopK and L1-based approaches achieve different effective sparsity levels. However, the paper does not show the detailed comparison data that would make the "controlling for sparsity" claim fully transparent. Figure 4 (middle) shows that varying k changes overlap for TopK SAEs, and Figure 5 (right) shows different architectures at presumably comparable sparsity levels on GPT-2, but the paper does not present a controlled experiment where TopK and ReLU SAEs are matched on both number of active latents (or average L0) and total latent count, trained on the same data, with the same model and layer.

A missing experiment: side-by-side TopK and ReLU SAEs with identical width, identical effective sparsity (matching k to the average number of active latents in the ReLU SAE), trained on the same model and data, with multi-seed overlap reported for both. The current results strongly suggest the architecture effect is real, but the evidence is presented across different figures and partially different experimental configurations, making precise comparison difficult.

Claim 4: "Orphan features are often quite interpretable, so that an individual SAE training run is likely 'missing out' on a number of interpretable features."

This claim is supported with specific examples (Table 1, Figure 6) but is based on a small sample. The interpretability analysis in Section 5 uses two seeds of a 32K-latent SAE (the Pythia 160M configuration), and the automated interpretability scores come from an external pipeline (Paulo et al., 2024) that has its own limitations and failure modes. The detection score metric, while providing a quantitative comparison, is not independently validated against human judgments of interpretability for SAE features specifically.

The examples in Table 1 are compelling and well-chosen to illustrate the phenomenon — particularly the "Supreme Court case citations" orphan with a 0.865 score matched to a 0.56-scoring counterpart — but they are selected examples rather than a random sample, and the selection criterion ("maximizing the score of both explanations if the cosine similarity between the latents is > 0.7 and by maximizing the score of the explanation on seed 1 and minimizing the score on seed 2 if the cosine similarity is < 0.7") is designed to highlight the most dramatic cases. A more systematic analysis would report the full distribution of interpretability scores for orphans vs. shared features with statistical comparisons (e.g., median scores, interquartile ranges, fraction of orphans above a high-score threshold), which Figure 6 provides to some extent but without numerical summary statistics.

Additionally, the claim that an SAE run is "missing out" on interpretable features implies that these features are "really there" in some sense, but the paper's own framework (Section 7) argues that features are constructed rather than discovered. An orphan feature in one SAE is not "missing" from the other SAE — it is represented differently (possibly split, absorbed, or distributed). The "missing out" framing is slightly in tension with the paper's philosophical position, or at least uses language that connotes discovery rather than construction.

Claim 5: "Feature overlap decreases as SAE width and the number of active latents increase."

This is strongly supported by Figure 4 (left and middle), Figure 5 (left), and Figure A3 (left and middle), which show clear monotonic trends across multiple configurations. The relationship is robust to the choice of metric (shared fraction or mean cosine similarity) and to the matching method (Hungarian or max cosine). The paper correctly does not claim a functional form (e.g., power law) for these relationships, only the direction and monotonicity.

One nuance: the claim that overlap decreases with increasing number of active latents (k) is demonstrated for TopK SAEs specifically. The paper does not show that the same relationship holds for ReLU SAEs with varying L1 coefficients — it's possible that the relationship between sparsity and overlap is architecture-specific rather than general. The paper's statement that "even when controlling for the level of sparsity" TopK SAEs are more seed-dependent than ReLU SAEs suggests an interaction: sparsity level and architecture type both matter, but the architecture effect persists at matched sparsity.

Claim 6: "This phenomenon [was observed] across multiple layers of three different LLMs, two datasets, and several SAE architectures."

The claim of breadth is partially supported by the experiments shown. Multiple layers: Figure 5 (left) shows results across all layers of Pythia 160M, confirming the phenomenon is not limited to a single layer. Multiple LLMs: the paper reports results on Pythia 160M, GPT-2, SmolLM, and Llama 3 8B (Figure 5, middle). Several SAE architectures: TopK, ReLU, and Gated are compared (Figure 5, right). However, "two datasets" is not clearly supported. The paper trains all SAEs on the Pile (Section 3); there is no mention of a second dataset used for SAE training. The 10M tokens used for firing frequency analysis (Figure 3) are also from the Pile. It is possible the paper is counting the model's pretraining data as one dataset and the SAE training data as another, but these are the same dataset (the Pile) for Pythia 160M. For GPT-2, SmolLM, and Llama 3 8B, the SAEs are also trained on Pile activations, but these models may have been pretrained on different data, creating a train-test distribution shift for the SAE. If so, the paper does not analyze whether this shift affects overlap.

Missing Experiments and Design Limitations

No ensemble or combined analysis across the nine seeds for Llama 3 8B. The asymptotic analysis (Section 4, Figure 2) is conducted only on Pythia 160M SAEs. We do not know whether the nonzero-asymptote finding generalizes to larger models, nor whether the power-law-plus-offset fit remains significantly better than pure power law at the Llama 3 8B scale.

No human validation of interpretability. The detection scoring metric from Paulo et al. (2024) is automated and has not been independently validated against human judgments for SAE features specifically. The claim that orphan features are "quite interpretable" rests entirely on this automated pipeline, and the paper does not report any human evaluation of the features or their explanations.

No analysis of whether orphan features in seed A appear as split/merged features in seed B. The Hungarian matching enforces one-to-one correspondence, but a feature might genuinely be represented as two latents in one SAE and one latent in another (feature splitting). The current framework would classify the one-latent version as orphan (no single counterpart with high similarity) even though the same information content exists in the other SAE in a different form. The paper acknowledges splitting (citing Chanin et al., 2024 and Karvonen et al., 2024) but does not attempt to detect or correct for it.

No analysis of whether overlap varies with the specific data seen during training. All SAEs see identical data in identical order. This isolates the seed effect cleanly but leaves open the question of how much overlap would decrease if SAEs were trained on different data samples from the same distribution — which is the more realistic deployment scenario, since in practice different groups train SAEs on different data slices. The identical-data-order control is excellent for isolating initialization effects but makes the overlap numbers potentially optimistic relative to real-world SAE comparison scenarios.

Limited reporting of training hyperparameters. The paper uses the sae library (Belrose, 2024) but does not report learning rate, batch size, optimizer settings, or convergence criteria. This makes exact replication difficult and prevents analysis of whether the seed dependence is sensitive to optimization details (e.g., whether different learning rates or optimizers would change the degree of overlap).

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For

The assumption or constraint. The entire compute-optimal framework rests on the ability to estimate prompt difficulty before allocating the inference budget, but the paper's method for doing so is extraordinarily expensive, and this cost is explicitly excluded from the efficiency calculations. The authors acknowledge this directly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The method generates 2048 samples per question and scores them with the PRM (for predicted difficulty) or checks them against ground truth (for oracle difficulty). This means the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it.

The consequence. In any realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter. The entire framework becomes circular if applied naively: you spend 2048 generations to decide whether to use 16 or 64 generations, when you could have just used 64 generations of best-of-N on every problem and still come out ahead on total compute. The paper's headline efficiency numbers (e.g., "16 generations matching 64 generations of best-of-N" in Figure 4) are best understood as upper bounds on achievable efficiency that assume difficulty is known for free, not as realized deployment gains.

What evidence exists in the paper. The 2048-sample difficulty estimation procedure is described in Section 3.2, and the gap between oracle and predicted difficulty bins (Figures 4 and 8) shows that even the cheaper predicted method still requires the same 2048 samples — it just substitutes PRM scoring for ground-truth checking. The paper does not report the computational cost of difficulty estimation in any unit (FLOPs, generations, wall-clock time), nor does it include a break-even analysis showing at what total budget the compute-optimal approach recovers its estimation overhead.

Mitigation status. The paper flags this as "a key avenue for future work" (Section 3.2) and suggests three directions: training models to predict difficulty directly from question text, using the verifier's scores on a small number of initial samples as a quick difficulty signal, and developing adaptive policies that interleave difficulty assessment with problem-solving. None of these are explored or evaluated in the paper. The mitigation is entirely speculative, and the paper provides no evidence that cheap difficulty estimation is possible while preserving the benefits of compute-optimal allocation.

Hard Problems Remain Essentially Unsolved Regardless of Budget

The assumption or constraint. The compute-optimal framework assumes that test-time compute can improve performance if the base model has some non-trivial probability of producing a correct answer. The paper explicitly acknowledges that this assumption fails on the hardest problems:

"On the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated" (Section 5.3, discussion of Figure 3 right).

Across all methods — PRM search, iterative revisions, and their compute-optimal combinations — difficulty bin 5 shows near-zero accuracy regardless of how much compute is deployed. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%.

The consequence. This establishes a fundamental capability boundary: test-time compute can amplify existing capability (improving the probability of finding correct answers the model already produces at some low rate) but cannot create capability from nothing. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help — there are no correct solutions in the proposal distribution to find or refine. This means the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, scaling pretraining remains the only viable path, and the FLOPs-matched comparison in Section 7 confirms this: on hard problems at high inference-to-pretraining ratios (R ≫ 1), test-time compute shows a −37.2% to −52.9% relative disadvantage compared to the ~14× larger model (Figure 1 bar charts, Figure 9).

What evidence exists in the paper. The bin 5 results are consistent and striking across every experiment: Figure 3 (right, search), Figure 7 (right, revisions), and Figure 9 (FLOPs-matched). The paper reports the pass@1 rate for the base model on bin 5 questions as being in the low single digits (Section 3.2 describes binning by pass@1 into quintiles, with bin 5 being the lowest). The paper is transparent about this limitation in the Section 7 takeaway box and in the abstract-level framing.

Mitigation status. The paper does not attempt to address this limitation — it is presented as an inherent property of test-time compute rather than a problem to be solved. The authors acknowledge that for the hardest problems, "pretraining is almost always more effective" (Section 7). No method is proposed for extending test-time compute to problems where the base model's pass@1 is near zero, and the paper's framework provides no guidance for this regime other than to recognize it and allocate compute elsewhere. This is not a failure of the paper — it is a clean empirical characterization of where test-time compute can and cannot substitute for pretraining — but it means the approach is only applicable to a subset of problems, and that subset shrinks as the base model's capability decreases relative to the problem distribution.

The FLOPs-Matched Baseline Uses a Weak Pretraining Comparison

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time strategies against a model with approximately 14× more parameters, but that larger model is not trained in a compute-optimal manner and uses no test-time compute itself. The authors acknowledge the first issue explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

Specifically, the comparison scales only model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than the Chinchilla-optimal approach (Hoffmann et al., 2022) where both parameters and data are scaled. Additionally, the ~14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search, no revisions.

The consequence. Both choices make the pretraining baseline weaker than it needs to be, potentially inflating the reported advantages of test-time compute. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, shrinking or reversing the reported advantages. Giving the larger model even a modest test-time compute budget — say, best-of-8 with the same verifier, or a few revision steps — would create a much stronger baseline. The paper's central claim that "test-time compute can substitute for pretraining" is therefore qualified: it can substitute for a particular kind of pretraining scaling (parameter-only, no inference augmentation) under specific conditions (easy-to-medium problems, low R). Against a stronger pretraining baseline, the substitution range would likely be narrower.

What evidence exists in the paper. The paper provides the FLOP accounting formulas (Section 7: X = 6ND_pretrain, Y = 2ND_inference) and explicitly states the parameter-only scaling assumption in the quoted sentence above. The FLOPs-matched results (Figures 1 and 9) are reported with the caveat that they apply to "a canonical approach to scaling pretraining compute." The paper does not include an ablation where the larger model receives any test-time compute budget, nor does it compare against a Chinchilla-optimal training recipe. The absence of these comparisons means the paper cannot distinguish between "test-time compute is generally preferable to pretraining on easy problems" and "test-time compute is preferable to a specific, potentially suboptimal pretraining strategy."

Mitigation status. The paper flags the parameter-only scaling choice as a limitation and suggests future work on the compute-optimal pretraining comparison (Section 7). It does not address the lack of test-time compute for the larger model baseline. Both are significant gaps for a paper making claims about the pretraining-inference tradeoff, and the reported numbers should be interpreted as reflecting the comparison against a specific (weaker) pretraining baseline rather than a general superiority of test-time compute over pretraining.

Sequential Revisions Introduce Latency That Is Not Accounted For

The assumption or constraint. The paper measures compute in "generations" — the number of complete solutions sampled — which serves as a reasonable proxy for total FLOPs but completely ignores wall-clock latency. Sequential revisions are inherently serial: each revision depends on the output of the previous one, so they cannot be parallelized. A strategy that allocates 128 generations as 64 sequential revisions × 2 parallel chains takes roughly 64× longer wall-clock time than one that runs 128 parallel samples simultaneously with sufficient hardware.

The consequence. For latency-sensitive applications — interactive assistants, real-time decision-making systems, any deployment where the user is waiting for a response — the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their FLOPs efficiency advantages. The paper's finding that fully sequential revisions are optimal for easy problems (Figure 7, right panel) and at low budgets (Figure 7, left panel) means the strategies that look best by the generation-count metric may be the worst by the latency metric. In a production setting where the model must respond within a fixed time budget (e.g., 2 seconds), a strategy requiring 64 sequential forward passes through an LLM could be orders of magnitude too slow even if its total FLOPs are modest.

What evidence exists in the paper. The sequential vs. parallel tradeoff is extensively analyzed by generation count (Figures 5, 6, 7, 8), but latency is never mentioned anywhere in the paper. The generation budget is treated as the sole cost metric, and the distinction between serial and parallel generation is collapsed into a single number. The paper does not report wall-clock times for any experiment, nor does it provide latency-normalized comparisons that would show, for example, how much accuracy a fully parallel strategy can achieve in the same wall-clock time as a sequential strategy.

Mitigation status. The paper does not address latency at all. This is a significant omission because the practical deployment of these methods requires optimizing a multi-dimensional cost function (FLOPs, latency, memory) rather than a single metric. A practitioner reading the paper would need to independently determine whether the reported accuracy improvements from sequential revisions are achievable within their latency constraints, and the paper provides no guidance for making this determination. The revision model's 38% correct-to-incorrect reversion rate (Section 6.1) further complicates the latency picture — if early stopping criteria or within-chain selection can detect when a revision chain has stabilized, that could reduce latency, but this is not explored.

Results Are Limited to a Single Benchmark and Model Family

The assumption or constraint. All experiments are conducted using the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an untested assumption. The 500-question MATH test set covers high-school competition-level math problems requiring multi-step symbolic reasoning — a specific type of cognitive task that may not generalize to other domains where test-time compute could be deployed.

The consequence. Several aspects of the findings could be model-specific or domain-specific in ways that the paper cannot assess:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different error patterns, or different base MATH performance could exhibit qualitatively different difficulty-dependent scaling curves — for instance, a stronger base model might have fewer "bin 5" problems and different optimal allocation patterns.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families (some models are much better at few-shot learning than others, which would affect the revision training data quality).
The MATH benchmark consists of problems with unambiguous, verifiable answers and clear multi-step solution structures. It is unclear whether the difficulty-dependent patterns — beam search hurting easy problems due to verifier over-optimization, revisions helping easy problems, search helping medium problems — generalize to code generation (where correctness is testable but solution structure differs), logical reasoning (where step validity is harder to verify), scientific QA (where factual knowledge matters more than reasoning chains), or open-ended generation tasks with no ground-truth correctness signal.

What evidence exists in the paper. The paper acknowledges the single-benchmark limitation implicitly by conducting all experiments on MATH, but does not provide any cross-domain results or discuss how the findings might transfer. The experiments with multiple difficulty bins (Section 5.3, 6) show that behavior varies substantially within MATH itself, which suggests domain-specificity is plausible — if difficulty-dependent patterns are strong within a single benchmark, different benchmarks with different difficulty distributions and reasoning types could show different overall patterns. The revision model training uses MATH training data and the edit-distance heuristic specific to math solutions, which may not transfer to other domains.

Mitigation status. The paper does not attempt to address this limitation. The authors suggest testing on code generation and other domains as future work in Section 8, but no preliminary results are provided. This is a standard single-benchmark limitation that is common in the field, but it means the paper's quantitative findings (4× efficiency gains, 30% shared features, optimal sequential-to-parallel ratios) should not be assumed to transfer to other models or tasks without replication. The paper's conceptual contributions — the compute-optimal allocation framework, the difficulty-dependent behavior, the over-optimization phenomenon — are more likely to generalize than the specific numbers.

The Revision Model and PRM Are Not Combined

The assumption or constraint. The paper studies two complementary mechanisms for test-time compute — PRM-guided search (Section 5) and iterative revisions (Section 6) — but studies them independently, never combining them into a single system. The authors acknowledge this explicitly in Section 8:

"we did not experiment with PRM tree-search techniques in combination with revisions"

This means the paper evaluates the two mechanisms as separate pipelines, each with their own compute-optimal allocation strategies, but does not explore whether their benefits are additive, redundant, or synergistic when deployed together.

The consequence. The reported results represent a lower bound on what test-time compute could achieve, because the two mechanisms have complementary strengths that the paper itself documents:

Revisions improve the proposal distribution — they generate better candidates by conditioning on previous attempts, which is most effective on easy-to-medium problems where the model can refine its answers (Section 6, Figure 7 right: sequential revisions dominate on easy bins).
PRM search improves candidate selection — it finds the best among generated candidates using a learned verifier, which is most effective on medium problems where the model needs to explore different high-level approaches (Section 5.3, Figure 3 right: beam search outperforms best-of-N on bins 3–4).

A combined system could use the revision model as the proposal distribution within beam search — at each step of the search tree, the model conditions on previous rejected branches as context, potentially producing higher-quality candidate steps than the base model alone. Alternatively, the PRM could guide which revisions to pursue, truncating revision chains when the verifier's score stops improving. Without testing these combinations, the paper cannot determine whether the two mechanisms provide independent gains (in which case a combined system would substantially outperform either alone) or whether their benefits overlap (in which case the optimal combination would look like one mechanism or the other depending on difficulty).

What evidence exists in the paper. The paper provides indirect evidence that the mechanisms are complementary: the difficulty-dependent optimal strategies differ between search and revisions (beam search best on medium problems in Figure 3 right, revisions best on easy problems in Figure 7 right). The FLOPs-matched comparison (Figure 9) shows that revisions outperform PRM search overall for the FLOPs-matched tradeoff. The paper's own data suggests that a combination could outperform either alone by applying the right mechanism per difficulty level or by combining them within a single problem. But no combined experiment is reported.

Mitigation status. The paper identifies this as a direction for future work (Section 8) but provides no analysis of whether the combination would be straightforward or would face obstacles (e.g., distribution shift between the revision model's outputs and the PRM's training distribution, which Appendix J, Figure 15a shows is already a problem for the base PRM on revision outputs). The omission of combined experiments is understandable given the scope of the paper — each mechanism is studied in substantial depth — but it means the paper provides an incomplete picture of what test-time compute can achieve, and the optimal allocation policy for a combined system might differ from either of the independent policies studied.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally challenges a presupposition that has shaped much of the mechanistic interpretability research program: that sparse autoencoders converge to something like the "true" features inside a language model, and that training a single large SAE on a model's activations gives us an exhaustive inventory of what that model represents. The finding that only ~30% of features are shared across seeds in a 131K-latent TopK SAE on Llama 3 8B — and that this fraction decreases as SAEs get larger and use the architectures the field has converged on — is not an incremental refinement. It is a reframing of what SAE features are.

The reframing shifts the conceptual status of SAE features from discovery to construction. If features were being discovered, we would expect convergence: different runs of the same measurement instrument on the same object should yield the same measurements. The fact that they do not — that overlap is low, that highly interpretable features appear as orphans in single seeds (Section 5, Table 1), and that the asymptotic analysis (Figure 2) shows a persistent nonzero fraction of seed-unique latents — implies that an SAE is better understood as imposing a particular kind of structure (sparse linear decomposition) on activation space, one that admits multiple valid solutions. This is not a failure of SAEs. It is a characterization of what they do.

This reframing has several concrete consequences for the research landscape:

It shifts the burden of proof for safety-verification claims. The hope that SAEs could "identify and enumerate over all features in a model" (Elhage et al., 2022) and thereby verify properties like "a model will never lie" (Olah, 2023) implicitly assumed feature uniqueness and exhaustiveness. The paper's results show that any single SAE training run is a sample from a distribution of possible decompositions, and that sample can miss interpretable features that other runs would have found (the orphan phenomenon). This does not make safety verification impossible, but it means that a safety audit based on a single SAE cannot claim completeness — a dangerous feature might be represented cleanly in one seed's decomposition and be absent or distributed across multiple latents in another's. Ensemble approaches, where multiple independently seeded SAEs are trained and their feature sets are combined, become a methodological necessity rather than an optional enhancement. The paper provides the measurement tools (Hungarian matching, shared/orphan classification) to determine how many seeds are needed to approach feature-set saturation.

It reconciles contradictory prior findings. Before this paper, the literature contained a genuine tension. Braun et al. (2024) and Anonymous (2024) found that ReLU SAEs were highly stable across seeds (>90% overlap), while Marks et al. (2024) found that TopK SAEs needed explicit alignment procedures to achieve good matching, and Karvonen et al. (2024) found that TopK and JumpReLU architectures had higher feature splitting rates than ReLU SAEs. The present paper resolves this tension by showing that the discrepancy is not about methodology or experimental error — it is about architecture. ReLU SAEs trained with L1 sparsity penalties genuinely do produce more consistent feature decompositions across seeds than TopK SAEs (Section 6, Figure 5 right). The earlier results were both correct, but they applied to different architectural regimes. The paper also reveals that a key finding that appeared to support convergence — Balagansky et al. (2024)'s alignment of SAE features across Gemma 2 layers — was confounded by the use of identical random seeds across all SAEs in the GemmaScope collection (Section 2). This is a striking example of how an unnoticed hyperparameter choice shaped the field's assumptions.

It adds feature universality as a new axis for SAE evaluation. The field has primarily evaluated SAEs on reconstruction fidelity (how well the SAE reconstructs the original activations) and sparsity (how few latents are active per input), with interpretability as a secondary but important metric. This paper introduces cross-seed feature overlap as an additional dimension that should be tracked and reported. The finding that TopK SAEs — which are state-of-the-art on reconstruction-sparsity tradeoffs — are the worst on feature overlap (Section 6) means that universality may trade off against other desirable properties. Researchers developing new SAE architectures should measure and report cross-seed overlap alongside reconstruction error and L0, making the tradeoff explicit rather than invisible.

It redirects research attention from "finding the right features" to "understanding the space of possible decompositions." If features are constructed rather than discovered, then the interesting scientific question shifts from "what are the features in this model?" to "what is the structure of the space of possible decompositions, and what does that structure tell us about the model?" The paper's asymptotic analysis (Figure 2) — showing that the number of seed-unique latents follows a power law with a nonzero offset — is a first step in this direction, characterizing how quickly the feature set saturates as more seeds are added. Future work could extend this by studying whether different seeds' decompositions are related by structured transformations (e.g., linear combinations, rotations within subspaces), whether the orphan features from different seeds collectively span the full activation space, and whether the degree of non-uniqueness varies across different regions of activation space (e.g., are features in certain semantic domains more universal than others?).

It connects SAE non-uniqueness to broader deep learning theory. The paper attributes the seed dependence to "the nonconvexity of the SAE loss function, which gives rise to many local optima" (Section 7). But it adds an important observation: "One might have expected a priori, however, that different local optima would have more feature overlap than we found in this study." This suggests that the SAE loss landscape has more distinct basins — corresponding to more qualitatively different ways of decomposing activation space — than standard intuitions about local minima in neural networks would predict. This observation connects to ongoing work on mode connectivity, linear interpolation between solutions, and the geometry of neural network loss landscapes, but casts it in a new light: here, the multiplicity of solutions is not a problem to be solved (as in, "how do we ensure convergence to a unique minimum?") but a feature of the representational space that should be characterized and understood in its own right.

It challenges the "flat" SAE design. The paper explicitly connects its findings to observations from Chanin et al. (2024) that features can split and be absorbed across different SAE sizes, and from Anonymous (2024) that "meta SAEs" can decompose individual SAE features into more specific sub-features. The seed-dependence results reinforce the concern that the standard "flat" SAE architecture — where all latents are at the same level of abstraction — may not accommodate the hierarchical structure of concepts. If a concept like "legal citation" can appear as a single feature in one seed and as multiple sub-features in another, then the decomposition is sensitive to how the flat architecture resolves hierarchical structure. This strengthens the case for hierarchical or multi-scale SAE architectures (as suggested by Ayonrinde et al., 2024) and suggests that seed dependence might be reduced — though probably not eliminated — by architectures that explicitly model feature hierarchies.

One should be precise about what the paper does and does not claim. It does not claim that SAE features are arbitrary or meaningless — the interpretability scores in Section 5 show that even orphan features are often highly interpretable. It does not claim that SAEs are useless for understanding models — the features they find are pragmatically useful decompositions that capture real structure. What it claims, and what the evidence supports, is that the feature set is non-unique, that the degree of non-uniqueness is large enough to matter for applications that assume completeness, and that this non-uniqueness should change how the field thinks about and uses SAEs.

Follow-Up Research This Work Enables

Measuring whether orphan features in one seed appear as split or merged features in another seed. The Hungarian matching enforces a one-to-one correspondence between latents, but a genuinely important feature might be represented as a single latent in SAE A and as two or more latents in SAE B (feature splitting) or might be absorbed into a broader feature in SAE B. The current shared/orphan framework would classify the single-latent representation as orphan even though the same information content is present in the other SAE in a different arrangement. A direct follow-up would take orphan latents from SAE A and attempt to reconstruct their activation patterns using linear combinations of the top-k most similar latents in SAE B, measuring the reconstruction quality (fraction of variance explained) as a function of k. If orphans can be reconstructed with high fidelity from a small number of other-SAE latents, that would indicate splitting/merging rather than genuine absence, and would provide a more nuanced measure of "feature overlap" that accounts for distributed representations. This experiment is newly tractable because the paper provides the matching infrastructure and the orphan/shared classification that identifies which latents to target for the reconstruction analysis.

Characterizing whether features in certain semantic domains are more universal than others. The paper shows that firing frequency predicts shareability (Figure 3) but also notes that some frequently-firing features remain orphans. A natural follow-up would annotate features by semantic domain — using the automated interpretability pipeline from Paulo et al. (2024) to cluster feature explanations into categories like "syntax," "semantics," "factual knowledge," "formatting," etc. — and test whether cross-seed overlap varies systematically by domain. One plausible hypothesis: features encoding surface-level patterns (punctuation, formatting, token identity) might be more universal because they correspond to clear, well-separated directions in activation space, while features encoding abstract semantic concepts (legal reasoning, emotional valence, narrative structure) might show more seed dependence because they admit multiple valid decompositions. A strong experiment would train 5–10 seeds of a medium-scale SAE (e.g., 32K latents on Pythia 160M, where the paper already has 9 seeds), generate explanations for all features, categorize them by domain, and compare the shared-fraction within each domain. A finding that certain domains have systematically lower overlap would have direct implications for which types of features can be reliably audited in single-SAE studies.

Training an SAE ensemble and measuring whether cross-seed feature union enables more complete model auditing. If orphan features in individual seeds are interpretable and capture real structure (as Section 5 and Table 1 demonstrate), then the union of feature sets across multiple seeds should provide more complete coverage of the model's representational space than any single seed. A concrete experiment: train 5–10 SAEs on the same model and data with different seeds, identify all shared and orphan features across the ensemble, and for a held-out set of model behaviors (e.g., specific reasoning patterns, factual recall tasks, bias manifestations), test whether probes or classifiers trained on the ensemble's full feature union outperform those trained on any single SAE's features. Additionally, measure how many "novel" orphan features are added by each additional seed — the paper's asymptotic analysis (Figure 2) provides a starting point for estimating this curve, but extending it to measure not just overlap but also downstream task coverage would connect the measurement framework to practical auditing applications. The key question: does auditing with an ensemble of N SAEs find dangerous or unexpected features that auditing with a single SAE would miss, and at what rate?

Determining whether the degree of seed dependence varies with the SAE training objective, particularly for newer loss functions. The paper compares TopK, ReLU+L1, and Gated architectures and finds that ReLU+L1 shows higher overlap. But the SAE training literature has continued to evolve, with new objectives like BatchTopK (Bussmann et al., 2024), JumpReLU (Lieberum et al., 2024), and various auxiliary losses designed to reduce dead latents or improve reconstruction. A systematic follow-up would take a fixed model, layer, and SAE width (e.g., Pythia 160M layer 6 MLP, 32K latents, following the paper's primary configuration), train 5 seeds each for a range of modern SAE training objectives, and measure cross-seed overlap using the paper's shared-latent definition. The hypothesis from the paper's results is that objectives with "harder" sparsity constraints (discrete selection, like TopK or BatchTopK) will show lower overlap than objectives with "softer" sparsity (continuous penalties, like L1 or JumpReLU). Testing this would provide practical guidance for researchers choosing SAE architectures: if you need feature universality (for auditing), prefer softer sparsity; if you need reconstruction quality, harder sparsity may be worth the universality cost.

Stress-testing whether higher overlap at convergence can be achieved with longer training or better optimization. The paper shows that overlap increases with training duration (Figure 4, right: from ~35% at 1B tokens to ~42% at 8B tokens) but does not establish whether this trend continues or plateaus. A direct follow-up would extend training to much longer horizons (e.g., 32B, 64B, or 128B tokens of Pile activations) for a fixed SAE configuration and measure overlap at logarithmically spaced checkpoints. If overlap continues to increase and approaches, say, 70–80% at very long training horizons, that would suggest that the seed dependence observed in the paper is partly a finite-training artifact — different seeds take different amounts of time to converge to the same basin. If overlap plateaus at ~45–50% and stays there regardless of training duration, that would strengthen the paper's interpretation that the loss landscape has genuinely distinct basins corresponding to qualitatively different decompositions. The paper's asymptotic analysis (Figure 2) already provides evidence for persistent non-overlap across different seeds trained for the same duration; extending this to different training durations within each seed would test whether "training longer" and "training more seeds" are independent or interacting factors.

Testing whether activation-space geometry predicts which features will be orphans. The paper notes that firing frequency correlates with shareability but that the relationship is imperfect (Figure 3). A more mechanistic investigation would examine the local geometry of activation space around orphan vs. shared features. For a given latent, one can measure: (1) the cosine similarity between its encoder and decoder vectors (how aligned the "when to fire" and "what to reconstruct" directions are), (2) the distribution of cosine similarities to other latents in the same SAE (how "crowded" that region of activation space is), and (3) the variance explained by the latent on typical inputs. The hypothesis: orphan features correspond to regions of activation space where the decomposition is genuinely ambiguous — where multiple directions explain similar amounts of variance and the SAE must "choose" which one to assign to a latent, with different seeds making different choices. This experiment is made tractable by the paper's shared/orphan labeling: one can take a pair of SAEs, classify each latent as shared or orphan, and then compare the geometric properties of the two populations in a single SAE (since the classification is about cross-SAE status, but the geometric properties are measured within one SAE). A finding that orphans systematically differ from shared features on geometric measures would illuminate the mechanistic origin of seed dependence.

Practical Applications and Downstream Use Cases

Ensemble-based auditing for AI safety. If a research group is using SAEs to audit a language model for potentially dangerous features — deception, situational awareness, power-seeking reasoning, or other safety-relevant patterns — the paper's results imply that training a single SAE is insufficient. A practical deployment would train 5–10 independently seeded SAEs on the same model and layer, use the Hungarian matching and shared/orphan classification from this paper to align their feature sets, and then audit the union of features across all seeds, with particular attention to orphan features that appear in only one or two seeds. The paper's finding that orphan features include highly interpretable patterns (Table 1: "Abbreviated country name in United States Supreme Court case citations," detection score 0.865, found in only one seed) directly demonstrates the risk: a dangerous feature could be similarly "orphaned" in a single-seed audit. The measurement framework from this paper provides the operational tools (matching algorithm, similarity thresholds, asymptotic analysis to determine how many seeds are enough) to implement such an ensemble audit. The cost is linear in the number of seeds — training 10 SAEs instead of 1 is a 10× increase in training compute — but for high-stakes safety applications where missing a feature could be catastrophic, this overhead is likely justified.

Cross-institutional SAE comparison and replication. Different research groups training SAEs on the same model (e.g., multiple independent efforts to interpret Llama 3 or GPT-4) will produce different feature decompositions due to seed dependence, even if they use identical architectures and training data. The paper's Hungarian matching framework provides a standard method for aligning and comparing feature sets across institutions, enabling questions like "did group A's SAE find the same deception-related features as group B's SAE?" to be answered quantitatively rather than through anecdotal feature inspection. A practical implementation would involve each group publishing their trained SAE weights along with a standardized "feature similarity matrix" computed using the paper's matching protocol against a reference SAE (or against all other published SAEs for that model). This would make cross-study comparisons systematic rather than ad hoc and would allow the field to track which features are robust across independent efforts versus which are idiosyncratic to particular training runs. The paper's finding that TopK SAEs are more seed-dependent than ReLU SAEs (Section 6) also suggests that cross-institutional comparisons should report the architecture used and should prefer architectures with higher overlap when comparability is a priority.

Improved feature coverage for model editing and steering. Techniques that modify model behavior by intervening on SAE features — such as clamping or amplifying specific latents during generation to steer the model's output — rely on the SAE having captured the relevant feature in a manipulable form. If a single SAE is missing interpretable features (as the orphan phenomenon demonstrates), then steering based on that SAE will have blind spots: there may be activation patterns relevant to the target behavior that the SAE simply does not represent as individual, manipulable latents. A practical approach, suggested by the paper's results, is to train an ensemble of SAEs (3–5 seeds), identify features relevant to the steering target (e.g., "sycophancy," "refusal," "truthfulness") across all seeds, and steer using a consensus or union approach — clamping the feature in all SAEs where it appears. This increases the probability that the relevant activation pattern is captured in at least one SAE's decomposition. The Hungarian matching enables tracking "the same" steering feature across seeds, so that consistent steering can be applied even though the feature appears at different latent indices in different SAEs.