Evaluating SAE interpretability without explanations

🎯 Pitch

Sparse autoencoder interpretability can be reliably measured without generating natural-language explanations at all. Simple tasks like spotting a non-activating sentence among four that trigger a latent achieve 78% human accuracy on the top activations, and an LLM evaluator correlates at 0.85 with human judgments—cutting out the unreliable explanation-generation step entirely.

1. Executive Summary

This paper introduces two methods for evaluating the interpretability of sparse autoencoder (SAE) latents without requiring natural language explanations as an intermediate step—intruder detection (identifying a non-activating sentence among four activating examples for a given latent) and example embedding scoring (measuring whether activating examples cluster in embedding space relative to non-activating examples)—tested on SAEs trained on SmolLM2 135M MLP outputs at layers 9, 15, 21, and 27. Human intruder detection accuracy averages 65% across 56 latents and reaches 78% on the highest-activating decile, with Claude Sonnet 3.5 achieving a Spearman correlation of 0.85 with human scores, establishing that LLM evaluators can serve as reliable proxies for human judgment of latent interpretability without the confounding hyperparameters of explanation generation. Example embedding scoring yields weaker but non-random AUROCs (0.64–0.70 on the highest decile) and correlates only modestly with human intruder scores (r = 0.48), establishing that small embedding models offer a fast but less faithful alternative to LLM-based evaluation only when the activating token signal is strong.

2. Context and Motivation

The Core Problem: Current SAE Evaluation Is Tightly Coupled to Explanation Generation

The fundamental problem this paper addresses is a measurement confound in the evaluation of sparse autoencoder (SAE) interpretability. Almost all existing methods for assessing whether SAE latents are interpretable follow a two-stage pipeline: (1) generate a natural language explanation for each latent, then (2) evaluate the quality of that explanation—typically by testing how well it enables an LLM to predict the latent's activation in new contexts. The problem is that this pipeline entangles the quality of the explanation with the underlying interpretability of the latent. If an evaluation produces a low score, it is impossible to determine whether the latent itself is uninterpretable, or whether the explanation generation process simply failed to capture an interpretable latent in words. As the authors put it:

"The use of natural language explanations is problematic... It complicates the evaluation pipeline, introducing additional hyperparameters and prompt choices which are likely to affect the final results."

This is more than a methodological quibble—it means the field lacks a clean, direct measurement of latent interpretability that isolates the property of interest from the machinery used to assess it. Every design decision in the explanation pipeline (how many examples to show the explainer model, which activation quantiles to sample from, how to structure the explanation prompt, which LLM to use for explanation generation) becomes a potential source of variation in the final interpretability score, making it difficult to compare results across papers or to draw robust conclusions about which SAE architectures and training procedures produce more interpretable latents.

Why This Problem Matters

This measurement problem has both practical and theoretical consequences for the interpretability research community.

Reproducibility and standardization. As SAEs have grown from an exploratory technique into a widely adopted tool—deployed on language models (Lieberum et al., 2024; Gao et al., 2024; Templeton et al., 2024), vision models (Surkov et al., 2024; Gorton, 2024), and protein models (Adams et al., 2025)—the need for standardized evaluation benchmarks has become acute. Karvonen et al. (2025) developed SAEBench specifically to address this gap, but even comprehensive benchmarks that use explanation-based scoring inherit the confound described above. Two groups training different SAE variants and evaluating them with different explanation-generation hyperparameters cannot meaningfully compare their interpretability scores, because any difference could reflect either genuine differences in latent quality or merely differences in the explanation pipeline. A method that removes the explanation step entirely eliminates an entire class of nuisance variables and moves the field closer to standardized, comparable measurements.

The philosophical assumption that interpretability implies verbalizability. The explanation-centered paradigm implicitly assumes that "an interpretable latent must have a meaning succinctly expressible in words" (Section 1). The authors question this assumption directly. Consider a latent that activates on a particular visual pattern in images, or a latent in a language model that captures a subtle syntactic construction that linguists have a name for but that most people could not articulate. A human might reliably distinguish activating from non-activating examples of this latent—demonstrating genuine understanding—while being unable to produce a satisfactory verbal description. If interpretability is about human understanding, and understanding can exist without verbalization, then explanation-based evaluation systematically underestimates the interpretability of latents whose meaning is intuitive but hard to articulate. The authors state their alternative definition clearly:

"We deem a latent interpretable if a human can intuitively understand its meaning, and thereby distinguish activating from non-activating examples."

This shifts the evaluation target from "can we describe this latent?" to "can we recognize this latent?"—a behavioral test of understanding rather than a linguistic one.

Computational cost. Explanation-based evaluation is expensive. Generating explanations typically requires collecting activations over a large corpus, presenting them to a capable LLM (often a frontier model), and then running a separate evaluation phase where another LLM uses the explanation to predict activations. Simulation scoring, the standard approach introduced by Bills et al. (2023), is "computationally expensive and requires fairly heavyweight language models to produce meaningful results" (Section 2.2). For researchers iterating on SAE architectures and training procedures, this cost is a practical bottleneck. A faster evaluation method—even one that is somewhat less faithful than LLM-based approaches—would enable more rapid experimentation cycles. This motivates the paper's exploration of example embedding scoring, which uses a 22M-parameter embedding model rather than a large language model.

Prior Approaches and Their Limitations

The paper surveys the existing evaluation landscape and identifies specific shortcomings in each approach.

Explanation-based simulation (Bills et al., 2023). The dominant paradigm works as follows: for each latent, collect contexts where it activates strongly, present these to an LLM to generate a natural language explanation, then test the explanation by having another LLM predict the latent's activation strength on new contexts given only the explanation. This approach has been widely adopted (Templeton et al., 2024; Paulo et al., 2024) and extended to causal evaluation settings where the model intervenes on latent activations and measures whether the explanation predicts the intervention effect (Paulo et al., 2024; Gur-Arieh et al., 2025). While influential, this approach conflates explanation quality with latent interpretability and introduces hyperparameter sensitivity at every stage—how examples are sampled (stratified by quantile? how many?), how explanations are prompted, how the simulation task is structured, and which LLM performs the evaluation.

Simplified classification tasks. In response to the computational cost of full simulation, researchers developed simpler proxies: detection (predict whether a given latent activates anywhere in a passage), fuzzing (predict whether a specific highlighted token activates the latent), and rubric scoring (rate the relevance of an explanation to observed activations). These reduce the evaluation burden but retain the fundamental dependence on generated explanations. They are, as the authors describe them, "more suitable for human evaluation due to their simplicity" (Section 2.2), but they do not escape the explanation step.

Concept bottleneck approaches. Paulo & Belrose (2025) attempted to replace SAE latents with human-predicted activations during a forward pass, measuring how much model performance degrades. This approach directly tests whether latent meanings are sufficiently well-understood to substitute for the encoder, but the results were "mostly negative"—humans could not predict activations accurately enough to maintain model performance. This finding underscores the difficulty of the evaluation problem and the gap between "interpretable" and "predictable from verbal descriptions."

Pre-neural interpretability evaluation. The paper draws on two established methods from the pre-LLM interpretability literature that evaluated representations without requiring explanations, repurposing them for SAEs. The word intrusion task (Chang et al., 2009) asked human participants to identify an out-of-place word within a set associated with a topic model topic; high accuracy indicated that the topic had a coherent, interpretable meaning. Subramanian et al. (2018) adapted this for sparse word embeddings. The two-alternative forced choice task (Borowski et al., 2020; Zimmermann et al., 2023) showed human participants pairs of images where one strongly activated a CNN feature map and the other did not; participants identified which was which, with accuracy reflecting the feature's human-interpretable coherence. These methods share a key property: they test behavioral understanding (can you distinguish activating from non-activating examples?) rather than verbal understanding (can you describe the feature?). The paper explicitly builds on this intellectual lineage, adapting both methods to the language model SAE setting.

How This Paper Positions Itself

The paper positions its contribution as methodological infrastructure for the SAE evaluation ecosystem. It does not claim that explanation-based evaluation should be abandoned—explanations serve genuine purposes, such as enabling humans to reason about model behavior and communicate findings. Rather, it argues that the field needs complementary methods that measure interpretability directly, without the explanation bottleneck, for use in benchmarking, architecture comparison, and rapid iteration.

The two proposed methods fill different niches. Intruder detection is designed to be the high-fidelity standard, correlating strongly with human judgment (r = 0.85 with Claude Sonnet 3.5) while eliminating the explanation step. It can be run with either human or LLM evaluators, with the LLM version serving as a scalable, automated proxy for human evaluation. Example embedding scoring is designed for speed—it uses a 22M-parameter embedding model and can be computed over many latents quickly—at the cost of lower correlation with human judgment (r = 0.48). This speed-accuracy tradeoff is deliberate and pragmatically motivated: not every use case requires human-level fidelity, and rapid feedback during SAE development may be worth the loss in measurement precision.

The paper also positions itself as contributing evidence about the relationship between activation strength and interpretability. By evaluating latents separately for each decile of the activation distribution, both methods provide a richer picture than aggregate scores: they confirm that higher-activating examples are systematically more interpretable (consistent with Templeton et al., 2024 and Paulo et al., 2024), but also show that for the most interpretable latents, even low-activation examples remain distinguishable from non-activating ones (Figure A1). This granularity is difficult to achieve with explanation-based methods, which typically require generating and evaluating separate explanations for each activation regime.

Finally, the paper explicitly addresses the validity of LLMs as evaluators. By comparing LLM intruder detection accuracy against human accuracy on the same latents and finding strong correlation (0.85 Spearman), the authors provide evidence that the evaluation pipeline can be fully automated without losing fidelity to human judgment—a claim that prior SAE evaluation work did not support with direct human baselines on identical tasks. The finding that "the human mostly finds latents more interpretable than their LLM counterparts" is taken as good news, since it suggests LLMs are not picking up on spurious patterns invisible to humans, but rather are conservative estimators of human-interpretable structure.

3. Technical Approach

3.1 Reader Orientation

This paper develops two evaluation protocols—intruder detection and example embedding scoring—that assess how interpretable a sparse autoencoder (SAE) latent is by directly testing whether its activation pattern is coherent enough to distinguish activating from non-activating text examples, rather than going through the intermediate step of generating and evaluating a natural language explanation. The system solves the problem of measurement confound in SAE evaluation: when explanation generation is part of the evaluation pipeline, low scores could mean either the latent is genuinely uninterpretable or the explanation generator simply failed to verbalize something a human could intuitively grasp; by eliminating the explanation step, the proposed methods isolate latent interpretability as the measured quantity.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three major stages, shared across both evaluation methods (see Figure 1 for a visual overview):

Activation Collection — For each SAE latent being evaluated, the system runs the base language model (SmolLM2 135M) over a corpus of text and records which tokens trigger the latent above a threshold, producing a distribution of activation strengths across contexts. This stage produces a pool of "activating examples" (text spans where the latent fires strongly) and "non-activating examples" (text spans where the latent does not fire, but other latents do).
Example Sampling — Given a specific latent to evaluate, the system samples a fixed number of activating examples from a chosen decile of the activation distribution (controlling for activation strength) and a fixed number of non-activating examples to serve as distractors. Activating tokens within examples are explicitly highlighted with << and >> delimiters to direct the evaluator's attention.
Interpretability Scoring — The sampled examples are presented to an evaluator (human, LLM, or embedding model) through a structured task that measures whether activating and non-activating examples are distinguishable. The evaluator's accuracy or discrimination score on this task becomes the interpretability metric for that latent. The two evaluation methods differ only in how the task is structured and what kind of evaluator is used.

Information flows as follows: SAE latent activations over a corpus → stratified sampling by activation decile → task construction (intruder detection or embedding comparison) → evaluator judgment → aggregated interpretability score per latent.

3.3 Roadmap for the Deep Dive

First, the SAE architecture and training setup, since the four SAEs evaluated in the paper (layers 9, 15, 21, 27 of SmolLM2 135M) are the objects being measured, and their architectural choices (TopK sparsity, k=32, no auxiliary loss) determine what "a latent" means in this context.
Second, the activation collection and example sampling pipeline, since both evaluation methods depend on the same procedure for constructing sets of activating and non-activating examples, and the choice of activation decile as the stratification variable is central to the paper's claims about interpretability varying with activation strength.
Third, the intruder detection method in full detail—its task structure, what the evaluator sees, how accuracy is computed, and why this particular design was chosen over alternatives like word-level intrusion or explanation-based classification.
Fourth, the example embedding scoring method—its mathematical formulation, how it differs from intruder detection in what it measures, and the deliberate speed-accuracy tradeoff it represents.
Fifth, the human evaluation protocol and cross-validation design, since the paper's central validity claim is that LLM scores correlate with human scores, and this requires careful construction of the human baseline task to match what the LLM sees.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a measurement methodology paper whose core idea is that latent interpretability can be directly assessed by testing whether activating and non-activating examples are behaviorally distinguishable, without requiring natural language explanations as an intermediate representation.

SAE Architecture and Training Setup

The four sparse autoencoders evaluated in this paper are trained on the MLP outputs of SmolLM2 135M (Allal et al., 2025), a small transformer language model with 135 million parameters. The SAEs are placed at layers 9, 15, 21, and 27 of the base model, providing coverage across the depth of the network. Each SAE takes as input the output of the MLP sublayer at its respective layer and attempts to reconstruct it through a sparse bottleneck.

TopK activation function. The SAEs use the TopK activation function introduced by Gao et al. (2024). Rather than applying an L1 sparsity penalty to the latent activations during training—which requires tuning a penalty coefficient $\lambda$ that trades off reconstruction quality against sparsity, often resulting in dead latents or poor reconstruction if not carefully calibrated—TopK directly enforces a hard sparsity constraint: for each input, only the $k$ latents with the highest activation values are retained, and all others are set to exactly zero. The paper uses $k = 32$ , meaning that for any given MLP output vector being encoded, exactly 32 latents (out of the total number available in the overcomplete dictionary) are active.

This choice matters for evaluation because it determines the nature of sparsity in the latent representations: with TopK, every input produces exactly $k$ active latents, and the activation values of those $k$ latents come from the raw pre-activation scores before the TopK selection (not from a soft-thresholding operation like ReLU). This means that "activation strength" for a latent refers to its pre-selection score conditional on being in the top $k$ , and the distribution of activation strengths across examples will reflect both the latent's typical firing magnitude and its competition with other latents for inclusion in the top $k$ .

Training objective and hyperparameters. The SAEs are trained to minimize the mean squared error (MSE) between their output reconstruction and the original MLP output, with no auxiliary loss terms—unlike many SAE training setups that add an L1 penalty on latent activations or an auxiliary loss to encourage specific properties. The specific hyperparameters reported are:

Optimizer: Adam (Kingma & Ba, 2017)
Sequence length: 2049 tokens
Batch size: 64 sequences
Training data: 10 billion tokens sampled from a reconstruction of the SmolLM2 training corpus

Training on 10 billion tokens is substantial relative to the model size (roughly 74,000 tokens per parameter), which means the SAEs have seen enough data diversity that their latents should have converged to stable, data-driven features rather than being artifacts of limited training.

Why this SAE setup matters for interpretability evaluation. The authors deliberately choose a standard, well-understood SAE configuration—not a novel variant—because the goal is to evaluate the measurement method, not to claim that any particular SAE architecture is more interpretable. The TopK activation function with $k = 32 ensures that sparsity is consistent across all inputs, which means differences in interpretability scores across latents cannot be attributed to some latents being evaluated in a fundamentally different sparsity regime than others. Training on a reconstruction of SmolLM2's training corpus also means the SAEs are evaluating in-distribution latents, not latents that might behave erratically on out-of-distribution text.

Activation Collection and Example Sampling Pipeline

Before any evaluation can occur, the system needs a database of which text spans activate each latent and how strongly. This pipeline is shared across both intruder detection and example embedding scoring.

Collecting activations. The base model (SmolLM2 135M) is run over a corpus of text, and for each token position in each sequence, the output of the MLP sublayer at the target layer is computed. The SAE encoder processes this MLP output, producing an activation value for each latent in the dictionary. For the TopK SAE, only the top $k=32$ latents receive non-zero activations; all others are implicitly zero.

For each latent, the system records every token-level activation across the entire corpus, along with the surrounding context (the text window around the activating token). This produces, for each latent, a large set of (context, activation_strength) pairs where the latent fired, and an even larger set of (context, 0) pairs where it did not fire (or, more precisely, was not in the top $k$ —since a latent could have a non-zero pre-activation value but still be zeroed out by TopK if 32 other latents scored higher).

Decile-based stratification. Rather than sampling activating examples uniformly from all activation strengths, the paper stratifies by activation decile. For each latent, the system sorts all recorded activations by strength, divides them into ten equally sized bins (deciles), and samples activating examples from a specific decile. The decile is a design choice, not a property of the data: decile 1 contains the weakest 10% of activations for that latent, decile 10 contains the strongest 10%.

This stratification is methodologically critical because it enables the paper to study interpretability as a function of activation strength—a question that explanation-based methods struggle to address, since they would require generating separate explanations for each decile. By evaluating each latent separately for each decile, the paper can answer questions like "are low-activation examples still interpretable?" and "does the meaning of a latent change qualitatively as activation strength increases?"

Sampling procedure for a single evaluation. For a given latent and a given decile, the evaluation procedure constructs a set of examples as follows:

Select the target decile. The system chooses which decile of the activation distribution to sample from. In the paper's standard setup, all activating examples come from the same decile, ensuring that the evaluator sees examples of roughly uniform activation strength.
Sample four activating examples. Four text spans are randomly drawn from the pool of contexts in the chosen decile. Each example is a contiguous text window of 32 tokens centered on (or near) an activating token. The activating token(s) within the example are highlighted using << and >> delimiters. The number of highlighted tokens in each example is not necessarily one—if the latent activates on multiple adjacent tokens or on a multi-token expression, all activating tokens are highlighted. The 32-token context length is chosen to "maintain consistency" across examples and to provide enough surrounding context for the evaluator to understand what the latent is responding to, without making the example so long that the activating signal is diluted.
Sample one non-activating (intruder) example. A single text span is drawn from a pool of examples that do not activate the latent being evaluated but do activate other latents. This is a deliberate design choice: the intruder is not a random sentence from the corpus, but rather a sentence that is "interesting" to the model (it triggers some latent) but is not in the activation set for the specific latent under evaluation. This makes the task harder than using completely random sentences as intruders, because the evaluator cannot simply look for "sentences the model doesn't care about"—both activating and non-activating examples contain model activations, just for different latents.
Handle token highlighting in the intruder. Since activating examples have highlighted tokens (the <<token>> markers) and the intruder has no genuine activating tokens, the system highlights a random selection of tokens in the intruder example. The number of highlighted tokens matches the average count from the activating examples, rounded down. This prevents the evaluator from trivially identifying the intruder by counting highlighted tokens (e.g., "every example has exactly one highlighted token except this one, which has none").
Assemble the prompt. The five examples (four activating, one intruder) are presented as a numbered list (typically 1 through 5) with their highlighted tokens. The ordering of the five examples is randomized, so the intruder can appear in any position. Each evaluation prompt is independent—the evaluator sees exactly five examples and must identify the one that does not belong.

Why this sampling design? Several choices are motivated by specific concerns:

32-token windows balance sufficient context for interpretation against the risk of including irrelevant text that might distract or confuse the evaluator. The choice is empirical and not extensively ablated in the paper, but the principle is that too little context makes the task impossible (the evaluator can't tell what a latent responds to from a single word) while too much context makes the activating token hard to find.
Non-activating examples that activate other latents (rather than random text) raise the difficulty floor. If the intruder were random text, the evaluator could solve the task by noticing that four sentences have model-interpretable content and one is gibberish or irrelevant. By making all five sentences "model-relevant," the evaluator must attend specifically to the latent's activation pattern.
Matching highlighted token counts removes a superficial cue that would otherwise make the task trivially easy and not a test of latent understanding.
Sampling from a single decile controls for activation strength as a confound. If activating examples were drawn from different deciles, the evaluator might learn to distinguish examples based on some property of the text that correlates with activation strength (e.g., very high-activation examples might all be about the same narrow topic) rather than based on the latent's actual semantic content.

Intruder Detection Method

Intruder detection is the paper's primary proposed method. It adapts the word intrusion task from the topic modeling literature (Chang et al., 2009) and the sentence-level intrusion approach used by Klindt et al. (2025) to the SAE setting, with modifications to handle the fact that SAE latents often activate on spans of multiple tokens rather than individual words.

Task structure. The evaluator (human or LLM) is presented with a numbered list of five text examples, each 32 tokens long with certain tokens highlighted by << and >> delimiters. Four of the examples contain text that activates the latent being evaluated (drawn from the chosen activation decile); one example (the intruder) does not activate the latent at all. The evaluator must identify the index (1 through 5) of the intruder example.

For LLM evaluators, the prompt includes a few-shot demonstration showing synthetic examples of the task format, so the model understands what is being asked without needing the concept of "SAE latent activations" explained in detail. The few-shot examples are synthetic (not drawn from real SAE latents) to avoid biasing the evaluator toward specific types of latents or text patterns.

Accuracy metric. For a given latent and a given decile, the system constructs multiple intruder detection prompts (each with a different random draw of four activating examples and one intruder from the non-activating pool). The evaluator's accuracy is the fraction of these prompts where it correctly identifies the intruder's index. The paper's standard setup uses:

LLM evaluation: 50 prompts per latent per decile (100 prompts per latent total when averaging across deciles—though the exact number varies by experiment)
Human evaluation: 8–20 prompts per latent, due to the cost and time constraints of human annotation

Aggregation across deciles. The "overall" interpretability score for a latent is the accuracy averaged across all ten deciles. The paper also reports per-decile accuracies to study how interpretability varies with activation strength.

Why intruder detection rather than binary classification? An alternative design would be to present the evaluator with single examples and ask "does this activate the latent, yes or no?" This would be a binary classification task. Intruder detection is operationally a five-way forced choice, which has several advantages for this use case:

Relative judgment is easier than absolute judgment. A human might struggle to say "this sentence feels like it should activate latent 427" in the abstract, but when shown four sentences that do activate latent 427 and one that doesn't, the task becomes "which one of these is not like the others"—a relative comparison that humans are generally good at. This reduces the burden on evaluators and increases the signal-to-noise ratio.
The forced-choice format eliminates threshold calibration. In binary classification, the evaluator must set an internal threshold for "activating enough to say yes." Different evaluators (or the same evaluator on different days) might have different thresholds. The forced-choice format removes this degree of freedom: the evaluator must pick exactly one intruder from five options, and there is always exactly one correct answer.
Guessing baseline is known and interpretable. Random guessing yields 20% accuracy on a five-way choice. This provides a clean, unambiguous baseline: any latent with accuracy at or below 20% is "clearly uninterpretable" by the paper's definition. The paper bins latents into interpretability levels based on their distance from this baseline: 20–40% (low), 40–60% (moderate), 60–80% (high), 80–100% (very high).
It matches the human evaluation paradigm from prior work. The word intrusion task (Chang et al., 2009) used exactly this forced-choice structure to evaluate topic model interpretability, and its long history of successful use with human participants provides a methodological template.

What the evaluator must learn. To succeed at intruder detection, the evaluator must infer the latent's activation pattern from the four activating examples and then recognize that the fifth example does not fit that pattern. This does not require the evaluator to articulate the latent's meaning in words—it only requires being able to recognize the pattern and detect deviations from it. This is the key departure from explanation-based evaluation: the evaluator is tested on discrimination, not verbalization.

Relationship to prior SAE evaluation methods. The paper explicitly compares intruder detection to existing simplified classification tasks (Section 2.2, Table 2 correlations). In detection, the evaluator predicts whether a given latent activates anywhere in a passage (a binary decision made without seeing other examples for comparison). In fuzzing, the evaluator predicts whether a specific highlighted token activates the latent (a more localized binary decision). Intruder detection differs from both by being a relative comparison task with a forced choice among multiple alternatives, which the paper argues is better suited to capturing the kind of intuitive understanding that constitutes interpretability.

Example Embedding Scoring Method

Example embedding scoring is the paper's lightweight, fast alternative to intruder detection. It is inspired by the two-alternative forced choice task from the vision interpretability literature (Borowski et al., 2020; Zimmermann et al., 2024), but adapted to work with text embeddings rather than image presentation to human participants.

Core idea. Rather than asking an evaluator to explicitly judge whether examples are activating or non-activating, example embedding scoring measures whether activating examples naturally cluster together in a general-purpose text embedding space, such that they are closer to each other than they are to non-activating examples. If a latent represents a coherent, interpretable concept, then the sentences that activate it should be semantically similar to each other and semantically different from sentences that do not activate it—and an off-the-shelf embedding model should be able to detect this similarity structure without being specifically trained on the SAE's latents.

Mathematical formulation. The method computes two delta scores for each latent—one for the positive (activating) class and one for the negative (non-activating) class—and then aggregates them into an AUROC (Area Under the Receiver Operating Characteristic) curve.

For a given latent and a chosen activation decile, the system constructs four sets of examples:

$E^+ = \{e^+_1, e^+_2, \ldots, e^+_n\}$ : a set of $n$ activating examples sampled from the target decile
$E^- = \{e^-_1, e^-_2, \ldots, e^-_n\}$ : a set of $n$ non-activating examples (sentences where the latent does not fire)
$q^+$ : a single "positive query" example, also drawn from the activating pool
$q^-$ : a single "negative query" example, drawn from the non-activating pool

All examples are embedded using a sentence embedding model—specifically, all-MiniLM-L6-v2, a 22-million-parameter transformer that maps variable-length text to a 384-dimensional dense vector. This model was chosen for its small size (enabling fast computation) and its general-purpose semantic similarity capabilities.

For each example, the embedding is a single vector representing the entire 32-token sentence. The system then computes the average cosine similarity between the positive query and all examples in the positive set, minus the average cosine similarity between the positive query and all examples in the negative set:

$\Delta^+ = \frac{1}{n} \left( \sum_{e^+_i \in E^+} \frac{q^+ \cdot e^+_i}{\|q^+\| \|e^+_i\|} - \sum_{e^-_i \in E^-} \frac{q^+ \cdot e^-_i}{\|q^+\| \|e^-_i\|} \right)$

where $q^+$ is the embedded positive query, $e^+_i$ are the embedded activating examples, $e^-_i$ are the embedded non-activating examples, $\cdot$ denotes dot product, $\|\cdot\|$ denotes Euclidean norm, and $n$ is the number of examples in each set.

Similarly, the negative delta is:

$\Delta^- = \frac{1}{n} \left( \sum_{e^-_i \in E^-} \frac{q^- \cdot e^-_i}{\|q^-\| \|e^-_i\|} - \sum_{e^+_i \in E^+} \frac{q^- \cdot e^+_i}{\|q^-\| \|e^+_i\|} \right)$

where $q^-$ is the embedded negative query.

What these equations compute, operationally. For $\Delta^+$ : take an activating query sentence, compute its cosine similarity to every other activating sentence in the set (should be high if the latent captures a coherent concept), subtract its average similarity to all non-activating sentences (should be lower if the latent distinguishes meaningful differences), and average the difference. A large positive $\Delta^+$ means the query is substantially closer to its own class than to the opposite class. The $\Delta^-$ computation does the reverse: take a non-activating query, measure how much closer it is to other non-activating examples than to activating ones.

Why cosine similarity in this form? Cosine similarity ( $a \cdot b / (\|a\|\|b\|)$ ) normalizes for embedding magnitude, so the score captures only directional similarity in the embedding space. This matters because sentence embedding models can produce vectors of different lengths depending on sentence length or content, and we want similarity judgments to be about content, not vector norm artifacts. The difference-of-means formulation ( $\frac{1}{n}\sum \text{sim to same class} - \frac{1}{n}\sum \text{sim to opposite class}$ ) is a simple, interpretable measure of class separation: zero means the query is equidistant to both classes on average (random structure), positive means it leans toward its own class.

AUROC scoring. Rather than reporting $\Delta^+$ and $\Delta^-$ directly, the method computes an AUROC by treating the cosine similarities as prediction scores. For each example in the combined pool of activating and non-activating examples, the "prediction score" for it being activating is its cosine similarity to the positive query, and the "prediction score" for it being non-activating is its cosine similarity to the negative query. The AUROC measures how well these scores separate the two classes across multiple random draws of query and reference examples. An AUROC of 0.5 means the embedding model's similarity rankings are no better than random; an AUROC of 1.0 would mean perfect separation.

The paper iterates over different choices of positive and negative queries and averages the AUROC across iterations to reduce variance from any single query being unrepresentative.

Why this method is fast. The all-MiniLM-L6-v2 model has only 22 million parameters, making it roughly 6,000 times smaller than the LLMs used for intruder detection (e.g., Llama 3.1 70B, Claude Sonnet 3.5). Embedding a 32-token sentence takes on the order of milliseconds on standard hardware, and the cosine similarity computation is a simple dot product. Evaluating thousands of latents becomes feasible in minutes rather than hours. The tradeoff, as the results show, is lower fidelity to human interpretability judgments.

Key limitation of example embedding scoring. The embedding model operates on whole-sentence embeddings, not on token-level representations. This means it has difficulty attending specifically to the highlighted tokens (the <<token>> markers) within the sentence. The paper hypothesizes that this is the primary reason for the method's modest correlation with human scores:

"Example embedding is a difficult task for an off-the-shelf embedding model, since it is unlikely to understand what our highlight tokens << and >> mean."

If a latent activates on a specific word or short phrase (e.g., the word "democracy"), but the surrounding sentence context varies widely across examples (one sentence is about elections, another about ancient Greece, another about workplace decision-making), a whole-sentence embedding will see these as semantically diverse and will not detect the clustering around the activating token. An LLM evaluator sees the highlighted tokens explicitly and can focus on them; the embedding model sees only the undifferentiated text.

This limitation explains the pattern in the results: example embedding scoring performs best on the highest activation decile (AUROC 0.64–0.70), where the activating signal is strongest and the surrounding context is most likely to be topically consistent with the latent's meaning, and performs near random on lower deciles where the activating token signal is weaker relative to the varied context.

Human Evaluation Protocol

A central claim of the paper is that LLM evaluators are valid proxies for human judgment of latent interpretability. To establish this, the authors need two things: (1) human intruder detection scores on a set of latents, and (2) LLM scores on the same latents using the same task format. The human evaluation protocol is designed to match the LLM protocol as closely as possible while accommodating the practical constraints of human annotation.

Latent selection. The human evaluation covers 56 latents drawn from the four SAEs (layers 9, 15, 21, 27), with approximately 14 latents per layer. The paper does not specify the exact sampling procedure for selecting which latents to evaluate, only that they are drawn from the SAEs being studied. The sample size of 56 is a practical limitation acknowledged implicitly by the variable number of prompts per latent (8–20 rather than the 50–100 used for LLMs).

Task presentation to humans. The human participants (the authors themselves—the paper states "the authors manually performed the intruder task") are shown five text examples per prompt, with tokens highlighted via << and >> delimiters, exactly as the LLM sees them. Each example is 32 tokens long. The human must select which of the five is the intruder by index (1 through 5).

Controlling for learning effects across prompts. Unlike LLMs, which process each prompt in an isolated context window with no memory of previous prompts, human participants see a sequence of prompts and may unconsciously or consciously learn patterns across them. The paper attempts to mitigate this:

"In LLMs, each intruder test is done in a fresh context, and so there is no information that carries from one set of examples to the next. The same is not true for the human participants. To minimize this effect, we randomly select which latent to sample the examples from, such that there aren't many consecutive examples of the same latent."

By randomizing the order of latents presented—so that, for example, the human does not see ten consecutive prompts for latent 427 and gradually learn its pattern—the design reduces (but does not eliminate) the advantage that humans might have from accumulating information across prompts. A related choice: the decile from which examples are drawn is also randomized and not shown to the participant, so the human cannot use knowledge of "this is a low-decile prompt" to adjust their strategy.

Number of prompts per latent. Each latent receives 10–20 prompts in the human evaluation, compared to 50–100 prompts in the LLM evaluation. This means the human score per latent is a noisier estimate of the "true" interpretability than the LLM score. The paper represents this uncertainty visually in Figure 2 by varying the size of each dot according to the number of prompts the human saw for that latent—smaller dots represent fewer prompts and thus greater measurement uncertainty.

Inter-human agreement. To assess the reliability of human judgments themselves, the paper reports inter-human agreement on a subset of 40 latents where two human labelers independently performed the intruder task. Each of these 40 latents received only 3–5 prompts per labeler (an even noisier estimate per latent), but across latents, the Spearman correlation between the two labelers' accuracy scores is 0.87. This is higher than the correlation between either human and any LLM, which the authors interpret as evidence that (a) the intruder detection task elicits consistent human judgments, and (b) there is room for LLMs to improve as evaluators before reaching the human inter-rater ceiling.

One human participant sees variable amounts of data. The paper notes that "each latent was evaluated in 10-20 prompts by the human, and on 100 prompts by the LLMs" (Table 1 caption). The human data is thinner, which means the correlation numbers (while high) should be interpreted as lower-bound estimates of the true human-LLM agreement: if humans could be evaluated on 100 prompts per latent, their scores would be more stable, and the correlation with LLM scores might increase further.

Cross-Validation and Evaluation Design Choices

Several design choices in the evaluation protocol are motivated by the need for fair comparison across methods and meaningful interpretation of scores.

Decile as the independent variable. By evaluating each latent separately for each of the ten activation deciles, the paper can study interpretability as a function of activation strength without the confound of different latents having different overall activation distributions. A latent that fires on a wide variety of contexts might have a different decile profile than a latent that fires on a narrow set of contexts, and per-decile evaluation teases this apart.

Intruder decile detection as a probe for latent semantics. The paper extends the intruder detection paradigm to study intra-latent structure: rather than using non-activating examples as intruders, the modified task uses examples that all activate the same latent, but one example is drawn from a different activation decile than the other four. This allows the paper to ask: does the latent's meaning change qualitatively with activation strength, or is activation strength merely a quantitative modulation of the same underlying feature?

If a latent represents a perfectly monosemantic binary feature (the thing is either present or not), then examples from different deciles should be semantically indistinguishable—the latent fires when the feature is present, and the activation strength reflects something orthogonal to meaning (like noise or competing latents). In this case, intruder decile detection accuracy should be close to random. If a latent represents a scalar feature with degrees of intensity (e.g., "amount of formal language" or "strength of positive sentiment"), then examples from distant deciles should be distinguishable—very high-activation examples should look qualitatively different from very low-activation ones. The accuracy matrix in Figure 3 (right panel) maps out this structure: rows are the majority decile (where four examples come from), columns are the intruder decile (where the odd-one-out comes from), and cell values are detection accuracy.

Random guessing baseline and interpretability bins. The paper operationalizes "interpretable" via distance from the 20% random baseline. Latents are binned into five categories: below 20% (clearly uninterpretable—worse than random), 20–40% (low interpretability), 40–60% (moderate), 60–80% (high), and 80–100% (very high). This binning serves two purposes: it simplifies the comparison between human and LLM scores (right panel of Figure 2 shows a confusion matrix of binned judgments), and it provides an interpretable taxonomy for discussing results ("one third of latents had a score of over 80%").

Evaluator selection for LLM-based intruder detection. The paper tests multiple LLMs as evaluators: Claude Sonnet 3.5, Llama 3.1 70B, Llama 3.1 8B, QwQ 32B, and Gemini Flash 2.0 (Table 1). This is not an exhaustive survey of available models but covers a range of model sizes (8B to 70B+ parameters) and model families (Llama, Claude, Gemini, Qwen). The inclusion of a small model (Llama 3.1 8B) is particularly informative: it achieves only 27% accuracy on the intruder task (barely above the 20% random baseline) and has the lowest correlation with human scores (0.64), establishing a lower bound on the model capability needed for meaningful evaluation.

Why this particular set of SAEs? The paper evaluates on four SAEs trained on SmolLM2 135M MLP outputs at layers 9, 15, 21, and 27. These layers are roughly evenly spaced through the model's depth (SmolLM2 135M has approximately 30 layers, though the exact number is not specified in this paper). The choice provides coverage of early, middle, and late representations, but the paper does not focus on layer-wise comparisons—the primary contribution is the evaluation methodology, not insights about how interpretability varies with layer depth.

Relationship between the two methods. Intruder detection and example embedding scoring are presented as complementary, not competing. Intruder detection is the high-fidelity method—it correlates strongly with human judgment and can be used as the primary metric in careful evaluations. Example embedding scoring is the fast-screening method—it can be used during iterative SAE development when many configurations need quick feedback, with the understanding that its scores are noisier and less aligned with human interpretability. The paper recommends neither as a replacement for the other, but rather offers both as tools for different stages of the SAE development pipeline.

Why This Approach Over Alternatives

The paper's decision to eliminate explanations is not merely a matter of convenience—it reflects a specific philosophical stance about what interpretability means and how it should be measured.

Against explanation dependence. The dominant paradigm (Bills et al., 2023; Templeton et al., 2024; Paulo et al., 2024) assumes that an interpretable latent is one for which a good explanation can be generated and validated. The paper argues this conflates two separate properties: (1) the latent's intrinsic interpretability, and (2) the ability of an LLM to verbalize that interpretability. An explanation-based evaluation can produce a low score either because the latent is genuinely uninterpretable (the "correct" low score) or because the explanation generator failed to articulate something a human could intuitively understand (a "false" low score caused by the explanation pipeline). By removing explanations, the proposed methods measure only property (1).

Against concept bottleneck approaches. Paulo & Belrose (2025) attempted to directly replace SAE activations with human-predicted activations based on explanations, finding mostly negative results. The paper implicitly argues that this is too high a bar: a latent can be interpretable (a human can distinguish activating from non-activating examples) without being so precisely predictable that the model's forward pass is unaffected. The intruder detection task sets a lower, more achievable standard for interpretability—discrimination rather than reconstruction.

Against word-level intrusion. The original word intrusion task (Chang et al., 2009) presented lists of individual words and asked humans to identify the out-of-place word. The paper departs from this by showing full 32-token contexts because "a significant number of latents activate on more than one token in a context, sometimes on several adjacent tokens." A word-level task would fail to capture multi-token features (phrases, syntactic constructions, discourse-level patterns) and would lose the contextual information needed to interpret the latent's meaning. The sentence-level design is a necessary adaptation from topic models (where topics are distributions over individual words) to SAEs (where latents can respond to multi-word expressions, grammatical structures, or semantic relationships spanning multiple tokens).

4. Key Insights and Innovations

Innovation 1: Interpretability Can Be Measured Directly as Behavioral Discrimination, Without the Verbal Bottleneck

The paper's most fundamental conceptual move is to redefine the operationalization of interpretability from "can an LLM generate a good natural language explanation for this latent?" to "can an evaluator distinguish activating from non-activating examples for this latent?" This shifts the measurement target from a linguistic artifact (the explanation) to a behavioral capacity (the discrimination), breaking a tight coupling that the field has largely taken for granted since Bills et al. (2023).

What makes this more than a methodological convenience is the philosophical claim it encodes: that interpretability and verbalizability are separable properties. The authors state this explicitly—"we deem a latent interpretable if a human can intuitively understand its meaning, and thereby distinguish activating from non-activating examples"—and then build an entire evaluation framework around it. Prior work implicitly equated the two: Templeton et al. (2024) and Paulo et al. (2024) treat explanation quality scores as direct proxies for latent quality, assuming that a latent worth understanding is one that can be summarized in a sentence. This paper argues that equation is unwarranted. A latent could be genuinely interpretable—a human could learn to recognize when it fires—while resisting concise verbal description, perhaps because it captures a gestalt pattern, a fine-grained syntactic construction, or a contextual nuance that language handles poorly.

The practical consequence of this reframing is the elimination of an entire class of nuisance variables from the evaluation pipeline. Every explanation-based method introduces hyperparameters (how many examples to show the explainer model, which activation quantiles to sample from, what prompt template to use, which LLM to use for explanation generation, how to structure the simulation or detection task) that have nothing to do with the latent's intrinsic interpretability but that demonstrably affect the final score. By removing explanation generation entirely, the proposed methods isolate the signal of interest—"is this latent interpretable?"—from the noise of explanation engineering. This is a measurement improvement analogous to replacing an indirect assay with a direct one in experimental science: fewer confounds, cleaner causal attribution.

The paper does not argue that explanations are useless—they serve communicative purposes and enable downstream reasoning about model behavior that discrimination tasks alone cannot support. Rather, it argues that explanation-based evaluation conflates two questions (Is the latent interpretable? Can it be explained well?) that should be answered separately, and it provides tools for answering the first question cleanly. This is a foundational reframing of the evaluation problem, not merely a new metric. It changes what "interpretability" means operationally for the field.

The evidence that this reframing is productive comes from the strong human-LLM correlation (Spearman r = 0.85, Figure 2 left panel) combined with the observation that "the human mostly finds latents more interpretable than their LLM counterparts" (Section 4.1). This asymmetry is diagnostic: if explanation-based evaluation underestimated interpretability due to the verbalization bottleneck, we would expect exactly this pattern—humans perform better at the discrimination task than LLMs do, even though both are seeing identical input. The paper doesn't prove this causal chain, but the pattern is precisely what the reframing predicts, lending it credibility as an explanation for the field's inconsistent interpretability findings.

Innovation 2: Activation Strength as a Dimension of Interpretability, Not a Sampling Artifact

The paper's per-decile evaluation—computing separate interpretability scores for each of the ten activation deciles within a single latent—transforms activation strength from a sampling nuisance into a first-class object of study. Prior work treated activation strength implicitly as a confound to control for (Templeton et al., 2024 sample from "high-activating quantiles" to get clean examples; Paulo et al., 2024 mention stratified sampling from quantiles). This paper actively investigates how interpretability varies as a function of activation strength, making the relationship itself a diagnostic signal about latent structure.

The conceptual contribution is to recognize that the activation distribution of a latent contains information about what kind of feature it represents, and that different kinds of features should exhibit different interpretability-by-decile profiles. A perfectly monosemantic binary feature—something is either present or not—should show roughly equal interpretability across all deciles, because activation strength is just noise around the binary signal. A scalar feature with genuine degrees of intensity should show decreasing interpretability at lower deciles, because weakly-activating examples are genuinely intermediate cases that are harder to distinguish from non-activating ones. A mixed or polysemantic feature might show qualitatively different meanings at different activation strengths, producing different intruder patterns at different deciles.

What makes this innovation significant is that it provides a mechanism for diagnosing latent semantics without generating explanations. The intruder decile detection experiment (Figure 3, right panel) operationalizes this diagnosis directly: by testing whether examples from different deciles of the same latent are distinguishable from each other, it reveals whether the latent captures a graded or binary concept, and whether its meaning shifts qualitatively with intensity. The asymmetric accuracy matrix—where Llama 3.1 70B achieves 51% accuracy at distinguishing low-decile intruders among high-decile examples but essentially chance accuracy in the reverse direction—is not just a measurement artifact; it is a finding about the structure of SAE latents that was invisible to prior methods. It tells us that, for most latents, high-activation examples are more homogeneous and recognizable than low-activation examples, and that the relationship between activation strength and semantic content is not symmetric. This asymmetry would have been impossible to detect with aggregate interpretability scores or with explanation-based methods that treat activation strength as sampling parameter rather than an object of investigation.

The finding that "the most interpretable latents remain interpretable even in the lower activation deciles, with accuracies greater than 0.75" (Figure A1) further refines the picture: it suggests that there is a class of genuinely coherent features whose meaning is stable across the full activation range, and that these features can be identified by their flat interpretability-by-decile profile. This is a diagnostic concept—the "interpretability profile" of a latent—that the paper introduces and demonstrates but does not fully explore, leaving it as a tool for future SAE development and comparison.

Innovation 3: LLM Evaluators as Validated Proxies for Human Interpretability Judgment on an Identical Task

Prior work in SAE evaluation used LLMs as evaluators without establishing their validity against human judgment on the same task. Bills et al. (2023) introduced simulation scoring and validated it by showing that GPT-4 could predict neuron activations, but the validation was against ground-truth activations, not against human interpretability judgments. Templeton et al. (2024) used LLM-based rubric scoring and found it correlated with other automated metrics, but did not compare LLM scores to human scores on identical stimuli. Paulo et al. (2024) included human evaluation, but the humans performed a fundamentally different task (reading explanations and rating them) than the LLMs (predicting activations from explanations), making the comparison between human and LLM scores a comparison of different measurement instruments, not a validation of one against the other.

This paper closes that gap by having humans and LLMs perform exactly the same intruder detection task on exactly the same latents, and then measuring the correlation between their scores. The Spearman correlation of 0.85 between Claude Sonnet 3.5 and human accuracy (Figure 2, left panel) is a substantially stronger validation claim than any prior SAE evaluation work has been able to make, because it eliminates the task-difference confound: any remaining gap between human and LLM scores cannot be attributed to the LLM being asked a different kind of question. The high inter-human correlation (0.87 on a subset of 40 latents, Section 4.1.2) establishes a ceiling—the LLM is nearly as consistent with human judgment as a second human is—while the observation that LLMs systematically underestimate human interpretability (Figure 2, right panel, where humans bin latents into higher interpretability categories than LLMs) provides a directionality to the measurement error that future work can calibrate against.

This finding matters because it enables fully automated, scalable SAE interpretability evaluation with known fidelity to human judgment. If intruder detection with Claude Sonnet 3.5 (or another strong LLM) correlates at 0.85 with humans, then researchers can iterate on SAE architectures and training procedures using only the LLM evaluator, confident that improvements in the automated metric will correspond to improvements in human-perceived interpretability. The paper's comparison across LLMs (Table 1) further refines this by showing which models are adequate proxies: Llama 3.1 70B (r = 0.77), QwQ 32B (r = 0.78), and Gemini Flash 2.0 (r = 0.83) all achieve correlations above 0.75, while Llama 3.1 8B (r = 0.64) is below the threshold for reliable use. This provides practical guidance—roughly, models below ~30B parameters are not yet capable of meaningful SAE interpretability evaluation—while also suggesting that the evaluation capability scales with model capability, a finding with implications for future work as models improve.

The inter-LLM correlations in Table 1 reveal an additional, non-obvious pattern: LLMs agree with each other more than they agree with humans. Claude Sonnet 3.5 and QwQ 32B correlate at 0.91; Llama 3.1 70B and QwQ 32B correlate at 0.89. These are higher than the human-LLM correlations (0.84 and 0.78 respectively). This suggests that LLMs share systematic biases in their interpretability judgments that differ from human biases—they are consistent with each other but consistently slightly off from human judgment. This is a subtle but important methodological finding: it means that comparing two LLM-based evaluation methods against each other (which is common in the literature) can produce inflated agreement estimates that overstate alignment with human interpretability. The paper implicitly argues that human correlation must be the gold standard for validating any automated evaluation method, and that inter-LLM agreement is not a sufficient substitute for human validation.

Innovation 4: Example Embedding Scoring as a Deliberate Speed-Fidelity Tradeoff with Characterized Limitations

The paper introduces example embedding scoring not as a competitor to intruder detection but as a complementary tool for a different point in the speed-accuracy Pareto frontier. This framing—explicitly characterizing the tradeoff rather than presenting a new method as uniformly better—is methodologically mature and relatively rare in the ML interpretability literature, where papers typically argue that their proposed method dominates alternatives.

The conceptual contribution is to identify when a cheap embedding-based method can substitute for an expensive LLM-based one, and when it cannot. The AUROC of 0.64–0.70 on the highest activation decile (Figure 4, left panel) is significantly above the 0.5 random baseline, establishing that for strongly activating examples, the latent's semantic coherence is strong enough to be detected even by a 22M-parameter model that cannot attend specifically to highlighted tokens. The near-random performance on lower deciles and the modest 0.48 correlation with human intruder scores (Table 2) establish the boundary: example embedding scoring is a weak evaluator when the signal is subtle or when the embedding model cannot distinguish activating tokens from surrounding context.

What makes this more than a simple "we tried a cheap method and it was worse" result is the diagnosis of the failure mode. The paper hypothesizes that the embedding model's inability to understand the << and >> highlight delimiters is the primary bottleneck—if the model could attend specifically to the tokens on which the latent actually fires, rather than averaging over the entire 32-token context, its performance might improve substantially. This is a testable hypothesis that suggests a specific improvement (use token-level embeddings with attention masking at the highlighted positions, or fine-tune the embedding model to attend to delimiters) rather than a vague call for "better embedding models." The diagnosis also explains the decile-dependent performance pattern: in the highest decile, the activating token is so strongly correlated with its surrounding context that whole-sentence embedding still captures the relevant semantic cluster; in lower deciles, the activating token signal is diluted by more varied context, and the whole-sentence embedding fails to isolate it.

The practical significance of this tradeoff characterization is that it enables researchers to choose the right tool for their evaluation stage. During SAE architecture search, where hundreds of configurations need rapid feedback, example embedding scoring on the highest decile may provide sufficient signal to identify promising candidates, even with its noisy relationship to true interpretability. During final evaluation and paper writing, intruder detection with a strong LLM (validated against human baselines) provides the high-fidelity measurement needed for reliable comparisons. The paper does not fully develop this two-stage workflow, but the data in Table 2 and Figure 4 provide the evidence needed for researchers to adopt it.

Finally, the symmetric construction of example embedding scoring (Equations 1 and 2) is a minor but deliberate technical choice that facilitates the decile-comparison analysis in Figure 4 (right panel). Unlike intruder detection, where detection accuracy depends on which decile serves as the majority and which as the intruder (producing the asymmetry in Figure 3 right), embedding scoring treats activating and non-activating examples symmetrically by construction—Δ+ and Δ- are computed identically up to class label. This means the symmetric near-zero AUROCs between nearby deciles (Figure 4 right) are a genuine reflection of semantic similarity, not an artifact of the metric's directional bias. This property makes example embedding scoring a cleaner probe for the "does meaning vary with activation strength?" question than intruder decile detection, even though both methods address the same underlying question. The paper implicitly uses the two methods as convergent evidence: the asymmetry in Figure 3 (right) and the symmetry in Figure 4 (right) together constrain the space of possible explanations for how SAE latents represent graded features.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper does not use a standard benchmark dataset in the traditional sense—the "data" for evaluation consists of text spans sampled from a corpus over which SAE latent activations were collected. The SAEs were trained on 10 billion tokens sampled from a reconstruction of the SmolLM2 training corpus (Section 3). For evaluation, the system runs the base model over text and records per-token latent activations, then samples 32-token text windows from this activation database to construct evaluation prompts. There is no separate training/validation/test split for the evaluation itself, since the goal is measuring latent interpretability rather than training a model.
Base model(s). The SAEs being evaluated are trained on the MLP outputs of SmolLM2 135M (Allal et al., 2025), a small 135-million-parameter transformer language model. Four SAEs are evaluated, placed at layers 9, 15, 21, and 27, providing coverage across the model's depth. The SAEs use the TopK activation function (Gao et al., 2024) with k = 32, trained to minimize MSE reconstruction loss with no auxiliary loss terms (Section 3). The choice of a small base model is pragmatic: it makes training multiple SAEs and collecting activations over billions of tokens computationally feasible, and the paper's contribution is the evaluation methodology rather than claims about any particular SAE architecture's superiority.
Metrics. The paper uses two primary metrics, one per evaluation method. For intruder detection, the metric is accuracy—the fraction of prompts where the evaluator correctly identifies the intruder example from among five candidates. Random guessing yields 20% accuracy, providing a clean baseline. Accuracy is reported per latent (averaged across 50–100 prompts for LLMs, 8–20 prompts for humans) and can be broken down by activation decile. For example embedding scoring, the metric is AUROC (Area Under the Receiver Operating Characteristic curve), computed by treating cosine similarities to the positive and negative queries as prediction scores for classifying examples as activating or non-activating. An AUROC of 0.5 corresponds to random performance. The paper also reports Pearson and Spearman correlation coefficients between different evaluators (human vs. LLM, LLM vs. LLM) and between different evaluation methods (intruder detection vs. example embedding scoring, fuzzing, detection) as measures of inter-evaluator and inter-method agreement (Tables 1 and 2, Figure 2).
Baselines. The paper's primary baseline is random guessing at 20% for intruder detection and AUROC 0.5 for example embedding scoring. For method comparison, the paper evaluates the same 56 latents using four additional scoring techniques from prior work (Table 2): fuzzing (predict whether a specific highlighted token activates the latent; from Paulo et al., 2024 and Templeton et al., 2024), detection (predict whether a latent activates anywhere in a passage; from Paulo et al., 2024), intruder detection with LLM evaluators, and intruder detection with human evaluators (the gold standard). The human evaluation on 56 latents serves as the validation baseline against which all automated methods are compared—the paper's central validity claim is that LLM intruder detection correlates strongly with human intruder detection.
Generation budget / compute accounting. The paper does not use "generation budget" as a compute metric in the traditional sense, since the evaluation methods do not involve generating text. Instead, compute cost is implicitly measured by model size and inference cost of the evaluator. For intruder detection, the cost scales with the LLM used: Claude Sonnet 3.5 and Llama 3.1 70B are substantially more expensive per evaluation than Llama 3.1 8B or the all-MiniLM-L6-v2 embedding model. The paper's comparison of evaluators at different scales (22M parameter embedding model, 8B LLM, 32B LLM, 70B LLM, frontier API models) is itself an exploration of the accuracy-compute tradeoff in evaluation. Example embedding scoring is explicitly positioned as fast and cheap: embedding a 32-token sentence with a 22M-parameter model takes milliseconds, versus full autoregressive inference with a 70B+ parameter LLM for intruder detection.
Cross-validation / statistical protocol. The paper does not use cross-validation in the traditional ML sense, since no model is being trained during evaluation. The statistical protocol is instead based on repeated sampling: each latent is evaluated on multiple prompts (50–100 for LLMs, 8–20 for humans) with different random draws of activating and non-activating examples per prompt, and the accuracy is averaged. The paper represents measurement uncertainty visually: in Figure 2 (left panel), the size of each dot represents the number of prompts the human saw for that latent, with smaller dots indicating fewer prompts and thus greater uncertainty in the human accuracy estimate. For human evaluation, inter-rater reliability is assessed by having two human labelers independently perform the intruder task on a subset of 40 latents (3–5 prompts each), yielding a Spearman correlation of 0.87 (Section 4.1.2). No formal confidence intervals or statistical significance tests are reported.

Main Quantitative Results

Intruder Detection: Human and LLM Performance

Headline numbers. Human intruder detection accuracy averages 65% across all 56 evaluated latents, rising to 78% on the highest activation decile (Section 4.1). One third of latents achieve accuracy above 80%; only one in seven fall below 30%. The Spearman correlation between Claude Sonnet 3.5 and human accuracy is 0.85 (Figure 2, left panel), with LLMs systematically underestimating interpretability relative to humans—humans bin latents into higher interpretability categories than LLMs do (Figure 2, right panel).

Evaluator comparison (Table 1). On 56 latents evaluated by humans (10–20 prompts each) and five LLMs (100 prompts each), the Pearson correlations between each evaluator's accuracy scores reveal:

Human vs. Claude Sonnet 3.5: r = 0.84
Human vs. Gemini Flash 2.0: r = 0.83
Human vs. QwQ 32B: r = 0.78
Human vs. Llama 3.1 70B: r = 0.77
Human vs. Llama 3.1 8B: r = 0.64
Inter-human correlation: r = 0.87 (on a subset of 40 latents, Section 4.1.2)
Claude Sonnet 3.5 vs. QwQ 32B: r = 0.91 (highest inter-LLM correlation)
Llama 3.1 8B vs. all other evaluators: r = 0.58–0.65 (consistently lowest)

Interpretability by activation decile (Figure 3, left panel). Accuracy varies systematically with the activation decile from which examples are sampled. On the highest activation decile, accuracy is approximately 78%, while on lower deciles accuracy drops—though it remains significantly above the 20% random baseline across all deciles. The paper does not report exact per-decile numbers for all deciles, but the trend is visually clear in Figure 3 (left): a roughly monotonic increase in interpretability from the lowest to the highest decile, with the steepest gains occurring at the upper end of the distribution.

Interpretability bin stratification (Figure A1). When latents are divided into five interpretability bins based on their overall accuracy (0–20%, 20–40%, 40–60%, 60–80%, 80–100%), the most interpretable bin (80–100%) maintains accuracy above approximately 0.75 even in the lowest activation deciles. The lower bins show sharper drops—latents that are only moderately interpretable overall (40–60%) drop to near-random performance on low deciles. This indicates that "truly interpretable" latents have coherent semantics across their entire activation range, while marginally interpretable latents are only coherent at high activations.

Interpretability bins: human vs. LLM agreement (Figure 2, right panel). The right panel of Figure 2 shows a coarse-grained confusion matrix where both human and LLM (Claude Sonnet 3.5) scores are binned into the five interpretability categories. The key pattern: humans place latents in higher interpretability bins more often than the LLM does. Specifically, across the 56 latents, many latents that humans classify as "high interpretability" (60–80%) or "very high interpretability" (80–100%) are classified by Claude Sonnet 3.5 as one or two bins lower. The agreement is strongest in the extremes—latents that the LLM finds uninterpretable (0–20%) are also rated low by humans, and latents the LLM finds highly interpretable are generally also rated highly by humans—but the mid-range shows substantial bin-level disagreement. The paper interprets this as evidence that "LLMs are not finding convoluted patterns in the latent activations that humans cannot find" (Section 4.1) but rather are more conservative in their interpretability assessments.

Intruder Decile Detection: Probing Intra-Latent Structure

Headline finding (Figure 3, right panel). When all five examples in an intruder detection prompt activate the same latent, but four are drawn from one activation decile and one (the intruder) is drawn from a different decile, Llama 3.1 70B achieves accuracy above the 20% random baseline—but only when the majority decile is high. The highest accuracy is 51% when the majority comes from the highest decile and the intruder from the lowest decile. The task is not symmetric: when the majority decile is low, Llama "essentially never achieves higher than random accuracy, no matter the decile of the intruder" (Section 4.1.1). When the majority decile is low and the intruder decile is high, accuracy drops to approximately 13%—significantly worse than random guessing.

Interpretation. The paper interprets the asymmetry as evidence against the hypothesis that most SAE latents represent simple binary monosemantic features (which would produce near-random accuracy for all decile pairings) or symmetric scalar features (which would produce symmetric accuracy increasing with decile distance). Instead, the pattern suggests that high-activation examples are more homogeneous and recognizable than low-activation examples, and that the relationship between activation strength and semantic content is qualitatively asymmetric—a finding the paper presents as a diagnostic capability of the method rather than a fully explained phenomenon.

Example Embedding Scoring

Headline numbers (Figure 4). The AUROC for distinguishing activating from non-activating examples using whole-sentence embeddings from all-MiniLM-L6-v2 reaches 0.64–0.70 on the highest activation decile (left panel), dropping toward 0.5 (random) for lower deciles. When distinguishing between examples from different activation deciles of the same latent (right panel), AUROCs are close to 0.5 for most decile pairs, with a notable exception: the first decile shows noticeably higher discriminability from all other deciles (AUROC approximately 0.55–0.60 across most comparisons), suggesting examples from the lowest activation decile are semantically distinct from higher-activation examples in a way the embedding model can detect.

Correlation with human intruder scores (Table 2). On the 56 latents evaluated by humans, example embedding scoring achieves a Pearson correlation of only 0.48 with human intruder detection accuracy, substantially lower than LLM-based intruder detection (0.84 with Claude Sonnet 3.5). The correlation with LLM-based intruder detection is similarly modest at 0.51.

Comparison to other automated methods (Table 2). The full correlation matrix among five evaluation methods on the same 56 latents shows:

Example embedding vs. fuzzing F1: r = 0.46
Example embedding vs. detection F1: r = 0.58
Intruder (LLM) vs. fuzzing F1: r = 0.74
Intruder (LLM) vs. detection F1: r = 0.59
Intruder (Human) vs. fuzzing F1: r = 0.69
Intruder (Human) vs. detection F1: r = 0.50
Fuzzing F1 vs. detection F1: r = 0.71

Example embedding scoring has the lowest or second-lowest correlation with every other method, confirming that it measures a substantially different signal than the LLM-based and human-based approaches. The paper's interpretation is that example embedding scoring "tends to underestimate the interpretability of latents due to the small size of the embedding model and the difficulty of getting the model to attend to the tokens on which the latent is actually active, rather than the context as a whole" (Section 4.2).

Ablation Studies and Robustness Checks

The paper does not include systematic ablation studies in the traditional sense—there are no experiments that remove components of the evaluation method and measure the impact on scores. However, several comparisons serve as de facto sensitivity analyses:

Evaluator model scale variation (Table 1). Testing five LLMs ranging from 8B to frontier-scale parameters reveals that accuracy and human correlation increase with model scale, but not linearly. Llama 3.1 8B achieves only 27% average accuracy (barely above 20% random) and correlates at 0.64 with humans, establishing a lower bound: models below roughly 30B parameters (the smallest model with r > 0.75 is QwQ 32B at r = 0.78) are not reliable evaluators. Claude Sonnet 3.5, Llama 3.1 70B, Gemini Flash 2.0, and QwQ 32B all achieve correlations above 0.75 with humans, suggesting that evaluation capability saturates above a certain scale threshold. The gap between Llama 3.1 70B (r = 0.77) and Claude Sonnet 3.5 (r = 0.84) indicates that model family and training procedure matter beyond raw parameter count.

Inter-human reliability (Section 4.1.2). On a subset of 40 latents with only 3–5 prompts per labeler (increasing noise), two human labelers achieve a Spearman correlation of 0.87. This is higher than any human-LLM correlation, establishing a ceiling for automated evaluation and suggesting that even the best LLM evaluators leave approximately 3–5 percentage points of correlation unexplained relative to the human reliability ceiling. Notably, the two human labelers' accuracy scores differ in absolute terms (the paper does not report the mean accuracy per labeler), not just in ranking.

Full activation distribution vs. non-activating examples (Figure 3, left panel and Figure A1). The per-decile analysis reveals that even the lowest-activating deciles of interpretable latents produce accuracy significantly above 20%, contradicting the assumption (implicit in prior work that samples only from high quantiles) that low-activation examples are essentially random noise for all latents. The stratified analysis in Figure A1 refines this: only latents in the highest interpretability bins maintain near-flat accuracy profiles across deciles; latents in lower bins show sharp drops in interpretability at lower deciles. This suggests that the practice of sampling only from high quantiles for explanation generation (as in Templeton et al., 2024; Paulo et al., 2024) may systematically overestimate the interpretability of marginally interpretable latents while having little effect on the assessment of highly interpretable ones.

Negative result: example embedding scoring's failure mode. The near-random AUROCs for low-to-medium activation deciles (Figure 4, left panel) and the modest 0.48 human correlation constitute a negative result: off-the-shelf sentence embeddings from a 22M-parameter model are insufficient for reliable SAE interpretability evaluation, even on a task (clustering activating vs. non-activating examples) that seems well-suited to embedding-based approaches. The paper attributes this to the embedding model's inability to attend specifically to highlighted tokens, and the failure is diagnostic—it suggests that the interpretable signal in SAE latents is often localized to specific tokens or short spans within sentences, not distributed evenly across the entire 32-token context window.

Absence of layer-wise analysis. A notable omission: the paper evaluates 56 latents drawn approximately equally from four layers (9, 15, 21, 27) but does not report interpretability scores broken down by layer. This means the paper cannot address questions like "are later-layer latents more or less interpretable than earlier-layer ones?" or "does the decile-interpretability relationship vary with network depth?" The small sample size (~14 latents per layer) would make such comparisons noisy, but even a qualitative observation would have been informative given prior work finding that SAE interpretability varies with layer depth (Bricken et al., 2023; Templeton et al., 2024).

Critical Assessment

Does the paper demonstrate that interpretability can be measured without explanations?

The paper demonstrates that intruder detection accuracy correlates strongly with human judgment (r = 0.85) and that humans perform above chance on the task (65% average), which together establish that the method measures something humans agree about and that this something is related to latent interpretability. However, the paper does not directly demonstrate that the something being measured is the same construct that explanation-based evaluation claims to measure. No experiment compares explanation-based scores and intruder detection scores on the same set of latents with the same human participants—the correlations in Table 2 compare intruder detection to fuzzing and detection F1 scores (automated metrics from explanation-based pipelines), but these are not direct comparisons to the human-validated simulation scoring that Bills et al. (2023) and Templeton et al. (2024) use as their primary metric. The correlation between LLM intruder detection and detection F1 is only 0.59, and with fuzzing F1 it is 0.74 (Table 2), suggesting that intruder detection and explanation-based evaluation are measuring related but distinct constructs. This is not necessarily a weakness—the paper's argument is precisely that they should be distinct—but it means the claim that intruder detection "measures interpretability" rests on the human correlation as construct validation, without convergent validation against an established gold-standard interpretability measure that is not explanation-based. Since no such gold standard exists, this limitation is inherent to the problem, but it should be acknowledged: intruder detection measures a specific operationalization of interpretability (discriminability of activating from non-activating examples) that the paper argues is the right one, but it does not empirically demonstrate that this operationalization captures everything researchers care about when they call a latent "interpretable."

Does the paper demonstrate that LLMs are valid proxies for human interpretability judgment?

The Spearman correlation of 0.85 between Claude Sonnet 3.5 and human scores (Figure 2, left panel) is the strongest evidence the paper presents. This is genuinely strong—0.85 is a high correlation for any human-automated-metric comparison, and it is higher than correlations reported in prior SAE evaluation work (though, as the paper notes, prior work did not compare humans and LLMs on identical tasks). The inter-human correlation of 0.87 establishes an approximate ceiling: Claude Sonnet 3.5 is nearly as correlated with human judgment as a second human would be. This supports the claim that LLM intruder detection can substitute for human evaluation in practice.

However, three caveats weaken this claim. First, the human data is thin (8–20 prompts per latent vs. 50–100 for LLMs), meaning the human scores are noisier estimates of "true human interpretability" than the LLM scores are of "true LLM interpretability." This noise artificially depresses the correlation—if humans could be evaluated on 100 prompts per latent, the human-LLM correlation might be higher than 0.85, or it might not; the paper cannot distinguish these. Second, the human evaluators are the authors themselves (Section 3.1.1: "The authors manually performed the intruder task"), who are deeply familiar with SAEs, latent interpretability, and the goals of the paper. Their judgments may not represent those of naive human evaluators—the very people for whom "interpretability" is ultimately intended. If the goal of SAE interpretability is to enable non-expert humans to understand model internals, then author self-evaluation overestimates the general-population interpretability of the latents. Third, the 56-latent sample is small and selected without a clearly specified sampling strategy. If the 56 latents were chosen because they were interesting or varied (which the paper does not state, but also does not rule out), the correlation may not generalize to a random sample of SAE latents. A replication on 500+ randomly sampled latents with naive human evaluators (e.g., crowdworkers) would substantially strengthen the validity claim.

Does the paper demonstrate that interpretability varies meaningfully with activation decile?

The per-decile analysis (Figure 3, left panel; Figure A1) clearly shows that accuracy is higher for high-activation examples than low-activation ones. This is consistent with prior work (Templeton et al., 2024; Paulo et al., 2024) and is not surprising—highly activating examples are, by definition, the ones where the latent fires most strongly, and we would expect those to be the most prototypical examples of whatever the latent represents. The more interesting claim is the stratified finding: that latents in the highest interpretability bin maintain flat accuracy profiles across deciles, while latents in lower bins show sharp drops (Figure A1). This suggests that "truly interpretable" latents have stable semantics across their entire activation range, while marginally interpretable latents are only coherent at high activations.

The evidence for this claim is visually suggestive but statistically thin: Figure A1 divides latents into five bins, each containing roughly 11 latents on average (56/5), and plots accuracy trends across deciles. With ~11 latents per bin, the standard error of the mean at each decile is substantial, and the paper does not report confidence intervals or test whether the between-bin differences in slope are statistically significant. The qualitative pattern is plausible and theoretically interesting, but the paper presents it as an observation rather than a statistically validated finding. A larger latent sample with formal hypothesis testing (e.g., comparing the decile-accuracy slopes between high- and low-interpretability latents) would strengthen this claim considerably.

Does the intruder decile detection experiment support claims about latent semantics?

The asymmetric accuracy matrix in Figure 3 (right panel) is the paper's most intriguing finding: when the majority examples come from a high decile, the intruder from a low decile is somewhat detectable (51% accuracy); when the majority comes from a low decile, no intruder is detectable, and in fact performance drops below random to 13% when the intruder comes from a high decile. The paper uses this to argue that "the full distribution of the activations of the SAE latents remains somewhat interpretable" and that we can "rule out the hypothesis that most features are both binary and monosemantic" (Section 4.1.1).

The below-random accuracy (13% vs. 20% chance) is particularly striking and warrants scrutiny. The paper does not provide an explanation for this anti-detection effect—why would evaluators systematically pick the wrong example more often than chance when a high-activation intruder is placed among low-activation examples? Possible explanations include: (a) the LLM evaluator uses a heuristic like "the example with the most salient semantic content must be one of the four matches" and incorrectly selects the high-activation example as non-intruder, (b) the prompt format or few-shot examples inadvertently bias the LLM toward specific response patterns when activation strength cues are present but semantic cues are absent, or (c) there is a genuine semantic inversion where the high-activation example is less representative of what the majority low-activation examples share than a random non-activating example would be. The paper does not attempt to distinguish these explanations, and without them, the 13% accuracy is an interesting anomaly rather than a robust finding about latent structure. It could easily be an artifact of the LLM evaluator's response biases rather than a property of the SAE latents themselves. Running the same experiment with human evaluators would be the obvious control—if humans also show below-random accuracy in the same condition, it strengthens the claim about latent semantics; if humans revert to random guessing, it suggests the LLM's behavior is an artifact.

What experiments would have strengthened the paper?

Several missing experiments or analyses would substantially increase confidence in the paper's claims:

Direct comparison to explanation-based evaluation on the same latents. The paper compares intruder detection to fuzzing and detection F1 (Table 2) but does not compare it to the primary explanation-based metric from prior work—simulation scoring (Bills et al., 2023) or the replacement score (Paulo & Belrose, 2025). Including simulation scores for the 56 evaluated latents would allow direct assessment of whether explanation-free and explanation-based methods rank latents similarly, and whether the systematic differences between them (e.g., humans finding latents more interpretable than LLMs) are specific to intruder detection or generalize across methods. The paper implicitly argues that explanation-free evaluation is better, but without showing that it produces different rankings of latents or SAEs—and that these different rankings are more aligned with human judgment—the practical advantage remains theoretical.

Larger and more diverse latent sample. Fifty-six latents from one model family (SmolLM2 135M) is a small sample for validating a measurement methodology. The paper would be stronger with evaluation on (a) a larger sample from the same four SAEs (e.g., 500+ latents), (b) SAEs from a different model family (e.g., Gemma Scope, GPT-2 SAEs), and (c) SAEs with different architectures (e.g., ReLU-based SAEs, JumpReLU SAEs, transcoders). The strong human-LLM correlation might not generalize if, for example, SmolLM2 latents happen to be more or less interpretable than latents in other models, or if TopK SAEs produce different activation distributions than L1-penalized SAEs.

Naive human evaluators. Using the paper's authors as the only human evaluators is a limitation the paper does not acknowledge. Authors are incentivized (even unconsciously) to find their proposed method working well, and their deep familiarity with SAEs may make them more adept at the intruder task than typical humans interacting with interpretability tools. A crowdworker study with 50–100 participants who receive minimal training on the task would establish whether the reported 65% human accuracy and 0.85 human-LLM correlation hold for the target population of "humans who might use interpretability tools." The inter-author correlation of 0.87 suggests reliability between the two specific individuals, but does not guarantee that those individuals' judgments are representative of broader human interpretability perception.

Ablation of prompt design choices. The intruder detection prompt format involves several design choices that could affect scores: the number of examples (5), the length of each example (32 tokens), the highlighting mechanism (<< and >> delimiters), the matching of highlighted token counts in the intruder example, the few-shot prompt structure, and the use of activating examples from other latents as intruders (rather than truly random text). The paper does not ablate any of these. Without ablations, a researcher adopting the method cannot know which choices are critical and which are incidental. The matching of highlighted token counts is an especially important control—if the intruder example could be identified by having fewer highlighted tokens, even a weak evaluator would achieve high accuracy for the wrong reason. The paper implements this control but does not test whether it matters.

Layer-wise analysis. The paper collects data from four layers but does not report layer-stratified results. If, for example, later-layer latents are systematically more or less interpretable than earlier-layer ones—or show different decile-interpretability profiles—this would be methodologically important for researchers using the evaluation tools. The omission likely reflects the small sample size (~14 latents per layer), but even reporting qualitative differences would have added value.

Confidence intervals and significance tests. The paper reports point estimates for correlations, accuracies, and AUROCs without confidence intervals, standard errors, or significance tests. For the Spearman correlation of 0.85 between human and LLM scores on 56 latents, the 95% confidence interval is approximately [0.74, 0.92] (using Fisher transformation approximation). This wide interval means the true correlation could be as low as 0.74—still strong, but substantially weaker than the point estimate suggests. Reporting confidence intervals would provide appropriate caution about the precision of the estimates given the small latent sample.

Summary: what the experiments do and do not demonstrate

The experiments convincingly demonstrate that intruder detection is a feasible task for both humans and LLMs: accuracy is substantially above random, humans agree with each other, and LLMs agree with humans. This establishes the method as a viable alternative to explanation-based evaluation for some purposes, particularly for rapid, scalable, automated assessment where explanation generation is undesirable.

The experiments do not demonstrate that intruder detection is a better measure of interpretability than explanation-based methods—no head-to-head comparison against a human-validated explanation metric is performed. Nor do they demonstrate that the method generalizes beyond the specific SAE architecture (TopK, k=32), base model (SmolLM2 135M), and evaluator population (the paper's authors) tested. The sample size (56 latents) and single-model-family scope make the strong human-LLM correlation suggestive rather than definitive—replication with larger samples, diverse models, and independent human evaluators would be needed to establish the method as a standard benchmark.

The example embedding scoring experiments demonstrate that a small embedding model can detect above-random structure in SAE latent activations (AUROC 0.64–0.70 on the highest decile), but the weak human correlation (r = 0.48) means the method is not a reliable substitute for human judgment. It may still be useful for rapid iteration during SAE development, but the paper's evidence for this use case is indirect—no experiment shows that embedding scores are predictive of downstream human or LLM interpretability rankings in a way that would make early-stage screening reliable.

The decile analysis is the paper's most generative contribution for future research: the asymmetry in Figure 3 (right) and the flat interpretability profiles for highly interpretable latents in Figure A1 raise specific, testable hypotheses about SAE latent structure that the evaluation methods enable researchers to investigate. But these findings are presented as preliminary observations, not as robustly established facts, and they require replication with larger samples and human controls to move from "interesting pattern" to "established finding."

6. Limitations and Trade-offs

The Human Validation Study Is Thin—Small Sample, Author Evaluators, No Crowdworker Replication

The paper's central validity claim—that LLM intruder detection is a reliable proxy for human interpretability judgment—rests on a human evaluation study of 56 latents performed by the two authors themselves (Section 3.1.1):

"The authors manually performed the intruder task on a subset of examples to provide a human baseline."

This design introduces three interrelated weaknesses that limit the strength of the validity claim.

The consequence. The Spearman correlation of 0.85 between Claude Sonnet 3.5 and human scores (Figure 2, left panel) is the paper's headline reliability number, but it is measured on a small sample (56 latents) using evaluators who are deeply familiar with SAEs, the intruder task design, and the paper's goals. This means the correlation may not generalize in two critical ways. First, naive human evaluators—the people for whom "interpretability" is ultimately intended—may perform differently on the intruder task than SAE researchers. The authors can draw on their knowledge of how SAE latents behave, what kinds of patterns are typical, and what the evaluation is trying to measure, all of which could inflate their accuracy relative to a typical user of interpretability tools. If naive humans perform systematically worse or differently, the LLM-human correlation measured here would overestimate the method's real-world validity. Second, the small sample (56 latents, with only 8–20 prompts per latent for humans vs. 50–100 for LLMs) produces noisy human accuracy estimates (represented by the variable dot sizes in Figure 2, left panel), and the correlation of 0.85 has a wide confidence interval—roughly [0.74, 0.92] using standard approximations—meaning the true human-LLM agreement could be substantially weaker than the point estimate suggests.

What evidence exists in the paper. The inter-human correlation of 0.87 on a subset of 40 latents (Section 4.1.2) demonstrates that the two authors agree with each other, but this is reliability within a specific population (SAE researchers), not validity against a target population (naive users). The paper does not report any data from non-author human evaluators. The finding that "the human mostly finds latents more interpretable than their LLM counterparts" (Figure 2, right panel) is interpreted as good news—LLMs are conservative estimators—but this interpretation assumes that author judgments are calibrated to what naive humans would find interpretable, an assumption the paper does not test. The 56-latent sample is drawn from four SAE layers with approximately 14 latents per layer, but the paper does not specify how these particular 56 latents were selected from the thousands available in each SAE, leaving open the possibility that they were chosen (even unconsciously) to be especially varied or interesting, which could inflate correlations.

Mitigation status. The paper does not acknowledge author-evaluator bias as a limitation. The human sample size is noted implicitly through the variable dot sizes in Figure 2, but there is no discussion of how the thin human data affects the precision of the headline 0.85 correlation. The authors do not suggest future work with crowdworker evaluations or naive human participants. This is a significant gap given that the paper's primary contribution is establishing the validity of an automated evaluation method—the validation study itself is the product, and its generalizability is not scrutinized.

Example Embedding Scoring Fails to Attend to Activating Tokens, Limiting Its Utility to High-Activation Regimes

Example embedding scoring uses the all-MiniLM-L6-v2 model (22M parameters) to embed entire 32-token sentences, computing whether activating and non-activating examples cluster in embedding space. The paper acknowledges the key failure mode explicitly (Section 4.2):

"We suspect that example embedding scores tend to underestimate the interpretability of latents due to the small size of the embedding model and the difficulty of getting the model to attend to the tokens on which the latent is actually active, rather than the context as a whole."

This is not a minor implementation detail—it is a fundamental architectural mismatch between the evaluation method and the thing being evaluated.

The consequence. The embedding model cannot see the << and >> highlight delimiters that mark which tokens activate the latent, so it must infer latent semantics from undifferentiated whole-sentence context. When a latent activates on a specific word or short phrase (e.g., "democracy") but the surrounding sentences are topically diverse (elections, ancient Greece, workplace decisions), the whole-sentence embeddings will be scattered across the embedding space, making activating examples appear no more clustered than non-activating ones—even though a human or LLM who can see the highlighted tokens would recognize the latent's meaning immediately. This produces AUROCs near 0.5 (random) for low-to-medium activation deciles (Figure 4, left panel) and a human correlation of only 0.48 (Table 2), making the method unreliable for evaluating latents where the activating signal is localized to specific tokens rather than distributed across the whole context. The method works only on the highest activation decile (AUROC 0.64–0.70), where the activating concept is so semantically dominant that it saturates the entire 32-token window—the regime where evaluation is easiest and arguably least needed. This severely limits example embedding scoring's value as a screening tool, since the latents where evaluation is most uncertain (those with subtle or localized activation patterns) are precisely where the method performs near chance.

What evidence exists in the paper. The performance collapse across deciles in Figure 4 (left panel) from AUROC ~0.67 at the highest decile to near 0.5 at lower deciles demonstrates the failure mode quantitatively. The correlation matrix in Table 2 shows that example embedding scoring has the weakest relationship to every other method (0.46–0.58) and to human judgment (0.48), confirming that it measures a substantially different and weaker signal. The paper's diagnosis—that the embedding model cannot attend to highlighted tokens—is plausible and supported by the decile-dependence (high-activation = context is saturated with the concept = whole-sentence embedding works), but no experiment directly tests this hypothesis, for instance by comparing whole-sentence embedding to an approach that masks non-highlighted tokens or uses token-level embeddings at the activation positions.

Mitigation status. The paper acknowledges the limitation but does not attempt to fix it. No alternative embedding approach is tested (e.g., using token-level embeddings at highlighted positions, fine-tuning the embedding model to recognize << and >> tokens, or using a cross-encoder that takes both the sentence and the highlighted span as input). The paper positions example embedding scoring as a speed-accuracy tradeoff (Section 4.2: "a very fast and computationally efficient technique relative to other approaches"), but without solving the attention problem, the accuracy side of the tradeoff is so weak that the method's practical utility is questionable except for the narrow use case of evaluating highly-interpretable, high-activation latents where evaluation is already easy. A practitioner would reasonably ask: if the method only works well on the easiest cases, what value does it add over simply inspecting the latents manually?

Difficulty Estimation Has No Real-World Deployment Path—the Paper Does Not Evaluate Any Latent at Inference Time

This limitation concerns the paper's framing as a measurement tool. The entire evaluation pipeline—collecting activations over a corpus, stratifying by decile, constructing prompts, running evaluator models—is a retrospective analysis performed after the SAE is fully trained and its activations over a corpus have been exhaustively recorded. This is appropriate for a benchmarking setting, but the paper's title and framing ("Evaluating SAE interpretability without explanations") suggests a replacement for the evaluation component of the standard interpretability pipeline. In practice, the proposed methods require exactly the same infrastructure as explanation-based evaluation: you still need to run the base model over a large corpus, collect per-token latent activations, and process the resulting database to construct evaluation examples. The only step eliminated is the LLM call to generate the explanation text—a step that is computationally cheap relative to the activation collection that both pipelines share.

The consequence. The paper does not reduce the dominant cost in SAE evaluation, which is the forward pass over billions of tokens to collect activations. Both explanation-based and explanation-free methods must do this. What the paper eliminates—generating a one-sentence explanation per latent—is a small fraction of the total evaluation budget when using capable LLMs. For the 56 latents evaluated, the activation collection over 10 billion training tokens has already been done as part of SAE training (Section 3), so the marginal cost of evaluation is low. But for a practitioner training a new SAE and wanting to evaluate it, the activation collection step would need to be run on an evaluation corpus (not the training corpus, to avoid contamination), and this cost—not the explanation generation—dominates the pipeline. The paper's claim that example embedding scoring is "a very fast and computationally efficient technique" (Section 4.2) is true for the scoring step itself but misleading about the end-to-end cost: you still need to run the base model over a corpus and compute SAE activations for every token, which is orders of magnitude more expensive than embedding 32-token sentences with a 22M-parameter model.

What evidence exists in the paper. The paper does not compare the end-to-end computational cost of its proposed methods against explanation-based evaluation. It reports only the relative costs of the scoring step (intruder detection with Claude Sonnet 3.5 vs. example embedding scoring with all-MiniLM-L6-v2) and notes that the latter is faster, but both are trivial compared to the activation collection they share. There is no experiment measuring wall-clock time or FLOPs for the full pipeline, and no comparison to the cost of running explanation-based evaluation on the same latents. The paper's introduction frames the explanation step as introducing "additional hyperparameters and prompt choices which are likely to affect the final results" (Section 1), but the proposed methods introduce their own hyperparameters (number of examples per prompt, example length, decile sampling strategy, choice of intruder pool) that are equally consequential and equally unaudited.

Mitigation status. The paper does not acknowledge this cost-amortization issue. It treats the activation collection step as given—since the SAEs were trained on a corpus and activations were already collected for training monitoring—without addressing what a practitioner would need to do to evaluate a new SAE. A realistic deployment scenario would involve: (1) selecting an evaluation corpus, (2) running the base model + SAE over it to collect activations, (3) running the intruder detection or embedding scoring pipeline on the collected activations. Step (2) is the same cost for both explanation-based and explanation-free methods, and it dwarfs step (3). The paper's contribution is therefore better characterized as removing a source of measurement noise (the explanation generation hyperparameters) rather than reducing evaluation cost, but the paper frames it partly as an efficiency improvement without quantifying the actual cost breakdown.

No Head-to-Head Comparison Against Explanation-Based Evaluation on Identical Latents

The paper argues that explanation-free evaluation is preferable because it eliminates the confound between latent interpretability and explanation quality. But it never directly compares its methods against the dominant explanation-based metrics (simulation scoring, detection, fuzzing) on the same set of latents using the same human evaluators as the ground truth. The correlations in Table 2 compare intruder detection to automated detection and fuzzing F1 scores—not to human-validated explanation-based evaluation—and the moderate correlations (0.59 with detection, 0.74 with fuzzing) suggest that intruder detection and explanation-based methods are measuring related but distinct constructs.

The consequence. Without a head-to-head comparison, a practitioner cannot determine whether intruder detection would produce different rankings of SAEs or latents than explanation-based evaluation—and whether those different rankings would be more aligned with human judgment. If intruder detection and explanation-based evaluation rank latents similarly, then the proposed method is a valid but not clearly superior alternative; if they rank latents differently, then the field needs to adjudicate which ranking better reflects "true" interpretability, and the paper provides no evidence to resolve that question. The paper implicitly claims that explanation-free evaluation is better because it removes a confound, but "removes a confound" is a theoretical argument about measurement purity, not an empirical demonstration that the resulting measurements are more useful or more aligned with human understanding. A method can be purer in its construct validity while being worse in practice—for example, if the intruder detection task is harder for humans than reading explanations, or if it misses aspects of interpretability (like the ability to communicate a latent's meaning to others) that explanation-based evaluation captures.

What evidence exists in the paper. Table 2 provides correlations between intruder detection (human and LLM) and two explanation-based metrics (fuzzing F1 and detection F1), but these are automated metrics computed by LLMs, not human-validated explanation scores. The correlation between LLM intruder detection and fuzzing F1 is 0.74; between LLM intruder detection and detection F1, 0.59. These moderate correlations indicate that the methods partially agree but have substantial unexplained variance. The paper does not report simulation scores (the original Bills et al., 2023 metric), replacement scores (Paulo & Belrose, 2025), or any other explanation-based metric that has been validated against human judgment. Without this comparison, the paper's argument that explanation-free evaluation is an improvement rests on the conceptual critique of explanation dependence (Section 1, Section 2) rather than on empirical evidence that the proposed methods produce more valid or more useful rankings.

Mitigation status. The paper acknowledges that prior work has used explanation-based evaluation, but does not frame the absence of direct comparison as a limitation. It positions intruder detection as complementary rather than competitive ("We do not claim that explanation-based evaluation should be abandoned"), which is a reasonable stance but one that sidesteps the practical question facing an SAE developer: which method should I use to evaluate my SAE, and will the choice affect my conclusions? Answering this would require evaluating both methods on the same latents with the same human ground truth—an experiment that is feasible (given the 56 latents already have human intruder detection scores) but not performed.

The Single-Model, Single-Architecture Scope Leaves Generalization Uncharacterized

All experiments use four SAEs trained on SmolLM2 135M MLP outputs with TopK (k=32) activation and no auxiliary loss (Section 3). The paper tests multiple LLM evaluators (Table 1) and two evaluation methods, but the objects being evaluated are drawn from a single base model (SmolLM2), a single SAE architecture (TopK), a single sparsity level (k=32), and a single training objective (MSE only). The paper provides no evidence about whether the findings—the 0.85 human-LLM correlation, the 65% human accuracy, the decile-dependence pattern, the asymmetry in Figure 3 (right)—generalize to other SAE configurations.

The consequence. A practitioner using a different SAE setup—for example, a ReLU-based SAE with an L1 sparsity penalty, a JumpReLU SAE, an SAE with a different k value, a transcoder, or an SAE trained on a different model family (GPT-2, Gemma, Llama, Pythia)—cannot assume that the human-LLM correlation or the intruder detection accuracy patterns hold. The TopK activation function produces a specific sparsity pattern (exactly k latents active per token, with hard zeroing of all others) that may affect how activation distributions look and how coherent latents appear at different deciles. An L1-penalized SAE produces a different activation distribution (soft thresholding, variable number of active latents per token, potentially different dead latent rates), and it is unknown whether the intruder detection task would work similarly well or whether the human-LLM correlation would replicate. The decile stratification—which is central to the paper's analysis of interpretability vs. activation strength—assumes that activation strength is a meaningful axis along which to stratify, but this axis may have different properties in SAEs with different sparsity mechanisms. For example, in a ReLU SAE, many latents may have a large fraction of exactly-zero activations, making the decile binning less informative because a large fraction of examples would all fall into the zero-activation "decile."

What evidence exists in the paper. None. The paper does not evaluate any SAE with a different architecture, sparsity mechanism, or base model. The choice of SmolLM2 135M is acknowledged as practical (it enables training multiple SAEs on 10B tokens), and the choice of TopK is described as "state-of-the-art" (Section 3), but there is no discussion of how these choices might affect the evaluation results. The inter-LLM correlations in Table 1 suggest that the choice of evaluator model matters (Llama 8B vs. Claude Sonnet 3.5 differ substantially), implying that the choice of SAE architecture could also matter, but this is not investigated.

Mitigation status. The paper does not acknowledge the single-architecture scope as a limitation. The authors present their SAE setup as a standard configuration that is "representative of the capabilities of many contemporary" approaches (though this exact phrasing is not used—the paper describes the setup in Section 3 without generalization claims), but no evidence supports representativeness. Future work would need to replicate the human-LLM correlation study on SAEs with different architectures and base models to establish whether intruder detection is a generally valid evaluation method or whether its validity is contingent on specific SAE design choices. The method itself is architecture-agnostic (it only requires per-token latent activations, which any SAE can produce), but the behavior of the method—how well it correlates with human judgment—may be architecture-dependent in ways the paper does not explore.

The Below-Random Intruder Accuracy (13%) in the Decile-Detection Experiment Is Unexplained, Complicating Interpretation

The intruder decile detection experiment (Figure 3, right panel) reveals a striking anomaly: when the majority of examples come from a low activation decile and the intruder comes from a high activation decile, Llama 3.1 70B achieves only 13% accuracy—significantly below the 20% random guessing baseline. The paper reports this as an interesting observation but provides no explanation for the anti-detection effect:

"Interestingly, when the intruder decile is high and the majority decile is low, we see significantly worse than random accuracy (around 13%)."

The consequence. Without an explanation, the reader cannot determine whether the 13% accuracy reveals a genuine property of SAE latents (e.g., that high-activation examples are somehow anti-representative of what low-activation examples share, causing evaluators to systematically reject them as matches) or whether it is an artifact of the LLM evaluator's response biases. If the LLM uses a heuristic like "the example that looks most semantically coherent must be one of the four matches" when faced with an ambiguous set, it might systematically select the high-activation intruder as non-intruder, producing below-random accuracy that has nothing to do with latent structure. This distinction matters for the paper's claim that the decile-detection experiment provides evidence about latent semantics (Section 4.1.1: "we can rule out the hypothesis that most features are both binary and monosemantic, since we see far better than random accuracy for many (intruder, majority) pairs"). If the above-random accuracy in the high-majority, low-intruder condition (51%) also reflects LLM heuristics rather than genuine semantic discrimination, then the experiment tells us more about LLM behavior than about SAE latents. The paper's interpretation of the asymmetry as evidence about latent structure (asymmetry in representational homogeneity) is only valid if the LLM's performance reflects latent semantics rather than response biases, and the 13% result raises exactly this concern without resolving it.

What evidence exists in the paper. Figure 3 (right panel) shows the full accuracy matrix for Llama 3.1 70B on the decile-detection task. The paper notes the below-random accuracy in passing but does not analyze it further or propose an explanation. No human baseline is provided for the decile-detection experiment—the only evaluator is Llama 3.1 70B—so it is impossible to determine whether humans would also show below-random accuracy in the low-majority, high-intruder condition. The paper does not report whether other LLMs (Claude Sonnet 3.5, Gemini Flash 2.0) show the same pattern, which would help distinguish model-specific biases from task-general effects. There is no ablation testing whether prompt format, few-shot examples, or response interface design affect the below-random accuracy.

Mitigation status. The paper does not acknowledge the below-random accuracy as a potential artifact or as a threat to the validity of the decile-detection experiment's interpretation. The anomaly is noted but treated as a finding rather than a methodological concern. A human baseline for the decile-detection experiment would substantially clarify interpretation: if humans also show below-random accuracy in the same condition, the phenomenon is likely a property of the latents or the task structure; if humans revert to random guessing, the LLM's behavior is an artifact and the decile-detection conclusions are unsupported. The paper does not suggest this as future work.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new SAE architecture, a new sparsity mechanism, or a new interpretability insight about language models. It introduces something more infrastructural: a measurement instrument that isolates latent interpretability from the machinery used to verbalize it. The magnitude of this shift is best understood as a methodological reframing with immediate practical consequences, not a paradigm shift in how we think about SAEs or a new scientific discovery about neural network internals.

The core reframing is operational: interpretability measurement moves from "can an LLM generate a good explanation for this latent?" to "can an evaluator discriminate activating from non-activating examples for this latent?" This is not merely a new metric—it is a claim about what the construct of interpretability should mean in practice. Prior work (Bills et al., 2023; Templeton et al., 2024; Paulo et al., 2024) implicitly defined interpretability as explainability—a latent is interpretable if its behavior can be summarized in natural language and that summary predicts future behavior. This paper argues that interpretability is better operationalized as recognizability—a latent is interpretable if a human can learn to identify when it fires, regardless of whether they can articulate the rule. The difference is not academic: many patterns that humans can reliably perceive (faces, musical styles, grammaticality judgments) resist concise verbal description, and SAE latents may behave similarly.

This reframing changes the landscape in four concrete ways:

First, it provides a validation standard for existing evaluation methods. The paper's finding that Claude Sonnet 3.5 correlates with human intruder detection accuracy at Spearman r = 0.85 (Figure 2, left panel) on an identical task establishes a benchmark that explanation-based evaluation methods have not yet met—prior work compared LLM evaluators to ground-truth activations, not to human interpretability judgments. Any future evaluation method, explanation-based or not, can be validated by the same protocol: have humans and the method perform the same discrimination task on the same latents, and measure the correlation. This is a higher bar than inter-LLM agreement (which the paper shows is inflated—LLMs agree with each other at r = 0.89–0.91, higher than any LLM agrees with humans at r = 0.77–0.84, Table 1) and lower than the conceptually demanding standard of predicting exact activation values from explanations. It provides a tractable validation target that the field previously lacked.

Second, it reorients SAE evaluation toward behavioral testing. The intruder detection task is essentially a psychophysics experiment applied to neural network features: present a human (or LLM) with stimuli, ask them to make a forced-choice discrimination, and use accuracy as the dependent measure. This connects SAE interpretability research to a century of methodology in experimental psychology and psychometrics that has developed rigorous protocols for measuring perceptual and cognitive abilities. Concepts like inter-rater reliability (the paper's 0.87 inter-human correlation), guessing baselines (the 20% floor for 5-way forced choice), and accuracy-by-condition functions (the decile-dependence curves in Figure 3, left panel) are standard in that literature and now become available as conceptual tools for interpretability research. The paper gestures toward this connection by citing the word intrusion task from topic modeling (Chang et al., 2009) and the two-alternative forced choice task from vision interpretability (Borowski et al., 2020), but the deeper implication is that psychophysical methodology—staircase procedures, signal detection theory, just-noticeable-difference measurements—could be productively adapted to study the "perception" of neural network features.

Third, it makes rapid iteration on SAE architectures empirically feasible in a way that explanation-based evaluation does not. The paper does not fully develop this argument, but the data supports it. Example embedding scoring takes milliseconds per latent using a 22M-parameter model and achieves AUROC 0.64–0.70 on the highest activation decile (Figure 4, left panel). LLM intruder detection takes seconds to minutes per latent but correlates with humans at r = 0.85 (Figure 2). A practitioner developing SAEs could use a two-stage screening pipeline: run example embedding scoring on all latents to identify the most promising SAE configurations (cheap, noisy, but above-random), then validate the top candidates with LLM intruder detection (expensive, accurate, human-correlated). This is a standard engineering pattern (cheap proxy filter → expensive validation) that the paper's data enables but does not explicitly recommend. The 4× efficiency gain over best-of-N reported in prior SAE scaling work is not claimed here—the paper's contribution is enabling the measurement, not improving the measured object—but the downstream effect is that measurement speed directly enables faster SAE development cycles.

Fourth, it provides a diagnostic for latent quality that does not require human labels at evaluation time. The decile-interpretability profile (Figure A1)—where highly interpretable latents maintain flat accuracy across all activation deciles while marginally interpretable latents show sharp drops—is a structural signature that can be computed entirely automatically (LLM intruder detection at each decile) and that reveals information about latent coherence that aggregate accuracy alone obscures. Two latents with identical average intruder detection accuracy of 60% could have very different profiles: one flat across deciles (coherent feature, stable semantics), one with 90% accuracy at the top decile and 30% at the bottom decile (feature that only "means something" when strongly activated). The flat-profile latent is arguably more genuinely interpretable, and the paper's method reveals this distinction that a single number would miss. This makes the decile profile a candidate for inclusion in SAE benchmarking suites like SAEBench (Karvonen et al., 2025), where metrics that capture the structure of interpretability, not just its average level, could help distinguish SAE architectures that produce genuinely coherent features from those that produce features interpretable only in their most extreme activations.

The paper also resolves a tension in the interpretability literature that has been largely implicit. On one hand, explanation-based evaluation has become standard and has produced influential results (Templeton et al., 2024 finding monosemantic features at scale; Paulo et al., 2024 scaling explanation generation to millions of features). On the other hand, there has been persistent unease about whether explanation quality scores truly reflect latent quality—the hyperparameter sensitivity the paper cites is real, and every practitioner who has run explanation pipelines has experienced the frustration of tweaking prompts and watching scores shift. The paper provides a conceptual vocabulary for articulating this unease (the difference between latent interpretability and explanation quality) and a concrete alternative that sidesteps the issue entirely. It does not prove that explanation-based evaluation is wrong—the correlation between intruder detection and fuzzing F1 is 0.74 (Table 2), suggesting substantial overlap—but it gives the field a tool for asking whether explanation quality and latent interpretability diverge, and under what conditions. This converts an inchoate methodological concern into an empirically testable question.

Which directions become more attractive? The paper makes research on verifier-free evaluation methods more attractive—if intruder detection works, what other behavioral tasks could measure interpretability without explanations? The psychophysical connection suggests a rich design space: two-alternative forced choice between latent pairs, oddity tasks (which of three examples is from a different latent?), or similarity-rating tasks where evaluators judge whether two examples activate the same latent. The paper also makes research on the structure of activation distributions more attractive—the asymmetric decile-detection matrix (Figure 3, right panel) is an unexplained phenomenon that could reward investigation, and the connection between activation strength and semantic homogeneity is a question the method is well-suited to address.

Which directions become less attractive? The paper implicitly argues that fine-tuning explanation prompts and optimizing explanation-generation hyperparameters is a less productive use of research effort than it might have seemed. If explanation quality scores are contaminated by nuisance variables that have nothing to do with latent interpretability, then the marginal return to prompt engineering for explanation generation is lower than the marginal return to developing better discriminative evaluation methods. The paper does not make this argument explicitly, but it is a natural reading of the motivation: the "additional hyperparameters and prompt choices which are likely to affect the final results" (Section 1) are a cost of the explanation paradigm, and the proposed methods eliminate that cost rather than optimizing within it.

Follow-Up Research This Work Enables

Scaling human validation to 500+ latents with naive crowdworkers rather than author self-evaluation. The paper's strongest claim—the 0.85 Spearman correlation between LLM and human intruder detection—rests on 56 latents evaluated by the two authors (Section 3.1.1). A direct follow-up would recruit 50–100 crowdworkers via a platform like Prolific or Amazon Mechanical Turk, train them on the intruder task with 10–20 synthetic examples, and have each worker evaluate 20–30 latents (5–10 prompts each), targeting a total of 500–1000 latents across workers. Each latent would receive judgments from 3–5 workers to enable inter-rater reliability computation. The key question: does the LLM-human correlation hold at 0.80+ for naive humans, or does it drop substantially when evaluators lack SAE expertise? A secondary analysis: compare author scores to crowdworker scores on a shared subset of latents to quantify the expertise effect directly. The paper's existing infrastructure (the intruder detection prompt format, the 32-token example construction, the decile-based sampling) transfers directly—the only new component is the human subjects protocol. This study would either validate the paper's central claim for the target user population or reveal a significant expertise confound that would qualify all subsequent uses of the method.

Head-to-head comparison of intruder detection against simulation scoring on the same 56 latents with the same human evaluators. The paper compares intruder detection to fuzzing and detection F1 (Table 2) but not to the original simulation scoring metric from Bills et al. (2023) that remains the most widely cited automated interpretability evaluation. A direct comparison would: (1) generate natural language explanations for the same 56 latents using the standard Bills et al. protocol (collect top-activating examples, prompt an LLM to summarize, evaluate with simulation scoring), (2) compute simulation scores for each latent, (3) have the same human evaluators who performed intruder detection also perform a simulation-adjacent task (e.g., read the explanation, then predict the latent's activation on new examples), and (4) measure whether human simulation accuracy or human intruder detection accuracy better predicts which latents humans subjectively rate as "interpretable" in a post-hoc survey. The paper's motivating argument—that explanation-based evaluation conflates latent quality with explanation quality—predicts that human intruder detection will better predict subjective interpretability ratings than human simulation accuracy. If this prediction fails (i.e., simulation accuracy and intruder detection accuracy predict subjective ratings equally well), then the practical case for explanation-free evaluation weakens substantially—both methods would be valid, and the choice between them would reduce to convenience rather than validity. The existing 56-latent dataset with human intruder detection scores makes this experiment logistically feasible; it requires only adding the explanation generation and simulation scoring steps and the subjective rating survey.

Token-level embedding scoring to test whether the attention bottleneck explains example embedding scoring's weak performance. The paper hypothesizes that example embedding scoring's modest human correlation (r = 0.48) stems from the embedding model's inability to attend specifically to highlighted tokens—the whole-sentence embedding dilutes the signal from the activating span (Section 4.2). A direct test: instead of embedding the entire 32-token sentence, extract the embedding of only the highlighted tokens (mean-pooling over the <<token>> spans) from the same all-MiniLM-L6-v2 model, and recompute the AUROC and human correlation. If the token-level embeddings substantially improve correlation with human scores (e.g., from 0.48 to 0.65+), the diagnosis is confirmed and a practical fix is identified. If they do not improve, then the problem is not attention but something deeper—perhaps the 22M-parameter embedding space lacks the representational capacity to capture SAE latent semantics even when directed to the right tokens, or perhaps SAE latents respond to contextual interactions that token-level embeddings cannot capture. A positive result would make example embedding scoring a genuinely useful fast-screening tool; a negative result would clarify that small embedding models are fundamentally insufficient and that research effort should shift toward making LLM-based intruder detection faster (e.g., through distillation or speculative decoding) rather than trying to salvage embedding-based approaches.

Replication across SAE architectures: TopK vs. ReLU vs. JumpReLU on the same base model. The paper evaluates only TopK SAEs with k=32 trained on SmolLM2 135M (Section 3). The decile stratification that is central to the analysis assumes that activation strength is a meaningful axis along which to examine interpretability, but this axis may behave differently in SAEs with different sparsity mechanisms. A systematic replication would train three SAE variants on the same base model (SmolLM2 135M) and the same layer (e.g., layer 15): (a) TopK (k=32, as in the paper), (b) ReLU with L1 penalty (tuned to achieve comparable average sparsity to the TopK variant), and (c) JumpReLU with tuned threshold. For each variant, evaluate 100 randomly sampled latents using LLM intruder detection (Claude Sonnet 3.5) at all ten deciles. The key comparisons: (1) does mean intruder accuracy differ across architectures? (2) do the decile-interpretability profiles (flat vs. declining, as in Figure A1) differ across architectures? (3) does the asymmetric decile-detection effect (Figure 3, right panel) replicate across architectures? If TopK produces flatter interpretability profiles than ReLU (suggesting more coherent features across the activation range), this would be evidence that the TopK activation function produces structurally different—and arguably better—features than L1-sparse SAEs, a claim that has been made in the SAE literature but not tested with human-correlated interpretability metrics. If all architectures produce similar profiles, then the paper's findings about decile-dependence are architecture-invariant properties of SAE latents in general.

Dynamic intruder detection: using the first few prompts to adaptively select the most informative decile for evaluation. The paper evaluates each decile separately with a fixed number of prompts, but in practice, a researcher evaluating a new SAE does not need per-decile accuracy for every latent—they need a single interpretability score per latent that is informative and efficiently estimated. An adaptive procedure inspired by psychophysical staircase methods could work as follows: start evaluating a latent at the median decile (decile 5), and based on the accuracy after 10 prompts, decide whether to move up (if accuracy is low, suggesting the latent is only interpretable at high activations, test decile 8 next), move down (if accuracy is high, test decile 2 next to see if the flat-profile pattern holds), or stop (if accuracy is near ceiling or floor at all deciles, further testing provides diminishing information). The total prompts per latent could be substantially reduced relative to the paper's 50-prompt-per-decile-per-latent protocol, making LLM-based evaluation cheaper without sacrificing the structural information that makes the method valuable. The deliverable would be an adaptive testing algorithm with a stopping rule, validated by showing that the resulting interpretability scores and profile classifications match those from the full 500-prompt evaluation on a held-out set of latents.

Application to transcoders and cross-layer SAEs to test whether the method generalizes beyond MLP-output features. The paper evaluates SAEs trained on MLP outputs at individual layers. Transcoders (which model the input-output function of an MLP sublayer rather than its intermediate activations) and cross-layer SAEs (which pool latents across multiple layers) represent different approaches to sparse dictionary learning with different interpretability properties. A natural extension would apply intruder detection to (a) transcoders trained on the same SmolLM2 135M MLP sublayers as the paper's SAEs, and (b) a cross-layer SAE that pools activations across layers 9–27. If transcoder features are more interpretable than MLP-output SAE features (as the transcoder literature suggests), this should manifest as higher intruder detection accuracy and flatter decile profiles. If cross-layer features are less interpretable (because they mix representations from different processing stages), this should manifest as lower accuracy and steeper decile profiles. The paper's methodology provides the measurement tool; the experiment would provide substantive evidence about which sparse coding approach yields more human-interpretable features, moving beyond the reconstruction-loss-and-sparsity metrics that currently dominate SAE comparisons.

Practical Applications and Downstream Use Cases

SAE architecture search and hyperparameter tuning with fast, automated feedback. A practitioner developing SAEs for a new model or domain needs to evaluate dozens or hundreds of training configurations (sparsity level, dictionary size, activation function, auxiliary loss terms) and select the one that produces the most interpretable latents. Without the methods in this paper, the practitioner's options are: (a) look at a few latents manually (subjective, doesn't scale, prone to cherry-picking), (b) use reconstruction loss and L0 sparsity as proxies (weakly correlated with interpretability; a low-reconstruction-loss SAE can still have uninterpretable latents), or (c) run a full explanation-based evaluation pipeline (slow, expensive, confounded by explanation-generation hyperparameters). The paper offers a two-stage alternative: run example embedding scoring on the top activation decile for all latents across all configurations (minutes, using a 22M-parameter model, producing above-random AUROCs of 0.64–0.70 per Figure 4) to identify the 3–5 most promising configurations, then validate those with LLM intruder detection (hours, using Claude Sonnet 3.5 or Gemini Flash 2.0, producing human-correlated scores at r ≈ 0.85 per Table 1). The specific benefit: SAE development cycles that currently take days (waiting for explanation-based evaluation on all configurations) could compress to hours (embedding-based screening plus targeted LLM validation), enabling more thorough hyperparameter sweeps and faster iteration. The paper's evidence that this workflow is viable is the 0.48 human correlation for example embedding scoring (Table 2)—noisy, but above random, and thus usable as a screening filter when the alternative is blind guessing.

Filtering and prioritizing latents for manual interpretability analysis. A common workflow in mechanistic interpretability research is: train an SAE, then manually inspect latents to find ones relevant to a particular behavior or phenomenon (e.g., "which latents fire on indirect object identification?" or "which latents represent factual associations?"). With thousands of latents per layer, exhaustive manual inspection is infeasible. Intruder detection provides an automated pre-filter: evaluate all latents with an LLM (or, for speed, example embedding scoring on the highest decile), rank by interpretability score, and have the human researcher inspect only the top-ranked latents. The paper's data suggests this would be effective: one third of latents achieve intruder detection accuracy above 80% (Section 4.1), and these high-accuracy latents maintain flat interpretability profiles across all activation deciles (Figure A1), meaning they are genuinely coherent features rather than artifacts of the evaluation method or the sampling procedure. A researcher who uses intruder detection to select the top 20% of latents by score can be confident (based on the human correlation of 0.85, Figure 2) that they are disproportionately selecting latents a human would also find interpretable, saving substantial manual inspection time without introducing systematic bias toward latents that "look good to LLMs but not to humans"—the paper explicitly finds that LLMs underestimate human interpretability, not overestimate it (Figure 2, right panel).

Benchmarking SAE training techniques with a metric that isolates latent quality from explanation quality. The SAE literature has produced numerous training innovations—TopK activation (Gao et al., 2024), JumpReLU, Gated SAEs, auxiliary loss variants, Matryoshka SAEs—and a persistent challenge is determining which technical choices actually produce more interpretable latents, as opposed to which produce better reconstruction-sparsity tradeoffs that may or may not translate to human understanding. Existing benchmarks like SAEBench (Karvonen et al., 2025) use explanation-based evaluation, which the paper argues conflates latent quality with explanation-generation quality. Intruder detection offers a complementary metric for benchmark suites: it is fully automated (using LLM evaluators validated against humans at r = 0.85), it eliminates the explanation-generation confound, and it provides per-decile resolution that reveals structural properties of latent interpretability (flat vs. declining profiles) that aggregate scores miss. A benchmark that includes both intruder detection accuracy (all deciles) and simulation scoring would allow researchers to ask: do these two metrics rank SAE techniques the same way? If they do, the choice between them is a matter of convenience. If they don't, the disagreement is itself informative—it identifies SAE techniques that produce latents that are easy to explain but hard to discriminate, or vice versa, revealing a previously invisible dimension of SAE quality.

When to Prefer This Method

The paper explicitly positions intruder detection and example embedding scoring as alternatives to explanation-based evaluation, not as uniformly superior replacements. The tradeoff is articulated in terms of what is measured (discriminability vs. explainability), what confounds are eliminated (explanation-generation hyperparameters vs. verbalization bias), and what practical constraints apply (compute budget, need for human validation). The decision rule that emerges from the paper's results is:

Prefer LLM intruder detection when: (a) the primary goal is ranking latents or SAE configurations by how interpretable a human would find them, (b) you have access to a capable LLM evaluator (roughly 30B+ parameters; below this, the human correlation degrades—Llama 3.1 8B achieves r = 0.64, Table 1), (c) you need the measurement to be directly validated against human judgment on the identical task (the paper provides the 0.85 correlation benchmark for Claude Sonnet 3.5, Figure 2), and (d) you want per-decile structural information about interpretability (flat vs. declining profiles, Figure A1) that aggregate metrics cannot provide. The method is not appropriate when you need natural language descriptions of latents for communication, downstream reasoning, or hypothesis generation—intruder detection measures recognition, not description.
Prefer example embedding scoring when: (a) you need rapid, approximate screening across many latents or SAE configurations (milliseconds per latent using a 22M-parameter model vs. seconds to minutes for LLM intruder detection), (b) you are evaluating latents where the highest-activation examples are the primary target of analysis (AUROC 0.64–0.70 on the top decile, Figure 4, left panel), and (c) below-random performance on some latents is acceptable because you are using the scores for filtering (keeping the top-ranked latents) rather than for precise quantification. The method should not be used when human-level fidelity is required—the 0.48 correlation with human intruder scores (Table 2) means that example embedding scoring produces rankings that differ substantially from what a human would produce, and decisions based on small score differences between latents or configurations are unreliable.
Prefer explanation-based evaluation when: (a) the downstream use case requires natural language descriptions of latent semantics (e.g., for communication in a research paper, for building human-in-the-loop interpretability tools, or for generating hypotheses about model behavior), (b) you need to evaluate the causal effects of latents on model output (intruder detection is purely correlational; it does not test whether activating a latent causes specific model behaviors, which explanation-based causal evaluations like Paulo et al., 2024 and Gur-Arieh et al., 2025 address), or (c) you want to test whether a latent's meaning can be verbalized at all—the explanation-generation step is itself a diagnostic for whether a latent's semantics are linguistically articulable, and failure to generate a good explanation is informative even if intruder detection accuracy is high.

The paper does not argue for abandoning any of these methods. It argues that intruder detection fills a specific gap—measurement of latent interpretability uncontaminated by explanation-generation noise—and that example embedding scoring fills a different gap—rapid approximate screening where compute constraints preclude LLM evaluation. The appropriate method depends on the research question, the available compute, and the tolerance for measurement error, and the paper's contribution is providing the quantitative characterization of these tradeoffs (the correlation matrices in Tables 1 and 2, the decile-dependence curves in Figures 3 and 4) that enables practitioners to make informed choices.