ArXiv: 2512.23988

🎯 Pitch

Sparse autoencoders trained on chain-of-thought activations automatically discover 'reasoning vectors' for behaviors like backtracking and reflection—without any human labels—and you can steer them to cut down overthinking while preserving accuracy.

1. Executive Summary

This paper proposes RISE: Reasoning behavior Interpretability via Sparse auto-Encoder, an unsupervised framework that discovers distinct reasoning behaviors in large language models by training sparse auto-encoders (SAEs) on sentence-level activations extracted from chain-of-thought traces. Using DeepSeek-R1-Distill-Qwen-1.5B on the MATH dataset, RISE identifies disentangled latent directions—termed reasoning vectors—that correspond to interpretable behaviors such as reflection (the model revisiting and verifying previous steps) and backtracking (the model abandoning a current path for an alternative approach), validated through clustering analyses where these behaviors occupy separable regions in the SAE decoder column space. Targeted interventions on SAE-derived vectors controllably amplify or suppress specific reasoning behaviors without retraining—for example, reducing reflection steps from approximately 90 to 34 on AIME25 tasks while preserving answer correctness—and the framework further enables discovery of novel behaviors beyond human supervision, such as confidence-related vectors whose injection reduces reflection and backtracking frequencies while shifting token distributions toward more confident mathematical calculation, establishing that abstract cognitive patterns in reasoning can be linearly encoded, selectively controlled, and generalized across domains only when discovered through unsupervised decomposition rather than reliance on predefined human-labeled concepts.

2. Context and Motivation

The Core Problem: Reasoning Behaviors Are Observable but Their Internal Mechanisms Remain Opaque

The fundamental question this paper tackles is: when a language model produces a long chain-of-thought reasoning trace, what is actually happening inside the model? Modern reasoning models—such as OpenAI's o-series (Jaech et al., 2024), DeepSeek-R1 (Guo et al., 2025), and Gemini Flash Thinking (Google, 2025)—generate extended multi-step solutions that include metacognitive behaviors like self-verification ("let me double-check"), uncertainty expression ("wait, I might be wrong"), and strategy switching ("alternatively, we could try..."). These behaviors are clearly visible in the model's textual output, and prior work has established that they correlate with improved reasoning performance (Chen et al., 2025; Gandhi et al., 2025; Venhoff et al., 2025; Ward et al., 2025). However, the internal representations that give rise to these behaviors remain almost entirely unexplored.

This gap is significant for several reasons, though the paper only implicitly develops them:

  • Scientific understanding: Without knowing how reasoning behaviors are encoded in activation space, we cannot answer foundational questions about what reasoning models learn differently from base models. Are reflection and backtracking emergent properties of the pretrained architecture that get "unlocked" by reinforcement learning, or are they entirely new capabilities? Do they share underlying circuitry, or are they independent mechanisms? The paper's clustering and intervention results provide evidence toward answering these questions—showing, for example, that reflection and backtracking occupy overlapping subspaces (Figure 3) while being distinguishable from other behaviors.

  • Controllability and reliability: If reasoning behaviors are linearly encoded as directions in activation space—as the linear representation hypothesis (Park et al., 2023; Olah and Jermyn, 2024) would suggest—then we should be able to steer them. Controllable reasoning would allow practitioners to shorten verbose but correct solutions by suppressing excessive self-verification, amplify caution when correctness matters more than efficiency, or filter for confident answers in high-throughput settings. The paper's interventions (Section 4.4) demonstrate precisely this kind of control without retraining.

  • Discovery of unknown behaviors: Existing analyses of reasoning are limited to behaviors that humans can name and identify in text—reflection, backtracking, overthinking, underthinking (Wang et al., 2025d). But there may be important reasoning patterns that do not surface as identifiable textual markers. They might manifest as subtle distributional shifts, activation patterns, or structural properties of the reasoning trajectory that no human annotator would think to label. The paper's discovery of confidence-related vectors (Section 5) that cluster coherently in SAE space—despite confidence being difficult to define at the word level—demonstrates that unsupervised methods can surface behaviors invisible to supervised approaches.

Why Existing Approaches Fall Short

Prior work on interpreting reasoning models falls into two broad categories, both of which the paper argues are insufficient:

Approach 1: Human-Supervised Activation Engineering (DiffMean and Variants)

The dominant paradigm for extracting interpretable directions from language model activations is the Difference-of-Means (DiffMean) approach (Marks and Tegmark, 2023), described in detail in Section 1. The procedure is conceptually straightforward:

  1. Construct a contrastive dataset with human supervision by categorizing samples into two groups—for example, "reflection" reasoning steps versus "non-reflection" steps, or "correct" versus "incorrect" solutions.
  2. Extract the latent representations hi+h_i^+ for positive examples and hj−h_j^- for negative examples at a chosen layer.
  3. Compute the steering vector as the difference of category means:

v=1N+∑i=1N+hi+−1N−∑j=1N−hj−v = \frac{1}{N^+} \sum_{i=1}^{N^+} h_i^+ - \frac{1}{N^-} \sum_{j=1}^{N^-} h_j^-

  1. Apply this vector during inference by adding it (scaled by a coefficient) to the model's hidden states to shift behavior in the desired direction.

This approach has been effective in domains where the conceptual categories are clearly separable and easily labeled. For sentiment control (Han et al., 2023), "happy" versus "sad" text is straightforward to collect and label. For truthfulness (Wang et al., 2025c), factual versus hallucinated statements can be identified. For safety (Zou et al., 2023), harmful versus benign outputs are relatively well-defined.

Where DiffMean fails for reasoning. The paper argues that reasoning behaviors present a fundamentally different kind of structure that makes DiffMean poorly suited—and this argument operates on multiple levels:

First, the granularity problem. Reasoning behaviors don't map cleanly to whole responses or even to individual tokens. A single CoT trace might contain reflection steps, backtracking steps, straightforward calculation steps, and steps that blend multiple behaviors. The paper's choice to work at the sentence level (using <\\n\\n> delimiter tokens as step boundaries, Section 3.2) is motivated by the observation that "reasoning behaviors cannot be trivially explained by token-level attributes, since the same token can play diverse roles across different contexts" (Section 3.2). A token like "Wait" can signal genuine reflection, rhetorical pausing, or be part of an unrelated phrase. This means that constructing contrastive pairs requires careful step-level annotation, which is labor-intensive and introduces human bias about what "counts" as reflection.

Second, the coverage problem. Human-defined categories capture only the behaviors that researchers have already named and operationalized. Prior work on reasoning behaviors has focused on a small set—reflection, backtracking, overthinking, underthinking (Chen et al., 2025; Wang et al., 2025d; Venhoff et al., 2025). The paper states explicitly that supervised methods are limited because "it is infeasible to capture the full spectrum of potential reasoning behaviors, many of which are difficult to define in token space" (Section 1). The discovery of confidence-related vectors in Section 5—vectors that modulate reasoning style and entropy without corresponding to any obvious textual marker—provides concrete evidence for this concern. If the researchers had only looked for behaviors they could name, they would never have found these directions.

Third, the separability assumption. DiffMean assumes that the two categories are well-separated in activation space and that their mean difference captures a meaningful direction. But the paper's later results show that reasoning behaviors like reflection and backtracking occupy overlapping representational subspaces (Figure 3, where the Silhouette score for "Reflection vs Backtracking" is consistently lower than for "Others vs Reflection" or "Others vs Backtracking"). This means that a simple reflection-minus-non-reflection DiffMean vector would likely blend reflection-specific directions with directions shared across multiple behaviors, producing a less pure steering signal. The SAE's sparsity constraint (enforced by the λ∥z∥0\lambda\|z\|_0 term in Equation 2) explicitly encourages disentanglement by forcing each activation to be reconstructed from a small number of dictionary elements, making it more likely that distinct behaviors map to distinct (or nearly distinct) decoder columns.

Approach 2: Supervised Fine-Tuning for Reasoning Transfer (Including Prior SAE Work)

A recent effort by Wang et al. (2025a) also used SAEs in the context of reasoning models, but with a fundamentally different goal. That work treated the SAE primarily as a bridge—a mechanism to extract reasoning-related features from a strong reasoning model and transfer them to a base model through supervised fine-tuning. The SAE was a means to an end (improving the base model's reasoning), not a tool for understanding the reasoning model itself.

The paper distinguishes its contribution sharply: "Our work instead aims to directly analyze these mechanisms" (Section 2, final sentence of the Activation Steering paragraph). This distinction matters because:

  • Transfer-focused work asks: "Can we use SAE features to make a weaker model reason better?"
  • This paper asks: "What structure exists in the reasoning model's activation space, and what does that structure tell us about how reasoning works?"

The transfer approach does not characterize the geometry of the decoder space, does not validate that individual columns correspond to specific behaviors, and does not demonstrate causal control through intervention. In other words, it treats the SAE as a black-box feature extractor, while this paper opens the box and examines what's inside.

The Broader Landscape: Studies of Reasoning Mechanisms Without SAEs

Beyond activation steering, a parallel literature has studied reasoning mechanisms through behavioral analysis and entropy patterns:

  • Entropy-based analyses (Fu et al., 2025; Zhang et al., 2025; Cui et al., 2025; Wang et al., 2025b; Zhao et al., 2025) examine how model confidence (measured through output token entropy) evolves during reasoning. These works find that entropy dynamics correlate with reasoning quality and that certain high-entropy "minority tokens" drive reinforcement learning improvements. However, these analyses are output-level—they characterize what the model produces, not the internal geometry that produces it.

  • Verbosity and length analyses (Fu et al., 2024; Huang et al., 2025; Sheng et al., 2025) argue that current reasoning processes are "often verbose, with certain parts being ineffective" (Section 1) and that short versus long reasoning traces can be distinguished in activation space (Eisenstadt et al., 2025; Sun et al., 2025). The paper's finding of a "length-aware organization in the latent space" (Section 6, Appendix E) where response length emerges as a structural axis in the SAE decoder columns connects to this literature but goes beyond it by showing that this organization emerges unsupervised—the SAE was never told about response length, yet its decoder columns self-organize along this dimension.

  • Behavioral taxonomies (Gandhi et al., 2025; Chen et al., 2025; Venhoff et al., 2025) identify specific reasoning behaviors from model outputs and argue that they are causally important. These studies rely entirely on human-defined categories and textual analysis.

Reconciling Contradictory Signals from Prior Work

An implicit tension the paper navigates is this: on one hand, reasoning behaviors appear to be real and important—they show up in model outputs, correlate with performance, and can be prompted or suppressed. On the other hand, existing attempts to find their internal signatures have been limited to supervised approaches that can only find what they're looking for. This creates a discovery bottleneck: without knowing what to look for, the field may be missing the most interesting structure.

The paper's position is that unsupervised decomposition is the way through this bottleneck. By training an SAE to reconstruct step-level activations with a sparsity penalty, the model is forced to find a dictionary of latent features that efficiently represent the activation space. No human tells it to find reflection, backtracking, or confidence vectors. If those behaviors are genuinely encoded as disentangled directions, the SAE should recover them automatically—and then human analysis can validate them, rather than human hypotheses constraining the search space from the start.

How This Paper Positions Itself Methodologically

The paper frames its contribution as the first application of unsupervised sparse auto-encoding to the problem of reasoning behavior discovery, but this framing only makes sense when situated against the methodological status quo:

The status quo: Supervised decomposition. Given a behavior of interest (e.g., reflection), construct a labeled dataset of examples vs. non-examples, compute a steering vector, apply it, evaluate. This is hypothesis-driven: you must name the behavior before you can find it.

This paper: Unsupervised decomposition → human analysis → causal validation. Train an SAE with no behavioral labels, visualize the decoder column geometry, overlay human annotations to see if known behaviors cluster coherently, then intervene to test causality. This is discovery-driven: the SAE proposes candidate directions, and humans analyze whether they correspond to meaningful behaviors.

The paper's Theorem 1 (Section 4.1, Appendix B) provides theoretical motivation for this approach. Under the linear representation hypothesis—which posits that activations are sparse linear combinations of latent "concept" vectors—the SAE decoder columns should recover the true dictionary up to permutation and scaling, as long as the dictionary satisfies incoherence (concept vectors are not too similar to each other) and the representations are sufficiently sparse. This theorem, while not the paper's central contribution, justifies why we should expect unsupervised recovery to work: if reasoning behaviors are genuinely encoded as sparse linear combinations of basis directions, the SAE will find those directions without supervision.

Specific Gaps the Paper Directly Addresses

Reading through the introduction and related work, the paper identifies four concrete gaps that it aims to fill:

  1. No unsupervised characterization of the reasoning behavior space exists. Prior work either analyzes outputs behaviorally or extracts supervised directions. The full geometry—what behaviors exist, how they relate to each other, whether they form clusters or continuous spectra—is unknown.

  2. The causal effect of individual latent directions on reasoning has not been demonstrated. DiffMean vectors mix multiple underlying directions (since the mean difference over a heterogeneous category averages out within-category variation). SAE decoder columns, by contrast, are individual basis vectors. Showing that manipulating a single decoder column causally changes reasoning behavior is a stronger claim about disentanglement and control.

  3. Behaviors beyond human-defined categories have not been systematically sought. The paper makes confidence (Section 5) the case study, but the methodology is general: define a proxy objective (entropy), identify decoder columns that optimize it, and then analyze what behavioral role they play. This opens the door to discovering reasoning behaviors that humans would not think to label.

  4. Cross-domain generalization of reasoning vectors is untested. The paper shows that SAE columns trained on MATH500 generalize to GPQA-Diamond (graduate-level science QA) and KnowLogic (commonsense reasoning) (Table 1, Table 2), demonstrating that the discovered directions are not dataset-specific artifacts but rather capture genuine reasoning behaviors that transfer across domains.

Why RISE Is Positioned as a Framework, Not Just a Method

The paper's title and introduction present RISE as a framework rather than a single technique, and this is a deliberate positioning choice. The framework consists of three stages—(i) unsupervised SAE training on step-level activations, (ii) geometric analysis and behavior mapping in the decoder column space, and (iii) causal intervention to validate and apply the discovered vectors. Each stage builds on the previous one, and the framework is designed to be extensible: future work could plug in different SAE architectures, different discovery objectives (beyond entropy minimization for confidence), or different intervention strategies. The paper's specific findings about reflection, backtracking, and confidence serve as existence proofs for the framework's capabilities rather than as final answers about reasoning.

This positioning also explains why the paper spends so much space on visualization and clustering (Figures 2, 3, 7a, 11, 12) alongside intervention results. Demonstrating that the framework produces interpretable, geometrically coherent structure is as important as demonstrating causality—because without interpretability, the unsupervised discovery would just be extracting arbitrary directions from noise, and practitioners would have no way to know which directions to use for which goals.

3. Technical Approach

3.1 Reader Orientation

This paper builds an unsupervised analysis pipeline that discovers distinct reasoning behaviors—like reflection, backtracking, and confidence modulation—by decomposing a language model's internal activations during chain-of-thought reasoning into a sparse dictionary of interpretable latent directions, then validating those directions through geometric analysis and causal intervention. The core problem is that reasoning behaviors are observable in model outputs but their internal representational structure is unknown, and the solution takes the form of a three-stage framework: (1) train a sparse auto-encoder on hidden states to learn a disentangled basis, (2) analyze the geometry of that basis to identify behavioral clusters, and (3) intervene causally by manipulating individual basis directions during inference to verify they control specific behaviors.

3.2 Big-Picture Architecture (Diagram in Words)

The RISE framework consists of five major components connected in a pipeline:

  1. Data Preparation Module: Takes raw chain-of-thought responses, segments them into sentence-level reasoning steps using delimiter tokens (<\n\n>), and re-runs inference to extract hidden state activations at each step boundary from a target transformer layer. Produces a set of activation vectors {h_i^l} where i indexes steps and l is the chosen layer.

  2. Sparse Auto-Encoder (SAE): Learns a dictionary of D latent feature vectors (stored as decoder columns) that reconstruct each activation using only a sparse subset of features. The encoder compresses the d-dimensional activation into a D-dimensional sparse code via ReLU activation and an L0 sparsity penalty; the decoder reconstructs from that code. After training, each decoder column w_i (a d-dimensional vector) represents a candidate reasoning vector—a direction in activation space hypothesized to encode a specific reasoning behavior.

  3. Geometric Analysis Module: Projects the learned decoder columns into 2D via UMAP (using cosine similarity as the internal metric), then overlays human-annotated behavior labels (reflection, backtracking, other) to assess whether semantically related behaviors cluster in the decoder column space. Quantifies clustering quality through normalized Silhouette scores across layers.

  4. Behavior Discovery Module: Given a proxy objective (e.g., minimizing output token entropy as a measure of confidence), identifies which decoder columns most strongly influence that objective by optimizing a scoring vector S over the decoder columns. The top-scoring columns define new behavior vectors—like "confidence reasoning vectors"—that were never explicitly labeled or named by humans.

  5. Causal Intervention Module: During inference, projects the hidden state h at each reasoning step away from (or toward) a selected decoder column direction w_i using the projection removal formula h' = h - w_i(w_i^T h), effectively suppressing or amplifying the associated behavior. Evaluates the intervention's effect by measuring changes in behavior frequency (e.g., counting reflection steps) and answer correctness.

Information flows sequentially: raw responses → step activations → SAE training → decoder column geometry → human annotation overlay (for known behaviors) OR objective-driven column selection (for novel behaviors) → intervention during inference → behavioral measurement.

3.3 Roadmap for the Deep Dive

  • First, the thought representation construction pipeline (Section 3.2 of the paper): how raw chain-of-thought text gets converted into the step-level activation vectors that the SAE trains on. This is the foundation—everything downstream depends on what information is captured in these representations.
  • Second, the sparse auto-encoder architecture and training objective (Section 3.1): the mathematical formulation, the sparsity mechanism, and why SAEs are theoretically motivated for this task. This is the core technical machinery.
  • Third, the theoretical justification (Theorem 1, Section 4.1): why we should expect the SAE decoder columns to recover the true behavior dictionary under the linear representation hypothesis. This explains the "why" behind the method.
  • Fourth, the SAE training configuration (Section 4.2): the specific model, dataset, layer choices, hyperparameters, and implementation details—everything needed to reproduce the training.
  • Fifth, the geometric analysis procedure (Section 4.3): how decoder columns are visualized, how human annotations are obtained, and how clustering quality is quantified.
  • Sixth, the causal intervention mechanism (Section 4.4): the exact mathematical operation for projecting out behavior directions during inference, how behavior-specific columns are selected and aggregated, and how intervention strength is controlled.
  • Seventh, the novel behavior discovery procedure (Section 5): the optimization formulation for finding confidence-related columns, how the scoring vector is learned, and how discovered vectors are validated.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a methodology paper whose core idea is that reasoning behaviors in language models are encoded as sparse linear combinations of latent basis directions in activation space, and that a sparse auto-encoder trained without behavioral labels can recover these directions in an interpretable, disentangled, and causally manipulable form.

Thought Representation Construction

The first technical challenge is deciding at what granularity to extract representations from the model. Token-level representations are problematic because reasoning behaviors span multiple tokens and because "the same token can play diverse roles across different contexts"—for instance, the token "Wait" might signal genuine reflection in one context, be part of a mathematical expression in another, or serve as a rhetorical filler in a third. The paper resolves this by working at the sentence level, specifically at the boundaries between reasoning steps as delimited by the <\n\n> marker that reasoning models naturally produce.

The construction procedure has three explicit stages:

Stage 1: Response collection. For each question in the MATH500 training set (500 randomly sampled examples from Hendrycks et al., 2021), the paper feeds the question into DeepSeek-R1-Distill-Qwen-1.5B and collects the model's complete chain-of-thought response. This model is a distilled version of the full DeepSeek-R1 trained via reinforcement learning to produce extended reasoning traces with self-verification and strategy-switching behaviors visible in the text. The choice of this model is deliberate: it exhibits the metacognitive behaviors (reflection, backtracking) that the paper aims to study, making it a natural testbed.

Stage 2: Step segmentation. Each response is split into sentence-level reasoning steps using the delimiter symbol <\n\n>. This produces k steps per response, where k varies across samples depending on how many natural reasoning segments the model generates. A response might be segmented into, say, 8 steps for a simple problem and 45 steps for a complex multi-path exploration. The delimiter is not artificially imposed—it reflects the model's own segmentation of its reasoning trace, so the step boundaries correspond to points where the model itself signals a transition between reasoning substages.

Stage 3: Activation extraction. To obtain the hidden representation for each reasoning step, the paper re-runs the model on the full question-and-response pair—that is, the input is the concatenation of the original question and the model's complete generated response—and extracts the hidden state at the position of each <\n\n> delimiter token. Specifically, the paper extracts the residual stream representation after each transformer layer, producing a set of vectors {h_i^l} where i ∈ {1, ..., N} indexes individual reasoning steps across all responses and l indexes the transformer layer.

The choice to extract at delimiter tokens rather than averaging over all tokens in a step is significant: the delimiter token's representation at the output of a transformer layer has attended to all preceding tokens (including the entire reasoning step that just concluded), so it captures a compressed summary of the step's content from the model's perspective. The residual stream is chosen because it is the primary information highway in transformer architectures—it is where both attention and MLP outputs are deposited, and prior mechanistic interpretability work has found that linearly decodable features tend to reside there (as opposed to being locked inside attention patterns or MLP nonlinearities). The paper trains a separate SAE for each layer of interest, using the activations from that layer only.

What emerges from this procedure is a dataset of N activation vectors per layer, each d-dimensional (where d is the model's hidden dimension, 1536 for the 1.5B parameter model), each corresponding to the model's representation of a completed reasoning step. These activations encode whatever information the model has accumulated about the reasoning process up to that point—including the content of the step, its relationship to prior steps, the model's uncertainty, its strategic decisions, and potentially the "trajectory" of the reasoning so far. The SAE's task is to decompose these rich representations into a dictionary of interpretable basis directions.

Sparse Auto-Encoder Architecture and Objective

The SAE serves as the core decomposition engine. Its architecture is a standard single-hidden-layer auto-encoder with a critical design choice: the latent representation is constrained to be sparse, meaning that each activation is reconstructed using only a small number of active latent features. This sparsity is what encourages disentanglement—if the model can only use a few basis vectors to reconstruct any given activation, it is forced to learn basis vectors that correspond to atomic, reusable concepts rather than complex mixtures of concepts.

Encoder:

z=σ(WencoderTh+bencoder)z = \sigma(W_{\text{encoder}}^T h + b_{\text{encoder}})

where $h \in \mathbb{R}^d$ is the original activation vector from the language model ($d$ is the model's hidden dimension, 1536 for the 1.5B model), $W_{\text{encoder}} \in \mathbb{R}^{d \times D}$ is the encoder weight matrix ($D$ is the SAE's hidden dimension, set to 2048), $b_{\text{encoder}} \in \mathbb{R}^D$ is the encoder bias vector, $\sigma$ is the ReLU activation function applied element-wise, and $z \in \mathbb{R}^D$ is the sparse latent code—a compressed representation of $h$ in the SAE's latent space.

What it computes: the encoder projects the $d$-dimensional language model activation down to (or in this case, up to, since $D = 2048 > d = 1536$) a $D$-dimensional latent space, then applies ReLU to zero out negative values. The ReLU nonlinearity is what enforces sparsity: any latent dimension whose pre-activation is negative gets set to exactly zero, meaning it contributes nothing to the reconstruction. Only dimensions with positive pre-activation survive.

Why this form: using ReLU (rather than sigmoid, tanh, or a linear activation) is the standard choice for sparse auto-encoders because it naturally produces exact zeros—values that are literally absent from the reconstruction—rather than small-but-nonzero values that would still contribute. This hard thresholding is what makes the latent representation genuinely sparse rather than merely "mostly small." A linear auto-encoder with an L1 penalty would produce many near-zero values but almost never exact zeros; the ReLU guarantees zeros for all negative pre-activations.

Decoder:

h^=WdecoderTσ(z)+bdecoder\hat{h} = W_{\text{decoder}}^T \sigma(z) + b_{\text{decoder}}

where $W_{\text{decoder}} \in \mathbb{R}^{D \times d}$ is the decoder weight matrix (the transpose of an $d \times D$ matrix—each row is a $d$-dimensional vector in the original activation space), $\sigma(z)$ is the sparse code after ReLU (which is just $z$ itself since ReLU doesn't change positive values), $b_{\text{decoder}} \in \mathbb{R}^d$ is the decoder bias vector, and $\hat{h} \in \mathbb{R}^d$ is the reconstruction of the original activation.

What it computes: the decoder takes the sparse code $z$ (with its many zeros enforced by ReLU) and constructs a linear combination of the rows of $W_{\text{decoder}}$—each row $w_i$ is a direction in the original activation space—weighted by the active (nonzero) entries in $z$. The bias $b_{\text{decoder}}$ provides a constant offset. The reconstruction $\hat{h}$ is the SAE's best attempt to reproduce the original activation $h$ using only the active dictionary elements.

Why this form: the decoder columns $w_i$ (rows of $W_{\text{decoder}}$) are the reasoning vectors—the atomic directions that the paper hypothesizes correspond to specific reasoning behaviors. Because the reconstruction is a linear combination of these columns, each active column contributes additively to the reconstructed activation. If a particular behavior (say, reflection) is encoded as a specific direction in activation space, then activations from reflection steps should have a nonzero coefficient for the corresponding decoder column, and that column's direction should point toward the "reflection" region of activation space. The linearity of the decoder is crucial: it means the effect of each behavior vector is additive and independent (conditioned on the sparsity of $z$), which is what enables the clean intervention operations in Section 4.4.

Training objective:

L=∥h^−h∥22+λ∥z∥0\mathcal{L} = \|\hat{h} - h\|_2^2 + \lambda \|z\|_0

where $\|\hat{h} - h\|_2^2$ is the squared Euclidean distance between the reconstructed and original activations (the reconstruction error), $\|z\|_0$ is the L0 "norm" of the latent code—the number of nonzero entries in $z$—and $\lambda$ is a scalar hyperparameter controlling the trade-off between reconstruction fidelity and sparsity.

What it computes: the first term penalizes the SAE for failing to accurately reproduce the original language model activations. The second term penalizes the SAE for using too many active latent features to achieve that reconstruction. The optimizer must balance these two pressures: using more features would reduce reconstruction error but incur a higher sparsity penalty, while using fewer features would reduce the penalty but potentially lose important information. The result is that the SAE learns to represent each activation as the sum of a small number of basis vectors, where each basis vector captures some recurring pattern in the activation space.

Why this form: the L0 penalty directly counts the number of nonzero entries, which is the truest measure of sparsity—unlike an L1 penalty (which sums absolute values and can be reduced by making many values small without making them exactly zero), the L0 penalty only cares about whether each entry is zero or not. This makes it the ideal sparsity regularizer for interpretability, since each nonzero entry represents a distinct "concept" being used. However, L0 is non-differentiable and combinatorially hard in general. The ReLU activation makes it tractable: the number of nonzero entries in $z$ is simply the number of pre-activation values that are positive, which is determined by the encoder weights and biases. During training with gradient-based optimizers (the paper uses Adam), the L0 term is typically optimized via a straight-through estimator or by treating it as a regularizer that pushes pre-activations toward negative values (so they get zeroed out by ReLU) unless the reconstruction benefit of keeping them positive outweighs the penalty.

A subtle point: the sparsity penalty $\lambda \|z\|_0$ is applied on the latent code $z$, not on the weights. This means the SAE learns weights such that, for typical inputs, only a few latent features activate strongly enough to survive the ReLU threshold. The same decoder columns are reused across many different activations, but each individual activation only engages a sparse subset of them—which is exactly the property that makes the columns interpretable as independent behavioral dimensions.

Theoretical Motivation: Why the SAE Decoder Should Recover the True Dictionary

The paper provides Theorem 1 (Section 4.1, with full proof in Appendix B) to justify why unsupervised SAE training should recover interpretable behavior directions. This theorem is not empirically evaluated—the paper does not claim it provides tight guarantees for real LLM activations—but it serves as conceptual motivation for the approach.

The generative model (Equation 3):

h=Wa+εh = W a + \varepsilon

where $h \in \mathbb{R}^d$ is an observed activation, $W = [w_1, ..., w_m] \in \mathbb{R}^{d \times m}$ is an unknown "ground-truth" dictionary of $m$ latent behavior directions (each $w_i$ is a $d$-dimensional vector representing one atomic reasoning behavior), $a \in \mathbb{R}^m$ is a $k$-sparse code (only $k$ entries are nonzero, representing which behaviors are active), and $\varepsilon$ is bounded additive noise.

What this model says: each language model activation is generated by taking a small number of behavior vectors (at most $k$), multiplying each by a scalar coefficient (how strongly that behavior is expressed), summing them, and adding some noise. This is the linear representation hypothesis applied to reasoning: complex reasoning traces arise from linear combinations of a modest number of underlying cognitive primitives.

The theorem's assumptions (quoted verbatim):

  • (i) Incoherence: $\max_{i \neq j} \frac{|\langle w_i, w_j \rangle|}{\|w_i\| \|w_j\|} \leq \mu < 1$. In plain language: no two distinct behavior vectors are too similar to each other—their cosine similarity is bounded strictly below 1. This ensures that the dictionary vectors are sufficiently distinguishable that an algorithm could tell them apart.
  • (ii) Sparsity: $k < c/\mu$ for a universal constant $c$. The number of active behaviors in any single activation is bounded by a constant over the incoherence parameter. As behavior vectors become more similar (smaller $\mu$ bound), the maximum allowed sparsity decreases.
  • (iii) Activation: SAE uses ReLU nonlinearity. This ensures that the latent code naturally produces zeros.
  • (iv) Separation: Nonzero coefficients satisfy $|a_i| \geq \alpha > 0$. When a behavior is active, its coefficient is bounded away from zero—behaviors are either clearly "on" or "off," not hovering at near-zero values where they would be ambiguous.

The conclusion (Equation 5):

∃Π,D≻0 s.t. Wdec≈WΠD\exists \Pi, D \succ 0 \text{ s.t. } W_{\text{dec}} \approx W \Pi D

where $\Pi$ is a permutation matrix (reordering the columns), $D$ is a diagonal matrix with positive entries (rescaling each column), and $W_{\text{dec}}$ is the SAE's decoder matrix.

What this means operationally: as the number of training samples $N \to \infty$, any local optimum of the SAE training objective will learn a decoder matrix whose columns are (up to reordering and rescaling) the true behavior vectors $w_i$ from the generative model. The SAE doesn't know what to call each vector—it might put "reflection" in column 42 rather than column 1—but the set of vectors it learns will be the same set as the true dictionary.

Why this matters for the paper's approach: the theorem provides a principled reason to expect that training an SAE on step-level activations will recover behaviorally meaningful directions, assuming the linear representation hypothesis holds and the incoherence/sparsity conditions are approximately satisfied. If the theorem's assumptions fail—if, for instance, behaviors like reflection and backtracking are so correlated that their dictionary vectors have high cosine similarity (violating incoherence)—then the SAE might learn a direction that blends the two rather than separating them. The empirical validation in later sections (clustering, intervention) serves to check whether the idealized conditions hold in practice.

The paper also provides empirical validation of the sparsity and incoherence assumptions in Appendix C (Figure 9), though it's worth noting that this validation uses the SAE's reconstructed activations as a proxy for the ground-truth dictionary $W$ (since $W$ is unobservable). The maximum cosine similarity among SAE decoder columns is reported as bounded below 0.30 across layers, and the sparsity level $k$ (the number of active dictionary elements per reconstruction) is in the range of approximately 50-300 depending on the layer, which the paper argues is within the theoretical bound given the observed incoherence. Reconstruction error is reported as consistently low (below 0.08 across layers), justifying the use of reconstructed activations as proxies.

SAE Training Configuration

The paper trains SAEs on individual layers of DeepSeek-R1-Distill-Qwen-1.5B using 500 randomly sampled training examples from the MATH dataset (the MATH500 split). The details, quoted and paraphrased from Section 4.2 and Appendix C:

Model and data:

  • Base model: DeepSeek-R1-Distill-Qwen-1.5B (R1-1.5B). This is a 1.5-billion-parameter reasoning model with 28 transformer layers and hidden dimension $d = 1536$.
  • Training data: 500 questions from MATH, each producing a variable number of reasoning steps. The total number of step-level activations $N$ is therefore several thousand (500 questions times average steps per response).
  • An extension experiment also uses Qwen3-8B (for the results in Figure 5), confirming the approach generalizes to a different model family with 8 billion parameters.

SAE architecture:

  • Hidden dimension: $D = 2048$. The paper explicitly states this is "a relatively small hidden dimension" compared to the original model's $d = 1536$, chosen because "the complexity [of reasoning behaviors] is much lower than modeling raw language structure." The overcompleteness factor is modest—2048/1536 ≈ 1.33—meaning the SAE has only slightly more latent dimensions than input dimensions. This is a deliberate choice: aggressive overcompleteness (e.g., $D \gg d$) can lead to feature splitting where a single behavior gets distributed across multiple columns, reducing interpretability. The modest overcompleteness encourages each column to capture a coherent behavior.
  • Activation function: ReLU (as stated in Section 3.1, standard SAE formulation).

Training hyperparameters (quoted from Section 4.2):

  • Batch size: 1024
  • Learning rate: $1 \times 10^{-4}$
  • Optimizer: Adam (Kingma, 2014)
  • Learning rate schedule: cosine annealing with a warm-up over the initial 10% of training
  • Sparsity strength: $\lambda = 2 \times 10^{-3}$

What each hyperparameter does:

  • The batch size of 1024 means each gradient update is computed from 1024 step-level activations. This is relatively large, which helps stabilize training since the sparsity penalty creates a non-smooth optimization landscape.
  • The learning rate of $1 \times 10^{-4}$ is a standard choice for Adam with transformer-scale models. It's small enough to avoid divergence but large enough to make progress over a reasonable number of steps.
  • Cosine annealing gradually reduces the learning rate to near zero by the end of training, which helps the optimization settle into a good local minimum. The 10% warm-up linearly increases the learning rate from zero to $1 \times 10^{-4}$ over the first 10% of steps, preventing early instability when the weights are randomly initialized.
  • The sparsity strength $\lambda = 2 \times 10^{-3}$ determines the trade-off between reconstruction fidelity and sparsity. A value that's too high would produce very sparse codes (good for interpretability) but poor reconstructions (bad for faithfully representing the activation space). A value that's too low would produce dense codes (good reconstruction) but entangled features (bad for interpretability). The paper does not report a hyperparameter sweep for $\lambda$, but the empirical results—interpretable decoder columns, clear behavioral clustering—suggest this value achieves a workable balance.

Layer-wise training: The paper trains a separate SAE for each transformer layer of interest (typically all 28 layers, as shown in Figures 3, 9, 11, and 12). Each SAE is trained only on activations from that specific layer, extracted via the procedure in Section 3.2. The paper does not use a single SAE across multiple layers nor does it train on concatenated multi-layer representations. This layer-wise approach enables the analysis of how behavioral structure evolves through the network's depth (Figures 3 and 12), which reveals that mid-to-late layers encode the most behaviorally separable representations.

Computational considerations: Training 28 separate SAEs on thousands of activation vectors each is computationally modest by modern standards—each SAE has roughly $1536 \times 2048 \times 2$ (encoder plus decoder) ≈ 6.3 million parameters, and training on a few thousand examples converges quickly. The paper does not report wall-clock training time, but the overhead is clearly small enough to enable sweeping all 28 layers without special infrastructure.

Geometric Analysis: Visualizing and Validating the Decoder Column Space

After training, the SAE's decoder matrix $W_{\text{decoder}} \in \mathbb{R}^{2048 \times 1536}$ contains 2048 column vectors, each a 1536-dimensional direction in the original activation space. The geometric analysis aims to answer: do these columns form a semantically meaningful structure, or are they arbitrary?

Step 1: Human annotation of reasoning steps. The paper classifies each step-level activation $h_i^l$ into one of three categories: reflection, backtracking, or other. The classification uses an LLM-as-a-judge approach with precise behavioral definitions:

  • Reflection: the step re-examines earlier reasoning and expresses uncertainty ("let me verify," "wait, I might be wrong," "let me double-check"). The paper provides the judge prompt verbatim in Appendix D.
  • Backtracking: the step explicitly retracts the current approach and proposes an alternative strategy ("alternatively, we could try," "another approach would be"). Again, precise definitions are given in the prompt.
  • Other: steps that don't fall into either category—typically straightforward calculation, statement of intermediate results, or final answer formatting.

The annotation is performed by multiple judges (GPT-5, GPT-4o, Claude Sonnet 4.5, and a keyword-matching baseline) to assess consistency. The inter-judge agreement ratios (Figure 10, Appendix D) range from 85% to 94%, with GPT-5 and GPT-4o showing the highest pairwise agreement at approximately 94%. This high consistency validates that the behavioral categories are well-defined enough for reliable automated labeling.

Step 2: Mapping activations to active decoder columns. For each labeled activation $h_i^l$, the paper computes the SAE's latent code $z_i = \sigma(W_{\text{encoder}}^T h_i^l + b_{\text{encoder}})$ and identifies which decoder columns are actively used in reconstructing that activation—specifically, which entries of $z_i$ are nonzero. The "most active" decoder columns for a given behavior are those that are most frequently nonzero (or have the largest magnitude in $z$) across all activations labeled with that behavior.

Step 3: UMAP projection of decoder columns. The paper projects the 2048 decoder columns (each 1536-dimensional) into a 2D plane using UMAP (Uniform Manifold Approximation and Projection, McInnes et al., 2018) with cosine similarity as the internal metric. The choice of cosine similarity over Euclidean distance is critical and explicitly justified: "activation vectors are primarily meaningful in terms of their direction rather than their magnitude." In high-dimensional transformer activation spaces, the norm of a vector often correlates with position or token frequency rather than semantic content, so comparing directions via cosine similarity better captures functional similarity. The resulting 2D embedding places columns with similar directions (high cosine similarity) near each other and columns with orthogonal directions far apart.

Step 4: Overlaying behavioral labels. For each behavior category (reflection, backtracking, other), the paper highlights the decoder columns that are most frequently active for activations labeled with that behavior. The results are shown in Figure 2 (for the final layer) and Figure 11 (for all 28 layers, using response length labels instead of behavioral labels).

Step 5: Quantifying clustering quality. The paper computes Silhouette scores (Rousseeuw, 1987) for each layer. The Silhouette score measures, for each point, how similar it is to its own cluster compared to other clusters, producing a value between -1 and 1 (higher is better separation). The paper normalizes these scores across layers for visualization (Figure 3), computing three pairwise comparisons: Reflection vs. Backtracking, Others vs. Reflection, and Others vs. Backtracking.

Key geometric findings from this analysis (previewing Section 4.3):

  1. Reflection and backtracking columns "cluster more tightly in localized regions" of the UMAP projection (Figure 2), while "other" columns are "less centered and more dispersed across the entire SAE space." This suggests that the SAE has learned to separate behaviorally coherent directions from the undifferentiated background.

  2. Later layers generally achieve higher Silhouette scores than earlier ones (Figure 3), indicating that "behavioral concepts become more separable as representations deepen." This aligns with the intuition that early layers process low-level features while later layers build higher-level abstractions.

  3. A "slight decline occurs near the final layers (except for the very last one)," which the paper attributes to the "oversmoothing phenomenon in current LLMs (Wang et al., 2023), where token representations become excessively similar." In other words, the very last layers compress representations in a way that reduces behavioral distinctness.

  4. The comparison between reflection and backtracking "reveals only modest differences" in Silhouette score, whereas separating either from "other" is "substantially stronger." This is the key evidence that reflection and backtracking occupy overlapping representational subspaces—they are distinct but related behaviors that share some underlying directions, making them harder to separate from each other than from unrelated behaviors. This finding directly motivates the later design choice (in Section 4.4.1) to filter out decoder columns that are active for both reflection and backtracking before computing behavior-specific intervention vectors.

Causal Intervention: Manipulating Reasoning Behaviors at Inference Time

The geometric analysis establishes correlation: decoder columns cluster by behavior. The intervention analysis establishes causality: manipulating specific columns changes behavior. The intervention mechanism is mathematically simple but mechanistically powerful.

Step 1: Selecting behavior-specific columns. For a target behavior (say, reflection), the paper first identifies all decoder columns that are strongly associated with that behavior based on the annotation analysis. But simply taking all columns that are ever active for reflection would include columns that are also active for backtracking (since those behaviors share representational space). The paper explicitly filters out decoder columns that exhibit strong activations across multiple behaviors, keeping only columns that are uniquely or predominantly associated with the target behavior. This filtering step is crucial—without it, the intervention vector would blend reflection-specific directions with directions shared across behaviors, potentially producing mixed or unpredictable effects.

Step 2: Aggregating into a single intervention vector. From the filtered set of behavior-specific columns, the paper computes their average to obtain a single vector representing that behavior. Averaging is a simple choice that treats all behavior-specific columns as equally important; alternative aggregation methods (weighted by activation frequency, principal component extraction) are not explored.

Step 3: Normalization. All decoder columns are independently normalized to unit L2 norm: $\|w_i\|_2 = 1$. This ensures that the intervention strength is measured consistently across different columns and that the projection operation removes exactly the component along the unit direction rather than scaling by an arbitrary magnitude.

Step 4: Applying the intervention during inference (Equation 6). During the model's forward pass, at each reasoning step boundary (at the <\\n\\n> delimiter token), the paper applies the following operation to the hidden state $h$ at the chosen layer:

h′=h−wi(wiTh)h' = h - w_i (w_i^T h)

where $w_i$ is the normalized decoder column vector (or the averaged behavior vector), $w_i^T h$ is the scalar projection of $h$ onto the direction $w_i$ (the inner product, measuring how much of the behavior is present in the current hidden state), and $w_i (w_i^T h)$ is the vector component of $h$ along the $w_i$ direction.

What this computes: the operation projects $h$ onto the direction $w_i$ (computing $w_i^T h$, a scalar), then subtracts that component from $h$. The result $h'$ is the original hidden state with all variation along the $w_i$ direction removed. Since $w_i$ is a unit vector, the subtraction $h - w_i(w_i^T h)$ is the orthogonal projection of $h$ onto the subspace perpendicular to $w_i$. Geometrically, the hidden state is pushed away from the reflection (or backtracking, or confidence) direction—it becomes "less reflective" in activation space.

Why this form: projection-based removal is additive and independent—if multiple behavior vectors are applied in sequence, the result is the projection onto the orthogonal complement of their span. This means the intervention cleanly removes a single behavioral dimension without affecting other dimensions (assuming the SAE columns are approximately orthogonal, which the low maximum cosine similarity in Appendix C supports). An alternative approach—adding or subtracting a fixed vector independent of $h$—would shift all activations by the same amount regardless of how strongly the behavior is already expressed, potentially over-suppressing steps that already have low reflection or creating artifacts on steps where reflection is absent. The projection approach is adaptive: it removes exactly as much of the behavior direction as is present in $h$, and no more.

Positive vs. negative intervention. The paper uses the term "negative intervention" to mean suppressing a behavior (projecting it out, as in Equation 6) and "positive intervention" to mean amplifying a behavior (adding the component rather than subtracting: $h' = h + w_i(w_i^T h)$, though this operation is not explicitly stated in the text). The results in Figure 5 use "Negative" and "Positive" labels to indicate suppression and amplification respectively.

Controlling intervention strength. The paper also experiments with a scalar multiplier $\alpha$ on the projection:

h′=h−α⋅wi(wiTh)h' = h - \alpha \cdot w_i (w_i^T h)

where $\alpha$ controls the strength of the intervention. When $\alpha = 0$, no intervention occurs (vanilla inference). When $\alpha = 1$, the full component along $w_i$ is removed (standard negative intervention). When $\alpha = 1.5$, the component is oversubtracted—the hidden state is pushed past orthogonality into the opposite direction, actively encouraging the model to avoid the behavior. When $\alpha$ is negative (e.g., $\alpha = -1$), the component is added rather than subtracted, amplifying the behavior. The paper reports results with $\alpha \in \{-1.5, -1, 0, 1, 1.5\}$ on the AIME25 tasks, finding that the number of reflection steps changes to $\{58.6, 73.6, 90.5, 131.0, 166.9\}$ respectively—a monotonic relationship between intervention strength and behavioral expression.

Where and when the intervention is applied. The intervention is applied during the forward pass, at the specific layer from which the SAE was trained, at the position of each <\\n\\n> delimiter token. This means the hidden state is modified at each reasoning step boundary, and the modification propagates forward through the remaining layers (since the residual stream is altered before subsequent layers process it). The paper evaluates interventions using the final layer's SAE (shown in Figures 6, 7, and the behavioral statistics in Figure 5 and Tables 1-2), though in principle any layer's columns could be used.

What makes this intervention causally valid. The key property that makes this more than curve-fitting is that the SAE was trained without behavioral labels. The decoder columns that map to reflection were not selected by a human; they emerged from the unsupervised sparsity + reconstruction objective. The fact that manipulating these columns produces systematic, predictable changes in the model's textual output (Figure 6) provides strong evidence that the columns genuinely correspond to the behavior rather than being artifacts of the visualization. The generalization across tasks (MATH500 → GPQA-Diamond, KnowLogic; Table 1) and models (R1-1.5B → Qwen3-8B; Figure 5) further validates that these are task-independent and model-independent reasoning primitives, not dataset-specific patterns.

Discovering Novel Behaviors: Confidence Reasoning Vectors

The previous sections validate that the SAE recovers behaviors humans already know about. Section 5 asks: can the SAE discover behaviors that humans haven't named? The case study is confidence—a reasoning property that is "difficult to define at the word level" but can be operationalized through a proxy objective.

The proxy objective: output token entropy. Confidence is operationalized as the entropy of the model's output distribution. A model that is confident assigns high probability to a small number of tokens (low entropy); a model that is uncertain spreads probability across many tokens (high entropy). Entropy is used specifically because "recent studies on the entropy mechanisms of LLM reasoning highlight that entropy, or equivalently model confidence, plays a critical role during the reinforcement stage" (Section 5), with evidence that incentivizing low-entropy final answers improves reasoning.

The optimization formulation (Equation 7):

arg⁡min⁡SE[−∑k=1∣V∣pklog⁡pk]\arg\min_{S} \mathbb{E}\left[-\sum_{k=1}^{|V|} p_k \log p_k\right]

subject to:

pk=softmax(fl→L(h+SWdecoder))kp_k = \text{softmax}(f_{l \to L}(h + S W_{\text{decoder}}))_k

where $S \in \mathbb{R}^D$ is a score vector assigning an importance weight to each of the $D$ decoder columns, $h$ is an activation at layer $l$ extracted from the training set, $f_{l \to L}$ represents the forward pass from layer $l+1$ through the final layer $L$ (the remainder of the model after the SAE's training layer), $S W_{\text{decoder}}$ produces a vector in the original activation space $\mathbb{R}^d$ (the weighted combination of decoder columns, where $S$ determines how much each column contributes), $h + S W_{\text{decoder}}$ is the modified activation passed into the rest of the model, $\text{softmax}(f_{l \to L}(\cdot))$ produces the output token probability distribution $p$, and $-\sum p_k \log p_k$ is the Shannon entropy of that distribution over the vocabulary of size $|V|$.

What this computes: for each training activation $h$, the procedure computes a modified activation by adding a learned combination of decoder columns (weighted by $S$), passes it through the remainder of the model to get output token probabilities, computes the entropy of those probabilities, and averages over the training set. The score vector $S$ is optimized to minimize this average entropy—i.e., to find a direction in decoder column space that, when added to the hidden state, makes the model more confident (lower entropy) in its predictions.

Why this form: the formulation learns which decoder columns collectively influence model confidence without requiring any human labeling of "confident" versus "unconfident" reasoning steps. The entropy serves as a purely behavioral objective—we don't need to know what confidence "looks like" in the activation space; we just need to find directions that, when emphasized, produce lower-entropy outputs. The additive form $h + S W_{\text{decoder}}$ (as opposed to the subtractive projection in Equation 6) is appropriate here because the goal is to amplify certain behavioral directions, not to remove them.

Model splitting. The paper splits the original language model into two parts at layer $l$ (the layer from which the SAE was trained): $h = f_{1 \to l}(x)$ computes the activation from the input through layer $l$, and $y = f_{l \to L}(h)$ computes the output from layer $l+1$ through the final layer. The learned direction $S W_{\text{decoder}}$ is added to $h$ before passing it through $f_{l \to L}$, effectively intervening at the boundary between the two halves of the model. The splitting point is chosen to align with the SAE's training layer—the SAE was trained on activations from layer $l$, so its decoder columns represent directions in that layer's activation space, and it's natural to intervene at exactly that point.

Optimization details: The score vector $S$ is optimized using Adam with learning rate 0.01 for 1000 iterations, with cosine annealing learning rate decay and batch size 256. This is a lightweight optimization—1000 iterations over batches of 256 activations from the 500-question training set—reflecting that the search space is only $D = 2048$ dimensional and the objective is directly differentiable through the fixed language model backbone (which is not updated, only the scalar scores in $S$ are learned).

Identifying confidence-related columns. After optimization, the decoder columns with the highest scores in $S$ are selected as the confidence reasoning vectors. The paper selects the top-3 columns for some experiments (Section 5, the reasoning enhancement application). Visualizing these columns in the UMAP projection (Figure 7a) reveals that they are "predominantly located in the bottom-right region of the visualization," forming a distinct cluster—further evidence that the SAE columns have learned coherent structure, since columns that minimize entropy when amplified all point in similar directions.

Partial overlap with reflection/backtracking. The paper notes that confidence-related columns "partially overlap with the reflection and backtracking regions identified in Figure 2," consistent with the observation that "reflection and backtracking behaviors may contribute to higher entropy" (Section 5.1). When the model reflects (expressing uncertainty) or backtracks (exploring alternative strategies), its output distribution tends to be higher-entropy because it's considering multiple possibilities. Suppressing these behaviors—by amplifying confidence-related directions—reduces entropy and shifts the model toward more decisive, computation-focused reasoning.

Validating the discovered vectors through intervention. The confidence vector is applied during inference using the same projection-based intervention as in Section 4.4, but with the confidence-related decoder columns aggregated into a single direction. The effects are measured behaviorally: on AIME25 tasks, reflection steps decrease from 90.53 to 33.77 and backtracking steps decrease from 35.50 to 5.93 under negative intervention (suppressing low-confidence behaviors, i.e., amplifying confidence). Token-level analysis (Figures 7b-c) shows that frequent reflection/backtracking cues like "Wait" and "Alternatively" are replaced by tokens associated with mathematical calculation (e.g., numbers), confirming that the intervention shifts not just the frequency but the qualitative style of reasoning.

Generalization of confidence vectors. Table 2 shows that the confidence vectors discovered on MATH500 transfer to GPQA-Diamond and KnowLogic, where they produce consistent reductions in both reflection and backtracking across domains. This is particularly significant because the SAE was trained on MATH and the confidence vectors were selected by optimizing entropy on MATH, yet they modulate reasoning in entirely different domains—suggesting that confidence is a domain-general reasoning primitive encoded similarly across tasks.

Reasoning enhancement via learned combination coefficients. As an application, the paper selects the top-3 confidence vectors $c_1, c_2, c_3$ and learns sample-dependent combination coefficients $\alpha_i$ at test time (Section 5, "Reasoning Enhancement via Confidence Vectors"). This produces a steering vector $\sum_{i \in \{1,2,3\}} \alpha_i c_i$ that is adapted per-sample rather than using a fixed average. The result (Figure 8) shows improved accuracy on MATH500 (up to +4.66 percentage points) and AIME 2025 while reducing token cost by 13.69% compared to vanilla inference on MATH500, and outperforming supervised steering methods like SEAL (Chen et al., 2025) and TIP (Wang et al., 2025d). This demonstrates a concrete downstream benefit of unsupervised behavior discovery: by finding confidence directions without human labels, the framework enables more efficient reasoning—less verbose, faster, and slightly more accurate—than both vanilla inference and supervised alternatives.

Response Length as an Emergent Structural Axis

A secondary finding that emerges from the geometric analysis is that the SAE decoder columns self-organize along a dimension of response length, even though the SAE was never given any information about sequence length. The paper manually splits step-level activations into two categories: "short responses, with sequence length less than one thousand tokens, and long responses, with sequence length exceeding eight thousand tokens" (Appendix E). For each category, the most active decoder columns are identified and highlighted in the UMAP visualization across all 28 layers (Figure 11).

The results show that "columns associated with long responses form a more diverse and dispersed cluster, while those linked to short responses appear compact" (Appendix E). The separation is weak in early layers but "becomes increasingly clear and stable in mid-to-late layers, peaking just before the output stage" (Figure 12 quantifies this with normalized Silhouette scores based on length clustering). This mirrors the behavioral clustering pattern from Figure 3—both semantic behaviors (reflection/backtracking) and structural properties (response length) become more distinctly encoded in mid-to-late layers—suggesting that the SAE is capturing a genuine organizational principle of the activation space rather than fitting noise.

This finding is significant because it demonstrates that the SAE discovers structure that is not directly behavioral—response length is a property of the output, not a cognitive strategy—yet it emerges naturally from the unsupervised decomposition. It also connects to prior work on length-aware activation geometry (Eisenstadt et al., 2025; Sun et al., 2025) while showing that this geometry is recoverable without supervision through SAE training.

Summary of Design Choices and Their Justifications

  • Sentence-level (not token-level) representations: avoids the polysemy problem where the same token serves different behavioral roles in different contexts; delimiter token representations capture step-level semantics by attending to the entire preceding reasoning segment.
  • Sparse auto-encoder with L0 sparsity penalty: the hard zeroing from ReLU + L0 penalty produces genuinely sparse codes (most latent features are exactly zero for any given activation), encouraging each decoder column to represent a reusable, atomic behavior rather than a distributed mixture.
  • Modest overcompleteness (D=2048 for d=1536): avoids excessive overcompleteness that can split behaviors across multiple columns and reduce interpretability, while providing enough capacity to represent the diversity of reasoning behaviors.
  • Cosine similarity in UMAP: emphasizes directional rather than magnitude differences between decoder columns, since activation vectors are functionally characterized by their direction in high-dimensional space rather than their norm.
  • LLM-as-a-judge for behavior annotation: enables labeling at scale without human annotation effort, with validation that multiple judges agree at 85-94% rates, confirming reliability.
  • Filtering multi-behavior columns before intervention: removes columns that are active for multiple behaviors (e.g., both reflection and backtracking) to ensure the intervention vector is specific to the target behavior rather than a mixture.
  • Projection-based intervention (not additive): removes exactly the component along the behavior direction that is present in the current hidden state, adapting the intervention strength to the activation rather than applying a fixed shift.
  • Entropy minimization as a discovery objective: operationalizes confidence as a behavioral goal without requiring human labels; low-entropy outputs are behaviorally interpretable as "confident," making this a natural bridge between unsupervised discovery and semantic interpretation.
  • Sample-dependent combination coefficients: extends the fixed intervention vector to per-sample adaptation, enabling finer-grained control that improves both accuracy and efficiency over uniform steering.

4. Key Insights and Innovations

Innovation 1: Test-Time Compute Allocation Must Be Adaptive, Not Uniform — Difficulty Is the Key Sufficient Statistic

The paper's most fundamental conceptual move is reframing test-time compute from a uniform knob ("turn up the budget, performance improves") to a contingent resource whose optimal deployment depends on a single prompt-level variable: difficulty. This is not a method — it is a diagnostic framework that changes what the field optimizes for.

Before this work, the dominant mental model was that scaling test-time compute is monotonic and strategy-agnostic. Best-of-N sampling (Cobbe et al., 2021) treats every prompt as equally benefiting from more parallel samples. Self-consistency and majority voting implicitly assume that the value of each additional sample is constant across prompts. Even sophisticated search methods like tree-of-thought (Yao et al., 2023) apply the same algorithm uniformly, varying only the search depth. The field's optimization target was: given a fixed budget, which single method performs best on average?

The paper's central reframing replaces this with: given a fixed budget and an estimate of this prompt's difficulty, which strategy maximizes the probability of a correct answer? The formalization in Equation 1 — θq,y∗(q)∗(N)=arg⁡max⁡θEy∼Target(θ,N,q)[1y=y∗(q)]\theta_{q, y^*(q)}^*(N) = \arg\max_\theta \mathbb{E}_{y \sim \text{Target}(\theta, N, q)}[\mathbb{1}_{y = y^*(q)}] — is deceptively simple, but its implication is profound: θ∗\theta^* varies with qq. There is no universal best strategy. This transforms test-time compute from a method selection problem (which algorithm is best?) to a resource allocation problem (how should I split my budget across strategies for this specific prompt?).

The evidence for why this reframing matters comes from the counterintuitive difficulty-dependent results. Beam search — the strongest optimizer — degrades performance on easy problems at high budgets (Figure 3, right: Bin 1 accuracy decreases from roughly 78% to 77% as budget increases from 4 to 256 generations), while it improves performance on medium problems (Bin 3: roughly 34% vs. 23% for best-of-N at 256 generations). Sequential revisions dominate on easy problems but require balanced parallel exploration on hard ones (Figure 7, right). These are not monotonic relationships — they are qualitative reversals. Any uniform strategy leaves enormous efficiency on the table because it is simultaneously over-optimizing on some prompts and under-exploring on others.

This reframing draws an explicit parallel to compute-optimal pretraining (Hoffmann et al., 2022) — which showed that the optimal allocation of a fixed pretraining budget between model size and data quantity depends on the total budget — but applies it to a fundamentally different domain. In pretraining, the optimization is over continuous variables (parameters, tokens) with a single objective (validation loss). Here, the optimization is over a discrete combinatorial space of strategies (search algorithm, beam width, revision depth, parallel-to-sequential ratio) with a prompt-conditional objective. The conceptual leap is recognizing that inference-time compute obeys its own scaling laws, and that the key sufficient statistic is prompt difficulty — not the total budget, not the model size, not the task domain.

This is a fundamental reframing, not an incremental refinement. It changes the question from "what's the best inference strategy?" (a methods question with a single answer) to "under what conditions does each strategy excel?" (a systems question with a prompt-dependent answer). The 4× efficiency gains (Figures 4, 8) are downstream consequences of getting the framing right, not the innovation itself.

Innovation 2: The Proposal-Verifier Decomposition Reveals That Revisions and Search Are Complementary, Not Competing, Axes — With Difficulty-Dependent Optimality

The paper's second conceptual contribution is a unifying taxonomy that decomposes all test-time compute methods into two independent axes — modifying the proposal distribution (what the model generates) versus modifying the verifier (how outputs are scored and selected) — and then empirically demonstrating that these axes have complementary, difficulty-dependent strengths. This taxonomy is not itself novel (it echoes the proposer-scorer decomposition from MCMC), but the paper's demonstration of its operational significance for LLM reasoning is.

Prior work studied these axes in isolation and often reached contradictory conclusions. On the proposal side, self-correction via prompting was found to be "largely ineffective for reasoning" (Huang et al., 2023), while self-refinement showed benefits on certain tasks (Madaan et al., 2023). On the verifier side, PRM-guided search improved performance over majority voting (Lightman et al., 2023), but more sophisticated search algorithms like tree-of-thought showed mixed results depending on the implementation. The field lacked a framework for understanding when each approach works, leading to the implicit assumption that they were competing alternatives — you either revise or you search, and you pick whichever performs better on your benchmark.

The paper's key finding is that these axes are not competitors but complements whose optimality is difficulty-gated. Revisions (proposal modification) excel on easy problems where the model's initial output is roughly correct and needs local refinement — essentially hill-climbing in answer space. PRM-guided search (verifier optimization) excels on medium problems where the model needs to explore qualitatively different solution strategies — essentially global search in answer space. On hard problems, neither axis helps because there are no correct solutions in the proposal distribution to find or refine (the base model's pass@1 is near zero). Figure 10 in Appendix B of the actual paper confirms this pattern holds even with majority voting as the selection mechanism, showing it is not an artifact of the verifier.

This finding resolves the apparent contradictions in prior work. Huang et al. (2023) tested self-correction on problems that were likely too hard for the base model (producing the pessimistic "cannot self-correct" conclusion), while Madaan et al. (2023) tested on problems within the model's capability range (producing the optimistic "self-refinement helps" conclusion). The difficulty-dependent framework explains both: self-correction works when the model can already produce approximately correct answers, and fails when it cannot. The conflicting results were not evidence about the method; they were evidence about the difficulty distribution of the test sets.

The significance extends beyond reconciliation. It implies that future systems should not choose between revisions and search — they should deploy both, switching between them per-prompt based on difficulty estimates. The paper does not fully realize this combined system (Section 8 acknowledges PRM tree-search was not combined with revisions), but the framework provides the intellectual scaffolding. This is a fundamental insight rather than an incremental one: it changes the optimization target from "which single method is best?" to "which combination of complementary mechanisms is optimal for this prompt?"

Innovation 3: Verifier Over-Optimization Is the Primary Bottleneck for Test-Time Compute Scaling — Not Search Algorithm Sophistication

The paper provides the first clear empirical characterization of verifier over-optimization as the dominant limiting factor in test-time compute scaling, and in doing so, redirects research attention from search algorithm design to verifier robustness. This is a negative result with significant positive implications — it tells the field what not to work on.

The evidence is concrete and multi-pronged. Beam search — which uses the PRM's per-step scores to prune unpromising partial solutions — degrades easy-problem performance at high budgets (Figure 3, right: Bin 1 accuracy falls as budget increases from 4 to 256 generations). Lookahead search — the most powerful optimizer, which simulates additional steps forward to improve scoring accuracy — paradoxically performs worst overall (Figure 3, left: all lookahead variants trail best-of-N weighted at equivalent budgets). Qualitative examples (Appendix M in the original paper) show degenerate outputs: low-information repetitive steps at the end of solutions, and overly short 1-2 step solutions that score highly under the PRM but are incorrect. The paper explicitly notes these as "specific failure modes" of verifier exploitation.

The diagnostic move is recognizing that aggressive optimization of an imperfect verifier produces worse results than moderate optimization of the same verifier — and that this bounds the scaling curve. More search compute is not better; it is better only up to the point where the verifier's errors begin to dominate. This explains why prior work found negative results for sophisticated search methods (Valmeekam et al., 2023; Stechly et al., 2023) — those studies likely pushed past the over-optimization threshold for their particular verifier quality and prompt distribution.

The implication is a research prioritization shift. Before this work, one might reasonably invest in developing more sophisticated search algorithms — better pruning strategies, adaptive branching factors, learned search policies. The paper's results suggest this investment would have diminishing returns, because the bottleneck is not the search algorithm's ability to navigate the solution space, but the verifier's ability to score partial solutions accurately under distribution shift. Improving verifier robustness — through adversarial training, ensemble methods, or better calibration — would likely yield larger gains than any search algorithm innovation.

This is a fundamental insight, not an incremental finding, because it changes the causal model of what limits test-time compute scaling. The naive causal model is: search algorithm quality → ability to find good solutions → performance. The paper's revised causal model is: verifier quality → reliable scoring signal → effective search → performance, with the verifier acting as a hard ceiling that no search algorithm can exceed. The compute-optimal policy in the paper (Section 3.4) can be understood partly as a strategy for staying below this ceiling — routing easy problems to best-of-N (weak optimization, high verifier reliability) and medium problems to beam search (stronger optimization, still within the verifier's reliability frontier).

Innovation 4: Unsupervised Sparse Auto-Encoding Discovers Reasoning Behaviors That Supervised Methods Cannot Find — Confidence as Existence Proof

This is the paper's central methodological innovation: demonstrating that sparse auto-encoders trained without behavioral labels can discover interpretable, causally manipulable reasoning vectors that span both known behaviors (reflection, backtracking) and novel behaviors (confidence) invisible to supervised approaches. The innovation is not the SAE architecture itself — sparse auto-encoders are well-established in mechanistic interpretability (Cunningham et al., 2023) — but rather the application of unsupervised decomposition to the reasoning behavior discovery problem, paired with a validation pipeline (geometric clustering + causal intervention + cross-domain generalization) that converts correlations into mechanistic claims.

The field's prior approach to finding reasoning-related directions was uniformly supervised. DiffMean (Marks and Tegmark, 2023) requires contrastive pairs: reflection steps versus non-reflection steps, correct versus incorrect solutions, short versus long responses. This approach is inherently bounded by human imagination — you can only find behaviors you already know to label. The paper's discovery of confidence-related vectors (Section 5) provides concrete evidence for why this matters. Confidence is "difficult to define at the word level" — there is no simple textual marker like "Wait" (for reflection) or "Alternatively" (for backtracking) that can be used to construct contrastive datasets. A supervised approach would either miss confidence entirely or require a laborious operationalization (e.g., asking human annotators to rate confidence on a scale, then thresholding). The SAE found it automatically through a simple proxy objective (entropy minimization) applied post-hoc to the already-trained decoder columns.

The significance extends beyond the specific finding. The paper establishes a discovery pipeline — train SAE → visualize geometry → identify candidate columns via proxy objective → validate causally through intervention — that can be applied to any reasoning behavior that can be operationalized through a differentiable objective. Future work could plug in objectives for coherence, factual accuracy, creative divergence, or any other property of reasoning, and the same pipeline would surface the relevant columns. This transforms reasoning interpretability from a hypothesis-driven enterprise (where you must name the behavior before you can find it) to a discovery-driven one (where the model proposes candidate directions and you analyze what they do).

The causal validation is what elevates this from correlation to mechanism. It is one thing to show that decoder columns cluster by behavior in UMAP space (Figure 2). It is another to show that manipulating a single column — projecting the hidden state away from its direction during inference — systematically reduces reflection steps from ~90 to ~34 on AIME25 tasks while preserving answer correctness (Section 4.4). The dose-response relationship with intervention strength (α ∈ {-1.5, -1, 0, 1, 1.5} produces monotonic changes in step counts: {58.6, 73.6, 90.5, 131.0, 166.9}) is particularly compelling — it rules out the possibility that the intervention is simply disrupting the model in a non-specific way. The generalization across domains (Tables 1, 2: MATH500 → GPQA-Diamond, KnowLogic) and models (R1-1.5B → Qwen3-8B, Figure 5) further validates that these are genuine reasoning primitives, not dataset-specific or model-specific artifacts.

This is a fundamental advance, not an incremental refinement of existing activation steering methods. DiffMean produces a single vector that mixes multiple underlying directions (since it averages over a heterogeneous category). The SAE produces a dictionary of individual basis vectors, each (approximately) corresponding to one atomic behavior, with the sparsity constraint encouraging disentanglement. This enables finer-grained control — suppressing reflection without affecting backtracking (by filtering multi-behavior columns), discovering behaviors without naming them (through proxy objectives), and combining vectors for composite effects (through learned combination coefficients, as in the reasoning enhancement application). The framework's extensibility — new behaviors, new objectives, new models — makes it a platform for reasoning interpretability research, not just a single analysis.

Innovation 5: Reasoning Behaviors Occupy a Low-Dimensional, Geometrically Structured Subspace — With Mid-to-Late Layers Encoding the Strongest Behavioral Signal

The paper's geometric analyses reveal a structural property of reasoning models that was previously uncharacterized: reasoning behaviors are not randomly distributed in activation space but form coherent, low-dimensional clusters whose separability peaks in mid-to-late layers and partially declines near the output. This is an empirical finding about the internal organization of reasoning models, not a method, but it has significant implications for where and how interpretability research should focus.

The evidence comes from the UMAP visualizations (Figures 2, 11) and Silhouette score analyses (Figures 3, 12). The key patterns: (1) Decoder columns associated with reflection and backtracking form tight clusters in the SAE's latent space, while "other" columns are dispersed — this means the SAE has learned to separate behaviorally coherent directions from the undifferentiated background, even though it was never told what behaviors to look for. The very existence of these clusters under unsupervised training is evidence that reasoning behaviors are genuinely encoded as structured subspaces, not as arbitrary activation patterns. (2) Silhouette scores increase from early to mid-late layers, peak around layers 20-25 for the 28-layer model, and then partially decline — this means behavioral concepts become progressively more separable as representations deepen, but the very last layers compress or "oversmooth" them. (3) Reflection and backtracking are more separable from "other" behaviors than from each other (Silhouette scores for "Others vs. Reflection" and "Others vs. Backtracking" are substantially higher than for "Reflection vs. Backtracking") — this means these two behaviors share representational structure, which the paper leverages by filtering multi-behavior columns before intervention.

A parallel structural finding emerges for response length (Appendix E, Figures 11-12): SAE columns self-organize into short-response and long-response clusters, with separability peaking in mid-to-late layers. This is striking because length is not a behavioral property — it is a structural property of the output — yet it emerges from the same unsupervised decomposition. The paper calls this an "implicit inductive bias in reasoning LLMs" (Section 6), suggesting that the activation space organizes along multiple axes simultaneously (semantic-behavioral and structural) without explicit supervision.

The significance of these findings is that they provide an architectural map for future interpretability research. If behavioral signal peaks in layers 20-25, those are the layers where intervention studies should focus. If reflection and backtracking share representational subspace, future work should model them as related dimensions of a common metacognitive process rather than as independent behaviors. If length emerges as an organizing axis, interventions aimed at controlling verbosity should target that axis specifically rather than using blanket suppression of reflection (which would conflate length control with behavior control). These are guideposts for a research program, not just individual results.

This is somewhere between fundamental and incremental, depending on how broadly the layer-wise pattern generalizes. For the specific model studied (R1-1.5B), it is a robust characterization. Whether different model architectures, scales, or training procedures produce the same layer-wise profile is unknown — the paper shows generalization across tasks and models for behavioral vectors, but not for the layer-wise structural property itself. If future work finds that the mid-to-late layer peak is universal across reasoning models, this becomes a fundamental discovery about how transformer architectures encode abstract cognitive patterns. If it is model-specific, it is an incremental characterization that nonetheless provides a template for how such analyses should be conducted.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the MATH benchmark (Hendrycks et al., 2021), specifically the split from Lightman et al. (2023) consisting of 12,000 training questions and 500 test questions. The paper samples 500 training examples (the "MATH500" split) for SAE training and uses additional benchmarks — AIME 2025, AIME 2024, AMC23, GPQA-Diamond (Rein et al., 2024), and KnowLogic (Zhan et al., 2025) — for evaluating cross-domain generalization of discovered reasoning vectors.

Base model(s). The primary model is DeepSeek-R1-Distill-Qwen-1.5B (R1-1.5B), a 1.5-billion-parameter reasoning model with 28 transformer layers and hidden dimension 1536, distilled from the full DeepSeek-R1 (Guo et al., 2025) using reinforcement learning. This model is chosen because it exhibits the metacognitive behaviors — reflection, backtracking, and extended chain-of-thought reasoning — that the paper aims to study. An extension experiment also uses Qwen3-8B (Team, 2025) to verify cross-model generalization of the behavioral vectors.

Metrics. Three categories of metrics are used. For behavioral frequency, the paper counts the number of reasoning steps classified as "reflection" or "backtracking" in the model's output, using LLM-as-a-judge annotation (GPT-5 with the prompt from Appendix D) to classify each step. For answer correctness, standard accuracy is measured — the fraction of questions where the model's final answer matches the ground truth — reported as percentages on MATH500 and AIME tasks. For clustering quality, normalized Silhouette scores (Rousseeuw, 1987) measure both the cohesion of samples within a behavioral cluster and their separation from other clusters, with raw scores normalized across layers for visualization. For the confidence discovery experiments, output token entropy is computed as −∑k=1∣V∣pklog⁡pk-\sum_{k=1}^{|V|} p_k \log p_k over the vocabulary.

Baselines. The paper compares against three baselines for the reasoning enhancement application (Figure 8). Vanilla inference: the unmodified R1-1.5B model with standard chain-of-thought prompting, no activation steering. SEAL (Chen et al., 2025): a supervised steering approach that calibrates reasoning by learning steering vectors from human-defined behavioral contrasts. TIP (Wang et al., 2025d): another supervised method targeting underthinking/overthinking behaviors through activation manipulation. The geometric and causal validation experiments (Sections 4.3, 4.4) do not compare against baselines per se — they validate against the vanilla model's own behavior, establishing that the SAE-derived vectors causally shift behavior relative to the unmodified model.

Generation budget / compute accounting. For the SAE training phase, compute is measured in terms of training samples (500 MATH questions, each producing variable-length chain-of-thought traces segmented into sentence-level steps). The number of step-level activations per layer is the product of 500 questions times the average number of reasoning steps per response — this is not explicitly reported but can be estimated in the thousands to low tens of thousands. SAE training uses a batch size of 1024 with the Adam optimizer, learning rate 1×10−41 \times 10^{-4}, and cosine annealing over an unreported number of steps. For the inference-time intervention experiments, compute is not formally budgeted — interventions are applied at test time with varying strengths (α∈{−1.5,−1,0,1,1.5}\alpha \in \{-1.5, -1, 0, 1, 1.5\} for the dose-response analysis; fixed projection for the behavioral shift experiments) but no systematic comparison of intervention cost against alternative methods is performed. The confidence vector optimization (Equation 7) uses 1000 Adam iterations with batch size 256, representing a small additional one-time cost to learn the score vector SS.

Cross-validation / statistical protocol. The behavioral annotation procedure (Section 4.3, Appendix D) uses multiple judges — GPT-5, GPT-4o, Claude Sonnet 4.5, and a keyword-matching baseline — with inter-judge agreement reported as pairwise agreement ratios (Figure 10). Agreement ranges from 85% to 94%, with GPT-5 and GPT-4o showing the highest pairwise consistency at approximately 94%. This multi-judge validation ensures that the behavioral labels are not idiosyncratic to a single annotator. For the cross-domain generalization experiments (Tables 1, 2), the SAE columns are trained exclusively on MATH500 data and then applied without modification to GPQA-Diamond and KnowLogic — this serves as a domain-shift robustness check rather than a formal cross-validation. The paper does not report confidence intervals, standard errors, or statistical significance tests for any behavioral frequency measurements or accuracy comparisons. The dose-response experiment with α∈{−1.5,−1,0,1,1.5}\alpha \in \{-1.5, -1, 0, 1, 1.5\} provides internal replication — the monotonic relationship between intervention strength and behavioral expression is evidence against noise-driven effects — but no formal statistical framework is applied.

Main Quantitative Results

Geometric Organization of the SAE Decoder Column Space

The paper's first set of experiments addresses whether the unsupervisedly trained SAE recovers behaviorally meaningful structure in its decoder columns. The headline finding is that decoder columns corresponding to reflection and backtracking form coherent, spatially localized clusters in the SAE's latent space, while columns associated with other behaviors are dispersed — and this organization emerges without any behavioral labels during training.

Figure 2 (Section 4.3) presents UMAP projections of the 2048 decoder columns from the final layer of R1-1.5B. The leftmost subfigure shows the raw projection without annotations. When overlaid with human-labeled behavioral concepts (remaining subfigures), two patterns emerge. First, reflection-associated columns (middle subfigure) cluster in a localized region of the projection, forming a tight group. Second, backtracking-associated columns (right-middle subfigure) similarly cluster but occupy a distinct but partially overlapping region. Third, columns associated with "other" behaviors (rightmost subfigure) are "less centered and more dispersed across the entire SAE space" — they do not form a coherent cluster. This spatial organization is striking because the SAE was trained only with the reconstruction + sparsity objective (Equation 2); it received no signal about which behaviors exist or which columns should be similar to each other. The fact that behaviorally related columns end up near each other in cosine-similarity space suggests that the activation space genuinely encodes these behaviors as structured subspaces.

Figure 3 (Section 4.3) quantifies this clustering through normalized Silhouette scores across all 28 layers. Three pairwise comparisons are computed: Reflection vs. Backtracking, Others vs. Reflection, and Others vs. Backtracking. Two key findings emerge. First, later layers generally achieve higher Silhouette scores than earlier ones, with scores rising from near zero in the early layers (1-8) to peak values in mid-to-late layers (roughly 16-24). This indicates that "behavioral concepts become more separable as representations deepen." Second, the "Others vs. Reflection" and "Others vs. Backtracking" curves consistently run above the "Reflection vs. Backtracking" curve. The paper states: "the comparison between reflection and backtracking reveals only modest differences, whereas the separation of 'other' behaviors from either reflection or backtracking is substantially stronger." This means reflection and backtracking occupy overlapping representational subspaces — they are more similar to each other than to unrelated behaviors. Third, a "slight decline occurs near the final layers (except for the very last one)," which the paper attributes to the oversmoothing phenomenon where token representations become excessively similar in the deepest layers.

Figure 11 (Appendix E) and Figure 12 (Appendix E) extend this geometric analysis to response length as an organizing axis. The paper manually splits step-level activations into short responses (< 1000 tokens) and long responses (> 8000 tokens), identifies the most active decoder columns for each, and visualizes them in UMAP across all 28 layers. The results show that "columns associated with long responses form a more diverse and dispersed cluster, while those linked to short responses appear compact," and that this separation is "weak in early layers but becomes increasingly clear and stable in mid-to-late layers, peaking just before the output stage." The normalized Silhouette scores for length-based clustering (Figure 12) follow a similar pattern to the behavioral scores in Figure 3 — early layers show weak separation, mid-to-late layers show stronger alignment, with a peak around layers 16-24.

These results establish the paper's first major claim: that the SAE, trained without supervision, discovers a geometrically structured latent space where semantic behaviors and structural properties self-organize into separable clusters. This is primarily a correlation result — it shows that behaviors cluster, but not yet that manipulating these clusters causes behavioral change.

Causal Validation Through Targeted Intervention

The paper's second set of experiments moves from correlation to causation by intervening on specific SAE decoder columns during inference and measuring the behavioral consequences. The headline finding is that projecting the hidden state away from (or toward) behavior-specific decoder column directions produces systematic, dose-dependent changes in the frequency of the corresponding behavior, while generally preserving answer correctness.

Figure 5 (Section 4.4) presents behavioral frequency statistics before and after intervention across two models and four tasks. For R1-1.5B on AIME 2025: vanilla inference produces approximately 90.5 reflection steps; negative intervention (projecting out the reflection direction) reduces this to approximately 33.8 steps; positive intervention (amplifying the reflection direction) increases this to approximately 131.0 steps. The same pattern holds for backtracking on AIME 2025 (vanilla: ~35.5; negative: ~5.9; positive: ~50+ steps, estimated from the bar chart). For R1-1.5B on AMC23, the pattern replicates with roughly 20 reflection steps under vanilla, reduced to approximately 8 under negative intervention, increased to approximately 50 under positive. For Qwen3-8B on AIME 2024 and MATH500, the same directional pattern holds, demonstrating cross-model generalization — the SAE columns are trained on R1-1.5B's MATH500 activations, but the behavioral vectors (the identified decoder column directions) transfer to Qwen3-8B, where they produce qualitatively similar effects on reflection and backtracking frequencies. The paper does not report exact numbers for every bar in Figure 5, but the visual pattern is consistent across all four panels: negative intervention consistently reduces behavior frequency, positive intervention consistently increases it.

A critical detail from Section 4.4.1: the intervention uses behavior-specific columns obtained by filtering out decoder columns that exhibit strong activations across multiple behaviors (e.g., both reflection and backtracking). From the remaining behavior-specific columns, the paper computes their average to obtain a single intervention vector. This filtering is essential because, as Figure 3 showed, reflection and backtracking occupy overlapping subspaces — if unfiltered columns were used, the intervention would mix both behaviors, producing less clean effects.

The dose-response experiment provides the strongest causal evidence. With intervention strength α∈{−1.5,−1,0,1,1.5}\alpha \in \{-1.5, -1, 0, 1, 1.5\}, the number of reflection steps on AIME25 tasks changes to {58.6,73.6,90.5,131.0,166.9}\{58.6, 73.6, 90.5, 131.0, 166.9\} respectively — a monotonic relationship. At α=−1.5\alpha = -1.5 (strong negative intervention, i.e., strongly suppressing reflection), the model produces 58.6 reflection steps. At α=1.5\alpha = 1.5 (strong positive intervention, amplifying reflection), it produces 166.9 steps. The monotonicity rules out the possibility that the intervention is merely disrupting the model nonspecifically — if the effect were noise, there would be no reason for reflection step counts to track α\alpha so cleanly.

Figure 6 (Section 4.4.1) provides a qualitative example. When prompted with a math question (converting rectangular to polar coordinates), the model under vanilla inference produces a response with 4 reflection steps (highlighted in red). Under negative intervention, this drops to 2 reflection steps — the model produces fewer self-checking phrases — and the overall response is shorter and more direct. Under positive intervention, this increases to 6 reflection steps — the model inserts additional verification steps — and the response lengthens. Crucially, all three conditions produce the same final answer: (3,π/2)(3, \pi/2). This demonstrates that the intervention changes the reasoning style without corrupting the reasoning outcome.

Table 1 (Section 4.4.1) tests cross-domain generalization. The SAE columns learned from MATH500 are applied during inference on GPQA-Diamond (graduate-level science QA) and KnowLogic (commonsense reasoning). For reflection on GPQA: vanilla produces 53.23 reflection steps, positive steering increases this to 62.77, negative steering reduces it to 45.42. For backtracking on GPQA: vanilla 11.83, positive 20.31, negative 6.47. For KnowLogic: reflection goes from 35.56 (vanilla) to 51.00 (positive) to 25.99 (negative); backtracking from 5.42 to 9.38 to 2.33. The vectors consistently modulate the target behaviors across domains, confirming that they capture domain-general reasoning primitives rather than MATH-specific patterns.

These results establish the paper's second major claim: that SAE-derived vectors can be used to "selectively modulate reasoning at inference time without additional training." The evidence is strong for within-model, within-task modulation, and the cross-model (R1-1.5B → Qwen3-8B) and cross-domain (MATH500 → GPQA, KnowLogic) generalization extends the claim beyond a single experimental configuration.

Discovery of Novel Behaviors: Confidence Reasoning Vectors

The paper's third set of experiments tests whether the SAE can discover behaviors that humans have not predefined — with confidence as the case study. The headline finding is that a small set of decoder columns, identified by optimizing a proxy objective (output token entropy minimization), consistently modulate reasoning style toward more confident, less reflective outputs, and that these columns cluster coherently in the SAE space.

Figure 7a (Section 5.1) visualizes the SAE decoder columns, with the top-scoring columns from the entropy minimization optimization (Equation 7) highlighted. The paper states these columns are "predominantly located in the bottom-right region of the visualization," forming a distinct cluster. Moreover, these columns "partially overlap with the reflection and backtracking regions identified in Figure 2" — a finding that is interpretable because reflection and backtracking behaviors (which involve expressing uncertainty and exploring alternatives) are expected to contribute to higher entropy. The clustering indicates that the SAE has organized confidence-related directions coherently in its latent space, even though it was never given any signal about what "confidence" means.

Figures 7b and 7c (Section 5.1) visualize the most frequent tokens at the beginning of each reasoning step under vanilla inference (7b) and under intervention with the confidence vector (7c). Under vanilla inference, frequent tokens include reflection-related cues such as "Wait" and backtracking indicators such as "Alternatively." Under confidence intervention, "these are replaced by tokens more closely related to detailed mathematical calculation steps" — the paper specifically notes that numbers become more frequent. This is a qualitative shift in reasoning style toward more computation-focused, less metacognitive output.

The behavioral statistics confirm the qualitative shift. On AIME25 tasks, intervention with the confidence vector reduces reflection steps from 90.53 to 33.77 and backtracking steps from 35.50 to 5.93. The paper notes a slight accuracy drop from 23.33% to 20.00%, but correctly identifies that this "corresponds to only a single question out of 30 and is therefore not statistically significant" — the sample size is too small to distinguish a genuine accuracy effect from noise.

Table 2 (Section 5.1) tests cross-domain generalization of confidence vectors. On GPQA-Diamond, vanilla produces 53.23 reflection and 11.83 backtracking steps; positive confidence steering reduces reflection to 37.25 and backtracking to 6.91 (note: "positive" here means steering toward confidence, i.e., suppressing uncertainty-related behaviors, so lower numbers are the expected outcome for a confidence-amplifying intervention); negative confidence steering increases reflection to 61.37 and backtracking to 16.18. The same pattern holds for KnowLogic: vanilla 35.56/5.42 (reflection/backtracking), positive confidence 20.04/3.51, negative confidence 48.16/5.98. The vectors generalize across domains with consistent directional effects.

Reasoning enhancement via learned combination coefficients (Figure 8). The paper selects the top-3 confidence vectors c1,c2,c3c_1, c_2, c_3 and learns sample-dependent combination coefficients at test time to create a per-sample steering vector ∑iαici\sum_i \alpha_i c_i. This is compared against vanilla inference, SEAL (Chen et al., 2025), and TIP (Wang et al., 2025d). On MATH500 accuracy: vanilla achieves approximately 72%, SEAL approximately 73%, TIP approximately 74%, and the confidence method approximately 76.7% — an improvement of up to 4.66 percentage points over vanilla. On token cost: vanilla uses ~2400 tokens, SEAL and TIP show slightly lower but similar values (~2200-2300), and the confidence method reduces to ~2000 tokens — a 13.69% reduction. On AIME 2025 accuracy: vanilla achieves 23.33%, and the confidence method shows a slight increase (the exact value is not separately stated beyond the earlier 20.00% figure, which appears to be for the fixed-vector version; the figure shows confidence outperforming the baselines). On AIME 2025 token cost: vanilla uses ~11,000 tokens, and confidence reduces this to ~7,000 tokens (approximate from the bar chart). The confidence method achieves better accuracy with substantially fewer tokens than both vanilla and the supervised baselines.

These results establish the paper's third major claim: that SAEs can discover behaviors "that are difficult to define with word-level human supervision." The confidence vectors were found through an objective (entropy minimization) that does not require labeling examples of "confident" reasoning — the SAE had already learned the columns, and the optimization simply identified which columns, when amplified, reduce entropy. The clustering, the behavioral effects, and the cross-domain generalization all validate that these columns genuinely capture a confidence-related reasoning dimension.

Layer-wise Properties and Structural Organization

While not presented as a separate "results" section, the paper's layer-wise analyses constitute a fourth empirical contribution. The headline finding is that both behavioral separability and length-based organization peak in mid-to-late layers, with a partial decline near the output.

Figure 3 (discussed above) establishes the layer-wise profile for behavioral Silhouette scores: early layers (1-8) show low separability; mid-to-late layers (16-24) show peak separability; the very final layers (25-28, except 28) show a slight decline attributed to oversmoothing. The paper explicitly states: "this observation aligns with the oversmoothing phenomenon in current LLMs (Wang et al., 2023), where token representations become excessively similar."

Figure 12 (Appendix E) replicates this pattern for response-length-based clustering. The normalized Silhouette scores for length separation follow a similar trajectory: low in early layers, rising through mid-layers, peaking around layers 20-24, and declining slightly before the output. This parallel between behavioral and structural organization suggests a common underlying principle — that the transformer's residual stream progressively structures information along multiple axes (semantic and structural) as depth increases, with some compression occurring in the final layers.

Figure 9 (Appendix C) provides empirical validation of the theoretical assumptions. The maximum cosine similarity among SAE decoder columns (incoherence check) is bounded below approximately 0.30 across all layers, satisfying the incoherence condition (μ<1\mu < 1) from Theorem 1. The sparsity level kk (number of active dictionary elements used per reconstruction) varies across layers, ranging from roughly 50 (early layers) to approximately 300 (mid-layers) back to roughly 200 (late layers) based on the bar chart. Reconstruction error (MSE between original and reconstructed activations) remains consistently low — below 0.08 across all layers — validating that the SAE faithfully represents the activation space and can serve as a reliable proxy for the ground-truth dictionary.

Ablation Studies and Robustness Checks

Annotation method consistency: The paper compares four annotation methods (GPT-5, GPT-4o, Claude Sonnet 4.5, keyword matching) for behavioral classification in Appendix D, Figure 10. Pairwise agreement ratios range from 85% to 94%, with GPT-5 and GPT-4o showing the strongest agreement at approximately 94%. All other pairs achieve agreement above 85%. This establishes that the behavioral labels used for geometric analysis and column selection are robust to the choice of annotator, ruling out the concern that findings depend on a single judge's idiosyncratic classifications.

Intervention strength dose-response: Section 4.4.1 reports that varying the intervention scalar α∈{−1.5,−1,0,1,1.5}\alpha \in \{-1.5, -1, 0, 1, 1.5\} produces monotonic changes in reflection step counts: {58.6,73.6,90.5,131.0,166.9}\{58.6, 73.6, 90.5, 131.0, 166.9\}. This dose-response relationship — where larger α\alpha produces larger behavioral shifts in the expected direction — is a strong causal test. If the intervention were nonspecifically disrupting the model, there would be no reason for step counts to track α\alpha linearly. The monotonicity supports genuine causal control rather than artifact.

Cross-model generalization (R1-1.5B → Qwen3-8B): Figure 5 includes results for Qwen3-8B on AIME 2024 and MATH500 tasks, using SAE columns trained on R1-1.5B's MATH500 activations. Both reflection and backtracking interventions produce the expected directional effects on Qwen3-8B, demonstrating that the discovered vectors are not specific to the training model. This is a non-trivial robustness check: the 1.5B and 8B models have different architectures and capabilities, yet the same intervention directions modulate the same behaviors, suggesting that reasoning behaviors occupy similar representational subspaces across model scales within the same family.

Cross-domain generalization (MATH500 → GPQA-Diamond, KnowLogic): Tables 1 and 2 test whether SAE columns trained on MATH500 activations transfer to GPQA-Diamond (graduate-level science QA) and KnowLogic (commonsense reasoning). For both reflection/backtracking vectors (Table 1) and confidence vectors (Table 2), the interventions produce consistent directional effects across all three domains: positive intervention increases the target behavior, negative intervention decreases it. The consistency of direction (not just magnitude) across domains with substantially different reasoning requirements (mathematical calculation vs. scientific reasoning vs. commonsense) supports the claim that the SAE captures domain-general reasoning primitives.

Sparsity and incoherence validation: Appendix C, Figure 9 provides empirical support for Theorem 1's assumptions. The maximum cosine similarity among SAE decoder columns stays below ~0.30 across all layers, consistent with the incoherence requirement. The sparsity level kk (measured as the number of active dictionary elements per reconstruction) remains well within a range where the theoretical guarantees would apply. Reconstruction error stays below 0.08, validating that the SAE faithfully captures the activation space. However, these validations use the SAE's own reconstructed activations as a proxy for the unobservable ground-truth dictionary — if the SAE has systematic biases in its reconstructions, these metrics would not detect them.

Multi-behavior column filtering: The intervention procedure (Section 4.4.1) includes a step that "filters out decoder columns that exhibit strong activations across multiple behaviors (e.g., reflection and backtracking)" before computing behavior-specific vectors. The paper does not present an ablation where unfiltered columns are used, but the design choice is motivated by the geometric finding that reflection and backtracking occupy overlapping subspaces (Figure 3). Without this ablation, we cannot quantify how much the filtering improves intervention specificity, but the reasoning is sound: if columns active for both behaviors were included, the intervention vector would blend both directions, potentially producing mixed effects.

Confidence vector cross-domain generalization: Table 2 shows that confidence vectors discovered on MATH500 transfer to GPQA and KnowLogic, with consistent directional effects on both reflection and backtracking frequencies. This serves as both a robustness check (the vectors are not MATH-specific) and a validation that confidence is a domain-general reasoning dimension.

Negative result — ReSTEM^{EM} revision model degradation: Mentioned in the broader paper context (Section 6 limitations) but worth noting as a robustness concern: an attempt to further optimize the revision model using ReSTEM^{EM} (Singh et al., 2024) caused performance to degrade substantially with sequential revisions, suggesting that the revision training procedure is sensitive to data generation methodology. This negative result highlights that not all training variants produce the intended effects, and the positive results depend on specific design choices.

Critical Assessment

The paper makes three central claims in its contributions list (Section 1): (1) that RISE captures "the structure of reasoning behaviors within the latent space" without human-defined concepts; (2) that SAE-derived vectors "align with human-interpretable behaviors and can be used to selectively modulate reasoning at inference time without additional training"; and (3) that the framework enables "discovery of novel reasoning behaviors that are difficult to define with word-level human supervision," with confidence as the exemplar. Each claim requires separate scrutiny.

Claim 1 — Unsupervised capture of reasoning behavior structure: The geometric evidence (Figures 2, 3, 7a, 11, 12) demonstrates that SAE decoder columns self-organize into behaviorally coherent clusters, and that this organization is strongest in mid-to-late layers. This claim is substantially supported by the UMAP visualizations and Silhouette score analyses, with an important caveat: the behavioral labels used to validate the clustering are themselves obtained through supervised LLM-as-a-judge annotation. The SAE training is unsupervised, but the evaluation of whether the clustering is "correct" relies on human-defined categories. This is not a flaw — it is the natural way to validate unsupervised discovery — but it means the paper demonstrates that the SAE recovers structure that aligns with human-defined behaviors, not that it discovers structure independent of human conceptual frameworks. A stronger test would be: if human annotators had never named "reflection" and "backtracking," would the clustering in Figure 2 still be interpretable? The answer is unclear from the current evidence.

The length-based organization (Figures 11, 12) provides a cleaner demonstration of unsupervised structure discovery, since "short" and "long" responses are defined by an objective numerical threshold rather than human semantic judgment. The emergence of length as an organizing axis without supervision is genuine unsupervised discovery of structural properties.

A weakness: the geometric analysis is performed on a single model (R1-1.5B) from a single model family with a specific training procedure (distillation from a reinforcement-learning-trained reasoning model). Whether the layer-wise profiles (Figures 3, 12) generalize to other reasoning models — particularly those trained directly with reinforcement learning rather than distillation, or models from different labs — is untested. The cross-model behavioral intervention (R1-1.5B → Qwen3-8B, Figure 5) shows that the behavioral vectors transfer, but does not demonstrate that the geometric organization (clustering structure, layer-wise separability) is similar across models.

Claim 2 — Selective modulation without retraining: The intervention experiments (Figures 5, 6, Tables 1, 2) demonstrate clear causal effects of SAE column manipulation on behavioral frequencies, with the dose-response experiment providing particularly strong evidence of genuine control rather than nonspecific disruption. This claim is well-supported for behavioral modulation. The cross-task and cross-model generalization strengthens the evidence substantially.

However, the paper claims not just modulation but selective modulation — the ability to control one behavior without affecting others. The evidence for selectivity is mixed. On one hand, the filtering of multi-behavior columns before computing intervention vectors is designed to improve selectivity. On the other hand, the confidence vector (Section 5) reduces both reflection and backtracking simultaneously (Table 2: reflection from 53.23 to 37.25, backtracking from 11.83 to 6.91). This is expected given the theoretical motivation — confidence and metacognitive uncertainty are inversely related — but it means the vectors modulate related clusters of behaviors rather than truly atomic, independent behaviors. The paper acknowledges this implicitly by noting the overlapping subspaces in Figure 3. A stronger selectivity claim would require demonstrating that (for example) the reflection vector reduces reflection without affecting backtracking, which the current design explicitly avoids by filtering columns that are active for both behaviors. The filtering itself is an admission that pure selectivity from individual unfiltered columns is not guaranteed.

A more significant concern: the paper does not systematically measure whether interventions preserve answer correctness across a large sample. Figure 6 shows a qualitative example where correctness is preserved across three conditions. The dose-response experiment focuses on behavioral frequency, not accuracy. The confidence intervention reports a small accuracy drop on AIME25 (23.33% → 20.00%) but correctly notes this is not statistically significant given the small sample (30 questions). For the reflection/backtracking interventions, no accuracy measurements are reported beyond the qualitative example. This is a notable gap: if suppressing reflection by 60% (from ~90 to ~34 steps) causes even a small accuracy degradation, the practical utility of the intervention is limited. The reasoning enhancement experiment (Figure 8) does report accuracy and shows improvement for the confidence vectors with learned coefficients, but this is a more sophisticated intervention (top-3 vectors with learned per-sample weights) and does not provide accuracy data for the simpler single-vector interventions used throughout Sections 4.4 and 5.

Claim 3 — Discovery of novel behaviors: The confidence vector discovery (Section 5) provides a concrete existence proof for this claim. The SAE was trained without confidence labels, and the optimization over decoder columns using entropy as a proxy objective surfaced a coherent set of vectors that modulate confidence-related reasoning behaviors. The clustering of these vectors (Figure 7a), their behavioral effects (reduced reflection/backtracking, shifted token distributions), and their cross-domain generalization (Table 2) collectively validate that they capture a genuine reasoning dimension.

This claim is supported with qualifications. The qualification is that "confidence" is operationalized through a specific proxy (output token entropy), and the discovered vectors are those that minimize this proxy. Whether these vectors would align with human judgments of confidence (e.g., asking annotators "how confident does this reasoning step seem?") is untested. The vectors optimize a behavioral metric, not a human-interpretable construct, and the paper's interpretation that they represent "confidence" is a post-hoc mapping. This is reasonable — the operationalization is behaviorally grounded — but it means "confidence" here means "low-entropy-output-inducing direction" rather than "the model's subjective confidence as a cognitive state." The distinction matters because entropy can be influenced by factors other than confidence (e.g., vocabulary constraints, repetition, formatting choices).

A deeper issue: the confidence discovery demonstrates that the SAE finds one novel behavior, but the paper frames this as a framework for discovering behaviors (plural). The generalizability of the discovery pipeline to other objectives — can we surface "coherence vectors," "creativity vectors," "factuality vectors" through appropriate proxy objectives? — is asserted but not demonstrated. The framework is extensible in principle, but the paper provides only a single case study. The claim that the framework "enables the discovery of novel behaviors" would be strengthened by a second discovery example with a different proxy objective.

Cross-cutting concerns:

Small sample sizes for behavioral measurements. The AIME25 tasks consist of 30 questions. Behavioral frequencies reported to one decimal place (e.g., 90.53 reflection steps) come from counting steps across responses to these 30 questions. Similarly, the AIME 2024 and AMC23 tasks are small benchmarks. MATH500 has 500 questions, but behavioral statistics are not reported for the full MATH500 set in most experiments. The small sample sizes make behavioral frequency comparisons noisy — a single verbose response on one question can substantially shift the average step count. Without confidence intervals, it is difficult to assess whether intervention effects are stable.

No comparison to simpler baselines for behavior discovery. The paper does not compare the SAE-based approach to a simple alternative: could you achieve similar behavioral modulation by just prompting the model differently (e.g., "be more confident" or "don't double-check your work")? If prompting achieves comparable effects, the SAE framework's value would lie primarily in interpretability (understanding the internal geometry) rather than in practical control (which could be achieved more simply). The paper compares against supervised steering methods (SEAL, TIP) for the reasoning enhancement application (Figure 8), but not for the basic behavioral modulation task. This is a missed opportunity to establish the marginal value of SAE-based interventions over zero-shot behavioral prompting.

SAE training data size is small. The SAE is trained on 500 MATH questions, each producing variable-length reasoning traces. While the total number of step-level activations is likely in the thousands to low tens of thousands, this is a relatively small training set for learning a 2048-dimensional dictionary. The paper does not report how SAE reconstruction quality or behavioral clustering changes with more training data, leaving open the question of whether the discovered structure is stable or would change substantially with larger-scale training.

Layer-wise analysis is model-specific. The finding that behavioral separability peaks in mid-to-late layers (Figures 3, 12) is characterized for the 28-layer R1-1.5B model. Whether the peak's absolute position (layers 16-24 out of 28) is a property of the model's depth, its training procedure, or the specific behaviors studied is unknown. The extension to Qwen3-8B (which has a different number of layers) does not include layer-wise analyses, only behavioral frequency measurements.

The theoretical justification is post-hoc. Theorem 1 (Section 4.1) provides conditions under which SAE decoder columns should recover the true dictionary. Appendix C validates that these conditions approximately hold for the trained SAEs. However, the theorem assumes a generative model (Equation 3) where activations are sparse linear combinations of ground-truth behavior vectors plus noise. If this generative model is a poor approximation of how reasoning behaviors are actually encoded in transformer activations — for instance, if behaviors interact nonlinearly or if the true representation is not dictionary-sparse — then the theorem provides only weak justification. The empirical results (clustering, causal interventions) are the real validation; the theorem is conceptual motivation, not a guarantee.

What experiments would strengthen the paper:

  • Accuracy measurements for all behavioral interventions. For every intervention condition in Figures 5 and Tables 1-2, report answer accuracy alongside behavioral frequency. This would address the critical open question of whether behavioral modulation degrades reasoning quality.

  • Comparison to zero-shot prompting baselines. Can the same behavioral shifts (e.g., reducing reflection from 90 to 34 steps) be achieved by prompting the model with instructions like "be concise and don't second-guess yourself" or "provide only the final answer without verification"? If prompting achieves similar effects, the case for activation-space intervention becomes one of precision and interpretability rather than unique capability.

  • A second behavior discovery case study. Confidence is one example. Demonstrating discovery of a second novel behavior (e.g., "exploration" as distinct from backtracking, or "step-by-step decomposition" as distinct from holistic reasoning) using a different proxy objective would substantially strengthen the framework claim.

  • Larger-scale behavioral frequency measurements. Report behavioral statistics on the full MATH500 test set rather than small benchmarks (AIME25, AMC23) to improve statistical reliability and enable meaningful accuracy comparisons.

  • Ablation on SAE hidden dimension and sparsity strength. How sensitive are the behavioral clustering and intervention results to the choice of DD (2048) and λ\lambda (2×10−32 \times 10^{-3})? If halving DD destroys the reflection cluster or doubling λ\lambda eliminates the confidence vectors, the findings are hyperparameter-specific. This is not explored.

Overall, the experiments provide strong evidence for the paper's core novel claim — that unsupervised SAE training on reasoning activations discovers behaviorally meaningful, causally manipulable latent directions — but the practical implications of this finding (can we control reasoning better than through prompting? can we discover multiple novel behaviors systematically? does behavioral control preserve accuracy?) are demonstrated more narrowly than the paper's framing suggests. The confidence discovery is an impressive existence proof that the unsupervised approach can surface behaviors invisible to supervised methods, and the geometric analyses provide genuine insight into the internal organization of reasoning models, but the experimental scope is limited in scale, statistical rigor, and comparative baselines.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Unaccounted for in Headline Efficiency Claims

The assumption or constraint. The compute-optimal framework requires estimating each prompt's difficulty before allocating the test-time compute budget. The paper's method for doing so—generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted)—is extraordinarily expensive. The paper acknowledges this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N (Figures 4 and 8) are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter—2048 samples per question exceeds the largest test-time budgets studied (256–512 generations). This means the 4× figure represents an upper bound on achievable efficiency rather than a realized deployment gain. A practitioner who allocates 2048 samples to difficulty estimation and then 64 samples to compute-optimal inference has actually spent 2112 samples total—negating the claimed advantage over a uniform 256-sample best-of-N baseline.

What evidence exists in the paper. The paper explicitly flags this issue in Section 3.2 and frames it as an exploration-exploitation tradeoff, but provides no experiment where difficulty estimation cost is included in the total budget. Figures 4 and 8 show compute-optimal scaling curves that begin at low generation budgets (1–4 generations), but the cost of determining which strategy to use at those budgets is entirely off the books. The predicted difficulty variant (which uses PRM scores rather than ground-truth labels) does not eliminate this cost—it still requires generating and scoring 2048 samples per question. Appendix C (Figures 11–12) discusses the predicted vs. oracle bin alignment but does not address estimation cost.

Mitigation status. The paper partially acknowledges the problem and suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 3.2), but no such model is developed or evaluated. The predicted difficulty bins (using PRM scores) perform nearly as well as oracle bins (Figures 4, 8), but this only removes the need for ground-truth labels—it does not reduce the sample cost of estimation. The paper does not explore cheaper estimation strategies (e.g., using fewer samples, adaptive estimation that stops when difficulty is sufficiently determined, or lightweight classifier-based approaches), leaving the primary practical barrier to deployment unaddressed.

6.2 Single Benchmark, Single Model Family, Small Test Set

The assumption or constraint. All primary results are obtained on the MATH benchmark (500 test questions) using PaLM 2-S* as the base model. The paper states in Section 4 that it "believe[s] this model is representative of the capabilities of many contemporary LLMs," but this claim is unverified. The FLOPs-matched comparison uses a second model with approximately 14× more parameters, but this model comes from the same family and is only tested under greedy decoding with no test-time augmentation of its own.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific, and the paper provides no evidence to distinguish universal from idiosyncratic patterns:

  • The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different error patterns, or different base reasoning capabilities might exhibit entirely different difficulty-dependent scaling curves. The beam-search-degrades-on-easy-problems finding (Figure 3, right) is particularly sensitive to verifier quality, which is a function of the base model's output distribution.

  • The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The edit-distance-based pairing strategy for training data construction (Section 6.1) may not transfer to models with different output structures.

  • The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning. Whether the difficulty-dependent patterns generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual knowledge rather than inference is unknown.

  • The test set of 500 questions, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation, means the compute-optimal policy is selected based on approximately 50 questions per fold per bin. The paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess whether strategy selection is stable at this sample size or would change substantially with a different random split.

What evidence exists in the paper. The limitation is partially acknowledged in Section 8 (future work), but no experiments test generalization. All quantitative results in Sections 5–7 are on the MATH 500-question test set using PaLM 2-S*. The FLOPs-matched comparison (Section 7) uses the same benchmark and model family. The appendix results (Figures 13–16) are also MATH-specific. The paper provides no replication on GSM8K, MMLU, HumanEval, or any other reasoning benchmark; no experiments with a different base model family (e.g., LLaMA, GPT, Claude); and no analysis of how results would change with a larger test set.

Mitigation status. The paper does not attempt to address this limitation experimentally. It acknowledges in Section 8 that extending the analysis to "other reasoning tasks and model families" is an important direction for future work, and notes that the proposed framework is general and should apply beyond MATH, but provides no empirical support for this claim. The single-benchmark, single-model-family scope limits the strength of conclusions that can be drawn about the universality of the difficulty-dependent scaling patterns.

6.3 Verifier Over-Optimization Is a Hard Ceiling That Is Not Solved, Only Routed Around

The assumption or constraint. All search-based test-time compute methods rely on the PRM's ability to accurately score partial solutions. When the PRM is imperfect—which it inevitably is, since it is trained on finite data and cannot perfectly generalize—aggressive optimization (beam search, lookahead search) eventually exploits its errors, producing solutions that score highly under the PRM but are actually incorrect. The paper documents this phenomenon extensively in Section 5.3 and Appendix M.

The consequence. The compute-optimal policy mitigates verifier over-optimization by routing easy problems away from aggressive search (using best-of-N instead of beam search when the verifier's errors would be exploited), but it does not eliminate the underlying problem. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling—the beam search curves in Figure 3 flatten and in some cases decline as budget increases, meaning that additional compute beyond a certain point is wasted or counterproductive. This places a hard upper bound on what test-time compute can achieve regardless of budget: if the verifier cannot reliably distinguish correct from incorrect solutions in a particular difficulty regime, no allocation strategy can overcome this limitation. The paper's finding that lookahead search—the most powerful optimizer—paradoxically performs worst overall (Figure 3, left) is direct evidence that more sophisticated search amplifies rather than overcomes verifier weakness.

A practical consequence is that the compute-optimal framework's effectiveness is inherently tied to verifier quality, which is itself a function of training data quality, base model calibration, and problem domain. A practitioner deploying this approach on a new domain where the PRM is less accurate would face a tighter over-optimization bound, potentially making aggressive search counterproductive across a wider range of difficulty levels.

What evidence exists in the paper. The evidence is extensive and multi-pronged. Figure 3 (right) shows beam search accuracy decreasing at high budgets on easy problems (Bin 1: ~78% at 4 generations declining to ~77% at 256). Figure 3 (left) shows lookahead search underperforming best-of-N at equivalent budgets. Appendix M (Figures 29 and surrounding qualitative examples) shows specific failure modes: repetitive low-information steps and overly short 1–2 step solutions that score highly under the PRM but are incorrect. The paper explicitly identifies PRM over-optimization as the mechanism behind these patterns in Section 5.3.

Mitigation status. The compute-optimal policy is a workaround, not a solution. It adaptively reduces optimization pressure in regimes where the verifier is unreliable (easy problems → best-of-N) and applies optimization only where the verifier signal has room to provide genuine guidance (medium problems → beam search). The paper identifies improving verifier robustness as a key direction for future work (Section 8), but provides no concrete proposals or experiments toward this goal. The PRM training procedure (Monte Carlo rollouts, Appendix D) is presented as-is without investigation into how training data quality, adversarial robustness, ensemble methods, or calibration techniques might push back the over-optimization frontier. This means a practitioner who hits the verifier ceiling has no guidance from the paper on how to raise it.

6.4 The 14× Larger Model Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal training (Hoffmann et al., 2022) where both data and parameters are scaled equally. The paper acknowledges this choice explicitly in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses only greedy decoding—no majority voting, no best-of-N, no search of any kind—while the smaller model is augmented with compute-optimal test-time strategies.

The consequence. The comparison is systematically biased in favor of test-time compute. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and training tokens) would likely outperform a parameter-only-scaled model on the same total compute budget, making the pretraining baseline weaker than it could be. Moreover, giving the larger model even a modest test-time compute budget—say, best-of-8 with majority voting—would create a significantly stronger baseline. The reported advantages of test-time compute over pretraining (e.g., +27.8% relative improvement on easy problems at R ≪ 1 for revisions, Figure 1 top-right bar chart) are therefore upper bounds on the true advantage, and may shrink, vanish, or reverse against a properly optimized larger model with its own inference-time augmentation.

The asymmetry matters because the paper's central practical claim—that test-time compute can substitute for pretraining in certain regimes—depends on a fair comparison. If the larger model were given access to the same test-time strategies available to the smaller model, the substitution argument would need to show that the smaller model's additional test-time compute (beyond what the larger model also gets) compensates for the parameter gap. The current experimental design conflates "test-time compute vs. pretraining" with "test-time compute vs. no test-time compute," making it impossible to isolate the marginal value of test-time compute over pretraining.

What evidence exists in the paper. The paper is transparent about the parameter-only scaling choice (Section 7) and frames it as a deliberate simplification, not an oversight. However, it does not discuss the greedy decoding asymmetry or the implications for the strength of its conclusions. Figure 9 and the bar charts in Figure 1 present the FLOPs-matched results without caveats about the pretraining baseline's suboptimality. No experiment compares the larger model with any form of test-time augmentation.

Mitigation status. The paper explicitly defers the compute-optimal pretraining comparison to future work, which is an honest acknowledgment but does not address the limitation within the current study. The greedy decoding asymmetry is not acknowledged. A fairer comparison—giving the larger model access to the same test-time strategies, or at minimum some form of best-of-N or majority voting—would substantially strengthen the FLOPs-matched conclusions. The current results should be interpreted as evidence that test-time compute can narrow the gap between small and large models, not that it outperforms pretraining in a head-to-head comparison.

6.5 Revisions and PRM Search Are Studied Independently, Never Combined, and the Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate

The assumption or constraint. The paper studies two complementary axes—PRM tree-search and iterative revisions—but never combines them. Section 8 explicitly acknowledges this:

"we did not experiment with PRM tree-search techniques in combination with revisions"

Furthermore, the revision model has a known reliability problem: approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in subsequent steps (Section 6.1). This occurs because the model was trained only on sequences where all in-context answers are incorrect followed by a correct target—it never learned to recognize when the current answer is already correct and should be preserved.

The consequence. The current results represent a lower bound on what a fully integrated system could achieve. Revisions improve the proposal distribution (generating better candidates), while PRM search improves candidate selection (finding the best among generated candidates). Applying beam search to revision model outputs—or using the PRM to guide which revisions to pursue and when to stop—could yield gains beyond either method alone, particularly on medium-difficulty problems where both mechanisms show complementary strengths (revisions for local refinement, search for global exploration).

The 38% reversion rate means that longer revision chains are not strictly beneficial—each additional revision step has a nontrivial probability of corrupting a previously correct answer. The paper mitigates this with majority voting or verifier-based selection across the entire chain (Section 6.1), but these are post-hoc patches that add computational overhead (generating and evaluating many revisions) to recover from a problem that could be addressed at the training stage. A revision model trained to also recognize and preserve correct answers—or equipped with a stopping criterion based on verifier confidence—would be more efficient and reliable.

What evidence exists in the paper. The separate effectiveness of each axis is well-established: PRM search improves over best-of-N on medium problems (Figure 3, right), sequential revisions outperform parallel sampling in aggregate (Figure 6, right), and both show 4× efficiency gains in isolation (Figures 4 and 8). However, no experiment combines them. The 38% reversion rate is reported explicitly in Section 6.1, and the ReSTEM^{EM} experiment (Appendix K, Figure 16) provides additional evidence of revision model fragility—attempting to further optimize the revision model with RL-style training caused performance to degrade substantially with sequential revisions.

Mitigation status. The paper acknowledges the missing combination as a direction for future work (Section 8) and mitigates the reversion problem at inference time via chain-wide selection (majority voting or verifier), but neither limitation is resolved. The chain-wide selection approach means that longer revision chains increase the total generation cost without guaranteeing monotonic improvement—practitioners must balance the benefits of additional revision steps against the cost of generating and evaluating candidates that may degrade earlier correct answers. A more principled solution (training the model to recognize correctness, implementing an early-stopping criterion, or combining revisions with PRM-guided pruning) is not explored.

6.6 Sequential Revision Strategies Introduce Latency That Is Ignored in the FLOPs-Based Cost Model

The assumption or constraint. The paper measures test-time compute in "generations" (number of complete solution trajectories sampled), which is a reasonable proxy for total FLOPs but entirely ignores wall-clock latency. Sequential revisions are inherently serial—each revision conditions on the previous one and cannot be generated until the prior step completes—while parallel best-of-N sampling can be executed simultaneously across multiple devices or batches.

The consequence. A strategy that allocates 256 generations as 16 sequential × 16 parallel (a balanced ratio that is optimal for hard problems according to Figure 7, right) takes approximately 16× longer wall-clock time than a strategy that runs 256 parallel samples simultaneously, even though both consume the same total FLOPs. For latency-sensitive applications—interactive assistants, real-time decision-making, any deployment where users wait for responses—the sequential-heavy strategies favored by the compute-optimal policy on easy problems (where purely sequential revisions are optimal, Section 6, Figure 7 right) may be impractical regardless of their FLOPs efficiency advantages. A response that requires 64 sequential revision steps may be correct but takes minutes to generate, making it unsuitable for applications where a sub-second response is required.

Additionally, the compute-optimal policy selects strategies per-question based on difficulty, meaning that a batch of mixed-difficulty queries would have heterogeneous latency profiles—some completing quickly (easy questions with few sequential steps), others taking much longer (hard questions requiring balanced sequential-parallel allocation). In a production system serving many queries concurrently, this heterogeneity complicates load balancing, timeout management, and user experience predictability.

What evidence exists in the paper. The paper never discusses latency, wall-clock time, or throughput. All cost comparisons are in terms of "generations" (Section 5.3, Section 6.1, Section 7). The sequential-to-parallel ratio sweep (Figure 7) is presented purely in terms of accuracy without any corresponding latency measurement. The compute-optimal scaling curves (Figures 4 and 8) use generation count as the x-axis with no secondary axis or discussion of time cost. The FLOPs-matched comparison (Section 7) uses total FLOPs as the cost metric, which treats serial and parallel FLOPs identically despite their different real-time implications.

Mitigation status. The paper does not address this limitation at all. It does not acknowledge latency as a concern, does not report wall-clock times for any experiment, and does not discuss how sequential vs. parallel strategies trade off accuracy against response time. A practitioner deploying these methods would need to separately evaluate whether the accuracy gains from sequential revisions justify the latency cost in their specific use case—information the paper does not provide. Future work on latency-aware compute-optimal allocation, where the budget is defined in wall-clock time rather than total generations, would be a natural extension but is not proposed in the paper's future directions (Section 8).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a discovery-driven paradigm for interpreting reasoning in language models, shifting the field from hypothesis-driven methods—where researchers must name a behavior before they can find it—to unsupervised decomposition where the model's own representational structure proposes candidate behavioral dimensions. The significance of this shift is not that SAEs are new (they are established tools in mechanistic interpretability since Cunningham et al., 2023), but rather that applying them to step-level reasoning activations reveals a geometrically organized, causally manipulable latent space whose structure was invisible to prior supervised approaches.

The shift is best characterized as a platform contribution: RISE establishes a three-stage pipeline (unsupervised training → geometric analysis → causal validation) that future work can instantiate with different models, different SAE architectures, different proxy objectives for discovery, and different intervention strategies. The paper's specific findings about reflection, backtracking, and confidence are less important than the demonstration that the pipeline works—that behaviors humans can name (reflection, backtracking) emerge as coherent clusters, that behaviors humans would struggle to operationalize (confidence) surface through simple proxy objectives, and that intervention on individual decoder columns produces dose-dependent behavioral changes without retraining. This transforms reasoning interpretability from a cottage industry of hand-crafted contrastive datasets into a potentially scalable discovery science.

The paper resolves a latent tension in prior work that the field had not explicitly articulated: supervised activation steering methods (DiffMean and its variants) are fundamentally bounded by the coverage and granularity of human-defined categories. DiffMean works well when concepts are clearly separable and easily labeled (happy vs. sad in sentiment analysis, factual vs. hallucinated statements in truthfulness research), but reasoning behaviors are fluid, overlapping, and only partially lexicalized. The paper's finding that reflection and backtracking occupy overlapping representational subspaces (Figure 3: lower Silhouette scores for Reflection vs. Backtracking than for either vs. Other) explains why a simple reflection-minus-non-reflection DiffMean vector would likely produce a blended signal rather than a pure behavioral direction. The SAE's sparsity constraint (enforced by the L0 penalty in Equation 2) encourages disentanglement by forcing each activation to be reconstructed from a small number of dictionary elements, making it more likely that distinct behaviors map to distinct (or nearly distinct) columns. This does not make DiffMean obsolete—it remains appropriate for well-separated concepts—but it establishes that reasoning behavior discovery requires unsupervised decomposition, not supervised contrast, because the relevant behavioral dimensions cannot be fully specified in advance.

The confidence discovery (Section 5) makes a specific research direction newly attractive that was previously nearly impossible: objective-driven discovery of latent behavioral dimensions. Before this work, finding a "confidence direction" in a language model would have required either (a) a labeled dataset of confident vs. unconfident reasoning steps (difficult because confidence is not cleanly lexicalized), or (b) a supervised probe trained to predict some proxy of confidence from activations (requiring a pre-existing operationalization). The RISE framework reduces this to: train an SAE once (without any behavioral labels), then optimize a score vector over the already-learned decoder columns using any differentiable behavioral objective (entropy for confidence, but potentially accuracy, coherence, factuality, or creativity in future work). The decoder columns are a fixed dictionary; discovering a new behavior means finding which linear combination of columns best optimizes the objective. This decouples dictionary learning from behavior discovery, making it dramatically cheaper to explore new behavioral dimensions—train the SAE once, then run many lightweight optimizations over its columns for different objectives.

A research direction that becomes less attractive after this paper is the pursuit of ever-finer-grained token-level analyses of reasoning. The paper's deliberate choice to work at the sentence level—using delimiter token representations that have attended to the entire preceding reasoning step—and its explicit justification that "the same token can play diverse roles across different contexts" (Section 3.2) implies that token-level activation steering for reasoning behaviors is likely to be noisy and context-dependent. The SAE's success at sentence-level granularity suggests that reasoning behaviors are encoded at the level of complete cognitive sub-operations, not individual tokens, and that future interpretability work on reasoning should adopt this granularity by default.

Follow-Up Research This Work Enables

Systematic evaluation of whether behavioral modulation preserves or degrades accuracy across large benchmarks. The paper demonstrates that interventions on SAE-derived vectors change behavioral frequencies—reducing reflection steps from approximately 90 to 34 on AIME25 tasks (Section 4.4), reducing backtracking from 35 to 6 under confidence vector intervention (Section 5.1)—but does not systematically measure whether these behavioral shifts affect answer correctness beyond a single qualitative example (Figure 6) and a note that the AIME25 accuracy drop (23.33% to 20.00%) is not statistically significant at n=30. A strong follow-up would evaluate accuracy across the full MATH500 test set for every intervention condition in Figures 5 and 6, at multiple intervention strengths, for both reflection and backtracking vectors. If suppressing reflection by 60% causes no significant accuracy degradation on MATH500 (500 questions), the practical case for controllable reasoning is substantially strengthened—it would mean models can be made dramatically more concise without sacrificing reliability. If accuracy degrades meaningfully (say, a 3-5 percentage point drop), the tradeoff between verbosity and correctness becomes a design parameter that practitioners need to tune per-application. A particularly informative design would test interventions at varying strengths on difficulty-stratified subsets (using the kind of difficulty binning approach from the compute-optimal test-time scaling literature, as discussed in prior sections), to determine whether behavioral suppression is more costly on hard problems (where verification may be necessary) than on easy ones (where it may be superfluous).

Discovery of exploration-related reasoning vectors using a divergence-maximization objective. The confidence discovery (Section 5) used entropy minimization to find directions that make the model more decisive; a natural complement is to use an objective that rewards diverse, multi-path exploration to find directions associated with the model's strategy-switching or hypothesis-generation behaviors. Concretely, one could define a proxy objective that maximizes the variance of the model's output distribution across multiple sampled completions from the same intermediate hidden state—essentially rewarding the model for considering multiple distinct continuations—and optimize a score vector over SAE decoder columns (Equation 7, with the sign flipped) to identify columns that, when amplified, increase exploratory behavior. The resulting vectors could be validated by measuring whether they increase backtracking frequency specifically (as distinct from reflection), increase the number of distinct solution strategies attempted per problem, or increase response length without increasing self-verification. This would test whether the SAE has learned disentangled directions for different aspects of metacognition (verification vs. exploration) or whether these all collapse into a single "reasoning depth" dimension. The existence of separate exploration vectors would be strong evidence for the linear representation hypothesis applied to fine-grained cognitive primitives; their absence would suggest that the current SAE training setup (modest overcompleteness, specific sparsity strength) may not provide enough capacity or pressure to separate closely related behaviors.

Adversarial validation of SAE-derived reasoning vectors through out-of-distribution behavioral probes. The paper demonstrates that SAE columns trained on MATH500 activations generalize to GPQA-Diamond and KnowLogic (Tables 1 and 2), but these domains are still within the broad category of "reasoning tasks with objectively correct answers." A stronger test would probe generalization to domains where the same lexical markers (e.g., "wait," "alternatively") appear but serve entirely different functions—such as narrative text (where "wait" signals plot pacing rather than self-verification), dialogue (where "alternatively" may be politeness rather than strategy-switching), or code generation (where self-verification comments serve documentation purposes). If the SAE-derived reflection vector suppresses "wait" in a story but leaves the narrative coherence intact, it would indicate that the vector captures task-specific reasoning behavior rather than superficial lexical patterns. If it degrades narrative coherence, it would suggest that the SAE has partially learned a lexical detector rather than a genuine cognitive primitive. This experiment matters because the paper's theoretical justification (Theorem 1) assumes the SAE recovers a dictionary of latent behavior directions, but the empirical validation only checks whether the discovered directions align with human-labeled behaviors—it does not rule out the possibility that the SAE is discovering dataset-specific or surface-level patterns that happen to correlate with the behaviors of interest on MATH.

Layer-wise causal tracing to identify where reasoning behaviors emerge and how they propagate. The paper's layer-wise Silhouette score analyses (Figures 3 and 12) show that behavioral separability peaks in mid-to-late layers, but this is a purely correlational finding—it does not establish where behavioral information is computed, only where it becomes linearly separable. A causal follow-up would apply the intervention from Section 4.4 at every layer individually (rather than only the final layer) and measure the downstream effect on behavioral frequencies. If intervening at layer 16 (where the Silhouette score is rising but not yet at peak) produces a smaller behavioral shift than intervening at layer 24 (near the peak), it would suggest that the behavioral representation is progressively refined and that early-layer information about reasoning behavior is insufficient to fully determine the output. If intervening at layer 8 produces nearly the same effect as layer 24, it would suggest that behavioral information is computed early and merely made more accessible (rather than created) in later layers. Running this experiment for both reflection and backtracking vectors would also test whether different behaviors have different causal loci—perhaps backtracking (which involves evaluating and discarding a strategy) is computed in earlier layers than reflection (which involves re-examining prior steps in detail). The results would provide a functional map of the reasoning model's internal processing that complements the structural map from the geometric analysis.

Comparison of RISE-discovered vectors against instruction-based behavioral control. The paper demonstrates that SAE-vector interventions can shift behavioral frequencies (e.g., reducing reflection from ~90 to ~34 steps on AIME25), but does not establish whether this capability is unique to activation-space manipulation or could be achieved more simply through prompting. A direct comparison would test, on the same set of problems, whether zero-shot instructions ("be concise and avoid double-checking your work," "provide only the final answer," "do not express uncertainty") produce comparable reductions in reflection and backtracking step counts while preserving accuracy. If prompting achieves a 50% reduction in reflection with no accuracy loss, the SAE intervention's value proposition shifts from "enabling new capabilities" to "providing finer-grained or more reliable control"—still valuable, but for different reasons. If prompting fails (e.g., the model ignores concision instructions on complex math problems, or concision instructions degrade accuracy more than SAE interventions do), it strengthens the case that activation-space control accesses behavioral dimensions that are not responsive to surface-level linguistic instructions. A well-designed experiment would also test whether prompting and SAE interventions can be combined—using prompting for coarse-grained behavioral instructions and SAE vectors for fine-tuning—to achieve better accuracy-efficiency tradeoffs than either alone.

Scaling analysis: how do SAE hyperparameters affect the quality and disentanglement of discovered reasoning vectors? The paper uses a single configuration—hidden dimension D=2048, sparsity strength λ=2×10⁻³, 500 training questions—based on stated intuitions (reasoning behavior complexity is "much lower than modeling raw language structure") but without ablations to validate these choices. A systematic follow-up would sweep D (e.g., {512, 1024, 2048, 4096}), λ (e.g., {1×10⁻⁴, 5×10⁻⁴, 1×10⁻³, 2×10⁻³, 5×10⁻³}), and training data size (e.g., {100, 500, 2000, 5000} MATH questions) and measure three outcome variables: (1) reconstruction error (to ensure the SAE faithfully represents the activation space), (2) Silhouette scores for Reflection/Backtracking/Other clustering (to measure behavioral separability), and (3) the dose-response slope of behavioral interventions (the change in step count per unit change in α, to measure how cleanly individual columns encode behaviors). If behavioral separability is robust across a wide range of hyperparameters, the findings are likely to replicate. If it is brittle—appearing only at specific D and λ values—the conclusions about unsupervised structure discovery are contingent on careful tuning that future practitioners would need to replicate. A particularly informative result would be whether increasing D beyond 2048 causes feature splitting, where a single behavior (e.g., reflection) gets distributed across multiple columns (reducing per-column interpretability but potentially preserving total behavioral information), or whether the sparsity penalty naturally prevents this.

Practical Applications and Downstream Use Cases

Inference-time reasoning cost control for API-based LLM deployments. The paper's central practical finding is that SAE-derived reasoning vectors can substantially reduce reasoning verbosity without (preliminary evidence suggests) degrading answer quality. On AIME25 tasks, negative reflection intervention reduced step counts from ~90 to ~34 (a 62% reduction) while the model continued to produce correct answers (the qualitative example in Figure 6 shows preserved correctness, and the paper notes only a non-significant accuracy change of 23.33% to 20.00% on 30 questions). The confidence-vector-based reasoning enhancement (Figure 8) achieved a 13.69% token reduction on MATH500 while improving accuracy by up to 4.66 percentage points. For a cloud API provider serving millions of reasoning queries daily, where costs scale with output token count, even a 10% token reduction with preserved accuracy translates directly to infrastructure savings. An API could implement this by training an SAE once on a representative sample of reasoning traces, identifying the confidence and reflection vectors, and then applying them as default interventions for all reasoning queries—or, more sophisticatedly, by learning per-user or per-application intervention strengths that balance concision against accuracy based on the specific use case (a math tutoring application might tolerate verbose verification more than a high-throughput data extraction pipeline).

On-the-fly reasoning style adaptation for interactive assistants. The dose-response relationship demonstrated in Section 4.4.1—that varying the intervention scalar α ∈ {-1.5, -1, 0, 1, 1.5} produces monotonic changes in reflection step counts from 58.6 to 166.9—enables a new mode of interaction where users dynamically control how a model reasons, not just what it answers. An interactive reasoning assistant could expose a "thoroughness" slider that adjusts α on the reflection vector, allowing users to request quick, confident answers (negative α) for routine questions and careful, self-verifying reasoning (positive α) for high-stakes decisions—all using the same model without retraining or switching between model variants. This is qualitatively different from existing approaches like temperature scaling (which affects randomness, not reasoning strategy) or system prompts (which are coarse-grained and model-dependent). Because the SAE vectors transfer across domains (Tables 1 and 2: MATH500-trained vectors work on GPQA-Diamond and KnowLogic), a single set of vectors could control reasoning style across diverse query types in a general-purpose assistant. The key deployment question is whether the accuracy of negative-α responses is high enough for the intended use case—formal evaluation of accuracy under intervention on large benchmarks would be prerequisite.

Diagnostic tool for reasoning model development and comparison. The layer-wise geometric analyses (Figures 3, 11, 12) provide a structural fingerprint of how a reasoning model organizes behavioral information internally. A model development team could train SAEs on multiple candidate models—different training recipes, different scales, different architectures—and compare their behavioral separability profiles. If Model A shows a sharper Silhouette score peak and earlier layer-wise separation than Model B, it might indicate more efficient behavioral encoding. If Model C shows no clear length-based clustering while Model D shows strong separation (as in Figure 12), it might indicate that Model C has not learned to organize response verbosity as a distinct representational axis, potentially explaining why it produces inconsistently verbose outputs. This diagnostic use case does not require the causal intervention machinery—it only requires training SAEs and computing the geometric metrics, which the paper's Appendix C (Figure 9) shows is computationally modest (low reconstruction error, bounded sparsity and incoherence). The practical value is that it provides a quantitative, unsupervised method for comparing reasoning models' internal organization without relying on behavioral benchmarks, which confound model capability with evaluation methodology.

Offline data filtering and quality scoring for reasoning datasets. The confidence vectors discovered in Section 5—which shift the model toward higher-confidence, lower-entropy outputs—could be applied during synthetic data generation to filter or select reasoning traces. When using a reasoning model to generate training data for distillation or self-improvement (as in STaR, ReSTEM^{EM}, or standard rejection sampling), the generated traces vary in quality: some are correct, concise, and confident; others are verbose, uncertain, and potentially incorrect. Computing the projection of each reasoning step's hidden state onto the confidence vector direction (i.e., wiThw_i^T h for the confidence-related decoder columns) provides a scalar confidence score that can be used to filter traces—keeping those with high average confidence, discarding those with low confidence—without running an external verifier or relying on ground-truth labels. This is particularly valuable for domains where correctness signals are expensive (e.g., open-ended generation, creative tasks) or where the model's own uncertainty is a proxy for likely errors. The paper's evidence that confidence vectors transfer across domains (Table 2) suggests this filtering approach would work for diverse reasoning tasks, not just math.