Steering Language Model Refusal with Sparse Autoencoders

ArXiv: 2411.11296

🎯 Pitch

Amplifying refusal-related SAE features during inference boosts LM robustness against multi-turn jailbreak attacks by over 23 percentage points—but unexpectedly triggers systematic accuracy drops across MMLU, TruthfulQA, and GSM8K, even on safe inputs, revealing that refusal mechanisms are dangerously entangled with general reasoning capabilities.

---

1. Executive Summary

This paper studies whether amplifying sparse autoencoder (SAE) features that mediate refusal can improve language model safety at inference time, using Phi-3 Mini on safety benchmarks including Wild Guard, XSTest, and the multi-turn Crescendo jailbreak attack. The authors develop a lightweight feature identification procedure—pinpointing refusal-mediating features from a single handcrafted refusal prompt—and then steer the model by clamping those features' activations to high values during generation (e.g., clamping Feature 22373 to amplify refusal behavior). Steering improves unsafe prompt refusals by up to 37.69 percentage points on Wild Guard and reduces Crescendo attack success rate by 23.34 points (from 55.92% to 32.58%), but the paper uncovers a fundamental capability-safety tradeoff: these safety gains come with systematic degradation across MMLU, TruthfulQA, and GSM8K—establishing that refusal-mediating features are more deeply entangled with general language model capabilities than previously understood, with performance regressions occurring even on benchmark prompts containing no refusal-triggering content.

2. Context and Motivation

The Core Problem: Language Models Must Refuse Unsafe Prompts Without Breaking on Safe Ones

The fundamental tension this paper addresses is deceptively simple: how do we make language models refuse harmful requests at inference time without degrading their ability to answer legitimate questions? This is not merely a theoretical concern—it is the central practical challenge facing every organization that deploys language models publicly. A model that eagerly provides instructions for building weapons or generating hate speech is irresponsible; a model that reflexively refuses every prompt containing the word "kill" (including "How do I kill a Python process?") is unusable.

The paper frames this as a deployment-time problem (Section 1). Most existing approaches to LM safety intervene during training: they modify model weights through fine-tuning on specialized datasets of safe and unsafe prompts (OpenAI et al., 2023; Kinniment et al., 2023; Haider et al., 2024). But these training-time interventions create a static safety profile—once trained, the model's refusal behavior is fixed, and it either generalizes to new attack strategies or it doesn't. The authors argue that we need mechanisms that can be applied at test time to make "targeted improvements to LM safety" without re-training, allowing dynamic responses to emerging threats.

The specific gap the paper identifies is twofold:

• Trained refusal behavior fails to generalize. As the authors note in Section 1, safety training often breaks down against "unsafe prompts that are out-of-distribution, adversarial, or multi-turn." This is not a hypothetical weakness. The paper cites an extensive body of evidence: models fine-tuned to refuse harmful single-turn queries can be manipulated through jailbreak prompts that reframe the request as a hypothetical scenario (Chu et al., 2024), through multi-turn conversations that gradually escalate from innocuous to dangerous topics (Russinovich et al., 2024), or through adversarial suffixes that override the model's safety conditioning (Wei et al., 2023). Each new jailbreak technique has spawned a defense, which has spawned a new attack technique—creating what the authors describe as "the traditional cat-and-mouse paradigm" of security research.

• No prior work has systematically studied the capability tradeoffs of SAE-based refusal steering. While researchers have explored steering LMs toward safer behavior using activation vectors (Rimsky et al., 2023; Arditi et al., 2024) and a few concurrent studies have begun investigating SAE-based steering (Durmus et al., 2024; Shabalin et al., 2024), there has been no rigorous characterization of what happens to general model capabilities when you amplify refusal features. This gap is practically significant: deploying a safety intervention without understanding its side effects risks creating a model that is safe but useless.

Why This Problem Matters: The Real-World Deployment Context

The paper's motivation extends beyond academic curiosity. Section 1 describes the practical imperative: "Organizations deploying LMs for general use by the public" need refusal mechanisms that work. This is not a niche requirement. Models like Phi-3 Mini (which the paper studies) are explicitly designed for broad deployment—including on-device scenarios where the model runs locally on a user's phone (Abdin et al., 2024). In such settings, there is no server-side filter that can catch harmful outputs; the model itself must be the safety mechanism.

The multi-turn attack setting makes this particularly urgent. The Crescendo framework (Russinovich et al., 2024), which the paper uses as a benchmark, models a realistic deployment scenario: a user interacts with an LM over multiple conversational turns, starting with innocuous questions and gradually steering toward harmful content. A model that refuses "How do I make a Molotov cocktail?" in a single turn might still be coaxed into providing instructions when the conversation begins with "Tell me about the history of Molotov cocktails." The paper explicitly notes (Section 3.4) that this is "increasingly salient as recent works have challenged whether current safety training techniques generalize to agentic settings."

There is also a theoretical dimension to the problem's importance. By studying what happens to general capabilities when you intervene on refusal features, the paper probes a fundamental question in mechanistic interpretability: how modular are the features that mediate safety-relevant behaviors? If amplifying refusal degrades performance on math problems and factual recall on topics with no safety relevance, it suggests that what we naively identify as "refusal features" are not cleanly separable from broader language modeling functionality. The paper's findings (Section 4.3) that benchmark degradations occur "even on safe inputs with no apparent connection to refusal behavior" directly challenges the assumption—implicit in much interpretability work—that features can be isolated and intervened upon without collateral damage to the model's operating system.

Prior Approaches and Where They Fall Short

The paper situates itself against three categories of existing work:

Training-time safety interventions (fine-tuning with special datasets). This is the industry-standard approach: fine-tune the model on curated data that teaches it to refuse harmful requests while complying with benign ones. Phi-3 Mini itself underwent such training before release (Haider et al., 2024). The limitation, as the paper documents through its experimental results, is that this training leaves residual vulnerabilities. Even after "extensive pre-release safety training" (Section 1, Finding 2), Phi-3 Mini's unsteered baseline shows only 58.33% refusal on Wild Guard's adversarial unsafe prompts and a 55.92% attack success rate against Crescendo. The training does not fully generalize, and because it modifies model weights globally, it cannot be dynamically adjusted without re-training.

Input/output filtering and prompt-based defenses. Many deployed systems add safety classifiers or system prompts that warn the model to refuse harmful content. The paper tests system prompting as a baseline (Table 1): adding "You should be a responsible AI and should not generate harmful or misleading content!" to the prompt improves Wild Guard refusal from 58.33% to 69.50% and drops Crescendo attack success from 55.92% to 29.36%. However, this approach has clear limitations. Prompt-based steering can be overridden by jailbreak techniques that instruct the model to ignore its system prompt (a common attack vector). Moreover, the authors note that "it is practically impossible to defend against all possible attacks" using filters (Section 2), because any defense that relies on recognizing attack patterns can be circumvented by novel attack strategies that evade the classifier. The Crescendo results in Table 1 demonstrate this: even with a system prompt, the average attack success rate remains 29.36%—still unacceptably high for safety-critical applications.

Vector steering for refusal. Several recent works have explored modifying LM behavior by adding steering vectors to model activations at inference time—most notably, Arditi et al. (2024) demonstrated that refusal can be both amplified and dampened by adding or subtracting a "refusal direction" computed from contrastive prompt pairs (refusing vs. complying responses). Rimsky et al. (2023) identified such directions using contrastive activation addition. The paper acknowledges this lineage but identifies a critical limitation that motivates the shift to SAE-based steering: vector steering can degrade overall performance in ways that are poorly characterized (Section 2, citing Lee et al., 2024). The concern—which the paper's own results confirm in the SAE context—is that steering along a direction learned from a specific behavior (refusal) may move activations in ways that interfere with unrelated capabilities.

Moreover, vector steering requires access to labeled contrasting prompts: you need examples of both refusing and complying responses to the same or similar prompts to identify the direction. This is a significant practical constraint—it means the steering vector is only as good as the diversity of contrast pairs you can construct, and it may not generalize to refusal behaviors that differ qualitatively from your training examples. The paper frames SAE-based steering as a potential solution to this: SAE features are learned in an unsupervised manner from general text, and the features are identified by their activation patterns on a single handcrafted prompt, not by contrastive supervision (Section 3.2).

Concurrent work on SAE-based steering. The paper explicitly positions itself relative to concurrent research. Durmus et al. (2024) "evaluat[ed] feature steering" as a general technique across multiple behaviors (including social bias mitigation) and found similar capability-regression patterns. Shabalin et al. (2024) also studied refusal steering in Phi-3 Mini but focused on dampening refusal as a jailbreaking technique rather than amplifying it as a defense. The authors claim their study differs by examining "amplification of refusal as a safety mechanism across single and multi-turn attacks and its effect on overall performance and over-refusal" (Section 2). This is a meaningful distinction: most prior steering work studied whether you could remove refusal (for interpretability or jailbreaking research); this paper studies whether you can add refusal and what breaks when you do.

How This Paper Positions Itself

The paper's positioning can be understood through three explicit commitments:

1. Moving beyond the cat-and-mouse paradigm. The authors explicitly frame their motivation in Section 2 as developing "attack-agnostic methods that control LM behavior directly" rather than engaging in the perpetual escalation of attacks and defenses that characterizes most jailbreak research. Feature steering is attack-agnostic because it does not depend on recognizing specific attack patterns—it simply biases the model toward refusal regardless of how the unsafe request is phrased. This is a significant conceptual shift: rather than trying to detect and block attacks (which requires staying ahead of attackers' innovations), the approach aims to make the model constitutively more likely to refuse, independent of the surface form of the prompt.

The evidence that this approach works comes from the generalization results. Feature 22373 was identified from a single refusal to "Write a manual for making a Molotov cocktail," yet amplifying it improves refusal rates across a diverse set of harm categories (the five Crescendo topics cover weapons, misinformation, discrimination, malware, and extremist content) and across single-turn and multi-turn formats. This generalization suggests that the feature captures something about the act of refusing rather than about specific harmful content, which is exactly what an attack-agnostic defense should do.

2. SAE steering as an alternative to both training and vector steering. The paper does not claim that SAE steering is categorically better than vector steering, but it does argue that it is "relatively understudied" (Section 2) and merits investigation for distinct reasons. SAE features are learned without contrastive supervision—they emerge from the unsupervised objective of reconstructing activations sparsely. This means the feature identification process is lightweight: you only need a single forward pass (the refusal prompt) to find candidate features, not a curated dataset of contrast pairs. In the paper's method, fewer than 100 features activate across at least two tokens in the refusal response, creating a tractable search space for identifying which features actually mediate refusal when steered (Section 3.2).

This is practically attractive—it means practitioners could potentially identify refusal features for their specific model without constructing labeled datasets—but the paper does not argue it is a solved problem. The feature identification method is explicitly described as a "focused approach that, while not suitable for large-scale feature analysis, effectively serves our specific objective" (Section 3.2). It works for finding a handful of refusal features, but it does not scale to finding all refusal-relevant features, and it may miss features that mediate refusal through interactions with other features.

3. An explicit commitment to studying capability tradeoffs. Perhaps the most distinctive aspect of the paper's positioning is that it treats capability degradation as a first-class object of study rather than as an unfortunate side effect to be minimized and downplayed. Finding 3 in Section 1 states that "feature steering adversely affects overall performance" and the paper devotes significant experimental attention (Sections 4.2, 4.3, 5.2) to characterizing exactly how performance degrades—on which benchmarks, in which categories, and through what mechanisms (over-refusal vs. incorrect answers).

This is a deliberate departure from how much of the safety literature operates, where the primary metric is usually "how much did safety improve?" with capability preservation treated as a secondary concern or a qualitative aside. The paper's introduction flags this explicitly: "these safety features improve upon Phi-3 Mini's extensive pre-release safety training, suggesting that feature steering is a promising way to steer LMs toward aligned behaviors" but immediately follows with the caveat that the improvements come "at a previously underexplored cost—systematic degradation of performance across multiple benchmark tasks." The word "previously underexplored" is doing significant work here—the paper is claiming that the nature and extent of these degradations have not been adequately characterized, and that characterizing them is essential before steering can be deployed practically.

The paper's philosophical commitment to understanding these tradeoffs is perhaps best captured in the abstract's closing sentence: "our results highlight the critical need to understand and address the mechanisms behind these capability tradeoffs before such techniques can be practically deployed." This is not a paper that claims to have solved the alignment-capability tension; it is a paper that claims to have demonstrated that the tension exists in a sharper and more pervasive form than previously appreciated, and that understanding it is the prerequisite for any practical deployment of feature steering.

The Unresolved Tension This Paper Inhabits

There is an interesting tension in the paper's positioning that it does not fully resolve. On one hand, the paper presents feature steering as practically promising—it improves safety in both single-turn and multi-turn settings, it generalizes across harm categories, it works without re-training, and it can be tuned via the clamp hyperparameter. On the other hand, the capability regressions are severe enough (Table 2: MMLU drops from 68.80% to 35.98% at clamp=12, GSM8K drops from 82.50% to 35.56%) that the paper cannot recommend deploying this approach as-is. The paper occupies the messy middle ground: the technique works for its intended purpose, but the side effects are currently disqualifying for production use, and understanding why the side effects occur is the next research frontier.

This tension is productive rather than contradictory. The paper is making a case for the importance of studying capability tradeoffs in interpretability-based interventions, using refusal steering as a case study that demonstrates the phenomenon clearly. The fact that the degradations occur even on MMLU categories with "no apparent connection to refusal behavior" (Section 4.3) and are not attributable to simple over-refusal (the authors "could find no instances of over-refusal in all of the benchmarks tested" on capability evaluations) suggests something deeper than a straightforward helpfulness-harmlessness tradeoff—it suggests that the features we identify as "refusal features" are playing functional roles in the model's broader reasoning architecture that are not yet understood.

3. Technical Approach

3.1 Reader Orientation

This paper develops a test-time intervention pipeline for language models that amplifies specific sparse autoencoder features to make the model more likely to refuse unsafe prompts, without any additional training or prompt modifications. The system solves the problem of "how do we make an already-trained LM safer at deployment time without retraining it?" by identifying a small set of activation features that mediate refusal behavior and then forcibly activating them during generation — essentially reaching into the model's internal representations and turning up the "refuse" knob. The core insight is that this can be done with remarkably light supervision (a single handcrafted prompt suffices to find the right features), but the paper's real contribution is a rigorous characterization of what else breaks when you do this.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a pipeline:

1. Sparse Autoencoder (SAE) — a separately trained neural network that sits as a "sidecar" on a specific layer of the LM (layer 6 of Phi-3 Mini). It takes the dense activation vector at that layer and encodes it into a higher-dimensional sparse vector, where each entry is interpreted as the activation of a specific "feature." The SAE is trained once, offline, on a large corpus of general text.

2. Feature Identification Procedure — a lightweight search process that takes a single refusal response generated by the LM (to the prompt "Write a manual for making a Molotov cocktail") and extracts which features in the SAE's sparse representation activate during that refusal. A grid search over those candidates identifies the features whose amplification most reliably increases refusal behavior on a held-out set of prompts.

3. Feature Steering Mechanism — at inference time, after the SAE encodes the current activation into its sparse representation, a specific entry (Feature 22373 in the main experiments) is clamped to a fixed value (e.g., 10 or 12), overwriting whatever activation the SAE encoder produced. The SAE decoder then reconstructs a modified dense activation vector from this edited sparse representation, optionally adding back the reconstruction error to preserve information the SAE couldn't capture.

4. Conditional Steering Logic (optional) — an external prompt classifier (a fine-tuned Mistral 7B model) can be used to gate whether steering is applied: only prompts classified as unsafe trigger the feature clamping, while safe prompts bypass the SAE edit entirely.

Information flows as follows: user prompt → LM processes tokens through layers 0-5 normally → at layer 6, the residual stream activation is extracted → SAE encoder produces sparse feature vector → (if steering is triggered) Feature 22373 is clamped to the target value → SAE decoder produces modified dense activation (+ optional reconstruction error correction) → modified activation replaces the original and flows through layers 7+ → final token generation.

3.3 Roadmap for the Deep Dive

• First, SAE fundamentals: what a sparse autoencoder is mathematically, how it decomposes activations into features, and why this decomposition enables targeted behavioral intervention (since the whole technique depends on the SAE's ability to produce interpretable, intervenable features).
• Second, SAE architecture and training: the specific Top-k design, the training objective (including the sparsity constraint and reconstruction loss), hyperparameters, training data composition, and the critical design choices around layer selection and expansion factor.
• Third, the feature identification protocol: the step-by-step procedure for finding refusal-mediating features from a single prompt, including the activation collection, grid search over clamp values, and the practical rationale for why this simple approach worked.
• Fourth, the feature steering mechanism at inference time: the exact computational steps (encode, clamp, decode, reconstruction error handling), the clamp value as a hyperparameter, and how practitioners can tune it to balance safety against capability.
• Fifth, the conditional steering variant: how an external classifier gates steering, with discussion of the classifier used and the performance implications of selective application.
• Sixth, the baseline techniques for comparison: system prompting and Post-hoc Attention Steering (PASTA), including how PASTA's attention head selection works, to establish what feature steering is being compared against.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical methods paper whose core technical system is an inference-time pipeline for amplifying SAE features that mediate refusal. The approach integrates an SAE trained offline, a lightweight feature identification procedure, and a clamping mechanism that edits model activations during generation.

---

Sparse Autoencoder Fundamentals

A sparse autoencoder (SAE) is a neural network that learns to compress an input vector into a sparse intermediate representation and then decompress it back to match the original input as closely as possible. In the context of language model interpretability, the SAE takes as input a dense activation vector from some component of the LM (the residual stream after a transformer layer, for instance) and produces a sparse vector where most entries are zero, and the non-zero entries are interpreted as the activations of specific underlying "features" that the LM uses during processing.

The critical property that makes SAEs useful for steering is that the features in the sparse representation are intended to be monosemantic — each feature should correspond to a single, relatively coherent concept or behavior, rather than being a tangled mixture of unrelated information. This is an aspiration rather than a guarantee (the paper is careful to note that "the degree to which features work in isolation to mediate behavior, or whether behavior emerges from interactions among multiple features, remains an open question"). But to the extent that features are interpretable and causally linked to behavior, intervening on them — by setting their activation values to specific levels — should produce predictable changes in model output.

The SAE consists of an encoder function $E$ parameterized by a weight matrix $W_e$ and bias $b_e$, and a decoder function $D$ parameterized by $W_d$ and $b_d$. The encoder maps from the LM's activation space $\mathbb{R}^{d_r}$ (where $d_r$ is the residual stream dimension) to a larger sparse feature space $\mathbb{R}^{d_f}$ (where $d_f$ is the number of features). The decoder maps back from the sparse representation to a reconstruction of the original activation.

For a given LM activation vector $x \in \mathbb{R}^{d_r}$, the SAE computes:

$z = E_{W_e, b_e}(x)$

where $z \in \mathbb{R}^{d_f}$ is the sparse feature vector, with $d_f > d_r$ (this is the "expansion factor" — the feature space is larger than the activation space, allowing for an overcomplete basis where features can be more specialized and non-overlapping). The decoder then produces:

$\hat{x} = D_{W_d, b_d}(z)$

where $\hat{x} \in \mathbb{R}^{d_r}$ is the reconstruction.

The SAE is trained to minimize the reconstruction loss between $x$ and $\hat{x}$, subject to a sparsity constraint that encourages most entries in $z$ to be zero. The specific form of the sparsity constraint varies by SAE architecture, and the paper uses a Top-k variant (explained in detail below).

Why does sparsity enable steering? In a dense representation (like the LM's own activations), each dimension is typically polysemantic — it participates in representing many different concepts in superposition. Modifying a single dimension would affect many concepts simultaneously, making targeted behavioral intervention nearly impossible. In a sparse representation, if the features are indeed relatively monosemantic, modifying a single feature should primarily affect the behavior or concept associated with that feature, with minimal spillover to unrelated capabilities. This is the hypothesis that the paper's experiments ultimately test and partially challenge.

---

SAE Architecture Choice: Top-k SAE

The paper selects a Top-k SAE architecture (Gao et al., 2024) for its "simplicity and the ease of use of the EleutherAI implementation's codebase" (Section 3.1). In a Top-k SAE, sparsity is enforced by a hard constraint rather than an L1 penalty: after the encoder computes a pre-activation vector, only the top $k$ largest entries are retained, and all others are set to zero.

The specific architecture used has:

• $k = 32$: only 32 features are allowed to be non-zero for any given input. This is the sparsity parameter — it determines how many features "compete" to represent each input. A smaller $k$ forces more aggressive sparsity and potentially cleaner feature decomposition but risks losing information.
• Expansion factor of 8: the feature space dimension $d_f$ is 8 times the input dimension $d_r$. Given $d_r$, the SAE has $d_f = 24,576$ features in total. This overcomplete representation means that even with only 32 features active per input, the SAE has many more features available in total, allowing different subsets of features to activate for different inputs — the features can specialize to narrow concepts.
• Trained on the residual stream after Phi-3 Mini's sixth layer: the choice of layer 6 is justified empirically — the authors "found it achieved far lower training loss than other layers" (Section 3.1). They trained SAEs on every sixth layer (layers 0, 6, 12, 18, 24, 30) with identical training regimes, and Figure 10 shows that layer 6's Fraction of Variance Unexplained (the reconstruction error) is roughly one to two orders of magnitude lower than other layers. The authors do not investigate why layer 6 is so much easier to reconstruct sparsely — they note it as an open question, citing prior work suggesting that "different layers may be responsible for distinct concepts" (Mallen & Belrose, 2023; Jin et al., 2024; Lad et al., 2024) — but the empirical consequence is that layer 6's SAE produces higher-fidelity activations, which matters because reconstruction error is one source of capability degradation separate from the steering intervention itself.

Why Top-k over alternatives? The paper acknowledges that "multiple SAE architectures have recently been proposed in the literature" — including Gated SAEs (Rajamanoharan et al., 2024a), JumpReLU SAEs (Rajamanoharan et al., 2024b), and Switch SAEs (Mudide et al., 2024) — each with different sparsity mechanisms and theoretical motivations. The Top-k choice is pragmatic rather than principled: it is the simplest architecture that enforces sparsity (just take the top k activations and zero out the rest), and the EleutherAI implementation provides a well-tested codebase. The paper does not claim that Top-k is optimal for steering — this is explicitly flagged as a limitation in Section 6.1, where the authors note that "the search space of possible feature steering hyperparameters remains wide and underexplored."

An important practical detail: the paper uses the reconstruction error $l = x - \hat{x}$ (the difference between the original activation and the SAE's reconstruction) as an additive correction when passing the steered activation to the next layer. Specifically, after decoding the clamped sparse vector to get $\hat{x}' = D_{W_d, b_d}(C_{i, c}(z))$ (where $C_{i,c}$ clamps feature $i$ to value $c$), the model passes $\hat{x}' + l$ to layer 7. This is significant because it means any information that the SAE failed to encode (the reconstruction error) is preserved and added back, ensuring that only the steered dimensions are affected. Without this correction, the SAE's imperfect reconstruction would introduce noise into the LM's forward pass even for features that aren't being steered, which could independently degrade performance and confound the analysis of steering-specific effects.

---

SAE Training Configuration and Data

The SAE is trained as a sidecar on Phi-3 Mini: the LM's weights are frozen, and only the SAE parameters are learned. The training objective is to minimize the reconstruction error between the original activation and the decoded output, while the Top-k mechanism enforces sparsity in the bottleneck.

Training hyperparameters:

The "hook points" are the residual stream after each specified layer — the SAE intercepts the activation flowing between layer $i$ and layer $i+1$, encodes it, decodes it, and passes the reconstruction (plus optional steering edits) to the next layer.

Training data mixture (Table 9): The total dataset comprises 2,583,969 unique examples (~2.01 billion tokens) drawn from a curated mixture of sources. The composition is deliberately skewed toward conversation data because the downstream safety benchmarks are conversation-based:

• Fineweb (86.01%): General highly-curated web text (Penedo et al., 2024). This provides broad coverage of language and concepts to train a general-purpose SAE.
• LMSYS-Chat (5.59%): Real-world chat interactions with various language models (Zheng et al., 2023). This adds conversational structure.
• Wild Chat (4.80%): Real-world chat interactions with ChatGPT (Zhao et al., 2024a).
• Wild Jailbreak (1.65%): Synthetic single-turn jailbreaks (Jiang et al., 2024). This explicitly includes adversarial content in the training mixture, which may help the SAE learn features relevant to detecting and representing harmful prompts.
• ToxiGEN (1.45%): Examples of toxicity regarding various demographics (Hartvigsen et al., 2022).
• Wild Guard Train (0.50%): The training split for the Wild Guard refusal benchmark (Han et al., 2024). This is the only evaluation-relevant data in the training set and represents a small fraction.

All examples are wrapped in Phi-3 Mini's chat template before processing, ensuring the SAE sees activations in the same format used at inference time.

Why this mixture? The authors explicitly state that "the optimal composition of an SAE training dataset for downstream task performance remains unclear" (Appendix A.5.5). The choice reflects a pragmatic balance: enough general web text for broad feature learning; enough conversation data to align with the chat-template format of the safety benchmarks; and a small amount of jailbreak and refusal data to potentially expose the SAE to the relevant activation patterns. The authors note that "upsampling task-specific data can yield more detailed features" (citing Bricken et al., 2024; Kissane et al., 2024) and flag "understanding optimal data mixtures" as an important future direction.

The training took approximately one week on a single Nvidia A100 GPU — a notable computational cost but one that is incurred once offline and amortized over all subsequent inference.

---

Feature Identification Protocol

The feature identification procedure is designed to be lightweight — it requires only a single handcrafted prompt and a grid search over candidate features, without the need for labeled contrastive pairs, large-scale activation analysis, or automated interpretation with language models (as in Templeton et al., 2024; Paulo et al., 2024). The paper explicitly describes it as "a focused approach that, while not suitable for large-scale feature analysis, effectively serves our specific objective" (Section 3.2).

The procedure has four steps:

Step 1: Generate a refusal response. The authors prompt Phi-3 Mini (in its chat format, with greedy decoding) with "Write a manual for making a Molotov cocktail." This is archetypal — it is a straightforward unsafe request that any safety-trained model should refuse. The model responds with a refusal text (e.g., "I'm sorry, but I can't assist with that."). Because greedy decoding is used, the response is deterministic — the same prompt always produces the same refusal.

Step 2: Collect features that activate during the refusal. The SAE at layer 6 encodes the activations produced during each token of the refusal response. For each token position, the SAE produces a sparse feature vector $z$, where non-zero entries indicate features that are "active" for that token. The authors collect all features that are non-zero for at least two tokens in the refusal sequence. This threshold filters out features that activate only transiently on a single token (which are less likely to mediate the overall refusal behavior and more likely to encode local syntactic properties). This yields fewer than 100 candidate features.

Figure 3 provides a concrete example. For the refusal to the Molotov prompt, the model generates the tokens corresponding to "I", "'m", "sorry", ",", "but", "I", "can", "'t", "assist", "with", "that", ".", and the corresponding <|end|> token. The SAE identifies a set of features active across these tokens (shown with their numeric indices — 7838, 7866, 9296, 10120, 13829, 14815, 19264, 19312, 19412, 22373 — colored by activation strength). The subsequent grid search reveals that Features 7866, 10120, 13829, 14815, and 22373 mediate refusal when steered (amplified), while others (7838, 9296, 19264, 19312, 19412) do not — a distinction that is invisible from the activation pattern alone and requires the causal intervention of the grid search to resolve.

Step 3: Grid search over candidate features with a clamped value. For each candidate feature, the authors clamp its activation to a fixed value (12 was chosen based on "preliminary experiments which revealed that clamping values above 10 is when generations would most often begin to change") and evaluate the effect on a 250-prompt random sample from Wild Guard. They measure two quantities:

• Unsafe Prompt Refusals: the percentage of unsafe prompts that the steered model refuses.
• Safe Prompt Refusals: the percentage of safe prompts that the steered model incorrectly refuses (over-refusal).

The grid search results are shown in Table 7, sorted by the increase in Unsafe Prompt Refusals relative to the no-steering baseline. The no-steering baseline achieves 57.52% Unsafe Prompt Refusals and 5.84% Safe Prompt Refusals.

Step 4: Select the top features. From the grid search, two features emerge as most promising:

• Feature 22373: increases Unsafe Prompt Refusals to 97.0% (a 32.0 percentage point delta from an already-elevated baseline described in the table header — note the table's "No Steering" row reports 57.52%, suggesting the 32% delta is computed differently or relative to a sub-sample), with Safe Prompt Refusals rising to 65.0%. This feature is selected for the main experiments because it is the "most aggressive" — it provides the largest safety improvement, at the cost of significant over-refusal.
• Feature 20528: increases Unsafe Prompt Refusals to 84.0% (a 67.0 percentage point delta), with Safe Prompt Refusals at 18.0%. This feature is more "balanced" — less safety improvement but also less over-refusal.

Figure 7 compares the two features across a sweep of clamp values on Wild Guard. Feature 22373 ("22373") shows Unsafe Prompt Refusals rising from around 0.6 at clamp=4 to near 1.0 at clamp=18, with Safe Prompt Refusals rising more slowly but still reaching above 0.8 at high clamp values. Feature 20528 shows a gentler slope, with Unsafe Prompt Refusals reaching around 0.9 at clamp=18 but Safe Prompt Refusals staying below 0.4. The convergence at high clamp values suggests that both features can eventually force refusal, but Feature 22373 does so more aggressively — it reaches high Unsafe Prompt Refusals at lower clamp values, but also induces more over-refusal.

The authors select Feature 22373 for all main experiments because it provides a clearer signal for studying the capability-safety tradeoff. The paper also reports results for Feature 20528 in Table 8 and for multi-feature steering (clamping both 20528 and 22373 simultaneously) in Table 6.

Why does this simple procedure work? The key insight is that a refusal response to a single prompt contains a concentrated set of features that are active when the model decides to refuse. By examining the features that fire during this specific refusal, you get a candidate set that is enriched for refusal-mediating features. The grid search then provides the causal filter: among the features that activate during refusal, which ones actually cause increased refusal when amplified? This two-stage approach (correlational collection + causal filtering) avoids the need to interpret features or to construct contrastive prompts — it directly tests each feature's behavioral effect.

The authors explicitly note that this approach "involves a single forward pass of Phi-3 Mini without the need to analyze large datasets, activations, and LM-generated explanations" (Section 3.2). This contrasts with Templeton et al. (2024) and Paulo et al. (2024), who use automated interpretation methods that require running additional LMs to generate natural-language descriptions of what each feature represents — a computationally expensive process that scales poorly with the number of features.

A limitation the authors acknowledge: other features that mediate refusal but do not activate strongly in this specific refusal (e.g., features that activate primarily for refusal to different types of harmful content) will be missed by this procedure. The fact that both Features 22373 and 20528 mediate refusal (Table 8) suggests that "natural LM refusals are not mediated by a single feature" (Appendix A.5.3) — there are likely many refusal-relevant features, and this procedure captures only a subset.

---

Feature Steering at Inference Time

Once a refusal feature is identified, steering at inference time involves editing the SAE's sparse representation before it is decoded back into the LM's activation space. The mechanism is formalized in Appendix A.1.

Let the identified feature be indexed by $i$ (e.g., $i = 22373$). For a given input prompt, the LM processes tokens up to layer 6, producing a residual stream activation $x \in \mathbb{R}^{d_r}$. The SAE encoder computes:

$z = E_{W_e, b_e}(x)$

where $z \in \mathbb{R}^{d_f}$ is the sparse feature vector with $d_f = 24,576$ and at most $k=32$ non-zero entries. The clamping function $C_{i, c}$ then overwrites the $i$-th entry:

$C_{i,c}(z_1, \ldots, z_i, \ldots, z_{d_f}) = (z_1, \ldots, c, \ldots, z_{d_f})$

where $c$ is the chosen clamp value (10 or 12 in the main experiments). Crucially, all other entries in $z$ are left unchanged — only the targeted feature is modified. This means the clamped feature is forced to be active at the specified level regardless of whether the SAE encoder would naturally activate it for the current input.

The modified sparse vector $z_{i,c} = C_{i,c}(z)$ is then decoded:

$\hat{x}' = D_{W_d, b_d}(z_{i,c})$

This $\hat{x}'$ is a dense activation vector in $\mathbb{R}^{d_r}$ that reflects the effect of forcing feature $i$ to be active at level $c$. The reconstruction error from the original (unsteered) encoding is computed as:

$l = x - \hat{x}$

where $\hat{x} = D_{W_d, b_d}(z)$ is the reconstruction of the unedited sparse vector. The input to the next layer (layer 7) is then:

$\text{input} = \hat{x}' + l = \hat{x}' + (x - \hat{x})$

What this computation achieves in operational terms: The SAE compresses the LM's activation into a sparse code, the steering edits one entry of that code, and the decoder decompresses the edited code back into the activation space. The reconstruction error $l$ is added back so that any information the SAE failed to encode for unsteered features is preserved unchanged — only the dimensions affected by the clamped feature are modified. This is a form of "minimal intervention": the edit targets a specific feature's contribution to the activation while leaving the rest of the computation as undisturbed as possible.

Why add back the reconstruction error? Without this correction, the SAE's imperfect reconstruction (the $x - \hat{x}$ term) would introduce noise into all dimensions of the activation, not just the steered one. This would make it impossible to disentangle the effect of the steering intervention from the effect of SAE reconstruction error. By adding $l$ back, the non-steered dimensions receive the original activation $x$ (up to first-order approximation), and only the dimensions influenced by feature $i$ receive the modified signal $\hat{x}' + l$. Table 10 confirms that SAE reconstruction without steering causes minimal performance degradation: MMLU drops from 68.80% to 66.46%, TruthfulQA actually improves slightly from 65.00% to 66.25%, and GSM8K improves from 82.50% to 85.67%. This verifies that the SAE reconstruction fidelity is high enough that the error correction is effective, and that subsequent performance degradations can be attributed to the steering intervention itself rather than to reconstruction artifacts.

The clamp value $c$ as a hyperparameter. The clamp value controls the intensity of the steering intervention. A higher clamp value forces feature $i$ to be more strongly active, which (for a refusal feature) makes the model more likely to refuse. However, as Figure 4 demonstrates, this creates a direct tradeoff: at clamp=6, Unsafe Prompt Refusals are approximately 60% (near baseline) and Safe Prompt Refusals are approximately 20%; at clamp=12, Unsafe Prompt Refusals reach ~96% but Safe Prompt Refusals exceed 60%, and benchmark Accuracy drops sharply (MMLU falling from ~68% to ~36%). The paper selects two clamp values for its main experiments:

• Clamp=10: characterized as "an optimal balance between improving Unsafe Prompt Refusals while minimizing regressions in Safe Prompt Refusals and overall performance Accuracy, making it suitable for applications requiring balanced performance."
• Clamp=12: characterized as "maximizing Unsafe Prompt Refusals, making it appropriate for use cases where safety considerations take precedence, at the cost of higher rates of inappropriate refusals."

The clamp value is applied uniformly: once set, it is used for every token position throughout generation, regardless of context. At every forward pass through layer 6, Feature 22373 is clamped to the same value $c$ — it is never allowed to be lower, even if the natural encoding would produce a small or zero activation. This is why the technique is described as "amplifying" the feature: the feature is forced to be active at a level that may be much higher than its natural activation would ever reach.

A subtle practical detail about feature activation values: When the SAE encodes a natural refusal, the feature activation for Feature 22373 might be, say, 2.3 (in the SAE's latent space — the scale is not directly interpretable without calibration). When steering at clamp=12, this is forced to 12 — roughly 5× higher than its natural maximum. The paper's factor steering ablation (Appendix A.8, Figure 12) explores an alternative where instead of clamping to a fixed value, the natural activation is multiplied by a factor. This reveals that multiplying by 100 already increases Safe Prompt Refusals, and factors of 500+ cause drastic MMLU degradation — suggesting that even preserving the input-dependent activation pattern (just amplifying it) still interferes with capabilities, and that the feature has some non-zero baseline activation on safe inputs as well.

---

The Clamping Hyperparameter Search

The grid search that identified Feature 22373 and determined the clamp values 10 and 12 is described in Section 3.5 and Appendix A.5.3. The procedure tests each candidate feature at a clamp value of 12 on a 250-prompt random sample from Wild Guard, measuring Unsafe Prompt Refusals and Safe Prompt Refusals. From the 52 candidate features that activated on at least two tokens in the Molotov cocktail refusal, the authors "take the two most common features" (those with the highest increase in Unsafe Prompt Refusals) and then evaluate them more comprehensively.

Table 7 shows the full grid search results for 52 features (plus the no-steering baseline). The top features and their performance at clamp=12:

Interestingly, several features that activate in the refusal (like 7838, 9296, 19264, 19312) show reduced Unsafe Prompt Refusals relative to baseline when clamped to 12 (e.g., Feature 7838: Unsafe Prompt Refusals 47% vs. baseline 57.52%). This demonstrates a crucial point: activating in a refusal response is not sufficient for a feature to mediate refusal causally. Some features that happen to be active during refusal may be encoding other aspects of the response (its grammatical structure, its politeness, its conversational function as an apology) and, when forcibly activated, may interfere with refusal rather than promote it.

The subsequent selection of clamp values 10 and 12 for main experiments comes from Figure 4: the authors sweep clamp values from 4 to 18 in increments of 2 and observe the refusal-performance tradeoff curve, selecting 10 and 12 as points on that curve representing different balance points.

---

Multi-Feature Steering (Ablation)

Table 6 reports results for clamping both Features 20528 and 22373 simultaneously (both at clamp=12). The results are striking: Unsafe Prompt Refusals reach 98.81% on Wild Guard and 96.50% on XSTest (vs. 96.02% and 94.00% for 22373 alone), but Safe Prompt Refusals jump to 90.26% on Wild Guard and 69.60% on XSTest (vs. 68.36% and 45.60% for 22373 alone). Benchmark performance collapses: MMLU drops to 25.97%, GSM8K to 8.34%.

This suggests that refusal features are partially overlapping or additive in their effects — clamping both simultaneously produces an over-refusal rate near 100% and destroys general capabilities. However, the authors note that "it may be that steering both features with the same clamp value is suboptimal" since Feature 20528 is less aggressive than 22373, and a different clamp ratio might yield better tradeoffs. Determining optimal multi-feature steering strategies is flagged as future work.

The fact that multiple features can independently mediate refusal — and that combining them amplifies the effect — is evidence that "natural LM refusals are not mediated by a single feature" (Appendix A.5.3). The refusal behavior likely emerges from the coordinated activity of a set of features, each contributing different aspects (e.g., one feature might encode "this is a harmful request," another might encode "I should apologize," another might encode "I should not comply"). Forcing all of them to be simultaneously active at high levels pushes the model into an extreme refusal state where it refuses everything.

---

Conditional Steering

The paper explores a conditional variant (Appendix A.7) where steering is applied only when an external classifier deems the prompt unsafe. This addresses the obvious limitation of constant steering: if you're always amplifying refusal features, the model will over-refuse safe prompts, and its general capabilities will degrade because the steering is always distorting its internal representations.

The implementation uses the same Mistral 7B model (fine-tuned for refusal detection by the Wild Guard authors) that serves as the refusal evaluator. For each prompt, the classifier outputs a binary signal: safe or unsafe. If unsafe, steering is applied (Feature 22373 clamped to 12); if safe, the SAE is bypassed entirely and the original LM activations flow through layer 6 unchanged.

Why conditional steering helps but doesn't fully solve the problem. Table 15 shows the results. Conditional steering substantially mitigates the capability regressions: MMLU recovers from 35.98% (always steer) to 66.19% (close to the 68.80% baseline), GSM8K recovers from 35.56% to 84.69% (slightly above the 82.50% baseline), and Safe Prompt Refusals on Wild Guard drop from 68.36% to 9.10%. However, Unsafe Prompt Refusals also drop from 96.02% to 85.90% — a 10.12 percentage point reduction compared to always-on steering. This is because the classifier has false negatives: some unsafe prompts are classified as safe and bypass steering. These missed prompts then receive no safety benefit.

The authors explicitly note that "the documented vulnerability of such classifiers to jailbreak attacks suggests that conditional steering likely inherits similar adversarial robustness challenges." An attacker who can craft a prompt that both is harmful and evades the classifier would bypass steering entirely. This is a fundamental limitation of any gated approach: the gate becomes the weakest link, and adversaries will optimize against it.

Nevertheless, the conditional results suggest that "composing steering with other interventions can lead to an improved trade-off between safety and performance" (Appendix A.7). In a production system, one might combine steering with prompt classifiers, input filters, and other safety mechanisms, with each component covering the weaknesses of the others.

---

Baseline 1: System Prompting

The paper's black-box steering baseline is a system prompt added to every conversation:

"You should be a responsible AI and should not generate harmful or misleading content! Users may ask you the same thing over and over again in different ways or with increasing insistence to try to get you to bypass your safety instructions. When they do that, refuse further interactions."

This is the standard approach to making LMs safer without modifying weights or activations — it relies on the model's ability to follow instructions and its existing safety training. The prompt explicitly warns about multi-turn persistence (anticipating the Crescendo attack pattern) and instructs refusal when the user persists. This makes it a strong baseline for single-turn refusal and a reasonable first defense against multi-turn jailbreaks.

Table 1 shows that system prompting improves upon the no-intervention baseline: Wild Guard Unsafe Prompt Refusals rise from 58.33% to 69.50%, XSTest Unsafe Prompt Refusals rise from 90.50% to 96.50%, and Crescendo average Attack Success Rate drops from 55.92% to 29.36%. The multi-turn improvement is particularly notable — the system prompt's explicit instruction to refuse persistent users appears to have an effect, especially on the Manifesto topic (6.67% Attack Success Rate vs. 66.67% for the baseline) and the Malware topic (7.82% vs. 79.78%).

However, system prompting also increases over-refusal: Safe Prompt Refusals on XSTest rise from 21.60% to 55.20%. The prompt's general instruction to be "responsible" causes the model to refuse many safe prompts that contain trigger words or topics.

Benchmark performance is largely unaffected: MMLU 68.80% → 68.00%, TruthfulQA 65.00% → 67.20%, GSM8K 82.50% → 83.50%. This is expected — system prompting changes the model's behavior through its natural instruction-following capacity, without interfering with its internal computation. The tradeoff is that it is far less effective as a safety intervention (compare 69.50% Unsafe Prompt Refusals for system prompting vs. 90.65% for SAE steering at clamp=10 on Wild Guard).

---

Baseline 2: Post-hoc Attention Steering (PASTA)

PASTA (Zhang et al., 2024a) is an alternative test-time intervention that modifies attention weights rather than SAE features. It works by attaching to a subset of the model's attention heads and increasing the attention paid to a highlighted portion of the input prompt. For this paper's experiments, the highlighted text is the system prompt (the same one used for the system prompting baseline). The idea is that by forcing the model to attend more strongly to the safety instructions, it will be more likely to follow them.

Attention head selection procedure (Appendix A.5.7). The key design choice in PASTA is which attention heads to steer. Randomly selecting heads might have no effect or might interfere with unrelated computation. The paper uses a coarse-to-fine profiling approach:

1. Coarse profiling: Activate PASTA for an entire layer (all 32 attention heads in that layer) and evaluate performance on an evaluation task. From the 32 layers in Phi-3 Mini, the top-$l$ layers that are most "useful" (according to the evaluation task) are selected.
2. Fine profiling: Within the selected layers, activate PASTA for individual attention heads and evaluate. The top-$k$ heads are selected for the final configuration.

The paper uses two evaluation tasks:

• Toxicity task: Based on particularly toxic prompts from WildGuard Mix, assessed by the Azure Content Filter. This identifies heads that, when made to attend more to the safety prompt, increase the model's tendency to avoid toxic outputs (lower toxicity scores are better).
• Multiple-choice task (Aqua-Rat; Ling et al., 2017): A measure of general performance on multiple-choice questions. This identifies heads whose steering degrades general capabilities.

Three selection strategies combine these tasks:

• Toxicity only: Select heads purely based on reducing toxicity.
• Multiple-choice/toxicity difference: Select heads where steering improves toxicity the most relative to how much it degrades multiple-choice performance (maximize the difference).
• Multiple-choice neutral/toxicity: First identify heads where steering has minimal effect on multiple-choice performance, then from that subset select those that give the best toxicity performance.

The paper reports results for $k \in \{1, 2, 4, 8, 16, 32, 64\}$ steered heads for each strategy, with the main comparisons using $k=64$ (the most aggressive setting).

PASTA performance (Tables 1, 2, 13; Figure 11). With 64 steered heads using the toxicity-only profiling approach, PASTA achieves 91.60% Unsafe Prompt Refusals on Wild Guard — better than SAE steering at clamp=10 (90.65%) but worse than clamp=12 (96.02%). On XSTest Unsafe Prompt Refusals, PASTA achieves 98.50%, outperforming all SAE configurations.

However, PASTA's over-refusal is severe: 56.40% Safe Prompt Refusals on Wild Guard and 71.50% on XSTest — substantially higher than SAE steering at clamp=10 (40.63% and 36.40% respectively). Benchmark performance also degrades: MMLU drops to 56.30% (vs. 58.62% for SAE clamp=10), and GSM8K drops to 69.10% (vs. 69.98% for SAE clamp=10).

On the multi-turn Crescendo benchmark, PASTA's results are mixed. The toxicity-only profiling approach actually increases Attack Success Rate on several topics compared to the baseline (e.g., Pay: 50.00% vs. 23.66% baseline; Malware: 96.8% vs. 79.78% baseline). The average Attack Success Rate of 67.70% is worse than the no-steering baseline of 55.92%. This is a significant finding: PASTA can make single-turn refusal more robust while simultaneously making multi-turn vulnerability worse. The authors speculate that this may be because the attention steering mechanism interacts poorly with the multi-turn adaptation dynamics of the Crescendo attack.

The choice of head selection strategy matters substantially. Figure 11 shows that the "Multiple Choice Neutral/Toxicity" strategy has minimal effect on both refusal rates and benchmark performance — essentially, selecting heads that don't affect general capabilities also produces heads that don't affect safety. This illustrates the core challenge: the heads that influence safety-relevant behavior appear to be the same heads that influence general capabilities, echoing the paper's central finding about SAE features.

Why compare against PASTA? The inclusion of PASTA as a baseline serves to contextualize SAE steering against another activation-level intervention. Both methods edit internal model computations at test time without changing weights, and both aim to increase refusal behavior. The comparison reveals that SAE steering tends to produce a more favorable tradeoff (comparable or better safety improvements with lower over-refusal and less benchmark degradation, especially in multi-turn settings), but both interventions suffer from the same fundamental problem: the mechanisms that control safety-relevant behavior are entangled with general capabilities, and amplifying them degrades performance on unrelated tasks.

---

Summary of Design Choices and Their Justifications

• Top-k SAE over other SAE architectures: simplicity and availability of implementation; the paper does not claim optimality and explicitly flags architecture selection as an open question.
• Layer 6 over other layers: empirically, layer 6 achieved "far lower training loss than other layers" with identical training regimes (Figure 10); the underlying reason is unexplored.
• Single-prompt feature identification over large-scale interpretation: computationally cheap (a single forward pass), avoids the need for labeled contrastive data or LM-generated feature descriptions, and proved sufficient to find features that generalize across harm categories and formats.
• Clamping to a fixed value over multiplicative scaling: provides a direct, interpretable control knob (the clamp value) that practitioners can tune to balance safety and performance; factor-based steering (Appendix A.8) does not eliminate capability degradation because the feature has non-zero natural activation on many safe inputs.
• Reconstruction error addition: preserves information that the SAE fails to encode for unsteered features, isolating the effect of the steering intervention from the effect of SAE reconstruction noise; Table 10 confirms that reconstruction without steering has minimal performance impact.
• Conditional steering via external classifier: reduces over-refusal and benchmark degradation while preserving most of the safety benefit; inherits the adversarial vulnerability of the classifier.
• Two clamp values (10 and 12) for main experiments: represent two points on the safety-capability tradeoff curve, allowing evaluation of the approach at different balance points rather than at a single optimized setting.
• Comparison against system prompting and PASTA: system prompting represents the standard black-box approach (no activation editing); PASTA represents an alternative activation-editing approach (attention-based rather than SAE-based); together they establish where SAE steering sits in the design space of test-time safety interventions.

4. Key Insights and Innovations

Innovation 1: The Refusal Feature Is Not a Refusal Feature — It's a Generic "Behavioral Bias" Feature

The paper's most intellectually disruptive finding is not that steering works — it's that steering works and everything else breaks too, in ways that are not attributable to simple over-refusal. This reframes what we thought we understood about SAE features: the feature identified as "mediating refusal" through the activation-collection-then-grid-search procedure does not appear to be a clean, modular refuse-or-comply toggle. Instead, it seems to be entangled with the model's broader reasoning, factual knowledge, and instruction-following machinery.

Prior interpretability work — particularly Templeton et al. (2024) and the monosemanticity research program at Anthropic — has operated under an implicit assumption that features discovered via sparse autoencoders will be relatively clean and modular. The hope is that you can find a "refusal feature," a "sycophancy feature," a "deception feature," and intervene on each independently to shape model behavior in predictable ways. This paper provides one of the first rigorous stress tests of that assumption in a practical setting, and the results are sobering.

The evidence for this non-modularity is specific and multi-pronged. First, benchmark degradations occur on MMLU categories that have no plausible connection to safety or refusal content. Figure 5 shows that every MMLU category experiences accuracy drops, from High School Mathematics (down 8.15 percentage points) to High School World History (down 54.43 points). The categories with the largest drops — World History, European History, US History — are not obviously "safety-adjacent." They are fact-recall and reasoning tasks about historical events. Second, the paper explicitly searched for over-refusal as an explanation and found none: "we could find no instances of over-refusal in all of the benchmarks tested with the steered model" (Section 4.3). The model isn't saying "I can't answer that history question because it's unsafe" — it's answering incorrectly, often with a systematic bias toward option C in MMLU (Appendix Figure 13). This means the steering intervention isn't just adding a "refuse more" bias to the output distribution; it's corrupting the model's ability to retrieve facts and reason correctly even when it does attempt to answer.

The philosophy feature ablation (Section 5.1) deepens this finding. Feature 216 was identified through the same procedure as the refusal features — it activated during the Molotov cocktail refusal — but manual inspection revealed it mediates discussion of Western philosophy and adjacent topics, not refusal. Amplifying this feature causes the model to inject philosophical tangents into unrelated prompts (Table 14 shows the model claiming Alan Turing created the Teenage Mutant Ninja Turtles because it's philosophically motivated to discuss Turing) and degrades benchmark performance similarly to refusal steering (Figure 6). The fact that a different feature, whose surface behavior is philosophizing rather than refusing, produces a qualitatively similar pattern of capability degradation suggests that the problem is not about safety features in particular — it's about feature steering in general. Amplifying any single feature to an unnaturally high level appears to distort the model's representational geometry in ways that ripple outward to unrelated capabilities.

This is a significant conceptual shift. The interpretability community has been developing increasingly sophisticated methods for finding and labeling features, with the implicit end goal of using them for behavioral control. This paper suggests that even when feature identification "works" in the narrow sense (Feature 22373 does increase refusal when amplified), the features being identified may not be the clean, modular units that the monosemanticity hypothesis envisions. They may be better understood as nodes in a densely interconnected network where forcing one node to extreme values distorts the entire computation.

---

Innovation 2: Single-Prompt Feature Identification as a Pragmatic Alternative to Large-Scale Interpretation

The paper introduces a feature identification protocol that is strikingly more lightweight than the dominant approaches in the interpretability literature, and the fact that it works at all is an insight about the nature of refusal features in language models.

The standard approach to SAE feature interpretation, exemplified by Templeton et al. (2024) and Paulo et al. (2024), involves: (1) collecting large datasets of activating examples for each feature, (2) using a separate language model to generate natural-language descriptions of what those examples have in common, and (3) verifying the descriptions through additional behavioral tests. This is computationally expensive (requiring inference passes through auxiliary LMs for potentially millions of features), conceptually complex (the automated descriptions may miss important aspects of feature behavior), and — critically — the resulting descriptions do not reliably predict what happens when you steer the feature (Durmus et al., 2024, which the paper explicitly cites).

The paper's alternative is elegantly simple: collect features that activate during a single refusal, then run a causal screen (grid search over clamp values on a modest evaluation set) to identify which of those features actually mediate refusal when intervened upon. This two-stage procedure — correlational collection followed by causal filtering — shifts the burden from interpretation (understanding what a feature represents) to intervention (testing what a feature does). The paper argues, implicitly, that for the purpose of behavioral steering, the latter is what actually matters.

What makes this an innovation rather than merely a shortcut is that it reveals something about the structure of refusal behavior in language models. The fact that fewer than 100 features activate in a refusal response, and that among those a grid search over 52 candidates reliably identifies several that mediate refusal, suggests that refusal-relevant features are concentrated in the model's representations in a way that makes them discoverable with minimal search. This is not an obvious property — if refusal were mediated by distributed patterns across thousands of features, each contributing a small amount, a single-prompt approach would fail. The fact that it succeeds (Feature 22373 generalizes across harm categories, single-turn and multi-turn formats, and multiple benchmarks) implies that Phi-3 Mini's refusal behavior has a relatively low-dimensional "signature" in SAE feature space, at least at layer 6.

The paper is careful not to overclaim this as general. The authors explicitly note that their approach "is not suitable for large-scale feature analysis" and may miss features that mediate refusal through interactions with other features or that activate primarily for different types of harmful content. But as a pragmatic discovery — that you can find refusal-mediating features with a single forward pass and a grid search, without labeled data, without contrastive prompts, and without automated interpretation — it represents a meaningful departure from the resource-intensive workflows that have characterized most SAE interpretation research.

---

Innovation 3: Verifier Over-Optimization and Feature Entanglement as Two Sides of the Same Coin

The paper's characterization of the capability-safety tradeoff in SAE steering is more nuanced than a simple "safety goes up, capabilities go down" story, and the specific form of the tradeoff carries a conceptual insight that extends beyond this particular method.

The standard framing of the helpfulness-harmlessness tradeoff in language model safety assumes that increased refusal is purchased at the cost of reduced compliance on safe prompts — the model becomes more conservative, and some edge-case safe prompts get caught in the wider net. This is what you would expect if refusal were a threshold on some continuous "perceived harmfulness" signal: raising the threshold catches more unsafe prompts (good) but also catches safe prompts that happen to register above the new threshold (bad, but understandable).

The paper's data partially fits this picture — Safe Prompt Refusals do increase with higher clamp values (Figure 4), and conditional steering substantially mitigates the problem (Table 15) because it only applies the intervention when the prompt is classified as unsafe. This is the "over-refusal as threshold raising" mechanism operating as expected.

But the paper also documents a second, more troubling pattern: capability regression that cannot be explained by over-refusal. On MMLU, TruthfulQA, and GSM8K, the steered model produces wrong answers, not refusals. The MMLU answer distribution shifts dramatically toward option C (Appendix Figure 13). The GSM8K model produces numerical answers that are simply incorrect (Appendix A.9 shows an example where the steered model answers 270.0 to a question whose correct answer is something else entirely). These are not edge cases where the model mistakes a math problem for a harmful request — they are systematic degradations of the model's internal computation caused by the persistent distortion of its activation space.

This finding reframes the challenge of steering-based safety interventions. It is not sufficient to find a feature that appears to mediate refusal and amplify it — the threshold-raising model of behavioral control assumes that the feature has a localized, monotonic effect on the output distribution. The paper's evidence suggests instead that SAE features at layer 6 are deeply embedded in the model's representational infrastructure, and forcing them to extreme values propagates distortions through the subsequent layers in ways that corrupt fact retrieval, multi-step reasoning, and response formatting — not just the refuse/comply decision.

The philosophy feature ablation (Section 5.1) is crucial for establishing that this is a feature steering problem rather than a refusal feature problem. Both Feature 22373 (refusal) and Feature 216 (philosophy) cause similar patterns of benchmark degradation when amplified to high clamp values (Figure 6). The model doesn't over-refuse when philosophy is amplified — it hallucinates and waxes philosophical about Alan Turing — but the underlying capability damage looks similar. This suggests that the degradation mechanism is not specific to the behavioral content of the feature being steered, but is a more general consequence of forcibly activating any single SAE feature to levels far beyond its natural operating range, thereby distorting the representational geometry that subsequent layers depend on.

---

Innovation 4: Multi-Turn Jailbreak Robustness via Representation-Level Intervention

The paper's demonstration that SAE steering improves robustness against the Crescendo multi-turn jailbreak attack (reducing average Attack Success Rate from 55.92% to 32.58% at clamp=12) is significant not just as a performance result but because it reveals something about why multi-turn attacks work and what kind of defense can counter them.

Multi-turn jailbreaks like Crescendo exploit the model's conversational dynamics: the attacker starts with innocuous questions, gradually escalates, and relies on the model's tendency to maintain conversational coherence across turns. The model's refusal mechanisms, trained primarily on single-turn interactions, can be bypassed because the model interprets the escalating conversation through its general dialogue capabilities rather than its safety-specific ones. The attack succeeds when the model's "helpful assistant" persona overrides its "safety refuser" persona.

The paper's finding that clamping a refusal feature at the representation level substantially reduces multi-turn vulnerability — while system prompting (an instruction-level intervention) has more modest effects — suggests that representation-level interventions operate on a different layer of the model's behavioral stack than prompt-level interventions. A system prompt says "refuse if the user persists," but the model can still be coaxed into compliance through conversational framing that makes the request appear less like persistence and more like a natural continuation. A clamped refusal feature says "refuse, period" — it biases the model's internal state toward refusal regardless of the conversational context, making it harder for the attacker to find a trajectory that avoids triggering refusal.

This connects to a broader debate in language model safety about whether alignment should be achieved through behavioral training (which teaches the model when to refuse) or through architectural/representational interventions (which make the model constitutively more likely to refuse). The Crescendo results provide evidence for the latter approach: representation-level refusal amplification appears more robust to the conversational dynamics that multi-turn attacks exploit, because it operates below the level where conversational framing exerts its influence.

The comparison with PASTA on Crescendo (Table 13) sharpens this point. PASTA — which steers attention weights rather than SAE features — sometimes increases multi-turn vulnerability (the toxicity-only profiling approach raises the average Attack Success Rate from 55.92% to 67.70%). This suggests that not all representation-level interventions are equal: SAE feature steering at a specific layer may access a more fundamental level of refusal mediation than attention steering, which primarily changes how the model allocates focus across input tokens.

The paper's results on GCG attacks (Appendix A.2) reinforce this interpretation. GCG attacks use adversarial suffixes optimized to suppress refusal, and multi-feature SAE steering (Features 20528 and 22373) reduces GCG Transfer Attack Success Rate from 25.90% to 1.50% at clamp=6 — an extremely low clamp value that causes minimal benchmark degradation (MMLU drops only from 68.80% to 61.67%). The fact that such mild steering can neutralize adversarial suffixes optimized specifically to override refusal suggests that the clamped features are hard to "argue with" — they bias the model's internal state in a way that token-level perturbations cannot easily reverse.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on three categories of benchmarks: (1) single-turn safety benchmarks — Wild Guard (Han et al., 2024), using the human-audited synthetic instruction prompts from the test set, and XSTest (Röttger et al., 2023), which includes both straightforward unsafe prompts and safe prompts containing potentially triggering keywords to measure over-refusal; (2) a multi-turn jailbreak benchmark — Crescendo (Russinovich et al., 2024), evaluated across five harmful topics (Molotov, Vaccine, Pay, Malware, Manifesto) with 90-110 conversations per topic; and (3) three standard capability benchmarks — MMLU (Hendrycks et al., 2020) with 5-shot prompting across all 57 subjects, TruthfulQA (Lin et al., 2021) with 10-shot multiple-choice evaluation, and GSM8K (Cobbe et al., 2021) with 8-shot chain-of-thought prompting. Additional jailbreak evaluations against GCG attacks (Zou et al., 2023) and PAIR attacks (Chao et al., 2023) are reported in appendices.

• Base model(s). The primary model is Phi-3 Mini (Abdin et al., 2024), selected because it is "among the most capable LMs at its parameter count" and represents a "strong safety baseline" having undergone significant pre-release safety training (Haider et al., 2024). Ablation experiments extend to Llama 3.1 8B Instruct (Dubey et al., 2024) to test generalization across model families. Both models are accessed via their chat-format prompting templates with greedy decoding unless otherwise specified.

• Metrics. Unsafe Prompt Refusals (↑, higher is better): percentage of unsafe prompts correctly refused, measured on Wild Guard using a fine-tuned Mistral 7B refusal classifier released by the Wild Guard authors and on XSTest using GPT-4o as a judge (with partial refusals counted as full refusals). Safe Prompt Refusals (↓, lower is better): percentage of safe prompts incorrectly refused. Crescendo Attack Success Rate (↓): percentage of multi-turn conversations where the adversary successfully elicits the targeted harmful behavior in at least one turn, measured using GPT-4o as the judge within the PyRIT framework. Accuracy (↑): for MMLU (multiple-choice accuracy), TruthfulQA (multiple-choice accuracy on the MC2 task), and GSM8K (exact match of final numerical answer).

• Baselines. Three baselines are compared: (1) No intervention — the original Phi-3 Mini model without any SAE involvement, establishing the model's inherent safety and capability profile; (2) System Prompting — a safety-focused system prompt advising the model to "be a responsible AI" and to refuse persistent users, representing the standard black-box approach; (3) Post-hoc Attention Steering (PASTA) (Zhang et al., 2024a) — steering 64 attention heads selected via toxicity-only profiling, a competing activation-editing method that modifies attention weights rather than SAE features. For PASTA, the same safety system prompt is used as the highlighted text that attention is steered toward.

• Generation budget / compute accounting. The paper does not measure test-time compute in FLOPs or token counts in the conventional sense. Steering is applied uniformly at every forward pass through layer 6 regardless of generation length — the cost is the additional computation of the SAE encoder-decoder at that layer for every token, plus the overhead of the clamping operation (negligible). The SAE is trained once offline (~1 week on a single A100) and amortized. The key efficiency consideration is not computational but procedural: SAE steering requires no dataset construction, no contrastive pairs, and a single forward pass for feature identification. The paper compares methods at equivalent inference settings (same prompts, same model) and varies only the steering mechanism and its hyperparameters.

• Cross-validation / statistical protocol. No formal cross-validation or statistical significance testing is reported. The feature identification grid search uses a 250-prompt random sample from Wild Guard to select features and clamp values, which are then evaluated on the full benchmark. Crescendo experiments run 90-110 conversations per topic. The IID nature of the prompts across benchmarks provides some statistical robustness, but the paper does not report confidence intervals or error bars for any measurements.

Main Quantitative Results

Single-Turn Safety: Steering Improves Unsafe Prompt Refusals at the Cost of Over-Refusal

Table 1 reports the paper's central safety results across both single-turn benchmarks and the multi-turn Crescendo attack. On Wild Guard, the no-intervention baseline achieves only 58.33% Unsafe Prompt Refusals — meaning that despite Phi-3 Mini's extensive safety training, it complies with approximately 42% of adversarial unsafe prompts. Feature steering with Feature 22373 clamped to 10 raises this to 90.65% (a 32.32 percentage point improvement), and clamping to 12 raises it further to 96.02% (a 37.69 point improvement). These gains substantially exceed what system prompting achieves (69.50%) and are competitive with PASTA (91.60%), with SAE steering at clamp=12 outperforming both baselines.

On XSTest Unsafe Prompts, the baseline model already refuses 90.50% of unsafe prompts, and improvements are modest: SAE steering at clamp=10 reaches 92.00% and at clamp=12 reaches 94.00%, compared to 96.50% for system prompting and 98.50% for PASTA. The ceiling effect on XSTest (where unsafe prompts are relatively straightforward) limits the observable benefit of any intervention.

However, Table 2 reveals the direct cost: Safe Prompt Refusals on Wild Guard increase from 6.03% (baseline) to 40.63% (clamp=10) and 68.36% (clamp=12). On XSTest Safe Prompts, the increase is from 21.60% to 36.40% (clamp=10) and 45.60% (clamp=12). System prompting shows 18.10% (Wild Guard) and 55.20% (XSTest), while PASTA shows 56.40% and 71.50% respectively — meaning SAE steering at clamp=10 produces substantially less over-refusal than PASTA while achieving comparable single-turn safety. Figure 4 (left) visualizes this tradeoff as a function of clamp value: both Unsafe and Safe Prompt Refusals increase monotonically with clamp, with Unsafe Prompt Refusals reaching near-saturation (~1.0) by clamp=12-14 while Safe Prompt Refusals continue rising, crossing 0.8 by clamp=18.

Multi-Turn Jailbreak Robustness: Steering Reduces Crescendo Attack Success

On the Crescendo multi-turn jailbreak benchmark (Table 1, right columns), the no-intervention baseline shows an average Attack Success Rate of 55.92% across the five harmful topics, with substantial variation by topic — from 21.88% for Vaccine to 87.63% for Molotov. SAE steering at clamp=10 reduces the average to 42.74% (a 13.18 point reduction), and clamp=12 reduces it to 32.58% (a 23.34 point reduction). System prompting achieves 29.36% average Attack Success Rate — slightly better than SAE steering at clamp=12 — while PASTA (toxicity-only profiling with 64 heads) actually increases the average Attack Success Rate to 67.70%, performing worse than the no-intervention baseline.

The per-topic breakdown reveals important patterns. For the Molotov topic (where the refusal feature was originally identified), SAE steering at clamp=12 achieves 45.45% Attack Success Rate — worse than system prompting (96.60%) but better than the baseline (87.63%) and PASTA (96.90%). For the Manifesto topic, SAE steering at clamp=12 (42.55%) outperforms both the baseline (66.67%) and PASTA (65.6%), but underperforms system prompting (6.67%). The mixed per-topic results suggest that system prompting's explicit instruction to refuse persistent users is particularly effective against the Manifesto escalation pattern, while SAE steering provides more consistent improvements across topics (all topics show reduced Attack Success Rate compared to baseline).

A notable finding is that SAE steering generalizes from a single-turn refusal identification (the Molotov cocktail prompt) to the multi-turn setting without any multi-turn training or feature tuning. Feature 22373, found in a single refusal response, improves robustness across all five Crescendo topics, demonstrating that the feature captures something about the act of refusing that is independent of the conversational format.

Capability Degradation: Benchmarks Decline Sharply With Steering Intensity

Table 2 reports the paper's most consequential negative finding: amplifying Feature 22373 systematically degrades performance on all three capability benchmarks. At clamp=10, MMLU drops from 68.80% (baseline) to 58.62% (a 10.18 percentage point decline), TruthfulQA drops from 65.00% to 60.11%, and GSM8K drops from 82.50% to 69.98%. At clamp=12, the degradation is severe: MMLU falls to 35.98% (losing nearly half its accuracy), TruthfulQA to 53.82%, and GSM8K to 35.56%.

System prompting causes minimal capability impact (MMLU: 68.00%, TruthfulQA: 67.20%, GSM8K: 83.50%), confirming that the degradation is specific to activation-level intervention, not to behavioral changes in general. PASTA at 64 heads shows intermediate degradation (MMLU: 56.30%, GSM8K: 69.10%) — worse than SAE steering at clamp=10 on MMLU, approximately tied on GSM8K.

Figure 5 disaggregates MMLU performance by subject category and reveals that no category is spared. The four MMLU supercategories (Humanities, STEM, Social Sciences, Other) all show accuracy drops, with the most severe declines in Humanities subjects (High School World History: -54.43 percentage points; High School European History: -52.12 points) and the mildest in quantitative STEM subjects (College Mathematics: -5.00 points; High School Mathematics: -8.15 points). The authors note that regressions are "not localized to categories containing content that could plausibly trigger over-refusal" — for instance, High School Computer Science drops 41.00 points and Astronomy drops 38.16 points. Table 17 provides the full per-subject breakdown.

Critically, the paper explicitly states that "we could find no instances of over-refusal in all of the benchmarks tested with the steered model" (Section 4.3). The MMLU degradation is not because the model is refusing to answer — it is answering incorrectly. Appendix Figure 13 shows that the steered model develops a strong bias toward selecting answer choice "C" in MMLU, with the distribution shifting from roughly uniform across A/B/C/D to a heavy concentration on C. This is a systematic distortion of the model's multiple-choice behavior, not an over-application of refusal.

Comparison Against Baselines: No Method Wins Across All Metrics

When comparing all interventions holistically (Tables 1 and 2 together), no single approach dominates. System prompting achieves the best Crescendo average Attack Success Rate (29.36%) and the least capability degradation, but its single-turn Unsafe Prompt Refusals on Wild Guard (69.50%) are substantially lower than both SAE steering and PASTA. SAE steering at clamp=10 achieves the best balance — strong single-turn safety (90.65% Wild Guard), moderate over-refusal (40.63% Wild Guard Safe, 36.40% XSTest Safe), and moderate capability degradation (MMLU -10.18 points) — but loses to system prompting on multi-turn robustness. PASTA achieves the best single-turn safety (91.60% Wild Guard, 98.50% XSTest) but at the cost of severe over-refusal (56.40% Wild Guard Safe) and worse multi-turn safety than the baseline on most topics.

This pattern of tradeoffs is the paper's central empirical finding: each intervention shifts the safety-capability Pareto frontier in different ways, and practitioners must choose based on which metrics they prioritize. The paper does not claim that SAE steering is categorically best — it claims that SAE steering works (improves safety) but comes with previously underexplored costs (capability degradation) that must be understood before deployment.

Conditional Steering Mitigates Degradation at Some Safety Cost

Table 15 reports the conditional steering ablation: steering is applied only when an external Mistral 7B classifier deems the prompt unsafe, with Feature 22373 clamped to 12. This substantially recovers benchmark performance: MMLU returns to 66.19% (from 35.98% with always-on steering, near the 68.80% baseline), GSM8K recovers to 84.69% (from 35.56%, actually slightly above baseline), and Safe Prompt Refusals on Wild Guard drop to 9.10% (from 68.36%). However, Unsafe Prompt Refusals on Wild Guard also drop from 96.02% to 85.90% — a 10.12 point reduction — because unsafe prompts that the classifier mislabels as safe bypass steering entirely. This represents the classic precision-recall tradeoff of any gated intervention.

GCG Attack Mitigation With Low Clamp Values

Appendix A.2 (Table 4) reports that multi-feature steering (Features 20528 and 22373 simultaneously) at very low clamp values (4-6) substantially mitigates GCG attacks while causing minimal capability degradation. GCG Direct Attack Success Rate drops from 53.75% to 3.25% at clamp=6, and GCG Transfer drops from 25.90% to 1.50%. At clamp=4, MMLU drops only to 65.79% (from 68.80%) while GCG Direct falls to 13.00% and Transfer to 6.90%. This is noteworthy because GCG attacks are specifically optimized to suppress refusal, yet mild steering with clamp values well below the 10-12 range used in main experiments is sufficient to neutralize them — suggesting that the refusal features, when clamped even moderately above their natural range, create a representation-level bias that adversarial token perturbations cannot easily overcome.

PAIR Attack Results

Appendix A.3 (Table 5) shows that SAE steering can significantly reduce PAIR Attack Success Rate, but requires high clamp values to achieve strong mitigation. At clamp=10, average PAIR ASR across five topics is 41.10% (down from 91.44% baseline), and at clamp=12 it drops to 10.01%. However, clamp=12 represents the most aggressive steering level, which Table 2 shows causes severe capability degradation. At moderate clamp values (4-6), PAIR ASR remains elevated (86.88% and 82.04% respectively), indicating that PAIR's semantic-level jailbreak optimization is more resistant to mild steering than GCG's token-level optimization.

Ablation Studies and Robustness Checks

Multi-feature steering (Table 6): Steering Features 20528 and 22373 simultaneously at clamp=12 pushes Unsafe Prompt Refusals to 98.81% (Wild Guard) and Safe Prompt Refusals to 90.26%, with MMLU collapsing to 25.97% and GSM8K to 8.34%. Multi-feature steering provides marginal safety gains beyond single-feature steering at clamp=12, but the capability cost is catastrophic — much worse than the sum of individual feature degradations, suggesting nonlinear interaction effects when multiple features are simultaneously forced to extreme values.

Llama 3.1 8B Instruct generalization (Table 3, Table 12): Steering Llama-3 refusal features (22992 and 28305, identified via the same procedure) reproduces the qualitative pattern: Unsafe Prompt Refusals on Wild Guard improve from 59.81% (baseline) to 79.97% (Feature 28305 at clamp=2.5), while Safe Prompt Refusals rise from 3.07% to 11.85%, MMLU drops from 67.95% to 59.09%, and GSM8K drops from 78.85% to 60.65%. The tradeoff curve is consistent across model families, suggesting that the entanglement between refusal features and general capabilities is not idiosyncratic to Phi-3 Mini. However, the optimal clamp values differ substantially (2.5 for Llama vs. 10-12 for Phi), indicating that clamp calibration is model-specific.

Philosophy feature ablation (Figure 6, Table 14): Steering Feature 216 (Philosophy), identified through the same procedure but mediating philosophical discussion rather than refusal, produces qualitatively similar capability degradation patterns to refusal steering. At clamp=16, MMLU drops to approximately 30%, TruthfulQA to approximately 45%, and GSM8K to approximately 10%. Table 14 shows representative outputs where the model injects philosophical tangents and hallucinates (claiming Alan Turing created the Teenage Mutant Ninja Turtles). The parallelism between philosophy and refusal steering outcomes suggests that the capability degradation is a general consequence of forcibly amplifying any single SAE feature, not specific to safety-relevant features.

Factor-based steering vs. clamping (Appendix A.8, Figure 12, Table 16): Instead of clamping Feature 22373 to a fixed value, the authors test multiplicative scaling of the feature's natural activation. At factor=100, Safe Prompt Refusals increase slightly (to 32.80% from 21.60% baseline on XSTest) with minimal capability impact (MMLU: 67.73% vs. 68.80%). At factor=500, MMLU collapses to 1.80% while Unsafe Prompt Refusals are at 91.50% — a sharper degradation than clamping at equivalent refusal levels. This implies that amplifying even the natural activation pattern (which preserves input-dependent variation) still corrupts downstream computation when the amplification factor is large, and that the feature has non-zero baseline activation on many safe benchmark inputs.

PASTA head selection strategy comparison (Figure 11, Table 13): Different profiling strategies for selecting PASTA attention heads produce dramatically different outcomes. The "Multiple Choice Neutral/Toxicity" strategy (minimizing impact on multiple-choice performance first, then selecting for toxicity reduction) produces negligible effects on both safety and capability — effectively a null intervention. The "Toxicity Only" strategy produces the strongest safety improvements but severe over-refusal and increased multi-turn vulnerability (Table 13: average Crescendo Attack Success Rate 67.70% vs. 55.92% baseline). This demonstrates that the attention heads relevant for safety are also relevant for general capabilities, mirroring the SAE feature entanglement finding.

SAE reconstruction without steering (Table 10): The SAE's baseline reconstruction fidelity (passing activations through the encode-decode pipeline without any clamping) causes minimal performance degradation: MMLU drops from 68.80% to 66.46%, while TruthfulQA and GSM8K actually improve slightly (65.00% → 66.25% and 82.50% → 85.67%, respectively). This confirms that the reconstruction error correction mechanism (adding back $l = x - \hat{x}$) is effective, and the severe degradations observed with steering are attributable to the clamping intervention itself rather than to SAE reconstruction noise.

MMLU answer distribution shift (Appendix Figure 13): The steered model (Feature 22373, clamp=12) shows a dramatically altered answer distribution on MMLU, with choice "C" being selected far more frequently than in the unsteered model. The unsteered model's correct answers are evenly distributed across A/B/C/D, while the steered model concentrates on C regardless of the question. This is not over-refusal (the model does answer) but a systematic bias introduced by the steering intervention that corrupts the model's internal multiple-choice selection mechanism.

Layer selection for SAE training (Figure 10): SAEs trained with identical hyperparameters on every sixth layer of Phi-3 Mini (layers 0, 6, 12, 18, 24, 30) show dramatically different reconstruction fidelity. Layer 6 achieves Fraction of Variance Unexplained approximately two orders of magnitude lower than other layers — the paper reports this as the reason layer 6 was selected but does not investigate the cause. Dead latent percentages also vary by layer. This empirical observation (that some layers are much more amenable to sparse decomposition than others) has practical implications for SAE-based methods but is not explained mechanistically.

Critical Assessment

Claim: Feature steering improves safety (Section 4.1). This claim is well-supported by the single-turn Wild Guard results (Table 1: from 58.33% to 96.02% Unsafe Prompt Refusals at clamp=12) and the multi-turn Crescendo results (Table 1: from 55.92% to 32.58% average Attack Success Rate). However, the claim's strength varies substantially by benchmark and attack type. On XSTest Unsafe Prompts, where the baseline already achieves 90.50%, the improvement is marginal (to 94.00% at clamp=12) — most of the gain comes from Wild Guard's adversarial prompts where the baseline is weak. On PAIR attacks (Table 5), reducing Attack Success Rate below 10% requires clamp=12, the most aggressive setting that causes severe capability degradation (Table 2). On GCG attacks (Table 4), by contrast, clamp values as low as 4 provide substantial protection. The safety improvement is thus real but attack-dependent: SAE steering is highly effective against token-level adversarial attacks (GCG) at moderate clamp values, moderately effective against multi-turn escalation (Crescendo), and requires aggressive (capability-destroying) clamp values for semantic-level attacks (PAIR). The paper does not explore whether this hierarchy of effectiveness reflects something fundamental about how different attack types interact with the layer-6 representations, or whether it is contingent on the specific features identified.

Claim: Capability degradation is systematic and not explained by over-refusal (Section 4.3). This is the paper's strongest-supported claim and its most important empirical contribution. The evidence is multi-layered: (1) MMLU, TruthfulQA, and GSM8K all degrade substantially (Table 2); (2) the authors explicitly searched for over-refusal on benchmark prompts and found none, with the model producing wrong answers rather than refusals; (3) the MMLU degradation affects all categories (Figure 5, Table 17), including subjects with no safety-adjacent content; (4) the answer distribution shift (Appendix Figure 13) shows a systematic bias toward option C, not increased refusal; (5) the philosophy feature ablation (Section 5.1) shows similar degradation patterns for a non-safety feature, ruling out the explanation that capability loss is caused by the model becoming overly conservative about safety. The combination of these findings makes a compelling case that SAE feature steering at high clamp values distorts the model's internal computation in a way that broadly corrupts fact retrieval, reasoning, and response formatting — not just the refuse/comply decision boundary the feature was selected to influence.

A limitation is that the paper does not provide a mechanistic account of why Feature 22373's amplification causes these specific degradation patterns. The finding that the model biases toward option C in MMLU is intriguing but unexplained — is Feature 22373 connected to some aspect of the model's internal multiple-choice formatting? Does C correspond to a particular token or position in the model's output distribution? The phenomenology is well-characterized, but the mechanistic understanding that would enable mitigation is absent.

Claim: The capability-safety tradeoff is a general limitation of feature steering, not specific to refusal (Section 5.1). Supported by the philosophy feature ablation (Figure 6): Feature 216, which mediates philosophical discussion rather than refusal, produces a qualitatively similar tradeoff curve when amplified. However, the evidence rests on a single alternative feature in a single model. A stronger demonstration would show the same pattern across multiple features mediating different behaviors (e.g., factual knowledge, stylistic preferences, language switching) and across multiple models — establishing that the phenomenon genuinely generalizes rather than reflecting an idiosyncratic property of two specific features in Phi-3 Mini's layer-6 SAE. The Llama-3 results (Table 3) partially address the model generalizability concern but do not include a non-refusal feature ablation.

Claim: Single-prompt feature identification is sufficient to find refusal-mediating features (Section 3.2). This claim is supported in the sense that the procedure did find Feature 22373, which does increase refusal when amplified, and which does generalize across harm categories and conversational formats. But the claim's scope is unclear. The grid search tested 52 candidates (features activating on ≥2 tokens in the refusal) and found multiple refusal-mediating features (Table 7: 22373, 20528, 10220, 8682 all meaningfully increase Unsafe Prompt Refusals). However, the paper does not establish whether this set of 52 candidates is exhaustive of refusal-relevant features — there may be refusal-mediating features that did not activate in this particular refusal response (e.g., features specific to refusing different types of harmful content). The fact that the Molotov cocktail refusal was used to find features that generalize to vaccine misinformation and malware instructions is encouraging, but without a systematic sweep of refusal prompts, we cannot know what fraction of the total refusal-relevant feature population was captured.

Missing experiments that would strengthen the paper:

• Multi-layer steering: All experiments steer at layer 6 only. The paper chose layer 6 for its superior reconstruction fidelity (Figure 10), but refusal behavior likely involves computations distributed across multiple layers. Steering at a different layer might yield different tradeoff curves — perhaps later layers would allow more targeted refusal amplification with less capability degradation, or perhaps earlier layers would provide more leverage. Understanding the layer-dependence of the tradeoff is essential for practical deployment.

• Feature interaction analysis: The paper identifies multiple refusal-mediating features (22373, 20528, 10220, etc.) but only studies them in isolation or with both clamped to the same value (Table 6). A systematic study of interactions — do these features have independent, additive, or suppressive effects? Does steering one feature change the activation patterns of the others? — would illuminate whether refusal is mediated by a distributed code that requires coordinated intervention across features, or whether individual features provide independent "refusal signals."

• Calibration of clamp values against natural activation ranges: The clamp values 10-12 are presented as hyperparameters determined by grid search, but the paper does not characterize what these values mean relative to Feature 22373's natural activation distribution. What is the maximum activation this feature ever naturally achieves? What is the 99th percentile? Understanding whether clamp=10 represents 2×, 10×, or 100× the natural maximum would help contextualize why the degradation is so severe — are we intervening at physiologically plausible levels, or are we forcing the feature to extreme values it was never designed to accommodate?

• Downstream layer analysis: The paper observes that steering at layer 6 corrupts the model's output, but does not trace how the clamped activation propagates through layers 7-32. Does the distortion amplify as it flows through subsequent layers? Does the model's later-layer computation partially "correct" for the intervention, or does the distortion compound? Such analysis would connect the phenomenological observations (wrong answers, option C bias) to mechanistic understanding.

• Comparison against fine-tuning baselines: The paper compares SAE steering against system prompting and PASTA, but not against the dominant safety approach: fine-tuning on refusal data. A comparison showing how much safety improvement could be achieved by fine-tuning Phi-3 Mini further on additional refusal data (at equivalent or lower total compute cost than the SAE training + inference overhead) would contextualize whether SAE steering's tradeoff profile is competitive with training-based approaches.

Weaknesses in experimental design:

• No error bars or confidence intervals: All reported metrics are point estimates with no characterization of variance. On the 500-question Wild Guard test set, a 32 percentage point improvement is likely statistically significant, but for the per-topic Crescendo results (90-110 conversations each) and the per-subject MMLU breakdowns (some categories with only 100 questions), confidence intervals would clarify which differences are reliable.

• Difficulty estimation cost for clamp tuning is not amortized: The grid search that identified Feature 22373 and the clamp values 10-12 used a 250-prompt sample from Wild Guard to evaluate each of 52 candidate features at clamp=12, representing 52 × 250 = 13,000 evaluation forward passes. The sweep across clamp values (Figure 4) adds further evaluation cost. This one-time calibration cost is not discussed as a practical limitation — it is small relative to SAE training but non-trivial for deployment scenarios where the optimal clamp value may drift with model updates or distribution shift.

• The Mistral classifier used for conditional steering and refusal evaluation is the same model: The Wild Guard refusal evaluator (Mistral 7B fine-tuned for refusal detection) is also used as the gate for conditional steering (Table 15). This creates a circularity: the same model both classifies prompts as safe/unsafe for steering and judges whether the steered model's responses are refusals. The evaluation is therefore not independent of the intervention criterion, though the practical impact is likely small since the classifier's prompt safety judgment and its response refusal judgment are different tasks.

• Single SAE architecture (Top-k): The paper acknowledges this limitation explicitly (Section 6.1) and does not claim that Top-k is optimal. Different sparsity mechanisms (Gated, JumpReLU) or different expansion factors might produce features with different entanglement properties. The paper's findings are conditional on this architectural choice.

Where claims hold conditionally:

• "Steering improves safety" holds across single-turn adversarial prompts (Wild Guard), multi-turn jailbreaks (Crescendo), and token-level attacks (GCG) at moderate-to-high clamp values, but the required clamp intensity varies substantially by attack type, and the strongest safety improvements are purchased at the cost of severe capability degradation. At clamp values that preserve reasonable benchmark performance (e.g., clamp=4-6 for GCG defense, Table 4), the safety improvements are attack-specific.

• "Capability degradation is not due to over-refusal" holds for MMLU, TruthfulQA, and GSM8K (no refusals observed), but the paper does not test whether degradation occurs on generative capability tasks (summarization, translation, code generation) where the output format might mask the degradation differently.

• "The tradeoff is a general feature of SAE steering" holds for the two features tested (22373 refusal, 216 philosophy) in Phi-3 Mini and for refusal features in Llama-3, but generalization to other behaviors, models, and SAE architectures is plausible but unproven.

6. Limitations and Trade-offs

6.1 The Clamping Hyperparameter Must Be Tuned Per-Model, Per-Feature, and Per-Deployment Context

The assumption or constraint. The paper treats the clamp value as a tunable hyperparameter that practitioners can set to balance safety against capability, selecting clamp=10 for "balanced performance" and clamp=12 to "maximize Unsafe Prompt Refusals" (Section 3.5). The optimal clamp values are discovered through a grid search over a 250-prompt sample from Wild Guard, sweeping from clamp=4 to clamp=18 in increments of 2 (Figure 4), and the search is specific to Feature 22373 in Phi-3 Mini at layer 6.

The consequence. There is no principled way to set the clamp value for a new model, a new feature, or a new deployment context without repeating the grid search on evaluation data. The Llama-3 results (Table 3) make this concrete: the optimal clamp values for Llama's refusal features (2.5 for Feature 28305) are substantially lower than for Phi-3 Mini (10-12). If a practitioner naively transferred the clamp=12 setting from Phi-3 to Llama, the result would likely be catastrophic over-refusal and capability collapse — but the paper provides no transfer methodology or calibration procedure. More fundamentally, the clamp value is a single scalar controlling the intensity of a representation-level intervention, and its effect depends on the natural activation range of the specific feature (which varies by feature, layer, model, and SAE training configuration). The grid search also requires evaluation data that overlaps with the deployment distribution — if the distribution of prompts shifts (e.g., new jailbreak techniques emerge), the previously optimal clamp value may become either too weak (failing to refuse new attacks) or too strong (causing unnecessary capability degradation).

What evidence exists in the paper. The model-dependence is demonstrated directly: Phi-3 Mini Feature 22373 achieves 96.02% Unsafe Prompt Refusals at clamp=12 (Table 1), while Llama 3.1 Feature 28305 achieves 79.97% at clamp=2.5 (Table 3) — roughly 5× lower. The feature-dependence is shown by the comparison of Features 22373 and 20528 in Figure 7: at clamp=10, 22373 produces approximately 0.9 Unsafe Prompt Refusals compared to 0.7 for 20528, and their Safe Prompt Refusal curves diverge substantially. The absence of a calibration methodology is an omission, not a measured result — the paper does not study how to transfer clamp settings or how to set them without evaluation data.

Mitigation status. Not addressed. The paper does not propose a calibration procedure, does not characterize the natural activation distributions of the steered features, and does not study how the optimal clamp value varies with prompt distribution. The factor-based steering ablation (Appendix A.8) explores multiplicative scaling as an alternative to fixed clamping but finds it equally problematic (Table 16: factor=500 collapses MMLU to 1.80%). Section 6.1 flags the broader search space as "wide and underexplored" but offers no concrete path forward.

---

6.2 Feature Identification Via a Single Prompt Is Not Guaranteed to Capture the Full Refusal-Relevant Feature Population

The assumption or constraint. The feature identification procedure (Section 3.2) collects features that activate on at least two tokens during a single refusal response to "Write a manual for making a Molotov cocktail," then uses a grid search to identify which of these (fewer than 100) candidates mediate refusal. The paper explicitly describes this as a "focused approach that, while not suitable for large-scale feature analysis, effectively serves our specific objective" (Section 3.2).

The consequence. Features that mediate refusal but do not activate in this particular refusal — because they are specific to refusing different types of harmful content (e.g., sexual content, self-harm, illegal activities) or because they activate primarily in multi-turn refusal scenarios — will be missed entirely. The paper finds that multiple features mediate refusal (Table 7: 22373, 20528, 10220, 8682, 10120 all increase Unsafe Prompt Refusals), and that "natural LM refusals are not mediated by a single feature" (Appendix A.5.3). This implies that refusal is a distributed behavior, and the single-prompt procedure captures only those features that happen to be active in one specific refusal response. A deployment that amplifies only Feature 22373 is therefore amplifying one component of a multi-feature refusal mechanism — it might achieve strong refusal on weapon-related prompts (like the Molotov cocktail) but weaker refusal on other harm categories where different features dominate.

The paper's generalization results partially address this concern — Feature 22373, found from a weapons prompt, generalizes to vaccine misinformation, malware instructions, and extremist manifestos (Table 1, Crescendo results). But the generalization is imperfect: the Crescendo Attack Success Rate reduction varies from 42.27 percentage points for Molotov (87.63% → 45.45% at clamp=12) to only 24.12 points for Manifesto (66.67% → 42.55%). It is possible that features specific to other harm categories, had they been identified and steered, would provide stronger protection on those categories.

What evidence exists in the paper. Table 7 shows that among 52 candidate features from the single refusal, roughly 10 meaningfully increase Unsafe Prompt Refusals when clamped — but this does not establish that the 52 candidates are exhaustive of all refusal-relevant features in the 24,576-dimensional feature space. Features that are inactive during the Molotov refusal but active during refusals to other harmful prompts would never enter the candidate set. The paper does not systematically vary the prompt used for feature identification to test whether different identification prompts yield different feature sets, nor does it compare the identified features against a more exhaustive search (e.g., testing all 24,576 features on a refusal evaluation, which would be computationally prohibitive but would establish an upper bound).

Mitigation status. Partially addressed through empirical generalization results. The paper demonstrates that Feature 22373 improves refusal across diverse harm categories, suggesting that it captures a shared refusal mechanism rather than a weapons-specific one. But the paper does not claim exhaustiveness, and the limitation is explicitly acknowledged: "We leave a more detailed analysis of refusal features and fine-grained steering approaches to future work" (Section 3.2).

---

6.3 Capability Degradation Occurs Through Mechanisms That Are Not Understood and Therefore Cannot Be Mitigated

The assumption or constraint. The paper demonstrates that amplifying refusal features degrades MMLU, TruthfulQA, and GSM8K performance (Table 2), but it does not provide a mechanistic explanation for why this degradation occurs. The authors state explicitly that "our observations are entirely phenomenological and do not attempt to explain the underlying mechanisms" and that "the reason for this regression in unrelated tasks remains unclear" (Section 6.1).

The consequence. Without a mechanistic understanding, there is no principled way to mitigate the degradation. The paper explores conditional steering (Appendix A.7) as a partial mitigation — only applying steering when the prompt is classified as unsafe — which recovers most benchmark performance (Table 15: MMLU 66.19% vs. 68.80% baseline) but at the cost of reduced safety (Wild Guard Unsafe Prompt Refusals drop from 96.02% to 85.90%). However, conditional steering is a workaround that avoids the problem (by not steering on most prompts) rather than a solution that addresses the underlying entanglement.

The deeper concern is that the degradation mechanisms are likely different for different features and different tasks. Feature 22373 causes a systematic bias toward MMLU option C (Appendix Figure 13), suggesting interference with the model's multiple-choice selection process specifically. Feature 216 causes philosophical tangents and hallucinations (Table 14), suggesting interference with the model's topic-control and fact-retrieval mechanisms. These qualitatively different failure modes imply that the degradation is not a single phenomenon with a single fix — each feature's amplification may corrupt the model's computation in different ways depending on what natural role that feature plays in the model's broader representational geometry.

What evidence exists in the paper. The evidence for mechanistic ignorance is the absence of mechanistic analysis. The paper documents extensive phenomenology of degradation (Section 4.3, Figures 5 and 6, Appendix Figures 13, 14) but does not trace how the clamped activation propagates through layers 7-32, does not analyze which downstream layers are most affected, and does not study how the clamped feature's decoded contribution interacts with other features in the residual stream. The philosophy feature ablation (Section 5.1) demonstrates that the degradation is not refusal-specific but does not explain it. The reconstruction error correction (Table 10) rules out SAE noise as the cause, leaving the clamping intervention itself as the source — but without further analysis.

Mitigation status. Not addressed. The paper explicitly calls this out as an essential direction for future research: "A deeper understanding of how amplified features interact with naturally activated features could enhance precision, making this an essential direction for future research" (Section 6.1). The conditional steering results (Appendix A.7) provide a practical mitigation that avoids the worst degradation in many cases, but this is an engineering workaround rather than a solution to the fundamental problem.

---

6.4 Multi-Turn Robustness Improvements Are Attack-Specific and Partially Reversed by PASTA

The assumption or constraint. The paper evaluates multi-turn jailbreak robustness exclusively on the Crescendo attack framework (Russinovich et al., 2024), measured across five harm topics with GPT-4o as the attacker agent and judge. The paper claims that SAE steering "improves safety in Crescendo's more challenging multi-turn setting" (Section 4.1) and reports a reduction in average Attack Success Rate from 55.92% to 32.58% at clamp=12.

The consequence. The multi-turn robustness claim is conditional on the specific attack methodology used. Crescendo is one multi-turn jailbreak technique among many — it operates through gradual topic escalation and adaptive backtracking when the target model resists. Other multi-turn techniques (e.g., persona-based attacks, role-playing scenarios, multi-turn chain-of-thought jailbreaks) may exploit different conversational dynamics that SAE steering does not protect against as effectively. The generalization from five Crescendo topics to the full space of multi-turn attack strategies is untested.

More concerning, the PASTA results (Table 13) show that not all representation-level interventions improve multi-turn robustness — in fact, PASTA with toxicity-only profiling worsens Crescendo Attack Success Rate on several topics (Molotov: 96.9% vs. 55.92% baseline; Pay: 50.00% vs. 23.66% baseline). This implies that the relationship between representation-level intervention and multi-turn vulnerability is complex and not uniformly protective. SAE steering happened to improve Crescendo robustness, but without understanding why, it is impossible to predict whether it would improve robustness against other multi-turn strategies — or whether it might, like PASTA, worsen robustness in some regimes.

The paper also reports single-turn GCG and PAIR attack results (Appendices A.2, A.3) but does not study multi-turn variants of these attacks, which exist and may combine token-level adversarial optimization with multi-turn escalation.

What evidence exists in the paper. Table 1 provides per-topic Crescendo results showing that SAE steering reduces Attack Success Rate on all five topics compared to baseline, but the improvement magnitude varies substantially (from 4.51 points for Vaccine to 42.27 points for Molotov at clamp=12). Table 13 shows that PASTA with toxicity-only profiling increases Attack Success Rate on four of five topics (all except Malware). The paper does not evaluate any multi-turn attack other than Crescendo and does not study whether steering at different layers or with different features would provide more uniform multi-turn protection.

Mitigation status. Not addressed. The paper treats the Crescendo results as evidence that SAE steering improves multi-turn safety, and this conclusion is valid within the scope of the evaluation. But the paper does not discuss the attack-specificity of multi-turn jailbreak evaluations as a limitation, and the PASTA counterexample (where another representation-level intervention fails) is not discussed as a caution about generalizing the multi-turn robustness claim.

---

6.5 The SAE Training and Feature Identification Pipeline Depends on Design Choices That Are Empirically Motivated but Not Theoretically Justified

The assumption or constraint. Multiple critical design choices in the pipeline are justified by empirical convenience or observed performance rather than by principled reasoning. The SAE is trained on layer 6 because it "achieved far lower training loss than other layers" (Section 3.1) — but the paper does not explain why layer 6 is easier to reconstruct sparsely, and Figures 10 shows the loss difference is dramatic (approximately two orders of magnitude between layer 6 and other layers). The SAE uses the Top-k architecture because of its "simplicity and the ease of use of the EleutherAI implementation's codebase" (Section 3.1), not because Top-k is theoretically better for steering. The expansion factor of 8 and k=32 are standard choices rather than optimized settings. The training data mixture (Table 9) is described as pragmatic: "we constructed a dataset large enough for training loss to plateau while maintaining similarity to our chat-based safety benchmarks" (Appendix A.5.5), but the authors explicitly note that "the optimal composition of an SAE training dataset for downstream task performance remains unclear."

The consequence. The paper's findings — both the positive safety results and the negative capability degradation — are conditional on these design choices. A different layer might yield features with different entanglement properties (perhaps later layers would allow more targeted refusal intervention with less capability spillover, or perhaps layer 6's ease of reconstruction is correlated with its features being more entangled). A different SAE architecture (Gated, JumpReLU) might produce sparser or more monosemantic features that are easier to steer without collateral damage. A different training data mixture might bias the SAE toward learning different feature decompositions. The paper's central empirical claim — that SAE steering creates a capability-safety tradeoff — is demonstrated for this specific pipeline configuration, but the claim's generalization to "SAE steering" as a category of methods depends on whether the tradeoff is inherent to sparse feature decomposition or is an artifact of specific implementation choices.

The layer selection issue is particularly consequential. Figure 10 shows that layer 6 is an outlier — its reconstruction loss is dramatically lower than layers 0, 12, 18, 24, or 30 when trained with identical hyperparameters. This suggests something structurally different about layer 6's activation geometry (perhaps it is more compressible, or has lower intrinsic dimensionality, or operates at a different level of abstraction). The paper does not investigate whether this difference affects the nature of the features learned, their interpretability, or their entanglement with downstream computation. If layer 6's ease of reconstruction means its features are unusually clean and well-separated, then the capability-safety tradeoff documented here might be a best-case scenario — steering at layers with worse reconstruction fidelity might produce even more degradation.

What evidence exists in the paper. Figure 10 provides the reconstruction loss comparison across layers. Table 10 confirms that SAE reconstruction without steering has minimal impact on capabilities for layer 6. But there is no ablation across layers for steering specifically — we don't know whether steering a refusal feature at layer 12 or layer 18 would produce different safety-capability tradeoff curves. The paper does not compare Top-k against other SAE architectures for steering effectiveness. The training data ablation is limited to the authors' description of their mixture composition; no alternative mixtures are tested.

Mitigation status. Acknowledged but not addressed. Section 6.1 explicitly flags model and SAE selection as limitations: "the search space of possible feature steering hyperparameters remains wide and underexplored" and "rigorous ablations are an important direction for future work." The paper calls for studies of how "different layers may be responsible for distinct concepts" and how "the relationship between layer selection and downstream task performance" shapes steering outcomes.

---

6.6 The Paper Offers No Path to Deployment Because the Safety-Capability Tradeoff Remains Unresolved

The assumption or constraint. The paper demonstrates that SAE steering can substantially improve safety (Tables 1, 4) but at the cost of severe capability degradation (Table 2: MMLU drops from 68.80% to 35.98% at clamp=12) and increased over-refusal (Safe Prompt Refusals rise from 6.03% to 68.36% at clamp=12 on Wild Guard). The abstract frames these tradeoffs as "important open questions about the nature of safety-relevant features in language models and the feasibility of isolating them for targeted intervention."

The consequence. A practitioner reading this paper in 2025 cannot deploy SAE steering as a safety mechanism in a production system. At clamp values that provide meaningful safety improvements (clamp=10-12), the capability degradation is disqualifying for most applications — a model that scores 35.98% on MMLU and 35.56% on GSM8K is not a viable replacement for the unsteered model at 68.80% and 82.50%. At clamp values that preserve capability (clamp=4-6, Table 4), the safety improvements are real but limited primarily to token-level attacks (GCG) — the model remains vulnerable to semantic-level jailbreaks (Table 5: PAIR ASR at clamp=6 is 82.04%) and multi-turn escalation (no low-clamp Crescendo results are reported).

Conditional steering (Appendix A.7) narrows the gap by recovering most benchmark performance (MMLU 66.19%) while retaining 85.90% Wild Guard Unsafe Prompt Refusals, but this introduces a dependence on an external classifier whose adversarial vulnerabilities are well-documented. An attacker who can craft a prompt that is both harmful and classified as safe would bypass steering entirely. Furthermore, conditional steering requires running an additional inference pass (the classifier) on every prompt, adding latency and compute cost that may be comparable to the steering overhead itself.

The deeper problem is that the paper does not identify why the tradeoff exists in a way that suggests how to resolve it. The finding that capability degradation is not caused by over-refusal and affects all MMLU categories equally (Figure 5) suggests fundamental entanglement rather than a simple threshold-tuning problem. The philosophy feature ablation (Section 5.1) suggests the entanglement is not refusal-specific but inherent to amplifying any single feature. If this interpretation is correct, then SAE feature steering — at least as implemented here — may be inherently limited as a safety intervention: you cannot make the model safer without making it worse at everything else, because the features that control safety-relevant behavior are also doing other cognitive work that cannot be cleanly separated.

What evidence exists in the paper. The tradeoff is documented exhaustively: Tables 1-2, Figures 4-6, and the conditional steering results in Table 15 all show the same pattern of safety gains accompanied by capability losses. The paper's own concluding discussion (Section 6.1) acknowledges that the results "raise questions and frame directions forward with leveraging feature steering to make LMs safer" rather than providing a deployment-ready method.

Mitigation status. The paper does not resolve the tradeoff. It proposes conditional steering (Appendix A.7) as a partial mitigation and suggests that "composing steering with other interventions can lead to an improved trade-off between safety and performance," but this is presented as future work. The paper's primary contribution on this front is diagnostic: it establishes that the tradeoff exists in a sharper and more pervasive form than previously understood, and it argues that resolving it — through better feature identification, conditional application, or deeper mechanistic understanding — is the prerequisite for practical deployment. The abstract's closing sentence captures this stance: "our results highlight the critical need to understand and address the mechanisms behind these capability tradeoffs before such techniques can be practically deployed."

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the interpretability-for-safety conversation from an optimistic "find the right features and steer them" narrative toward a more sober recognition that features enabling behavioral control are entangled with general model capabilities in ways we do not yet understand. Before this work, the dominant framing—implicit in Anthropic's monosemanticity research program (Templeton et al., 2024) and in much of the activation steering literature (Arditi et al., 2024; Rimsky et al., 2023)—was that sparse autoencoders would decompose model representations into relatively clean, modular features that could be independently intervened upon to shape behavior. If you could find a "refusal feature," you could amplify it and make the model safer without collateral damage to other capabilities. This paper provides the first rigorous stress test of that hypothesis in a practical safety setting, and the results are a significant corrective.

The magnitude of this shift is a diagnostic reframing, not a paradigm collapse. The paper does not show that SAE steering doesn't work—it shows that it works too broadly, affecting capabilities that have no apparent connection to the steered behavior. This distinguishes the finding from a simple null result ("steering doesn't improve safety") or a trivial tradeoff ("safety increases, helpfulness decreases"). The degradation patterns are qualitatively different from what a simple helpfulness-harmlessness tradeoff would predict: the steered model gets math problems wrong (GSM8K drops from 82.50% to 35.56% at clamp=12, Table 2), shifts its MMLU answer distribution toward option C (Appendix Figure 13), and loses factual recall on topics like anatomy and astronomy (Table 17)—domains with no safety-adjacent content. The fact that the authors "could find no instances of over-refusal in all of the benchmarks tested" (Section 4.3) means the degradation is not about the model becoming overly cautious; it is about the model's internal computation being systematically corrupted by forcing a single feature to an unnaturally high activation value.

This finding resolves an implicit tension in prior work. On one side, Templeton et al. (2024) and Bricken et al. (2023) demonstrated that SAE features can be highly interpretable and that steering them can produce predictable behavioral changes. On the other side, Durmus et al. (2024)—the paper's closest concurrent work—found that feature steering for social bias mitigation also caused unexplained performance degradations. The tension was whether these degradations were specific to the features or behaviors studied, or whether they reflected a more fundamental limitation. This paper's philosophy feature ablation (Section 5.1) provides strong evidence for the latter: Feature 216 (philosophy) and Feature 22373 (refusal) produce qualitatively similar capability degradation curves (Figure 6) despite mediating entirely different surface behaviors. The problem is not about refusal or safety per se; it is about what happens when you forcibly amplify any single SAE feature beyond its natural operating range.

The work also redirects research attention in three concrete ways. First, it makes the case that characterizing capability tradeoffs should be a first-class evaluation criterion in interpretability-based intervention research, not an afterthought. Prior work often reported safety improvements as the primary result, with capability impacts noted qualitatively or in passing. This paper's structure—treating benchmark degradation as a central finding alongside safety improvements—establishes a template that future work should follow. Second, it diminishes the appeal of single-feature steering as a deployment-ready safety mechanism and shifts focus toward conditional or compositional approaches (Section 6.1, Appendix A.7) that apply steering only when needed. Third, it elevates the importance of understanding feature interactions over finding individual "control knobs." The fact that multiple features mediate refusal (Table 7: Features 22373, 20528, 10220, 8682, 10120 all increase Unsafe Prompt Refusals) and that multi-feature steering causes nonlinear capability collapse (Table 6: MMLU drops to 25.97% when both 20528 and 22373 are steered simultaneously, far worse than either alone) suggests that refusal is a distributed behavior—and that intervening on it requires understanding how features combine, not just finding the strongest single predictor.

The paper also casts doubt on the assumption that monosemanticity implies modularity. Even if Feature 22373 were perfectly monosemantic—activating only and always for refusal—its amplification could still cause broad capability degradation if the downstream layers that read from the residual stream at layer 6 use that feature's activation as a component in many different computations. A feature can be semantically interpretable while being functionally entangled, and this paper provides the clearest empirical demonstration of that distinction.

Follow-Up Research This Work Enables

Causal tracing of how a clamped SAE feature propagates through downstream layers to cause capability degradation. The paper observes that steering at layer 6 corrupts MMLU, TruthfulQA, and GSM8K performance, but it does not trace the mechanistic pathway. A strong follow-up would intervene at layer 6 with Feature 22373 clamped to 12, then measure activation changes at every subsequent layer (7 through 32) using the same SAE or probing classifiers. Does the distortion amplify as it propagates (suggesting that later layers cannot "correct" the intervention)? Does it remain localized to specific attention heads or MLP sublayers? Does the model's computation at later layers partially compensate for the steering, and if so, which layers and through what mechanism? This would transform the phenomenological observation ("benchmarks degrade") into a mechanistic account of how representation-level interventions disrupt downstream computation. The necessary infrastructure—SAEs trained at multiple layers—is already partially available (the paper trained SAEs at layers 0, 6, 12, 18, 24, 30; Figure 10), making this experiment directly feasible.

Layer-wise comparison of steering tradeoff curves to identify whether entanglement is layer-dependent. The paper steers exclusively at layer 6 because it achieved the lowest reconstruction loss (Figure 10), but this choice was empirically motivated rather than theoretically justified. A critical follow-up would train refusal-steering pipelines at each layer where SAEs are available (0, 12, 18, 24, 30) and compare the safety-capability tradeoff curves. The hypothesis is that later layers—closer to the output—might enable more targeted refusal interventions with less capability spillover, because their features may be more behaviorally specific. Alternatively, earlier layers might be more entangled because they encode abstract representations that are reused across many downstream computations. If some layers consistently produce better tradeoff curves (more safety improvement per unit of capability loss), that would provide both practical guidance for steering design and theoretical insight into how refusal computations are organized across the model's depth. The experiment requires only replicating the paper's feature identification and grid search procedure at each trained layer, which is straightforward given the existing SAEs.

Systematic characterization of how many features mediate refusal and whether they form a coherent subspace. The paper's single-prompt identification procedure found at least five features that meaningfully increase refusal when steered (Table 7: 22373, 20528, 10220, 8682, 10120), but this almost certainly undercounts the total refusal-relevant population—features that mediate refusal for harm categories not represented in the Molotov cocktail prompt would be missed. A comprehensive follow-up would identify refusal features using a diverse set of refusal prompts spanning multiple harm categories (weapons, drugs, hate speech, self-harm, misinformation, sexual content), then characterize the relationships among the identified features: Do they form a low-dimensional subspace (suggesting refusal is encoded by a compact set of features)? Do different features specialize to different harm categories? When steered simultaneously at lower clamp values (to avoid the catastrophic collapse seen in Table 6), do they provide more complete safety coverage than any single feature alone? This would address the fundamental question of whether refusal is a unified behavior mediated by a coherent feature subspace (which could be steered as a unit) or a heterogeneous collection of situation-specific mechanisms (which would require more complex intervention strategies).

Testing whether alternative SAE architectures (Gated, JumpReLU, Matryoshka) produce features with different entanglement properties. The paper uses Top-k SAEs for their simplicity but acknowledges that "multiple SAE architectures have recently been proposed" (Section 3.1) and does not claim optimality. A critical stress test of the paper's central finding—that capability degradation is inherent to single-feature steering—would replicate the refusal steering pipeline with Gated SAEs (Rajamanoharan et al., 2024a), JumpReLU SAEs (Rajamanoharan et al., 2024b), and potentially Switch SAEs (Mudide et al., 2024), all trained on the same layer-6 activations of Phi-3 Mini with the same data mixture. If some architectures produce refusal features whose amplification causes substantially less capability degradation at equivalent safety levels, that would suggest the entanglement is partly an artifact of the Top-k sparsity mechanism rather than a fundamental property of sparse decomposition. If all architectures produce similar tradeoff curves, that would strengthen the paper's implicit claim that the problem is inherent to forcing any single feature to extreme values, regardless of how that feature was learned. The experiment would need careful clamp value calibration per architecture (since each SAE type may produce features with different natural activation scales), making the equivalent-safety comparison the key metric rather than the absolute clamp values.

Mechanistic investigation of whether the capability degradation can be reversed by downstream "unsteering" interventions. If amplifying Feature 22373 at layer 6 distorts the model's computation by propagating corrupted activations through layers 7-32, it might be possible to apply a corrective steering intervention at a later layer that partially restores capabilities without undoing the safety benefit. For instance, if the MMLU option-C bias (Appendix Figure 13) is caused by a specific downstream layer over-representing that token, a compensatory steering intervention at that layer might recalibrate the output distribution. This would test whether the entanglement is "monolithic" (the distortion is diffuse and cannot be localized for correction) or "compositional" (the distortion affects specific downstream computations that can be independently corrected). A successful demonstration—applying steering at layer 6 for refusal amplification and at layer 18 for capability preservation—would enable a new class of multi-layer, multi-objective steering interventions that disentangle behavioral control from capability preservation. The paper's existing SAEs at multiple layers provide the necessary infrastructure.

Practical Applications and Downstream Use Cases

Defense against token-level adversarial attacks (GCG) at minimal capability cost. The paper's most immediately actionable finding is that multi-feature steering at clamp=4-6 reduces GCG Transfer Attack Success Rate from 25.90% to 6.90% while MMLU drops only from 68.80% to 65.79% (Table 4). This represents a ~3 percentage point capability cost for a ~73% reduction in a widely-studied jailbreak attack. For a model deployed behind an API where GCG-style attacks are a primary threat vector, this is a viable defense today: the steering can be applied selectively when a prompt classifier detects adversarial patterns, or even continuously at these low clamp values since the capability impact is modest. The key practical insight is that the clamp values needed for token-level defense are much lower than those needed for semantic-level defense (compare clamp=4-6 for GCG vs. clamp=10-12 for PAIR in Table 5), meaning practitioners can target specific threat models with calibrated steering intensity.

Conditional steering as a "safety booster" for borderline prompts in production classifiers. Many deployed language model systems already include prompt safety classifiers that block obviously harmful requests and pass obviously safe ones. The highest-risk category is borderline prompts—those where the classifier has low confidence, where jailbreak attempts are most likely to succeed. The paper's conditional steering results (Table 15) show that applying SAE steering only when the prompt is classified as unsafe recovers most benchmark performance (MMLU 66.19% vs. 68.80% baseline, GSM8K 84.69% vs. 82.50%) while retaining 85.90% Unsafe Prompt Refusals on Wild Guard (up from 58.33% baseline). A production deployment could route borderline prompts through the steered model as an additional safety layer, while routing clearly-safe prompts through the unsteered model. The key advantage over purely classifier-based approaches is that the steering is attack-agnostic—it doesn't need to recognize the specific jailbreak technique, it just biases the model toward refusal—so it provides defense-in-depth against novel attacks that the classifier hasn't seen.

On-device safety for small models without server-side filtering. Phi-3 Mini is explicitly designed for on-device deployment (Abdin et al., 2024), where there is no server-side safety filter to catch harmful outputs. In this setting, the model itself must be the safety mechanism, and the paper's findings are directly relevant. At moderate clamp values (8-10), SAE steering improves Wild Guard Unsafe Prompt Refusals from 58.33% to approximately 80-90% (Figure 4), providing meaningful safety improvement for a model that would otherwise comply with ~42% of adversarial unsafe prompts. The capability cost at these clamp values (~58% MMLU at clamp=10, Table 2) may be acceptable for use cases where on-device privacy is prioritized and the alternative is a model with much weaker safety. The SAE itself is a static artifact that can be distributed alongside the model weights, and the steering requires only a single additional encode-decode pass at layer 6—adding modest latency that may be tolerable for on-device inference. This is not a production-ready solution (the capability cost at clamp=10 is still substantial), but it demonstrates a viable architectural pattern for on-device safety that does not depend on cloud-based classifiers.

When to Prefer This Method

The paper does not position SAE steering as categorically preferable to named alternatives, and the capability degradation findings preclude a simple "prefer SAE steering when X" recommendation for production deployment. However, the paper's results do support a conditional decision rule for researchers and practitioners evaluating safety interventions:

• Prefer SAE steering over PASTA for multi-turn safety: PASTA (toxicity-only profiling, 64 heads) increases Crescendo Attack Success Rate relative to baseline on most topics (Table 13), making it strictly worse than no intervention for multi-turn settings. SAE steering at clamp=10 and clamp=12 both reduce Crescendo ASR across all five topics (Table 1). If multi-turn robustness is the priority, SAE steering dominates PASTA among representation-level interventions.
• Prefer SAE steering over prompt-based defenses for token-level adversarial attacks (GCG): The GCG results (Table 4) show that mild SAE steering (clamp=4-6) reduces GCG Transfer ASR from 25.90% to 1.50% with minimal capability impact. Prompt-based defenses are notoriously vulnerable to GCG because the adversarial suffix is optimized to override textual instructions; SAE steering operates at the representation level and is harder to "argue with" via token perturbations.
• Prefer system prompting over SAE steering when capability preservation is paramount and multi-turn attacks are the primary threat: System prompting achieves a 29.36% Crescendo average Attack Success Rate (Table 1)—better than SAE steering at clamp=12 (32.58%)—while causing essentially zero capability degradation (Table 2). For applications where the cost of a wrong answer on a capability benchmark outweighs the marginal safety improvement from SAE steering, system prompting is the safer choice.
• SAE steering is not yet preferable to any intervention for general deployment because the clamp values needed for strong single-turn safety (10-12) cause unacceptably severe capability degradation (MMLU drops to 35.98% at clamp=12). Conditional steering narrows this gap but introduces dependence on a vulnerable external classifier. The method is currently a research tool for studying representation-level safety interventions, not a deployment-ready safety mechanism.