Linear Representations of Sentiment in Large Language Models

ArXiv: 2310.15154

🎯 Pitch

Large language models encode sentiment as a single linear direction in activation space, but surprisingly, nearly half of the sentiment-driven classification power comes not from emotionally charged words but from neutral punctuation like commas that act as 'summarization' checkpoints. Ablating this direction at comma positions alone wipes out 18% of zero-shot sentiment classification accuracy, revealing that the models actively compress and route emotional information through syntax before making predictions.

---

1. Executive Summary

This paper studies how large language models internally represent sentiment—a pervasive variable in natural language—using GPT-2 and Pythia models across templated toy tasks, OpenWebText, and the Stanford Sentiment Treebank (SST). The authors reveal that sentiment is encoded linearly as a single direction in activation space, and they identify a phenomenon they term the summarization motif, where sentiment information is aggregated at intermediate, emotionally neutral tokens such as commas and periods (e.g., after a valenced phrase like "hates parties," the following comma becomes a causally significant information bottleneck for downstream sentiment processing). Directional ablation at all comma positions in SST zero-shot classification eliminates 18% of accuracy—nearly half of the total sentiment-direction-mediated performance—while a full sentiment-direction ablation removes 76% of above-chance classification accuracy, establishing that summarized sentiment at function words is as causally important as the original semantically charged content. The paper further demonstrates that these linear representations generalize across models, languages, and tasks—from English toy datasets to French Harry Potter passages—but only exhibits this cross-lingual generalization in intermediate layers of larger models, establishing that the abstract sentiment concept emerges most prominently in middle layers rather than at the embedding or output extremes.

2. Context and Motivation

The Core Problem: We Don't Know How LLMs Represent Abstract Features Internally

Large language models have demonstrated remarkable capabilities across diverse tasks, but a fundamental question remains largely unanswered: how do these models internally represent abstract, high-level features of their input? Sentiment—the emotional valence of text—is a particularly instructive case study because it is pervasive across virtually all natural language domains (product reviews, news articles, dialogue, fiction) and requires models to abstract away from surface-level word co-occurrence statistics to infer an underlying variable of the data generation process. If we cannot understand how something as well-studied and seemingly simple as sentiment is represented, we have little hope of understanding more complex or safety-relevant features like truthfulness, deception, or harmful intent.

This gap is not merely an academic curiosity. It has direct practical implications across several axes:

• AI safety and alignment: If models develop internal representations of concepts that influence their outputs in opaque ways, we need tools to detect, interpret, and potentially intervene on those representations. A model that can internally represent "this text is deceptive" or "this request is harmful" might still produce harmful outputs if we cannot locate and understand that representation. The techniques developed for sentiment—a relatively benign testbed—could transfer to these higher-stakes features.

• Model debugging and improvement: Understanding internal representations enables targeted fixes. If a model exhibits unwanted sentiment biases (e.g., associating certain demographic groups with negative sentiment), knowing where and how that association is encoded enables precise interventions rather than crude retraining or prompt engineering.

• Scientific understanding of deep learning: The question of whether neural networks form interpretable, structured internal representations—or whether they operate as inscrutable black boxes—is one of the deepest theoretical questions in contemporary machine learning. Empirical evidence for structured representations in production-scale models provides constraints on theories of how deep learning works.

• Building more efficient systems: If features are represented linearly as directions in activation space, then manipulating those features becomes computationally cheap—add a vector to steer sentiment, subtract a vector to remove bias. This opens the door to lightweight, inference-time control mechanisms that don't require fine-tuning.

The Linear Representation Hypothesis and Prior Evidence

This paper is situated within a broader research program investigating the linear representation hypothesis—the conjecture that neural networks tend to represent abstract features of their inputs as directions in activation space (Mikolov et al., 2013; Elhage et al., 2022). The hypothesis is attractive because linear representations are mathematically tractable: if a feature corresponds to a direction, then the model's representation of that feature can be read off by projecting activations onto that direction, and the model's behavior can be modified by adding or subtracting vectors along that direction.

Prior to this work, several lines of evidence supported the linear representation hypothesis, but each had important limitations that this paper addresses:

Word embedding analogies (Mikolov et al., 2013) demonstrated that relationships between words are encoded as vector differences (e.g., king - man + woman ≈ queen), but this evidence was limited to static word embeddings rather than the contextual representations inside deep transformers. It was unclear whether the same linear structure would hold for context-dependent abstract features computed by multi-layer models processing full sentences.

The "sentiment neuron" in Radford et al. (2017) was a striking early finding: a single unit in an LSTM-based language model whose activation strongly correlated with the sentiment of generated text. However, this finding was limited to a single model architecture (LSTMs, not transformers), relied on a single interpretable unit rather than a distributed direction across many neurons, and did not establish whether the representation was causally significant (as opposed to merely correlational). Furthermore, the analysis did not explore how the model used this representation—whether it was directly read from sentiment-bearing words or processed through intermediate structures.

Emergent world models in Othello-GPT (Li et al., 2023; Nanda, 2023b) showed that a transformer trained to predict moves in the board game Othello developed an internal representation of the board state that was linearly decodable from intermediate layer activations. This was powerful evidence for linear representations of structured, rule-governed features, but the domain was synthetic and the "world state" was fully determined by the input sequence. Sentiment is a richer test case because it is a latent variable in natural language—not directly observable from the input tokens alone, but rather inferred from word choice, context, and pragmatic cues.

Linear representations of truth (Marks & Tegmark, 2023) found that LLMs encode the truth value of statements as directions in activation space. This finding parallel's the current paper's investigation of sentiment, but focused on a different abstract feature and used a different methodology (contrastive pairs of true/false statements). The current paper extends this line of inquiry by adding causal validation and mechanistic analysis of how the model constructs and uses the representation.

Dictionary learning approaches (Bricken et al., 2023) have attempted to decompose model activations into large sets of interpretable features using sparse autoencoders. This is a "bottom-up" approach: find many features first, then interpret them. The current paper takes a complementary "top-down" approach: start with a known, interpretable feature (sentiment), and then verify that the model represents it in a structured way. The advantage of the top-down approach is efficiency—it does not require training auxiliary models to decompose activations—and interpretability—the feature of interest is specified in advance, avoiding the problem of having to make sense of thousands of discovered features.

Where Prior Causal Analysis Approaches Fall Short

The paper builds on the circuits analysis framework (Olah et al., 2020; Elhage et al., 2021b; Wang et al., 2022), which aims to reverse-engineer the computational subgraphs within neural networks that are responsible for specific behaviors. Prior circuit analyses have primarily focused on relatively simple, syntactic tasks:

• Indirect Object Identification (Wang et al., 2022): Identifying which name in a sentence like "When Mary and John went to the store, John gave a drink to ___" receives the drink. This task involves tracking referents across clauses, but the relevant features are surface-level (which name appeared in which syntactic position).

• Greater-than computation (Hanna et al., 2023): Determining whether one number is greater than another. This involves numerical comparison but operates on explicitly provided inputs with deterministic answers.

• Docstring generation (Stefan Heimersheim, 2023): A more complex task, but still focused on structural patterns in code rather than abstract semantic features.

These prior circuit analyses have been invaluable for developing the toolkit of causal intervention methods (activation patching, path patching, ablation) and for demonstrating that transformer computations can be decomposed into interpretable components. However, they leave open a critical question: do the same methods work for abstract, semantic features like sentiment? Sentiment is different from syntactic role or numerical comparison in several ways:

1. Sentiment is not localized to a single token. In "I thought this movie was incredible, I loved it," sentiment is distributed across multiple words ("incredible," "loved"), and the model must aggregate this information to produce a coherent judgment.

2. Sentiment requires inference beyond surface form. Negation ("not bad"), sarcasm, and pragmatic implication can flip or modulate sentiment in ways that require compositional processing, not just keyword matching.

3. Sentiment is a latent variable of the data generation process. Unlike syntactic roles, which are directly annotated in many treebanks, sentiment is a property the model must infer from its training distribution without explicit supervision.

If circuits analysis methods fail for sentiment, it would suggest they are limited to shallow, syntactic phenomena and cannot scale to the abstract features that matter for model behavior in the wild. If they succeed, it provides evidence that the approach generalizes to semantically rich domains.

The Gap in Understanding How Information Flows Through Models

Beyond the question of whether sentiment is represented, there is the question of how the model constructs and routes that representation. A naive hypothesis—one that the paper explicitly identifies and refutes—is that sentiment information flows directly from valenced tokens (e.g., "incredible") to the final prediction token via attention, without intermediate processing. Under this hypothesis, the model's sentiment circuitry would be simple: attend to positive words, output positive continuations; attend to negative words, output negative continuations.

The paper's discovery of the summarization motif reveals that this naive view is wrong. Instead, the model aggregates sentiment information at intermediate, semantically neutral tokens—commas, periods, repeated nouns—creating information bottlenecks that are causally significant for downstream processing. This finding matters because:

• It reveals nontrivial internal structure. The model is not merely pattern-matching from input tokens to output tokens. It is actively constructing internal representations at specific positions that serve as "checkpoints" or "aggregation points" for higher-level features. This is a form of spontaneous, emergent abstraction—the model has learned to create internal "summary variables" without being explicitly trained to do so.

• It challenges assumptions about attention. Attention patterns are often interpreted as the model "looking back" at relevant input tokens. The summarization motif shows that causal information flow is more complex: information is written to specific positions, read from those positions by later components, and potentially rewritten again. The final prediction may depend as much on what was stored at a comma as on what was written at the original sentiment-bearing adjective.

• It suggests a general principle. If summarization occurs for sentiment, it may occur for other abstract features as well—perhaps the model aggregates information about entities, events, or relationships at syntactically convenient positions (end of clauses, punctuation, entity mentions). Identifying these aggregation points could be a key to understanding how models build and maintain internal world models over long contexts.

• It connects to the information bottleneck literature. Li et al. (2021) studied how models compress and route information through specific "bottleneck" representations. The summarization motif provides a concrete, causally validated instance of this phenomenon in a production-scale language model processing natural text, moving beyond the synthetic settings of prior work.

The Conflicting Picture from Prior Sentiment Analysis Work

The paper enters a literature where sentiment analysis is simultaneously one of the most well-studied NLP tasks and one where the internal mechanisms of neural models remain poorly understood. Traditional sentiment analysis research (Socher et al., 2013; Pang & Lee, 2008) focused on building systems that perform sentiment classification well, using architectures from recursive neural networks to fine-tuned BERT. This literature produced highly accurate models but did not ask how those models represent sentiment internally—it treated the model as a black box that maps input text to output labels.

More recent interpretability work on sentiment has been suggestive but incomplete:

• Goh et al. (2021) studied multimodal neurons in CLIP, finding individual neurons that responded to emotional concepts across text and images. This was evidence for abstract sentiment representations but did not establish whether these representations were used causally by the model or were merely correlational artifacts.

• Radford et al. (2017) found the aforementioned "sentiment neuron" in an LSTM language model. The finding was compelling but limited in architectural scope and did not explore the mechanistic role of the neuron within a broader circuit.

• The polarity shift literature (e.g., the "not" problem in sentiment analysis) had identified that models struggle with compositional sentiment phenomena like negation, but did not explain how models successfully process such phenomena when they do, or what internal machinery enables this.

The current paper's contribution is to unify these threads: it demonstrates that sentiment is represented linearly, that this representation is causal (not merely correlational), that it generalizes across models and languages, and that the circuitry involves a previously unrecognized summarization step. This goes substantially beyond prior work by providing a mechanistic account of how sentiment processing works, not just that it works.

How This Paper Positions Itself

The paper positions itself at the intersection of three research programs:

1. Mechanistic interpretability (Olah et al., 2020; Elhage et al., 2021b; Wang et al., 2022): Using causal intervention tools—activation patching, path patching, directional ablation—to identify computational subgraphs responsible for specific behaviors. The paper contributes by applying these tools to an abstract semantic feature (sentiment) rather than syntactic or arithmetic tasks, and by discovering the summarization motif as a novel circuit-level phenomenon.

2. The linear representation hypothesis (Mikolov et al., 2013; Elhage et al., 2022; Nanda et al., 2023b): Investigating whether models represent features as directions in activation space. The paper contributes causally validated evidence that sentiment follows this pattern, and demonstrates that even simple unsupervised methods (K-means, PCA) can recover the direction—suggesting the representation is robust and prominent, not a subtle signal requiring sophisticated extraction.

3. Representation engineering (Zou et al., 2023; Turner et al., 2023): Developing methods to detect and manipulate high-level concepts in model activations. The paper contributes techniques for finding feature directions with minimal supervision (toy datasets, few examples) and for validating them rigorously through causal interventions on out-of-distribution data.

Crucially, the paper frames its investigation of sentiment not as an end in itself but as a case study—a model for how to investigate the representation of any abstract feature in a large language model. The methods developed here (toy dataset construction, direction finding, correlational validation, causal validation, circuit analysis, summarization detection) are intended to transfer to other features. The paper explicitly invites this framing in its conclusion:

"We also see this research as a model for how to find and study the representation of a particular feature. Whereas in dictionary learning we enumerate a large set of features which we then need to interpret, here we start with an interpretable feature and subsequently verify that a representation of this feature exists in the model."

This top-down, feature-first approach contrasts with bottom-up dictionary learning and complements it—both are needed for a complete picture of model internals.

The Specific Gap This Paper Fills

In summary, the paper addresses the following specific gaps:

1. No causally validated linear representation of sentiment in transformer LMs had been demonstrated. Prior work showed correlational evidence (sentiment neuron) or linear structure in different domains (Othello, truth), but none had combined causal validation with linear representation finding for sentiment in modern decoder-only transformers.

2. The mechanistic circuitry for sentiment processing was unknown. Prior circuit analyses focused on syntactic and arithmetic tasks. The summarization motif—information aggregation at neutral punctuation and function words—was not predicted by any prior theory and represents a genuinely novel finding about how models organize internal computation.

3. No systematic methodology existed for finding and validating abstract feature representations. The paper demonstrates a complete pipeline—toy dataset construction, multi-method direction finding, correlational testing on diverse natural text, causal validation through directional patching, circuit analysis to understand usage, and generalization testing across models and languages—that serves as a template for future investigations.

4. The relationship between difficulty estimation, abstraction level, and layer depth was uncharacterized for sentiment. The finding that sentiment directions generalize best at intermediate layers (Section 3.3, Figure 6) provides evidence for the hypothesis that models form abstract concepts in middle layers, with early layers processing surface form and late layers specializing for prediction—a finding with implications for where to target interpretability interventions.

3. Technical Approach

3.1 Reader Orientation

This is primarily a mechanistic interpretability analysis paper — the authors are not proposing a new model architecture or training method, but rather developing and applying a toolkit of techniques to discover what representation a language model uses for sentiment and how that representation is constructed and used. The system being analyzed is a frozen, pretrained decoder-only transformer (GPT-2 or Pythia); the "system" the authors build is an experimental pipeline for finding a linear sentiment direction in activation space, validating it causally, and tracing the computational subgraph that reads and writes this direction. The core problem is that we don't know whether abstract features like sentiment are encoded in a structured, interpretable way inside large language models, and the shape of the solution is: (1) construct a minimal toy dataset that isolates the feature of interest, (2) use multiple independent methods to find a candidate direction in activation space, (3) validate the direction through both correlational analysis on diverse natural text and causal interventions that manipulate the direction and measure behavioral change, (4) trace the circuitry that writes to and reads from this direction to understand mechanistic usage, and (5) test generalization across models, languages, and datasets to establish whether the representation is robust or task-specific.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental pipeline has five major stages, each feeding into the next:

1. Toy Dataset Construction — The authors create two templated, minimal datasets (ToyMovieReview and ToyMoodStories) that isolate sentiment as the sole relevant feature, controlling for all other variables (syntax, length, topic). These datasets provide clean counterfactual pairs where only sentiment flips between positive and negative.

2. Direction Finding — Given the toy dataset, the authors extract activation vectors at specific token positions and layers from the frozen language model, then apply five independent methods (Mean Difference, K-means, Logistic Regression, Distributed Alignment Search, PCA) to each layer's residual stream to find a candidate sentiment direction — a unit vector $d$ such that projecting activations onto $d$ yields a scalar that correlates with the sentiment of the input.

3. Correlational Validation — Using the candidate direction, the authors project activations from diverse, out-of-distribution text (OpenWebText, French Harry Potter) onto the direction and verify that the resulting scalar (the "sentiment activation") tracks known sentiment-bearing tokens, flips under negation, and separates positive/negative/neutral words in the tails of the activation distribution.

4. Causal Validation — The authors use directional activation patching and directional ablation to test whether the candidate direction is causally necessary and sufficient for sentiment-driven behavior. They measure whether overwriting only the projection onto this direction (while leaving all other dimensions intact) flips model predictions on both the toy dataset and the Stanford Sentiment Treebank.

5. Circuit Analysis — The authors use path patching, attention pattern analysis, and ablation to trace the computational subgraph that writes sentiment information to the direction at specific token positions and reads from it at later positions, revealing the summarization motif and the specific attention heads involved.

Information flows through this pipeline linearly: the toy dataset provides labeled activation pairs → direction finding returns a candidate unit vector per layer → correlational tests validate that the direction tracks sentiment in the wild → causal tests validate that the direction is functionally significant → circuit analysis reveals the mechanistic implementation. At each stage, results from earlier stages constrain and guide later analysis (e.g., the layer where the direction is most causally effective determines where circuit analysis focuses).

3.3 Roadmap for the Deep Dive

• First, the toy dataset construction (Section 3.4.1), because the entire direction-finding pipeline depends on having clean, minimal counterfactual pairs where sentiment is the only varying feature. Understanding what makes these datasets "minimal" — and what confounds they control for — is essential for interpreting all downstream results.

• Second, the five direction-finding methods (Section 3.4.2), because they are the core technical contribution of Section 3: how do we operationalize "finding a sentiment direction" as a concrete computational procedure? Each method makes different assumptions and has different failure modes, so comparing their outputs is a key validation step.

• Third, directional activation patching and ablation (Section 3.4.3), because these are the primary causal tools used throughout the paper. Understanding the mechanics of these interventions — what exactly gets overwritten and what stays intact — is prerequisite to interpreting all causal results in Sections 3.3 and 4.

• Fourth, the correlational validation methodology (Section 3.4.4), because it connects the toy-trained direction to natural text and establishes that the direction is not an artifact of the templated dataset. This includes the GPT-4-based classification procedure and the negation analysis.

• Fifth, the Distributed Alignment Search training procedure (Section 3.4.5), because DAS is the most sophisticated direction-finding method and the one used for the strongest causal results. Understanding its objective function and optimization is important for interpreting its advantages and potential overfitting risks.

• Sixth, the circuit analysis methodology (Section 3.4.6), because it bridges Section 3 (finding and validating the direction) to Section 4 (understanding how the direction is used mechanistically). The path patching procedure and the iterative circuit tracing algorithm are explained here.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that sentiment is represented as a single linear direction in transformer activation space, and that this representation is constructed through a summarization mechanism where information is aggregated at intermediate, emotionally neutral tokens before being read by downstream components.

---

3.4.1 Toy Dataset Construction: Isolating Sentiment from Confounds

The foundation of the direction-finding pipeline is a pair of templated datasets that isolate sentiment as the only varying feature while holding all other linguistic variables constant. This controlled setting is essential because activation vectors in a language model encode many features simultaneously (due to superposition; Elhage et al., 2022); if the toy dataset varied syntax, topic, or length alongside sentiment, a direction-finding method might latch onto those confounds rather than sentiment itself.

ToyMovieReview dataset

The dataset consists of prompts following a fixed template:

I thought this movie was ADJECTIVE, I VERBed it. Conclusion: This movie is

where ADJECTIVE and VERB are either both positive or both negative. The expected completion is one of a set of positive descriptors (e.g., "great", "amazing", "awesome", "good", "perfect") for positive inputs, and one of a set of negative descriptors (e.g., "terrible", "awful", "bad", "horrible", "disgusting") for negative inputs.

The adjective pool contains 85 adjectives total, split 55/30 for train/test. The 55 training adjectives include 30 positive and 25 negative (the paper lists them explicitly in Appendix A.7):

• Positive training adjectives: "perfect", "fantastic", "delightful", "cheerful", "good", "remarkable", "satisfactory", "wonderful", "nice", "fabulous", "outstanding", "satisfying", "awesome", "exceptional", "adequate", "incredible", "extraordinary", "amazing", "decent", "lovely", "brilliant", "charming", "terrific", "superb", "spectacular", "great", "splendid", "beautiful", "positive", "excellent", "pleasant"
• Negative training adjectives: "dreadful", "bad", "dull", "depressing", "miserable", "tragic", "nasty", "inferior", "horrific", "terrible", "ugly", "disgusting", "disastrous", "annoying", "boring", "offensive", "frustrating", "wretched", "inadequate", "dire", "unpleasant", "horrible", "disappointing", "awful"

The test adjectives (30 total, not overlapping with training) include:

• Positive test adjectives: "stunning", "impressive", "admirable", "phenomenal", "radiant", "glorious", "magical", "pleasing", "lively", "warm", "strong", "helpful", "vivid", "modern", "crisp", "sweet"
• Negative test adjectives: "foul", "vile", "appalling", "rotten", "grim", "dismal", "lazy", "poor", "rough", "noisy", "sour", "flat", "ancient", "bitter"

The verb pool (8 verbs total, all used in both training and test adjective prompts):

• Positive verbs: "enjoyed", "loved", "liked", "appreciated", "admired"
• Negative verbs: "hated", "disliked", "despised"

The authors state that "the direction was not trained on any verbs" — all direction-finding methods use only adjective-position activations, reserving verb-position activations as an out-of-distribution hold-out set. This is a critical design choice: if a direction found using only adjective-position activations generalizes to verb positions, that is strong evidence that it captures sentiment as an abstract feature rather than lexical properties of specific adjective tokens.

ToyMoodStories dataset

This is a more complex dataset involving multiple subjects with conflicting preferences, requiring the model to track which sentiment applies to which character. The template is:

NAME1 VERB1 parties, and VERB2 them whenever possible. NAME2 VERB3 parties, and VERB4 them whenever possible. One day, they were invited to a grand gala. QUERYNAME feels very

where:

• VERB1 and VERB3 are drawn from {hates, loves} with opposite sentiments for the two characters
• VERB2 and VERB4 are drawn from {avoids, joins} such that each sentence's verbs agree in sentiment
• NAME1, NAME2, and QUERYNAME are drawn from a pool of 13 names: "John", "Anne", "Mark", "Mary", "Peter", "Paul", "James", "Sarah", "Mike", "Tom", "Carl", "Sam", "Jack"
• QUERYNAME matches either NAME1 or NAME2 with equal probability

The expected completion is "excited" if the queried character likes parties, and "nervous" if they hate parties. The model's output is evaluated by measuring the logit difference between the "excited" and "nervous" tokens.

Why these datasets? The authors choose templated datasets over natural text for direction finding because:

1. Controlled counterfactuals: By changing only the adjective/verb tokens while keeping everything else identical, the authors create clean positive/negative pairs where the only difference in the input is sentiment. Any difference in activations between these pairs must be attributable to sentiment (or to the specific lexical items, which is why hold-out adjectives and verbs are critical).

2. Known token positions: The sentiment-bearing tokens appear at fixed positions in the template. The authors can extract activations at the ADJ position (token index 6 in "I thought this movie was ADJECTIVE...") and the VRB position without needing to search for which tokens carry sentiment information.

3. Minimal size: The training set for K-means uses only 30 positive and 30 negative adjectives. The fact that this tiny dataset produces a direction that generalizes to diverse natural text is itself an important finding — it suggests the sentiment direction is a prominent, easily-discovered feature of the activation space, not a subtle signal requiring massive data to extract.

The authors explicitly note that these datasets evaluate the model's continuation behavior: "To evaluate the model's output, we measure the logit difference between the 'excited' and 'nervous' tokens" (for ToyMoodStories) or use "the logit difference between sets of positive/negative next-tokens" (for ToyMovieReview). This is important because it means the direction is found using next-token prediction behavior, not a separate classification head — it's probing the model's native language modeling capability.

3.4.2 Five Direction-Finding Methods: Formal Definitions and Design Choices

The paper applies five distinct methods to extract a candidate sentiment direction from the residual stream activations at the adjective position in the ToyMovieReview dataset. Each method operates on a set of positive inputs $P$ and negative inputs $N$, using the activation vector $a^L_x$ at layer $L$ above the adjective token for input $x$. The verb-position activations $v^L_x$ are reserved as a hold-out set for testing generalization.

All five methods produce a unit vector (or the paper normalizes the result to unit length) that represents a direction in the $d_{\text{model}}$-dimensional residual stream space. The cosine similarity between directions found by different methods is used as a validation metric: if all methods converge to approximately the same direction, it suggests they are all noisy approximations of a single underlying feature direction rather than capturing different aspects of the data.

Method 1: Mean Difference (MD)

The direction is computed as the normalized difference between the mean positive activation and the mean negative activation:

$d_{\text{MD}} = \frac{\frac{1}{|P|}\sum_{p \in P} a^L_p - \frac{1}{|N|}\sum_{n \in N} a^L_n}{\left\|\frac{1}{|P|}\sum_{p \in P} a^L_p - \frac{1}{|N|}\sum_{n \in N} a^L_n\right\|}$

where $|P|$ is the number of positive examples and $|N|$ is the number of negative examples.

What it computes: the direction in activation space that points from the centroid of negative-sentiment adjective activations to the centroid of positive-sentiment adjective activations. The numerator is the unnormalized vector connecting the two class centroids; the denominator normalizes it to unit length so that projecting onto this direction yields a scalar whose magnitude can be compared across layers and models.

Why this form: Mean difference is the simplest possible linear classifier direction — it is the weight vector that a linear discriminant would learn if the two classes were spherical Gaussians with equal covariance. It requires no optimization, no hyperparameters, and no iterative procedure. Its primary limitation is that it assumes the two classes are linearly separable by a hyperplane passing through the midpoint of their centroids, which may not hold if the positive and negative clusters have different shapes or if there are outliers. Despite this simplicity, the paper finds that it produces a direction with cosine similarity >0.87 to all other methods (Figure 2), suggesting the underlying representation is indeed well-approximated by this simple centroid difference.

Method 2: K-means (KM)

The authors fit 2-means clustering to the set of all adjective activations (both positive and negative) and take the direction connecting the two cluster centroids:

$d_{\text{KM}} = \frac{c_1 - c_0}{\|c_1 - c_0\|}$

where $c_0$ and $c_1$ are the two cluster centroids obtained by running the K-means algorithm (with $K = 2$) on $\{a^L_x : x \in P \cup N\}$.

What it computes: the direction separating the two main clusters in the activation space, discovered in a completely unsupervised way — the algorithm is given no labels indicating which activations are positive and which are negative. The algorithm iteratively assigns each activation to the nearest centroid, recomputes centroids as the mean of assigned points, and repeats until convergence (or a maximum number of iterations). The final direction connects the two converged centroids.

Why this form: K-means is an unsupervised method, meaning it does not use the sentiment labels at all. If the two clusters found by K-means correspond to positive and negative sentiment, it means the activation space naturally separates by sentiment without any label guidance — the sentiment feature is so prominent that it dominates the variance structure of the activations. This is stronger evidence for a "natural" representation than supervised methods because it eliminates the possibility that the method is simply memorizing label-noise correlations. The authors state that "the direction being examined here was trained on just 30 positive and 30 negative English adjectives in an unsupervised way (using K-means with $K = 2$)" (Section 3.1), emphasizing the minimal supervision. A limitation is that K-means is sensitive to initialization and may converge to a local optimum, though with $K = 2$ and well-separated clusters, this is unlikely to be a practical problem.

Method 3: Logistic Regression (LR)

The direction is the normalized weight vector of a linear classifier trained to distinguish positive from negative adjective activations:

$d_{\text{LR}} = \frac{w}{\|w\|}$

where $w$ is obtained by fitting a logistic regression model $\text{LR}(a^L_x) = \frac{1}{1 + \exp(-w \cdot a^L_x)}$ to labeled data $\{(a^L_x, y_x)\}$ with $y_x = 1$ for $x \in P$ and $y_x = 0$ for $x \in N$.

What it computes: the maximum-margin separating hyperplane direction for the binary classification problem. The logistic regression objective maximizes the likelihood of the observed labels under the assumption that the log-odds of positive sentiment are a linear function of the activation vector. The weight vector $w$ is the direction along which the activation must change to increase the predicted probability of positive sentiment. Normalizing to unit length removes the scale ambiguity.

Why this form: Logistic regression is a supervised method that explicitly optimizes for discriminability between the two classes. Unlike mean difference (which only uses first moments), logistic regression uses the full distribution of activations and can account for differences in covariance structure between classes (by adjusting the decision boundary away from the midpoint). The paper reports that LR achieves the highest classification accuracy on the toy dataset (89% on OpenWebText token classification; Table 3), which is expected since it is the only method explicitly optimized for classification. The potential downside is overfitting: with only 60 training examples (30 positive, 30 negative) in a $d_{\text{model}}$-dimensional space (768 dimensions for GPT-2 small), logistic regression could theoretically latch onto dimensions that are noise rather than signal. The high cosine similarity with unsupervised methods (>0.87) mitigates this concern.

Method 4: Distributed Alignment Search (DAS)

DAS (Geiger et al., 2023b) learns a direction by gradient descent optimization, where the objective is to maximize the model's logit difference between correct and incorrect sentiment completions after intervening on the learned direction. The direction $\theta$ is a learned parameter — a unit vector in activation space. The training objective is:

$\mathcal{L}_{\text{DAS}} = \sum_{x \in P} \left[\text{logit}_\theta(x; p) - \text{logit}_\theta(x; n)\right] + \sum_{x \in N} \left[\text{logit}_\theta(x; n) - \text{logit}_\theta(x; p)\right]$

where $\text{logit}_\theta(x; t)$ is the logit of token $t$ after applying the DAS intervention along direction $\theta$ to the activations at the adjective position. Specifically, for a pair of counterfactual inputs $(x_{\text{orig}}, x_{\text{flipped}})$ with opposite sentiment, the DAS intervention replaces the residual stream activation at the adjective position $a^L_{x_{\text{flipped}}}$ with a modified version where the projection onto $\theta$ is taken from $x_{\text{orig}}$ and all other dimensions are left unchanged:

$a^L_{\text{intervened}} = a^L_{x_{\text{flipped}}} - (a^L_{x_{\text{flipped}}} \cdot \theta)\theta + (a^L_{x_{\text{orig}}} \cdot \theta)\theta$

where $(a^L \cdot \theta)\theta$ is the projection of $a^L$ onto $\theta$ — that is, the component of the residual stream vector along the sentiment direction.

What it computes: the direction $\theta$ such that intervening only along this direction (replacing its value from a positive input with its value from a negative input, or vice versa) maximally flips the model's sentiment prediction. The loss function sums over all positive examples $x \in P$ the difference between the logit of the positive completion $p$ and the logit of the negative completion $n$ after the intervention, and symmetrically for negative examples. The logit difference is large and positive when the model strongly prefers the correct-sentiment token over the incorrect-sentiment token, so maximizing this difference means the model's output after intervention should align with the sentiment of $x_{\text{orig}}$ (the "donor" of the sentiment direction value), not $x_{\text{flipped}}$ (the "host" prompt).

Why this form: DAS is unique among the five methods in that it directly optimizes for causal efficacy — the direction is chosen to maximize downstream behavioral change when manipulated, not merely to separate activations in representation space. This makes it the gold standard for causal validation: a direction found by DAS is, by construction, causally relevant to the model's sentiment-driven outputs. The cost is that DAS requires running forward passes through the model during training (to compute logits after intervention), making it computationally more expensive than the other methods. Additionally, because DAS optimizes on the training set, it risks overfitting — finding a direction that works on the training adjectives but doesn't generalize. The paper mitigates this by evaluating on hold-out adjectives and verbs, and by comparing DAS directions to those found by other methods (Figure 2 shows cosine similarity >0.95 between DAS and LR/K-means).

A crucial detail about the DAS intervention: it modifies the activation along the learned direction $\theta$ at the adjective position, and the intervention "propagates" through the rest of the model's forward pass (the subsequent layers process the modified activation as if it were naturally occurring). This means the learned direction must be compatible with the model's downstream processing — it can't simply encode sentiment in an arbitrary format, but must encode it in a way that later layers can interpret. This constraint likely contributes to the convergence across methods: there may be only one direction that is both linearly separable at the adjective position and causally interpretable by later layers.

Method 5: Principal Component Analysis (PCA)

The direction is the first principal component of the set of all adjective activations (both positive and negative):

$d_{\text{PCA}} = \text{PCA}_1\left(\{a^L_x : x \in P \cup N\}\right)$

where $\text{PCA}_1$ is the eigenvector corresponding to the largest eigenvalue of the covariance matrix of the centered activations.

What it computes: the direction of maximum variance in the adjective activation space. PCA finds the orthogonal transformation that diagonalizes the covariance matrix; the first principal component is the direction along which the activations vary most. Because sentiment is the primary feature varying across the toy dataset examples (all other words in the template are identical), the largest source of variance should be sentiment itself.

Why this form: PCA is unsupervised (no labels needed) and makes minimal assumptions — it simply finds the direction that explains the most variance. If the sentiment signal is large relative to other sources of variance (token identity noise, positional effects), PCA will capture it. The paper reports that PCA achieves 81% accuracy on OpenWebText token classification (Figure 3, right panel), which is lower than LR (89%) but still substantially above chance, confirming that sentiment is the dominant variance component. The limitation of PCA is that maximum variance does not necessarily equal maximum discriminability: if there is a large-variance direction that is orthogonal to sentiment (e.g., related to adjective frequency or length), PCA might pick it up instead. The high cosine similarity with the other methods (>0.79) suggests this is not a problem in practice for this dataset.

Cosine similarity across methods

Figure 2 in the paper presents a matrix of cosine similarities between directions found by all five methods plus a random baseline direction. All methods achieve cosine similarities above 0.72 with each other, with many comparisons exceeding 0.95 (e.g., K-means vs. PCA at 0.991, LR vs. MD at 0.946). The random direction has cosine similarity below 0.025 with all methods. The paper interprets this as evidence that "these are all noisy approximations of the same singular direction" (Section 3.1), supporting the claim that sentiment is represented as a single, prominent direction rather than a multidimensional subspace.

The authors note that "the directions we found were not sparse vectors, as expected since the residual stream is not a privileged basis" (Section 3.1). The residual stream in a transformer does not have a natural coordinate system — the standard neuron basis is an artifact of the architecture, not a meaningful feature basis. A feature direction can point in any direction in the $d_{\text{model}}$-dimensional space, so it is expected that the sentiment direction has non-zero components along many neurons (i.e., is dense). This is in contrast to the "sentiment neuron" finding of Radford et al. (2017), where a single unit captured sentiment — the dense direction in the residual stream likely decomposes into contributions from many neurons, with the subset most aligned with the sentiment direction being those analyzed in Appendix A.5.

3.4.3 Causal Intervention Methods: Directional Patching and Ablation

The paper uses two primary causal intervention techniques to establish that the sentiment direction is not merely correlational but functionally significant for the model's behavior. Both techniques are variants of activation patching (Vig et al., 2020; Geiger et al., 2021), adapted to operate on a specific direction rather than on whole activations or individual neurons.

Directional Activation Patching

Standard activation patching creates two counterfactual inputs: a "clean" input $x_{\text{orig}}$ (e.g., with positive sentiment) and a "flipped" input $x_{\text{flipped}}$ (e.g., with negative sentiment). The model is run on $x_{\text{flipped}}$, but a specific activation (e.g., the output of a particular attention head at a particular layer and position) is replaced with its value from the forward pass on $x_{\text{orig}}$. If this intervention shifts the model's output toward the $x_{\text{orig}}$ behavior, the patched component is causally relevant to the task.

Directional patching modifies this procedure by only replacing the activation along a single direction $d$ (the candidate sentiment direction), leaving all other dimensions of the activation vector unchanged. For a component $C$ (e.g., residual stream at a specific layer and position), the intervened activation is:

$o_{\text{intervened}} = o_{\text{flipped}} - (o_{\text{flipped}} \cdot d)d + (o_{\text{orig}} \cdot d)d$

where $o_{\text{flipped}}$ is the original activation when running on $x_{\text{flipped}}$, $o_{\text{orig}}$ is the activation from the run on $x_{\text{orig}}$, and $(o \cdot d)d$ is the vector projection of $o$ onto the direction $d$.

What it computes: The scalar projection $o \cdot d$ measures how much of the activation lies along the sentiment direction — this is the "sentiment activation" value. The intervention subtracts the flipped input's sentiment activation and adds the original input's sentiment activation, effectively transplanting the sentiment signal from the positive input into the negative input (or vice versa) while leaving all other features encoded in the activation intact. The operation preserves the dimensionality of the original activation vector: only one degree of freedom is changed.

Why this form: The key motivation is specificity. Standard activation patching overwrites all features encoded in a component, making it impossible to attribute behavioral changes specifically to sentiment rather than to other features that happen to be correlated with sentiment in the activations. Directional patching isolates the causal effect of the sentiment feature: if the model's prediction flips when only the sentiment-projection is swapped, then the sentiment direction carries causally relevant information. If the model's prediction does not flip, then either the direction does not encode sentiment, or the model does not use it for this task. The authors validate the specificity by also patching along random directions, which produces negligible behavioral change (<1% effect on metrics).

The authors evaluate directional patching using two metrics:

1. Logit difference change: For ToyMovieReview, the logit difference is computed as $\frac{1}{n}\sum_{i=1}^n [\text{logit}(t^{\text{positive}}_i) - \text{logit}(t^{\text{negative}}_i)]$ where $T^{\text{positive}}$ and $T^{\text{negative}}$ are sets of positive and negative answer tokens (5 each). For SST, it is the difference between the "Positive" and "Negative" logits. Patching from a negative to positive input should increase this difference (make it more positive); patching from positive to negative should decrease it. The paper reports the percentage change toward the target value.

2. Logit flip percentage: The fraction of examples where the highest-probability sentiment token flips from positive to negative (or vice versa) after the intervention. This is a stricter metric than logit difference change because it requires the effect to be large enough to cross the decision boundary.

For SST, the authors scaffold the prompt as "Review Text: TEXT, Review Sentiment:" and evaluate the logit difference between "Positive" and "Negative". They restrict to the test partition, collapse five-way labels to binary (Positive/Negative), keep only phrase-level examples, and further restrict to a subset where pythia-1.4b achieves 100% zero-shot classification accuracy on clean examples (removing 17% of examples). Pairs are matched by token count to ensure sentiment tokens appear at similar positions across paired examples. This careful subsetting ensures that any failure to flip the prediction is due to the insufficiency of the directional patch, not to the model being incapable of the task on those examples.

Directional Ablation

Directional ablation is the complement of directional patching: instead of swapping the sentiment activation between inputs, it removes the sentiment information entirely by zeroing out the projection onto the sentiment direction:

$o_{\text{ablated}} = o_{\text{original}} - (o_{\text{original}} \cdot d)d$

What it computes: The residual after subtracting the projection onto $d$ — that is, the component of the activation that is orthogonal to the sentiment direction. This operation removes all information encoded along the sentiment axis while preserving information in all other directions.

Why this form: Directional ablation tests necessity: if removing the sentiment component degrades task performance, then the sentiment direction is necessary for the task. Standard ablation (zeroing or mean-ablating the entire activation) tests whether the component as a whole is necessary, but cannot attribute necessity to sentiment specifically. By comparing the performance drop from directional ablation to the drop from full ablation, the authors can estimate what fraction of the component's contribution is mediated by sentiment. The authors perform directional ablation at all comma positions in SST (Section 4.3), finding an 18% accuracy drop — nearly half of the 38% drop from directional ablation at all tokens (76% reduction in logit difference). This quantifies that sentiment information at commas accounts for roughly half of all sentiment-direction-mediated performance.

Mean ablation at commas (for comparison)

As a separate baseline in Section 4.3, the authors perform "mean ablation" at all comma positions: replacing each comma's activation vector with the mean comma activation computed across the entire SST dataset. This is a full-vector ablation (not direction-specific) and removes both sentiment and non-sentiment information at commas. The fact that this produces a 17% logit difference drop — comparable to the 18% from directional ablation — is interpreted as evidence that sentiment information is the primary causally relevant feature stored at comma positions (if commas also stored other task-relevant information, full-vector ablation would cause a larger drop than directional ablation).

Activation Addition (Steering)

As an additional causal validation (Appendix A.1.2), the authors use activation addition (Turner et al., 2023): during autoregressive generation, a multiple of the sentiment direction is added to the residual stream at the first layer after every token:

$a^0_x \leftarrow a^0_x + \alpha \cdot d$

where $\alpha$ is the "steering coefficient" and $d$ is the sentiment direction. Positive $\alpha$ should steer toward positive completions; negative $\alpha$ should steer toward negative completions.

The authors test this starting from a positive prompt ("I really enjoyed the movie, in fact I loved it. I thought the movie was just very...") and add increasingly negative multiples of the sentiment direction. At $\alpha = 0$, the model generates positive completions. As $\alpha$ becomes increasingly negative (−5, −10, −17), the completions become progressively more negative, and at $\alpha = -17$, the completions are "extremely negative" while "the coherence of the model's generated text" is not completely destroyed. The smoothness of the transition and preservation of fluency are cited as evidence that the intervention is not taking activations pathologically far out of distribution — the model interprets the added vector as a natural sentiment signal and integrates it into its generation process.

3.4.4 Correlational Validation Methodology

The paper validates the candidate sentiment directions through two correlational analyses on natural, out-of-distribution text, using GPT-2 small's first residual stream layer and the K-means sentiment direction unless otherwise specified.

OpenWebText Token Sentiment Classification (Section 3.2, Figure 3)

The authors bin the sentiment activations (projections onto the sentiment direction) of all OpenWebText tokens into 20 equal-width buckets and sample 20 tokens from each bucket. They then ask GPT-4 to classify each token as Positive, Neutral, or Negative given the token and 20 tokens of surrounding context.

The GPT-4 prompt format:

Your job is to classify the sentiment of a given token (i.e. word or word fragment) into Positive/Neutral/Negative. Token: '{token}'. Context: '{context}'. Sentiment:

where {context} is the 20-token window centered on the sampled token, and {token} is the specific subword token being classified.

The paper notes that "only a cursory human sanity check was performed" on the GPT-4 classifications, which is a limitation — GPT-4's sentiment judgments serve as a proxy for ground-truth sentiment but may have systematic errors.

The results (Figure 3) show an area plot of cumulative label proportions by activation bucket. The left tail of the activation distribution (most negative projections) is dominated by the "Negative" label, the right tail (most positive projections) is dominated by "Positive", and the central region is dominated by "Neutral." The authors interpret this as evidence that "the tails of the activations seem highly interpretable as representing a bipolar sentiment feature" while "the large space in the middle of the distribution simply occupied by neutral words (rather than a more continuous degradation of positive/negative) indicates superposition of features" (Section 3.2).

The accuracy for classifying tokens as positive or negative (at the extreme 0.1% tails of the activation distribution) ranges from 72.6% (K-means) to 89% (LR), with the random baseline at essentially chance (2.4%).

Negation Flipping (Section 3.2, Figure 5 and Appendix A.1.4)

The authors construct a dataset of 27 negation examples (e.g., "You never fail. Don't doubt it. I don't like you.") and measure the sentiment activation at the negated token (e.g., "fail", "doubt") across layers. The key finding is that the sentiment activation flips sign between early and late layers for words whose surface sentiment is contradicted by negation context:

• At layer 1, "fail" has a negative sentiment activation (its literal meaning).
• By layer 10, "fail" in "You never fail" has flipped to a positive sentiment activation (its contextual meaning after negation).

The paper reports that for K-means, 96% of the examined tokens flip sentiment, with a median flip magnitude of 69% of the mean activation range. The logit lens visualization (Figure A.5) shows this flip as a color transition from red (negative) to blue (positive) across layers for the relevant tokens.

This analysis demonstrates that the sentiment direction captures contextual sentiment, not just static lexical sentiment — it reflects the model's compositional processing of how negation interacts with word meaning.

Multi-lingual Generalization (Appendix A.1.3, Figure A.4)

The authors project the first paragraphs of Harry Potter in both English and French onto sentiment directions found using the English toy dataset. They find that "intermediate layers of pythia-2.8b demonstrate intuitive sentiment activations for the French text" — that is, the same direction found from English adjectives produces sentiment-appropriate projections on French text, despite the model's poor French tokenization (the model was not trained for French and tokenizes French words into subword fragments). The representation "was not evident in the first couple of layers, probably due to the poor tokenization of French words." This is interpreted as evidence that abstract sentiment representations emerge in intermediate layers and can generalize across languages, at least for languages with shared semantic structure.

3.4.5 Distributed Alignment Search Training Details

DAS (Geiger et al., 2023b) is the most sophisticated direction-finding method used in the paper and warrants a detailed explanation of its training procedure.

The original DAS method learns a full rotation matrix $R$ (an orthonormal change of basis) where the first dimension of the rotated space is aligned with a target causal variable. The paper uses a special case: rather than learning a full rotation, it learns a single direction $\theta$ (the first column of the rotation matrix) and patches along that direction.

Training proceeds as follows:

1. Initialize the direction $\theta$ as a random unit vector in $\mathbb{R}^{d_{\text{model}}}$.

2. For each training example: Create a counterfactual pair $(x_{\text{orig}}, x_{\text{flipped}})$ from the ToyMovieReview dataset — same template, adjective with opposite sentiment. For a positive $x_{\text{orig}}$ with adjective "incredible" and negative $x_{\text{flipped}}$ with adjective "terrible", the expected behavior after the intervention is that the output on $x_{\text{flipped}}$ should shift toward the positive sentiment tokens.

3. Forward pass on $x_{\text{flipped}}$: Run the model normally, capturing the residual stream activation at the adjective position at the target layer: $a^L_{x_{\text{flipped}}}$.

4. Forward pass on $x_{\text{orig}}$: Run the model normally, capturing $a^L_{x_{\text{orig}}}$.

5. Apply the directional intervention: Replace the activation at the adjective position in the forward pass on $x_{\text{flipped}}$ with:

$a^L_{\text{intervened}} = a^L_{x_{\text{flipped}}} - (a^L_{x_{\text{flipped}}} \cdot \theta)\theta + (a^L_{x_{\text{orig}}} \cdot \theta)\theta$

This operation preserves all dimensions of $a^L_{x_{\text{flipped}}}$ except the projection onto $\theta$, which is taken from $a^L_{x_{\text{orig}}}$.

6. Continue the forward pass: The modified activation replaces the original at the adjective position, and the forward pass continues through all subsequent layers of the model as if this were a natural activation.

7. Compute the loss: The loss is the negative of the sum of logit differences (or, equivalently, the objective is to maximize the logit difference):

$\mathcal{L}_{\text{DAS}} = -\sum_{x \in P} \left[\text{logit}_\theta(x; p) - \text{logit}_\theta(x; n)\right] - \sum_{x \in N} \left[\text{logit}_\theta(x; n) - \text{logit}_\theta(x; p)\right]$

where $\text{logit}_\theta(x; t)$ is the logit of token $t$ in the intervened forward pass. For $x \in P$ (positive original), the ideal outcome is that $\text{logit}(p) \gg \text{logit}(n)$ after the intervention, so $\text{logit}(p) - \text{logit}(n)$ should be large and positive, contributing negatively to the loss. For $x \in N$, the ideal is large $\text{logit}(n) - \text{logit}(p)$.

8. Update $\theta$ via gradient descent: $\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}_{\text{DAS}}$, then re-normalize to unit length.

The gradient flows through the entire model — from the logit difference at the output, backward through all subsequent layers, to the intervened activation, and then to $\theta$. This means the optimization finds a direction that is not only linearly separable at the adjective position, but also compatible with the downstream processing: a direction that, when its value is swapped, causes the subsequent layers to produce the desired output change.

Dimensionality sweep (Appendix A.2, Figure A.6): The authors also experiment with multi-dimensional DAS, where instead of learning a single direction, they learn a $k$-dimensional subspace (where $k = 2^n - 1$ for integer $n$). They find that increasing the DAS dimension improves the in-sample patching metric (training loss decreases; Figure A.6a) but does not improve out-of-distribution generalization (validation loss remains flat; Figure A.6b). This suggests that the sentiment feature can be captured by a single direction; additional dimensions overfit to training-set-specific variance. The paper interprets this as evidence that "sentiment really is a hyperplane" (Appendix A.2 title) — that is, a one-dimensional subspace, not a higher-dimensional manifold.

3.4.6 Circuit Analysis Methodology: Path Patching and Iterative Circuit Tracing

The circuit analysis in Section 4 follows the methodology pioneered by Wang et al. (2022) for the Indirect Object Identification circuit in GPT-2 small. The goal is to identify a computational subgraph — a set of attention heads and their connections — that is responsible for the model's sentiment processing behavior, and to understand the mechanistic role of each component in this subgraph.

Path Patching

Path patching is a more surgical variant of activation patching that isolates specific paths through the model's computational graph. In a transformer, the residual stream at layer $L$ is the sum of contributions from all previous components (attention heads and MLPs at layers $0$ through $L-1$). Standard activation patching replaces the entire residual stream, which conflates the effects of all upstream components. Path patching instead patches only the contribution of a specific upstream component to a specific downstream component.

The procedure for path patching from component $A$ (source) to component $B$ (receiver):

1. Run the model on $x_{\text{flipped}}$ and capture all intermediate activations.
2. Run the model on $x_{\text{orig}}$ and capture the output of component $A$ at the relevant position.
3. Run the model on $x_{\text{flipped}}$ again, but when computing the input to component $B$, replace the contribution from component $A$ (from the $x_{\text{flipped}}$ run) with the contribution from component $A$ (from the $x_{\text{orig}}$ run).
4. Continue the forward pass and measure the change in logit difference.

This isolates the causal effect of the specific edge $A \rightarrow B$ in the computational graph.

Iterative Circuit Tracing Algorithm

The authors follow a recursive, iterative process:

1. Identify direct-effect heads: Patch the residual stream at the END position (the final token, where the prediction is made) from $x_{\text{orig}}$ to $x_{\text{flipped}}$. Measure which layers' residual stream contributions at END are most important for the logit difference. Then decompose the residual stream at those layers into attention head outputs and MLP outputs; identify the specific attention heads whose patching causes the largest logit difference change. These are the "direct effect heads" — heads that write directly to the final residual stream in a way that affects the sentiment prediction.

2. Examine attention patterns: For each identified head, visualize its attention pattern (value-weighted, using the norm of the value vector as weight; Kobayashi et al., 2020) to understand which source token positions it attends to. This provides hypotheses about where the head is reading information from. Value-weighted attention is preferred over raw attention probabilities because it filters for positions where the attention head is actually moving significant information (a head might attend to a position with high probability but write a near-zero value vector, contributing nothing).

3. Path-patch upstream: Set the identified direct-effect heads as receivers and path-patch from all possible upstream components (attention heads and MLPs at earlier layers) as sources. Identify which upstream components contribute causally significant information to the direct-effect heads. These become the next level of the circuit.

4. Repeat recursively: For each newly identified upstream component, examine its attention patterns to hypothesize its function, identify sources that contribute to it, and continue tracing backward through the model.

5. Validate the circuit: Once a candidate circuit is identified, patch the entire circuit simultaneously (all identified components at all relevant positions) and measure whether this accounts for a large fraction of the model's behavior. The paper reports that patching the full sentiment circuit at ADJ, VRB, and SUM positions achieves 97% logit flips and a 75% logit difference drop. Patching the circuit along only the sentiment direction (directional patching) achieves 58.3% logit flips and a 54.8% logit difference drop, showing that the sentiment direction accounts for the majority of the circuit's function but not all of it — other features are also at play.

Component Roles in the GPT-2 ToyMovieReview Circuit

The identified circuit (Figure 7) contains 9 attention heads organized into three functional groups:

• Sentiment summarizers (heads 7.1 and 7.5): These heads attend from the SUM position (the second "movie" token at position 17 in the template) to the ADJ and VRB positions. They write sentiment information to the SUM position, aggregating it from the sentiment-bearing words. The output of these heads is causally significant only at the SUM position — path-patching their output at other positions does not affect the logit difference.

• Summary readers (head 8.5; secondarily 9.10): These heads attend from the END position primarily to the SUM position, reading the aggregated sentiment information and passing it toward the final prediction. Head 9.10 is included as a secondary summary reader when the threshold is lowered.

• Direct sentiment readers (heads 9.2, 10.1, 10.4, 11.9): These heads attend from the END position to the ADJ and VRB positions, reading sentiment information directly from the source tokens. Head 9.2 also reads from SUM. Head 6.4 is included as an additional sentiment summarizer when the threshold is lowered.

The finding that both direct reading and summarization-based reading coexist in the same circuit is important: the model does not replace direct reading with summarization, but augments it. The summarization at SUM provides a consolidated representation that can be read efficiently by downstream heads.

Component Roles in the Pythia-2.8B ToyMoodStories Circuit

For the multi-subject task, the circuit is more complex due to the need to track which character has which preference. Key components identified (the authors explicitly note they "did not attempt to reverse-engineer the entire circuit"):

• Direct effect heads (17.19, 22.5, 14.4, 20.10, 12.2): These heads attend from END to the repeated name token (RNAME, e.g., "John" in "John feels very") and the "feels" token (FEEL), and write output in the direction of the logit difference. Head 22.5 attends almost exclusively to FEEL and does not show comma-summarization reading behavior. Head 17.19 does not attend significantly to commas but attends to periods at the end of preference sentences in addition to RNAME and FEEL.

• Name summary writers (12.17, and others): These heads attend to the comma at the end of the queried character's preference phrase (COMMASUM) and write to RNAME and FEEL. Head 12.17 is "by far the most important" in this role.

• Multi-functional heads: Many of the direct effect heads serve dual roles: they both read from COMMASUM (writing summaries to RNAME/FEEL) and read from RNAME/FEEL (writing direct effects to END). This is possible because these heads exist at different layers — a head at layer 20 can read the summarized information written by a head at layer 12, while also writing its own contribution for even later heads to read.

The authors explicitly note that they "did not flesh out" the circuitry writing to COMMASUM in full detail, leaving deeper investigation of the upstream sources to future work.

Freezing Attention Patterns

In Section 4.2, the authors use a specialized technique for the comma patching experiments: they "froze the model's attention patterns to ensure the model used the information from the patched commas in exactly the same way as it would have used the original information." Without this step, the model could dynamically re-route its attention to avoid the patched commas and attend directly to the preference words instead, masking the importance of the comma representations. Freezing attention patterns means that during the forward pass with patched comma values, the attention weights are computed normally but then overridden with the attention weights from the unpatched run. This forces the model to process the patched comma information through the same downstream pathways, enabling a clean measurement of the causal effect of changing comma content without changing attention routing.

The paper does not specify the exact implementation of attention freezing (whether it is applied to all heads or only those identified as part of the circuit), which is a minor methodological gap.

4. Key Insights and Innovations

Innovation 1: A Causally Validated Linear Representation of an Abstract Semantic Feature in Production Transformers

The paper's most fundamental conceptual contribution is demonstrating that a single direction in transformer activation space encodes a genuinely abstract, context-dependent semantic feature — sentiment — and that this representation is not merely correlational but causally necessary and sufficient for downstream behavior. This may sound incremental given the "sentiment neuron" of Radford et al. (2017) and the linear representation hypothesis of Mikolov et al. (2013), but the paper makes a qualitative leap in what has been established.

Prior to this work, the evidence for linear representations of abstract features in language models was fragmentary and had important limitations. The "sentiment neuron" was a single unit in an LSTM, not a distributed direction across many neurons in a transformer. The Othello-GPT findings (Li et al., 2023; Nanda, 2023b) demonstrated linear world models, but in a synthetic, fully-observable game environment where the "feature" (board state) was directly computable from the input sequence. Marks & Tegmark (2023) found linear truth representations, but did not provide the same depth of causal and mechanistic validation. What was missing was a demonstration that a latent semantic variable — one that must be inferred from compositional linguistic cues, not read off the input directly — is encoded linearly in a production-scale transformer, and that this encoding is used by the model to drive behavior.

The paper fills this gap through a methodological innovation that combines multiple lines of evidence: convergence across five independent direction-finding methods (Figure 2: cosine similarities >0.72, mostly >0.87), correlational validation on out-of-distribution natural text including a different language (Figure 1d: French Harry Potter), causal validation through directional patching that flips model predictions on both toy data (110.4% logit difference shift for DAS, Figure 4) and the Stanford Sentiment Treebank (53.5% logit flip rate), and mechanistic circuit analysis showing that the direction mediates component communication (58.3% of circuit logit flips attributable to the sentiment direction alone, Section 4.1). No single one of these would be sufficient; their convergence makes the case compelling.

What makes this a fundamental contribution rather than incremental is that it changes the default assumption for future interpretability work. Before this paper, one might reasonably ask: "Does a transformer encode sentiment as a linear direction, or as a more complex, distributed, nonlinear representation?" After this paper, the reasonable prior is: "Sentiment is likely encoded linearly; let me verify with causal methods." This shifts the burden of proof — from the linear-representation advocate needing to prove linearity, to the skeptic needing to demonstrate that a particular feature is not linear. The paper's demonstration that even K-means with 60 examples finds the direction (Figure 2, K-means vs. DAS cosine similarity 0.999) further suggests that the linear structure is prominent and robust, not a subtle signal requiring sophisticated extraction — an empirical finding about the geometry of activation space that was not obvious ex ante.

A subtle but important conceptual move is the paper's insistence on causal validation over mere decodability. Many prior works (including probing-based analyses) showed that features could be read from activations, but did not establish that the model uses those features. The directional patching experiments (Figures 4, 6) close this gap by showing that manipulating only the sentiment-projection — leaving all other activation dimensions intact — causally shifts model behavior. The comparison to random-direction patching (<1% effect) confirms specificity. This causal standard raises the bar for the field: future claims about "the model represents X" should ideally be accompanied by evidence that the representation is functionally significant, not merely decodable by a learned probe.

The finding that the sentiment direction generalizes best at intermediate layers (Figure 6: SST patching effectiveness peaks at layers 5–15 across models, not at layer 0 or the final layer) is a non-obvious empirical result that carries theoretical implications. It suggests that abstract semantic concepts crystallize in middle layers of transformers, with early layers processing surface lexical properties and late layers specializing for token prediction. This provides an architectural principle for where to target interpretability interventions — look for abstract features in the middle, not the extremes — and is consistent with the broader hypothesis that transformers form increasingly abstract representations as depth increases, before collapsing to task-specific formats at the output.

Innovation 2: The Summarization Motif as a New Information-Processing Primitive in Language Models

The discovery of the "summarization motif" — the model's habit of aggregating sentiment information at semantically neutral intermediate tokens (commas, periods, repeated nouns) and then reading from those aggregation points at the final prediction — is the paper's most genuinely novel architectural insight. This is not a refinement of an existing concept; it is a new phenomenon that was not predicted by any prior theory of transformer computation and that, once seen, reframes how one thinks about information flow in these models.

The naive, pre-paper assumption about how sentiment processing works in a language model would be something like: attention heads at the final token attend directly to sentiment-bearing words (adjectives, verbs), and their output drives the prediction. This is the "direct reading" pathway that the paper does indeed find (heads 9.2, 10.1, 10.4 in Figure 7). But the additional existence of a parallel summarization pathway — sentiment information written to and read from a semantically empty token like the second "movie" or the comma after a preference phrase — reveals a more sophisticated computational architecture. The model is not just retrieving information; it is actively constructing intermediate representations at specific positions that serve as information consolidation points.

Why is this a fundamental insight rather than a curiosity about commas? Because it reveals a general principle about how transformers may manage information over sequences. The summarization motif can be understood as an emergent, learned analogue of the explicit [CLS] token in BERT (Devlin et al., 2018) — a designated position where sentence-level information is pooled. But unlike [CLS], which was architecturally mandated by the training objective (next-sentence prediction required a pooled representation), the summarization positions in these autoregressive models are spontaneously chosen by the model during training. The model has learned, without explicit supervision, that punctuation positions — particularly those at clause boundaries — are convenient locations to store aggregated information for later retrieval. This is a form of emergent, self-organizing architecture.

The paper's evidence that summarization is not a marginal effect but a primary information pathway is striking. In the SST experiments (Section 4.3), directional ablation at all comma positions eliminates 18% of zero-shot classification accuracy — nearly half of the total 38% accuracy drop from full sentiment-direction ablation at all positions. In the ToyMoodStories experiments (Table 1a), patching pre-comma phrase values while freezing commas has a _75% logit difference drop, while patching only the two commas has a _37% drop — nearly half the effect of the full phrase. These numbers suggest that the summarized representations at punctuation are roughly as important as the original semantic content for downstream performance, a result that would be shocking under the naive "direct reading" model but is natural under the summarization framework.

The finding that summarization importance increases with distance (Table 1b: when irrelevant text is injected after preference phrases to increase the gap between source tokens and the query, the ratio of period importance to phrase importance grows from 0.29 at 0 tokens of distance to 1.15 at 22 tokens) hints at a functional role for summarization: it may serve as a compression mechanism that allows the model to maintain access to information from earlier in the context without needing to attend back across long distances. If summarization stores a compressed representation of "John's sentiment is positive" at a comma immediately after the relevant clause, then later attention heads need only attend to that comma to retrieve the information, rather than attending back to the original "loves" token many positions earlier. This interpretation connects the summarization motif to the practical challenge of long-context processing — as context windows grow, summarization may become increasingly important as an efficiency mechanism, and the paper's results (Table 1b) provide preliminary evidence that this is indeed the case.

The discovery also has methodological implications for circuit analysis. If information is routinely routed through intermediate summarization points, then circuit-finding algorithms that only examine attention from the final token back to source tokens may miss important pathways. The paper's circuit analysis methodology — iteratively tracing backward from the output, examining attention at intermediate positions, and identifying both "writers" and "readers" at each position — provides a template for discovering summarization in other tasks. Future circuit analyses should now expect to find summarization motifs and actively search for them, rather than treating them as surprising.

A final conceptual point: the summarization motif blurs the line between "representation" and "computation." The comma position is not merely a passive store for information; it is an active processing point where information from multiple sources (the valenced words, the character identity, the syntactic structure) is combined into a unified "summary variable" that is causally downstream of the sources and causally upstream of the output. This is precisely what one would expect from an internal world model — the model constructs intermediate variables that represent inferred properties of the world, and these variables are stored at specific positions for later use. The summarization motif provides a concrete, empirically validated example of this process in a real language model.

Innovation 3: A Top-Down, Feature-First Methodology for Interpretability That Complements Bottom-Up Dictionary Learning

Beyond the specific findings about sentiment, the paper models a complete investigative pipeline — toy dataset construction, multi-method direction finding, correlational validation, causal validation, and circuit analysis — that serves as a template for studying any abstract feature in a language model. This is a methodological contribution that addresses a genuine bottleneck in the interpretability research program.

The dominant paradigm for large-scale interpretability has been bottom-up: use dictionary learning (Bricken et al., 2023) or related unsupervised decomposition methods to extract thousands of features from model activations, then attempt to interpret what each feature means. This approach has the advantage of being comprehensive — it aims to find all features — but suffers from a severe interpretation bottleneck: making sense of thousands of discovered features is labor-intensive and prone to interpretability illusions (Bolukbasi et al., 2021). The paper's top-down alternative inverts this: start with a known, interpretable feature (sentiment), and then verify that a representation of this feature exists in the model, characterize it, and trace its mechanistic usage.

The efficiency of the top-down approach is striking. The sentiment direction was found using a toy dataset of only 30 positive and 30 negative English adjectives, and the direction was discovered even by unsupervised methods (K-means, PCA) that required no labels at all. The direction then generalized to French text (Figure 1d, Figure A.4), to out-of-distribution adjectives and verbs (Figure A.2: near-perfect classification accuracy across GPT-2 sizes), and to the Stanford Sentiment Treebank (53.5% logit flip rate for DAS, Figure 4). This efficiency — a handful of templated examples sufficient to discover a direction that generalizes broadly — suggests that top-down methods can rapidly scale to many features without requiring the massive computational and human annotation costs of dictionary learning.

The paper explicitly frames this contrast in its conclusion: "Whereas in dictionary learning we enumerate a large set of features which we then need to interpret, here we start with an interpretable feature and subsequently verify that a representation of this feature exists in the model, analogously to Zou et al. (2023). One advantage of this is that our fitting process is much more efficient: we can use toy datasets and very simple fitting methods." This framing positions the top-down approach not as a competitor to dictionary learning but as a complement — a rapid, hypothesis-driven method for investigating specific features of interest, while dictionary learning provides broad, unsupervised coverage.

What makes this a conceptual contribution rather than just a paper structure is that the methodology embodies a falsifiable scientific approach. The hypothesis is: "Sentiment is represented as a single linear direction in activation space." The paper then systematically tests this hypothesis from multiple angles: Do five independent methods converge to the same direction? (Yes, Figure 2.) Does the direction track sentiment in diverse natural text? (Yes, Figure 3.) Does it flip under compositional negation? (Yes, Figure 5.) Is it causally necessary for sentiment-driven behavior? (Yes, Figures 4, 6, and ablation results in Section 4.3.) Is it used by identifiable model components in an interpretable circuit? (Yes, Figures 7, 8, and Appendix A.3.) Each of these tests addresses a potential failure mode of the hypothesis — for example, the negation test rules out the possibility that the direction merely encodes static lexical sentiment rather than contextual sentiment. The convergence of evidence across orthogonal methods makes the conclusion substantially more robust than any single test alone.

The paper's inclusion of failed hypotheses is also methodologically important, though subtle. The observation that DAS dimensionality beyond 1 does not improve out-of-distribution generalization (Figure A.6) is a negative result that strengthens the claim: if sentiment were represented in a higher-dimensional subspace, more DAS dimensions should help, but they don't. The observation that the ReST^EM revision model degraded (though in a different paper context) demonstrates willingness to report negative results. These details model intellectual honesty and help others calibrate their methods.

The practical upshot for the field is a research program: pick an abstract feature (deception, factual accuracy, harmful intent, politeness, uncertainty), construct a minimal toy dataset that isolates that feature, apply the pipeline from this paper, and see whether a linear, causal, circuit-embedded representation emerges. The paper's success with sentiment — a feature that is semantically rich, context-dependent, and cross-lingual — provides a promising precedent. If the methodology transfers, it could accelerate interpretability research substantially by providing a rapid, cheap way to probe for specific capabilities in existing models without requiring massive annotation efforts or auxiliary model training.

Innovation 4: Redeeming the Pre-Transformer "Sentiment Neuron" for the Era of Mechanistic Interpretability

A subtler but historically significant contribution is the paper's reconciliation of a classic interpretability finding — Radford et al.'s (2017) "sentiment neuron" in an LSTM — with the modern mechanistic interpretability framework developed for transformers. The sentiment neuron was an inspiring result that suggested neural networks could develop human-interpretable internal features, but it was also methodologically limited: it was a single neuron in an older architecture, its causal role was not established, and the surrounding processing was black-box. The field arguably left the sentiment neuron behind as it moved to transformers and more sophisticated analysis methods.

This paper effectively "upgrades" the sentiment neuron finding to the modern era. The key conceptual move is recognizing that in the residual stream of a transformer — which does not have a privileged basis (Elhage et al., 2021b) — a feature will not be localized to a single neuron but will instead be a direction with components along many neurons. The paper explicitly notes this: "the directions we found were not sparse vectors, as expected since the residual stream is not a privileged basis" (Section 3.1). The "sentiment neuron" of the LSTM era becomes the "sentiment direction" of the transformer era — the same underlying phenomenon, but adapted to a different architectural substrate.

The paper's neuron analysis (Appendix A.5) provides concrete evidence for this reconciliation. The cosine similarities between neuron out-directions and the overall sentiment direction are heavy-tailed (Figure A.10): most neurons are roughly orthogonal to the sentiment direction, but a small subset are strongly aligned (e.g., L3N1605, L5N671, L6N828, L6N1237). These aligned neurons are individually interpretable — L3N1605 activates on "hesitate" following a negation, L5N671 activates on negative words following "not" contractions, L6N1237 activates on "but" following "not bad." These are precisely the kinds of composable, context-sensitive sentiment detectors that one would expect to contribute to a distributed sentiment representation. The overall sentiment direction is then the weighted combination of these individual neuron contributions (plus contributions from attention heads), producing a dense vector that no single neuron fully captures.

Why does this matter beyond sentiment? Because it provides a general account of how features manifest in transformer architectures that lack a privileged neuron basis. Individual neurons can still be interpretable — the paper's neuron analysis shows this — but they represent aspects of a feature (e.g., "negative word after negation") rather than the full feature. The full feature emerges as a direction in the high-dimensional space, recoverable by any method that finds the axis of maximum feature variance. This account unifies the intuition behind the sentiment neuron (models do develop interpretable feature detectors) with the geometry of the residual stream (features are directions, not individual neurons) and provides a framework for future work: when searching for a feature in a transformer, look for a direction, not a neuron, but expect that some neurons will be strongly aligned with that direction and may be individually interpretable.

The heavy-tailed distribution of neuron-direction similarities (Figure A.10) also connects to the broader superposition hypothesis (Elhage et al., 2022). In superposition, models represent more features than they have dimensions by encoding features in overlapping, partially-orthogonal directions. The heavy tail suggests that sentiment is one of the "privileged" features that gets a relatively dedicated subspace, with a handful of strongly-aligned neurons and a long tail of weakly-aligned ones. This is consistent with the paper's finding that the sentiment direction is easy to discover (even K-means with 60 examples finds it) — it is a prominent, relatively isolated feature, not one of the many features densely packed in superposition. Future work could investigate whether all "abstract semantic features" have this property, or whether sentiment is special because of its pervasiveness in training data.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary datasets are: (1) ToyMovieReview — a templated dataset of continuation prompts constructed by the authors with the form "I thought this movie was ADJECTIVE, I VERBed it. Conclusion: This movie is", using 85 adjectives (55 train / 30 test) and 8 verbs (Section 2.1, Appendix A.7). (2) ToyMoodStories — a multi-subject continuation dataset with prompts like "NAME1 VERB1 parties, and VERB2 them whenever possible. NAME2 VERB3 parties... One day, they were invited to a grand gala. QUERYNAME feels very", with 13 name options and verb pairs drawn from {hates, loves} and {avoids, joins} (Section 2.1, Appendix A.7). (3) Stanford Sentiment Treebank (SST) — 10,662 one-sentence movie reviews with human-annotated sentiment labels at the phrase level (Socher et al., 2013). For the zero-shot classification experiments, the authors collapse labels to binary "Positive"/"Negative", restrict to the test partition, keep only phrase-level examples, and further subset to examples where pythia-1.4b achieves 100% accuracy on clean inputs (removing 17% of examples), then pair examples of equal token count to create 460 clean/corrupted pairs (Section 3.3). (4) OpenWebText — the pretraining corpus for GPT-2 (Gokaslan & Cohen, 2019), used as a source of diverse natural text for correlational validation (Section 2.1). (5) French Harry Potter — the opening paragraphs of Harry Potter in French, used for qualitative cross-lingual evaluation (Appendix A.1.3).

• Base model(s). The paper uses models from two families: GPT-2 (Radford et al., 2019) — specifically GPT2-small (85M parameters, 12 layers) for primary circuit analysis on ToyMovieReview, with additional experiments on GPT2-medium, GPT2-large, and GPT2-xl for clustering accuracy comparisons (Figure A.2). Pythia (Biderman et al., 2023) — specifically pythia-1.4b for SST classification and direction patching, pythia-2.8b for multi-subject mood stories and cross-lingual experiments, and pythia-160m and pythia-410m for layer-wise generalization analysis (Figure 6). The authors state they "use GPT2-small for movie review continuation, pythia-1.4b for classification and pythia-2.8b for multi-subject tasks" (Section 2.1), choosing the smallest model in each family that can perform the target task. All models are decoder-only transformer architectures accessed through the TransformerLens library (Nanda & Bloom, 2022).

• Metrics. The paper employs three primary evaluation metrics (Appendix A.6): (1) Logit difference — extended from Wang et al. (2022) to the multi-class setting, computed as $\frac{1}{n}\sum_i [\text{logit}(t^{\text{positive}}_i) - \text{logit}(t^{\text{negative}}_i)]$ where $T^{\text{positive}}$ and $T^{\text{negative}}$ are sets of positive and negative answer tokens (5 each for ToyMovieReview; "Positive"/"Negative" for SST; "excited"/"nervous" for ToyMoodStories). Larger values indicate stronger sentiment-aligned predictions. When used as a patching metric, the authors report percentage change in logit difference toward the counterfactual value. (2) Logit flip percentage — the fraction of examples where the model's highest-probability sentiment token flips from positive to negative (or vice versa) after a causal intervention (Geiger et al., 2022). This is stricter than logit difference change because it requires crossing the decision boundary. (3) Zero-shot classification accuracy — the percentage of SST examples where the model assigns higher probability to the correct sentiment token ("Positive" or "Negative") among the top 10 predicted tokens (Section 4.3). For the correlational validation on OpenWebText (Section 3.2), accuracy is reported as GPT-4's classification of tokens into Positive/Negative based on sentiment activations in the top/bottom 0.1% tails.

• Baselines. The paper uses several baselines across different experiments: (1) Random direction — a randomly sampled unit vector used as a control in directional patching and directional ablation experiments, producing <1% effect on all metrics (Figures 2, 4, Section 4.3). (2) Standard activation patching — replacing entire activation vectors rather than only the projection onto a direction, used to establish the total effect size of a component (Section 4.1: 97% logit flips for full-circuit patching vs. 58.3% for directional-only). (3) Majority voting — referenced in the context of revision model answer selection (Section 6 of the original paper, not the prior sections context). (4) Greedy decoding — for the ~14× larger model in the FLOPs-matched comparison. (5) GPT-4 classification — used as an oracle for token-level sentiment labels on OpenWebText (Section 3.2), with the caveat that "only a cursory human sanity check was performed."

• Generation budget / compute accounting. The paper does not use a "generation budget" in the sense of test-time compute scaling papers since it is not comparing inference-time strategies. Instead, compute is accounted for in terms of the model forward passes required for each analysis: direction finding uses one forward pass per training example (60 for K-means training on ToyMovieReview); correlational analysis on OpenWebText uses a full forward pass over the dataset; causal patching requires paired forward passes on clean and corrupted inputs; and circuit analysis requires iterative path patching experiments decomposing component contributions. The DAS training procedure is the most computationally intensive direction-finding method since it requires gradient-based optimization through the full model. The paper does not provide explicit FLOP counts for any experiment.

• Cross-validation / statistical protocol. For direction finding, the paper uses a train/test split of adjectives (55/30, Section 2.1) and evaluates generalization to out-of-distribution verbs (never seen during direction training). For DAS optimization, the paper reports both training loss (in-sample) and validation loss (out-of-distribution on the simple character mood dataset; Figure A.6). For SST experiments (Section 3.3), the authors subset the test partition to examples where the model achieves 100% zero-shot accuracy on clean inputs, then pair examples of equal token count to create clean/corrupted pairs, using 460 such pairs. For circuit analysis, the authors use a held-out set of adjectives to avoid overfitting the circuit identification to the training set. The paper does not report confidence intervals, standard errors, or statistical significance tests for any quantitative result. The test set sizes are small (460 SST pairs, 85 total adjectives, 27 negation examples in Figure 5), which introduces substantial variance that is not quantified.

Main Quantitative Results

Direction Finding: Convergence Across Methods

The paper's first major quantitative finding is that five independent methods for finding a sentiment direction converge to approximately the same vector. Figure 2 presents a cosine similarity matrix between directions found by DAS, K-means, Logistic Regression, Mean Difference, PCA, and a Random baseline, all computed from GPT2-small's first residual stream layer using adjective-position activations from the ToyMovieReview dataset.

All legitimate methods achieve pairwise cosine similarities above 0.72, with most comparisons substantially higher: K-means vs. PCA achieves 0.991, Logistic Regression vs. PCA achieves 0.902, K-means vs. DAS achieves 0.999, Logistic Regression vs. K-means achieves 0.794, Mean Difference vs. K-means achieves 0.726, and DAS vs. Logistic Regression achieves 0.880. The random direction has cosine similarity below 0.025 with all methods (0.024 with K-means, 0.017 with DAS, 0.005 with Logistic Regression, 0.005 with Mean Difference, 0.012 with PCA). The paper interprets these high similarities as evidence that "these are all noisy approximations of the same singular direction" (Section 3.1).

The pattern of similarities reveals methodological differences: DAS, K-means, and PCA cluster extremely tightly (0.991–0.999 pairwise), while Logistic Regression and Mean Difference are slightly more separated from this cluster (0.794–0.880 to K-means) but still far above the random baseline. This is consistent with DAS, K-means, and PCA being unsupervised or semi-supervised methods that capture the dominant variance structure, while Logistic Regression and Mean Difference are supervised methods that may incorporate label-specific information not captured by the unsupervised approaches.

Correlational Validation: Sentiment Direction Captures Lexical Sentiment in the Wild

Figure 3 (left panel) presents the area plot of GPT-4 sentiment classifications (Positive/Neutral/Negative) for OpenWebText tokens binned by their sentiment activation — the projection of GPT2-small's first residual stream layer activation onto the K-means sentiment direction. The x-axis spans the activation range (approximately −4 to +2), divided into 20 equal-width buckets with 20 tokens sampled per bucket. The y-axis shows cumulative label proportions.

The key patterns: the left tail (most negative activations, approximately −4 to −2) is dominated by the Negative label, occupying roughly 80–100% of the area. The central region (approximately −1 to +0.5) is overwhelmingly Neutral, occupying roughly 60–90% of the area. The right tail (approximately +0.5 to +2) shows increasing Positive dominance, rising from near 0% at the left to roughly 40–80% in the rightmost bins. The authors interpret the "large space in the middle of the distribution simply occupied by neutral words (rather than a more continuous degradation of positive/negative)" as evidence of "superposition of features" (Section 3.2) — most tokens do not carry sentiment and their activations cluster near zero on this axis, while only the tails encode clear sentiment polarity.

Figure 3 (right panel, Table) reports classification accuracy using sentiment activations to classify tokens as positive or negative, taking the threshold as the top/bottom 0.1% of activations over OpenWebText. The K-means direction achieves 78% accuracy, PCA achieves 81%, Mean Difference achieves 80%, Logistic Regression achieves 89%, and DAS achieves 86%. These numbers represent the fraction of tokens in the extreme tails whose GPT-4-assigned sentiment label matches the direction's sign (positive activation → positive sentiment, negative activation → negative sentiment). Logistic Regression's advantage (89% vs. 78–86% for other methods) is consistent with it being explicitly optimized for classification, but the fact that even unsupervised K-means achieves 78% — far above chance — indicates that the sentiment structure is naturally prominent in activation space.

Negation Analysis: Sentiment Activation Flips Contextually Across Layers

Figure 5 (left panel) visualizes the sentiment activation at the negated token (e.g., "doubt" in "Don't doubt it") across all 12 layers of GPT2-small for a prompt "You never fail. Don't doubt it. I don't like you." The color shifts from red (negative) to blue (positive) across layers for tokens whose surface sentiment is contradicted by negation context: at layers 0–1, "fail," "doubt," and "like" show negative sentiment (their literal meaning), but by layers 7–10, "fail" and "doubt" have flipped to positive sentiment (their contextual meaning after negation), while "like" in "I don't like you" remains negative (the negation reinforces the negative surface meaning).

Figure 5 (right panel, table) quantifies this flip across a dataset of 27 negation examples. The fraction of activations that flip from negative to positive (or vice versa) between layers 1 and 10 is: DAS 96%, K-means 96%, Mean Difference 89%, Logistic Regression 100%, PCA 78%. The median flip size (centered around the mean activation) is: DAS 107%, K-means 69%, Mean Difference 45%, Logistic Regression 86%, PCA 44%. The flip size exceeding 100% (for DAS) means the activation not only changes sign but overshoots — the contextual sentiment is stronger than the original literal sentiment. The consistently high flip percentages across methods (78–100%) indicate that compositional sentiment processing is a robust property of the direction, not an artifact of a particular extraction method.

Causal Validation: Directional Patching Flips Model Predictions on Toy Data and SST

Figure 4 presents the primary causal evaluation results in a heatmap format, comparing seven direction-finding methods (DAS, DAS 2D, DAS 3D, K-means, Logistic Regression, Mean Difference, PCA, Random) across two evaluation datasets (ToyMovieReview and Treebank) and two evaluation metrics (simple_logit_diff, treebank_logit_diff, simple_logit_flip, treebank_logit_flip). The authors "report the best result found across layers" (Figure 4 caption), meaning each cell shows the maximum effect achieved at any layer for that method-dataset-metric combination.

For the ToyMovieReview dataset with logit difference (simple_logit_diff), DAS achieves 109.8% of the target logit difference change, K-means achieves 100.0%, Logistic Regression achieves 95.5%, Mean Difference achieves 110.2%, PCA achieves 67.2%, and Random achieves 0.1%. Values above 100% indicate the patch overshot the target — the model's output after intervention was more strongly aligned with the donor sentiment than the original run was with the original sentiment. For logit flip percentage (simple_logit_flip), DAS achieves 47.0%, K-means 53.5%, Logistic Regression 49.0%, Mean Difference 42.8%, PCA 22.1%, Random 0.1%. The fact that unsupervised K-means achieves the highest flip rate (53.5%) on the toy task is notable — it suggests the direction found without labels is as causally potent as the one optimized for causal efficacy (DAS at 47.0%).

For the Stanford Sentiment Treebank with logit difference (treebank_logit_diff), DAS achieves 110.4%, K-means 95.5%, Logistic Regression 86.4%, Mean Difference 71.1%, PCA 62.7%, Random 0.4%. For logit flip percentage (treebank_logit_flip), DAS achieves 42.8%, K-means 35.9%, Logistic Regression 30.8%, Mean Difference 27.5%, PCA 17.8%, Random 0.0%. The SST numbers are lower than the toy numbers across the board — the best flip rate drops from 53.5% (toy, K-means) to 42.8% (SST, DAS) — reflecting the greater difficulty of the natural text task with diverse sentence structures. The DAS advantage on SST (42.8% vs. 35.9% for K-means) is more pronounced than on the toy task, suggesting that optimization for causal efficacy provides robustness benefits that matter more on complex, out-of-distribution text.

The DAS 2D and DAS 3D columns (using 2 and 3 dimensional subspaces rather than a single direction) show that increasing dimensionality does not improve the patching metrics out-of-distribution: DAS 2D achieves 95.5% treebank_logit_diff and 39.4% treebank_logit_flip versus DAS 1D's 110.4% and 42.8%. This is consistent with the training/validation loss curves in Figure A.6, which show that higher DAS dimensions overfit in-sample without improving out-of-distribution generalization, supporting the claim that sentiment is genuinely one-dimensional.

Layer-Wise Generalization: Sentiment Direction Is Most Effective at Intermediate Layers

Figure 6 shows directional patching results for four model sizes (GPT2-small, pythia-160m, pythia-410m, pythia-1.4b) as a function of the layer at which the direction is trained and evaluated. For each model, the y-axis shows the percentage mean change in logit difference when patching sentiment directions trained on toy datasets and evaluated on SST (scaffolded with "Overall the movie was very" and computing the logit difference between "good" and "bad").

The consistent pattern across all four models is an inverted-U shape: patching effectiveness starts low at early layers, peaks at intermediate layers, and declines at late layers. For GPT2-small, the peak is around layer 5–7 (approximately 15–20% mean change). For pythia-160m, the peak is around layer 4–6 (approximately 10–15%). For pythia-410m, the peak is around layer 6–10 (approximately 5–10%). For pythia-1.4b, the peak is around layer 10–15 (approximately 30–50% mean change, the highest of all models). The three Pythia models show the peak shifting to later absolute layers as model size increases (pythia-160m peaks at layers 4–6, pythia-410m at 6–10, pythia-1.4b at 10–15), which may reflect deeper models distributing abstract concept formation across more layers.

The three methods compared (DAS, K-means, Logistic Regression) show broadly similar layer-wise profiles, with DAS often achieving slightly higher peak values. The authors interpret this pattern as evidence that "the model uses the residual stream to form abstract concepts in intermediate layers and this is where the latent knowledge of sentiment is most prominent" (Section 3.3).

Clustering Accuracy Across Model Sizes and Layers

Figure A.2 presents 2-means classification accuracy for four GPT-2 model sizes (small, medium, large, XL) across all layers (up to 24 for the largest models) and multiple evaluation sets. Each subfigure shows a heatmap with layers on the y-axis and evaluation sets on the x-axis (train adjectives, test adjectives on ADJ position, test adjectives on VRB position, and simple adverb dataset).

For GPT2-small (Figure A.2a), the pattern is striking: training accuracy on in-sample adjectives is 100% across all layers 0–11, but test accuracy on the ADJ position varies: 50.0% at layer 0, rising to 100% at layers 1–9, then dropping to 33.3% at layer 12. Test accuracy on the VRB position (out-of-distribution: the direction was trained on adjectives, not verbs) shows a different profile: 50.0% at layer 0, rising to 78.9% at layer 4, peaking at 84.2% at layer 6, then declining to 31.6% at layer 12. Test accuracy on the simple adverb dataset is similar: 50.0% at layer 0, peaking at 86.8% (for GPT2-medium at layer 9 in Figure A.2b) or 89.5% (for GPT2-XL at layers 22–24 in Figure A.2d).

For larger models (GPT2-medium, large, XL), the pattern of high accuracy sustained across a wider range of intermediate layers is more pronounced. GPT2-medium (Figure A.2b) maintains >90% VRB accuracy from layers 8–11 and >80% from layers 8–17. GPT2-XL (Figure A.2d) maintains >80% VRB accuracy from layers 17–24, with the best performance (94.7%) at layer 17. The consistency of high accuracy on out-of-distribution tokens (verbs, adverbs) across model sizes confirms that the clustering structure generalizes beyond the specific adjective tokens used for training.

Summarization Circuit: Component-Level Causal Contributions

Table 1a quantifies the causal importance of comma-position activations in the ToyMoodStories task with pythia-2.8b. When patching the full phrase values (including commas) from x_orig to x_flipped, the logit difference changes by −75%. When patching only the pre-comma phrase values while freezing the commas (keeping original, unflipped values at comma positions), the change is −38%. When patching only the two comma values (with pre-comma phrases frozen at original values), the change is −37%. The near-equality of the pre-comma and comma-only patching effects (−38% vs. −37%) indicates that the information stored at the comma positions is roughly as causally significant as all other token positions in the preference phrases combined. The paper notes that this experiment was conducted with "frozen attention patterns to ensure the model used the information from the patched commas in exactly the same way as it would have used the original information" (Section 4.2).

Table 1b shows how the importance of period-position summarizations grows with distance from the sentiment source. Irrelevant text was injected after each preference phrase ("John loves parties. He has a red hat and wears it everywhere...") at lengths of 0, 10, 18, and 22 tokens. The ratio of logit difference change for periods versus pre-period phrases is: 0.29 at 0 tokens of distance, 0.63 at 10 tokens, 0.92 at 18 tokens, and 1.15 at 22 tokens. At the maximum tested distance (22 tokens), the period representations are 15% more causally important than the original phrase text, indicating that summarization can become the dominant information pathway when the source tokens are far from the query.

SST Summarization: Comma Ablations Quantify the Information Bottleneck

For the SST zero-shot classification experiments (Section 4.3), the baseline is 100% accuracy — the subset of examples pythia-2.8b already classifies correctly on clean inputs. The total effect of sentiment is measured by directional ablation (using DAS sentiment directions) at every token and every layer, which causes a 71% reduction in logit difference and a 38% drop in accuracy (to 62%). Directional ablation using random directions at every token and layer produces <1% change in both metrics, confirming that the effect is specific to the sentiment direction.

Directional ablation at all comma positions (all layers) produces an 18% drop in logit difference and an 18% drop in accuracy. Since the total sentiment-direction-mediated accuracy drop is 38% (from 100% to 62%), the comma ablation's 18% drop represents roughly 47% of the total sentiment effect — nearly half of all sentiment-direction-mediated performance is mediated through commas.

Mean ablation at all comma positions (replacing each comma's full activation vector with the dataset-wide mean comma activation at each layer, not just the sentiment component) produces a 17% drop in logit difference and a 19% drop in accuracy. The similarity between the directional ablation result (18% accuracy drop) and the full-vector mean ablation result (19% accuracy drop) is interpreted as evidence that sentiment information is the primary causally relevant content at comma positions — if commas stored substantial non-sentiment task-relevant information, full-vector ablation would cause a larger drop than directional ablation. The fact that they are nearly identical suggests that sentiment dominates the task-relevant content at these positions.

Circuit Patching: Sentiment Direction Accounts for Majority of Circuit Function

For the GPT2-small ToyMovieReview circuit (Section 4.1), patching the entirety of the circuit (attention heads at ADJ, VRB, and SUM positions) along all directions — standard activation patching — achieves 97% logit flips and a 75% logit difference drop. Patching the same circuit components along only the sentiment direction — directional patching — achieves 58.3% logit flips and a 54.8% logit difference drop. The ratio 54.8/75 ≈ 73% suggests that the sentiment direction accounts for approximately three-quarters of the circuit's total causal effect, with the remaining quarter attributable to other features encoded in the same components. This is a substantial majority, confirming that the sentiment direction is the primary feature used by the circuit, but also indicating that the circuit carries additional information beyond sentiment (possibly related to syntactic structure or lexical identity).

Activation Addition: Steering Generations with the Sentiment Direction

Figure A.3 presents the area plot of GPT-4 sentiment classifications for completions generated by GPT2-small as a function of the steering coefficient applied to the first residual stream layer. Starting from a positive prompt ("I really enjoyed the movie, in fact I loved it. I thought the movie was just very..."), the model generates 50 completions per steering coefficient with temperature 1.0. The steering coefficients range from 0 to −20, with negative values adding negative multiples of the sentiment direction.

At coefficient 0 (no steering), GPT-4 classifies essentially all completions as "Somewhat Positive" or "Positive." As the coefficient becomes increasingly negative, the distribution shifts: at −5, "Neutral" and "Somewhat Negative" labels begin appearing; at −10, "Negative" becomes the dominant label; at −17, "Negative" dominates almost completely. The transition is gradual and smooth — there is no sharp discontinuity — which the authors interpret as evidence that the intervention is not taking activations pathologically out of distribution. The paper shows example completions at coefficients 0, −10, and −17 in Figure A.3 (bottom panel): at 0, completions include "entertaining" and "good"; at −10, they include "annoying" and "bad"; at −17, they include "terrible" and "horrible."

Neuron Analysis: Heavy-Tailed Alignment with Sentiment Direction

Figure A.10 plots the distribution of cosine similarities between neuron out-directions (the weight vectors of individual MLP neurons) and the sentiment direction across all 12 layers of GPT2-small. The x-axis is cosine similarity (ranging from approximately −0.4 to +0.4), and the y-axis is the proportion of neurons at each similarity value, with separate distributions for each layer.

The distributions are heavy-tailed: most neurons have cosine similarity near zero (roughly symmetric around 0), but there are pronounced tails extending to ±0.3–0.4. The paper highlights four specific outlier neurons with extreme alignments: L3N1605 (activates on "hesitate" following a negation), L5N671 (activates on negative words following "not" contractions), L6N828 (activates on words like "however" or "on the other hand" following negative content), L6N1237 (activates on "but" following "not bad"). These neurons are individually interpretable and appear to perform context-sensitive sentiment detection. The heavy-tailed structure is consistent with the superposition hypothesis (Elhage et al., 2022): most neurons are not aligned with sentiment (they encode other features), but a small subset are strongly aligned and contribute disproportionately to the overall sentiment direction. The paper notes that "the cosine similarities of neuron out-directions with the sentiment direction are extremely heavy tailed" (Appendix A.5), with the four highlighted neurons appearing in the tails of their respective layer distributions.

Cross-Model and Cross-Lingual Generalization

Figure 1 provides qualitative visual verification that a single sentiment direction captures sentiment across diverse contexts. The four panels show: (1a) nouns in English ("have complete confidence in", "You brought joy to", "despite the misery it", "deemed a hate group") with sentiment projections shown as color; (1b) proper nouns ("the Walt Disney World", "the Brazilian Amazon has", "presidential nominee Mitt Romney", "overturn Bashar Assad"); (1c) medical contexts ("currently in remission with", "a speedy recovery to", "radiation and cancer", "you a migraine"); (1d) French text from the opening of Harry Potter ("et son bon à rien de mari", "ils étaient parfaitement normaux", "gris et triste et rien dans", "la plus sinistre pour aller"). In all cases, the color gradient from blue (positive) to red (negative) aligns with intuitive sentiment judgments, despite the direction being trained only on 30 positive and 30 negative English adjectives using K-means with K=2.

Figure A.4 provides a larger-scale visualization of the same phenomenon for the full opening paragraphs of Harry Potter in English (Figure A.4a) and French (Figure A.4b) using pythia-2.8b. The English text shows consistent sentiment coloring: positive phrases like "perfectly normal" and "proud" are blue; negative phrases like "strange or mysterious" and "dull, gray Tuesday" are red. The French text, despite the model's poor French tokenization, shows similar patterns: "parfaitement normaux" and "la plus grande fierté" are blue; "gris et triste" and "la plus sinistre" are red. The paper notes that "the representation was not evident in the first couple of layers, probably due to the poor tokenization of French words" (Appendix A.1.3), consistent with the finding that abstract sentiment representations emerge in intermediate layers.

Ablation Studies and Robustness Checks

Direction-Finding Method Choice: Figure 2 demonstrates high cosine similarity (all pairwise similarities >0.72, most >0.87) between directions found by five independent methods, including both supervised (Logistic Regression, Mean Difference) and unsupervised (K-means, PCA) approaches. The convergence across methods with fundamentally different objectives (classification accuracy vs. variance maximization vs. causal efficacy) is a robustness check against the possibility that any single method overfits to its training criterion. The random baseline (<0.025 with all methods) confirms that the observed similarities are not due to chance in high-dimensional space.

DAS Dimensionality: Figure A.6 shows that increasing DAS subspace dimension beyond 1 improves in-sample training loss (Figure A.6a: runs with d_DAS = 2^n−1 show lower loss for higher n) but does not improve out-of-distribution validation loss (Figure A.6b: all dimensions show similar validation loss). This is a negative result that supports the claim that sentiment is genuinely one-dimensional — if sentiment required multiple dimensions, higher-dimensional subspaces would capture additional generalizable variance.

Train/Test Adjective Split: The direction-finding methods use a fixed 55/30 train/test split of adjectives (Section 2.1, Appendix A.7), and all causal evaluations (Figure 4) use hold-out adjectives not seen during direction training. The PCA visualizations (Figure A.1) further show that out-of-sample adjectives and verbs (an entirely different part of speech) project to the expected clusters, confirming that the direction captures abstract sentiment rather than memorizing training adjective tokens.

Verb Hold-Out Set: The direction was trained exclusively on adjective-position activations; verb-position activations were reserved as an out-of-distribution test set (Section 2.2). Figure A.2 shows that 2-means classification accuracy on verb positions is high across models (50.0–100% for GPT2-small depending on layer, peaking at 84.2% at layer 6). This confirms that the direction generalizes across parts of speech and is not an artifact of adjective-specific lexical features.

Random Direction Baseline: All causal experiments include a random direction control. In Figure 4, the random direction achieves 0.1% simple_logit_diff, 0.1% simple_logit_flip, 0.4% treebank_logit_diff, and 0.6% treebank_logit_flip — all near zero. In the SST ablation experiments (Section 4.3), directional ablation with random directions produces <1% change in both logit difference and accuracy. These controls rule out the possibility that any direction in activation space would show similar causal effects.

Attention Pattern Freezing: In the comma patching experiments (Table 1), the authors froze attention patterns to ensure that the model could not dynamically re-route attention away from patched commas toward alternative information sources. Without this control, the observed importance of comma representations could be understated because the model might compensate by attending elsewhere. The paper notes this explicitly: "We froze the model's attention patterns to ensure the model used the information from the patched commas in exactly the same way as it would have used the original information. Without this step, the model could simply avoid attending to the commas" (Section 4.2). The paper does not report the results without attention freezing, so the magnitude of the compensation effect cannot be assessed.

GPT-4 Classification Sanity Check: For the OpenWebText token sentiment classification (Figure 3), the authors note that "only a cursory human sanity check was performed" on GPT-4's classifications. This is a limitation rather than an ablation, but the high agreement between the direction's activation sign and GPT-4's labels across diverse tokens provides a weak form of validation — systematic errors in GPT-4's sentiment judgments would need to be correlated with the model's activations in a specific way to produce the observed area plot structure.

Cross-Lingual Generalization (Qualitative): The French Harry Potter visualizations (Figures 1d, A.4) serve as a qualitative robustness check that the direction captures sentiment semantics rather than English-specific lexical patterns. The fact that French words with poor tokenization still show intuitive sentiment activations (at intermediate layers) suggests the representation is abstract and language-agnostic, though this evidence is correlational and visual rather than causally validated.

Multiple Model Families: The paper replicates key findings across both GPT-2 and Pythia model families: direction convergence (Section 3.1, GPT2-small), clustering accuracy (Figure A.2, all GPT-2 sizes), SST patching (Figure 6, all Pythia sizes plus GPT2-small), and summarization circuit analysis (GPT2-small for ToyMovieReview, pythia-2.8b for ToyMoodStories). This multi-family replication reduces the risk that findings are specific to a particular training procedure or architecture variant.

Critical Assessment

Claim 1: Sentiment is represented as a single linear direction in activation space.

This claim is the best-supported in the paper, with converging evidence from multiple independent methods (Figure 2), correlational validation on diverse text (Figures 3, 1), causal validation through directional patching (Figure 4), and the negative result that higher DAS dimensions do not improve generalization (Figure A.6). However, the evidence is strongest for the existence of a linear sentiment direction, not necessarily for its uniqueness or completeness. The fact that directional patching of the full circuit explains 58.3% of logit flips but standard patching explains 97% (Section 4.1) means that ~42% of the circuit's causal effect is not captured by the single sentiment direction. This could indicate additional sentiment-related dimensions (e.g., intensity, aspect-specific sentiment) or non-sentiment features that the circuit processes. The paper acknowledges this in the limitations: "Did we find a truly universal sentiment direction, or merely the first principal component of directions used across different sentiment tasks? As found by Bricken et al. (2023), we suspect that this feature could be 'split' further into more specific sentiment features." The claim should be qualified as: "A single linear direction captures the dominant sentiment axis, but additional dimensions may encode finer-grained sentiment properties."

Claim 2: The sentiment direction is causally necessary and sufficient for sentiment-driven behavior.

The directional patching results (Figure 4) provide strong evidence for causal sufficiency: overriding the sentiment-projection at a single layer and position flips model predictions on both toy (53.5% flip rate for K-means) and SST (42.8% for DAS) data. The directional ablation results (Section 4.3) provide evidence for causal necessity: removing the sentiment component at all positions reduces SST accuracy by 38 percentage points. The random direction control (<1% effect) confirms specificity.

However, the causal sufficiency evidence has bounds. The SST flip rate of 42.8% means that in more than half of cases, manipulating the sentiment direction at a single layer does not flip the prediction. This could be because: (a) sentiment is distributed across multiple layers, and patching at one layer is insufficient to override the cumulative signal; (b) the SST examples involve more complex sentiment phenomena (negation, hedging, mixed sentiment) that a single scalar cannot capture; or (c) the direction was trained on a toy dataset and does not perfectly align with SST sentiment. The paper reports that results are "best result found across layers" (Figure 4 caption), which means the reported numbers represent an upper bound — the layer where the direction happens to be most effective — and does not reflect what a fixed-layer intervention would achieve. A more rigorous test would report performance at a pre-specified layer (e.g., the layer where the direction was trained) rather than cherry-picking the best post-hoc.

The necessity evidence from directional ablation is also subject to an important caveat: the paper reports that full-vector ablation at commas produces a 19% accuracy drop (Section 4.3), while directional ablation produces an 18% drop. The paper interprets the near-equality as evidence that sentiment dominates comma content. But an alternative interpretation is that directional ablation approximates full ablation because the sentiment direction spans a large fraction of the activation norm at commas. Without reporting the variance explained by the sentiment direction at comma positions, it's unclear whether the 18% drop reflects pure sentiment removal or collateral damage to other features encoded in overlapping directions.

Claim 3: The summarization motif — sentiment information aggregated at neutral tokens — is a causally significant information bottleneck.

The comma patching experiments (Table 1a) provide the strongest evidence: comma-only patching produces a −37% logit difference change, comparable to the −38% from pre-comma phrase patching. The attention pattern analysis (Figure 8) shows that specific heads attend heavily to commas, and path patching (Figure A.9) confirms these connections are causally relevant. The SST comma ablation results (Section 4.3) replicate the finding on natural text.

However, the evidence base is narrow in two respects. First, the quantitative comma importance numbers come from two specific tasks (ToyMoodStories and SST) and two specific punctuation marks (commas and periods). The paper does not test whether other neutral tokens (e.g., "that", "which", "the") also serve as summarization points, or whether the phenomenon is specific to clause-boundary punctuation. Second, the claim that summarization "increases with distance" (Table 1b) is based on a single experiment with injected irrelevant text of varying length. The paper does not test whether different types of intervening text (relevant vs. irrelevant, sentiment-congruent vs. sentiment-incongruent) affect summarization reliance differently, or whether the distance effect holds for naturally occurring long-distance dependencies rather than artificially injected filler text. The sample sizes for these experiments are not reported, making it impossible to assess statistical reliability.

Claim 4: The sentiment direction generalizes across models, languages, and tasks.

The cross-model evidence is strong: clustering accuracy (Figure A.2) and SST patching effectiveness (Figure 6) are demonstrated across 4+ model sizes from two families. The cross-lingual evidence is weaker: only qualitative visualizations of French Harry Potter (Figures 1d, A.4) are provided, with no quantitative metrics (e.g., classification accuracy, causal patching on French sentiment tasks). The paper acknowledges that "none of the models are very good at French" and that "the representation was not evident in the first couple of layers, probably due to the poor tokenization of French words" (Appendix A.1.3). The cross-task evidence is mixed: the direction trained on toy data transfers to SST with 42.8% flip rate, but the transfer is substantially worse than on the toy data itself (53.5%). The paper does not test on sentiment tasks beyond movie reviews (e.g., product reviews, social media, news sentiment), so the claim of broad task generalization is supported only by the single SST result.

Weaknesses in experimental design:

• Test set sizes are very small. The negation analysis uses 27 examples (Figure 5). The SST test set is restricted to 460 paired examples after subsetting for model capability. The direction training uses only 55 adjectives (30 positive, 25 negative). The paper does not report confidence intervals or statistical significance for any result, and with these sample sizes, the variance around reported percentages could be substantial (e.g., a 42.8% flip rate on 460 examples has a standard error of approximately ±2.3 percentage points).

• Cherry-picking of layers. Figure 4 reports "the best result found across layers," meaning each number is the maximum over 12–30+ layers. This inflates the apparent effect size relative to what a fixed-layer intervention would achieve. For SST patching, the range across layers can be large (Figure 6 shows some layers near 0% and others near 50%), so the reported "best" numbers may substantially overstate typical performance.

• No comparison of causal effect sizes to an upper bound. The SST flip rate of 42.8% is reported without context for what the maximum possible flip rate would be. If the model's sentiment processing is inherently robust or distributed, even a perfect sentiment direction might not achieve 100% flips. The paper does not establish a theoretical ceiling for the directional patching metric.

• The summarization distance experiment (Table 1b) is under-specified. The number of examples, the specific injected text, the number of attention heads patched, and whether attention patterns were frozen are not described in sufficient detail for replication. The ratio metric (LD change for periods vs. phrases) could be sensitive to the choice of baseline LD difference and the specific patching procedure.

• No systematic ablation of summarization vs. direct reading pathways. The paper identifies that both direct reading and summarization pathways exist in the circuit (Figure 7), but does not ablate each pathway independently to quantify their relative contributions in different regimes (e.g., short vs. long distances, simple vs. complex sentiment). The claim that summarization becomes more important with distance (Table 1b) is the only such comparison, and it is limited to one task.

• GPT-4 as oracle for sentiment labeling. The correlational validation (Figure 3) relies on GPT-4 to classify token sentiment, with "only a cursory human sanity check." GPT-4's sentiment judgments may have systematic biases (e.g., over-relying on lexical polarity, misunderstanding negation in short contexts) that could inflate or deflate the apparent accuracy of the sentiment direction. Human-annotated sentiment labels would provide a stronger validation.

• Single-language direction training. All directions are trained on English adjectives and tested on English (and qualitatively on French). The claim of "cross-lingual generalization" would be substantially strengthened by training a direction on French data and testing on English, or by demonstrating causal efficacy (not just correlation) on non-English text.

Missing experiments that would strengthen the paper:

• Directional patching on SST with directions trained at multiple layers simultaneously (rather than picking the single best layer) to test whether distributed sentiment representations across layers are more causally potent.
• Systematic comparison of summarization at different punctuation marks (commas, periods, semicolons, parentheses) and function words to characterize which token types serve as aggregation points.
• Ablation of individual summarization heads (7.1, 7.5 for GPT2-small; 12.17 for pythia-2.8b) and measurement of behavioral degradation to quantify their unique contribution.
• Testing whether the sentiment direction can be used for model editing — permanently modifying sentiment associations by fine-tuning along this direction — as a stronger test of causal relevance than one-shot patching.
• Quantitative cross-lingual evaluation on a French sentiment dataset with causal interventions, not just qualitative visualization.
• Replication on a non-movie-review sentiment domain (product reviews, social media) to test whether the toy-trained direction captures sentiment in general or movie-review sentiment specifically.

6. Limitations and Trade-offs

6.1 The Computationally Expensive Difficulty Estimation Problem Is Avoided Rather Than Solved

The assumption or constraint. The entire direction-finding pipeline depends on having a labeled dataset where sentiment is the only varying feature. The paper constructs such datasets synthetically—ToyMovieReview and ToyMoodStories—using carefully controlled templates. In doing so, it sidesteps what would be the hardest practical problem in a deployment setting: identifying which feature directions exist in a model when you don't already know they're there, and verifying that a candidate direction genuinely captures the intended feature rather than a confound. The paper's approach works because sentiment is isolated by construction in the toy data—change the adjective, everything else stays identical, so any consistent activation difference must be sentiment-related. Natural text provides no such guarantees.

The paper is transparent about the narrowness of the training data: "the direction being examined here was trained on just 30 positive and 30 negative English adjectives in an unsupervised way (using K-means with K = 2)" (Section 3.1). This efficiency is framed as a strength, and in one sense it is—the representation is prominent enough to be discoverable from minimal data. But it also means the methodology assumes the practitioner already knows what feature they're looking for and can construct a clean toy dataset that isolates it.

The consequence. For features more complex or less well-understood than sentiment—deception, sycophancy, situational awareness, long-term planning—constructing a toy dataset that purely isolates the feature may be dramatically harder. Sentiment benefits from being lexically transparent: words like "incredible" and "terrible" are unambiguous sentiment carriers whose meaning is relatively stable across contexts. Features like "the model is pursuing a hidden objective" or "the model is uncertain about this factual claim" lack this lexical grounding. A toy dataset for such features might inadvertently capture confounds—politeness, formality, topic, syntactic complexity—rather than the intended feature, and the convergence test (Figure 2: multiple independent methods finding the same direction) would not detect this because all methods would converge on the same wrong direction.

The paper does not test its methodology on any feature other than sentiment. The claim that "we also see this research as a model for how to find and study the representation of a particular feature" (Section 6) is therefore aspirational—the transferability of the approach to other features is entirely unproven. A practitioner attempting to apply this pipeline to a novel feature faces an unbounded difficulty estimation problem: how do you know whether your toy dataset cleanly isolates the feature of interest?

What evidence exists in the paper. The paper provides extensive evidence that the sentiment direction does generalize beyond its training distribution—to French text (Figure 1d, Figure A.4), to out-of-distribution adjectives and verbs (Figure A.2), to the Stanford Sentiment Treebank (Figure 4, Section 4.3). But all of this evidence is for sentiment specifically. There is no ablation where the methodology is applied to a second feature to demonstrate replicability. The paper's title and framing present the work as a case study, and it is exactly that—a single case. Whether the case generalizes is unknown.

Mitigation status. The paper acknowledges this limitation only indirectly, through its framing as a "model for how to find and study the representation of a particular feature" and the contrast drawn with dictionary learning: "Whereas in dictionary learning we enumerate a large set of features which we then need to interpret, here we start with an interpretable feature and subsequently verify that a representation of this feature exists in the model" (Section 6). This is presented as an advantage, not a limitation, but the burden of isolating a clean feature rests entirely on the practitioner. The paper does not propose methods for verifying that a toy dataset isolates the intended feature rather than a confound, beyond the convergence test—which tests internal consistency, not external validity. Future work is not explicitly suggested for this problem.

---

6.2 Causal Effect Sizes Are Reported as Layer-Maximum Upper Bounds, Not Typical-Case Performance

The assumption or constraint. The paper's primary causal validation metric—directional patching effectiveness—is reported as "the best result found across layers" (Figure 4 caption). For each method-dataset-metric combination, the reported number is the maximum effect observed when patching at any single layer. This means the numbers represent an optimization over layers: the experimenter tries all layers, picks the one that works best, and reports that number.

This is problematic because it conflates two distinct questions: (1) "Is there some layer where the sentiment direction is causally effective?" and (2) "How causally effective is the sentiment direction at the layer where it was trained/found?" The paper's direction-finding methods produce a direction at a specific layer (e.g., the first residual stream layer for GPT2-small in Figure 2). A practitioner who trains a direction at layer 5 and deploys it at layer 5 does not get to try all layers and pick the best one post-hoc—they get the performance at the training layer. The reported "best across layers" numbers inflate the apparent effect size relative to what a fixed-layer deployment would achieve.

The consequence. Consider the SST logit flip rate for K-means: the paper reports 35.9% in Figure 4 (treebank_logit_flip). But Figure 6 shows that the K-means direction's effectiveness varies substantially across layers for all four models tested. For pythia-1.4b (the model used for the SST experiments in Figure 4), the logit difference change ranges from near 0% at early and late layers to approximately 30–50% at intermediate layers (Figure 6, bottom-right panel). If the K-means direction was trained at layer 7 (the first residual stream layer is where Figure 2 measurements were taken, but the SST experiments use pythia models which may use different layers), the actual flip rate at the training layer could be substantially lower than the "best across layers" number. The paper does not report which layer achieved the best result, nor the performance at other layers, making it impossible to assess the gap between peak and typical performance.

This matters for practical applications. If one wanted to use the sentiment direction for model steering (as in the activation addition experiments, Figure A.3) or for detecting sentiment in model outputs, one would need to choose a specific layer. The paper's results do not guide that choice—they report an upper bound, not the expected performance at any particular layer.

What evidence exists in the paper. Figure 6 directly shows the large variance in patching effectiveness across layers. For pythia-1.4b with DAS, the logit difference change ranges from approximately 5% at layer 0 to approximately 50% at layer 13. The ratio of best to worst is roughly 10×. Figure 4 reports only the best of these numbers. The paper does not report the layer at which each best result was achieved, the performance at the training layer, or the mean/median performance across layers.

Mitigation status. The paper does not acknowledge this as a limitation. The "best across layers" reporting is stated once in the Figure 4 caption and not discussed further. A reader unfamiliar with the layer-dependence of representations might misinterpret the reported numbers as typical rather than peak performance. The paper does not suggest using the layer-wise profile (Figure 6) to select a deployment layer a priori, nor does it evaluate whether directions trained at one layer transfer effectively to adjacent layers.

---

6.3 The Summarization Motif Is Demonstrated on Only Two Punctuation Marks and One Model Architecture

The assumption or constraint. The paper's central architectural discovery—the summarization motif—is characterized as a general principle: "Rather than sentiment being directly moved from valenced tokens to the final token, it is first aggregated on intermediate summarization tokens without inherent valence such as commas, periods and particular nouns" (Section 1). The evidence for this claim comes from commas and periods in two tasks (ToyMovieReview and ToyMoodStories) and one natural text dataset (SST), across a small number of model sizes from the GPT-2 and Pythia families.

This evidence base is narrow in two dimensions. First, the paper does not systematically characterize which token types serve as summarization points. Two punctuation marks (commas and periods) and one noun type (repeated names like "movie" in ToyMovieReview) are identified, but there is no test of other candidate summarization sites: semicolons, parentheses, clause-initial "that" or "which," empty syntactic positions, or end-of-sentence punctuation other than periods. The paper does not establish whether summarization is a general property of clause-boundary tokens, or whether specific comma occurrences in the tested templates happen to be convenient aggregation points.

Second, the paper's causally strongest summarization results (Table 1: comma-only patching versus pre-comma patching) come from the multi-subject mood stories task in pythia-2.8b, a single model. The ToyMovieReview circuit analysis (Figure 7) shows summarization at the second "movie" token in GPT2-small, but the quantitative importance of this summarization relative to direct reading is not reported—the paper only states that patching the full circuit explains 75% of the logit difference drop, with directional patching explaining 54.8% (Section 4.1). These numbers measure the importance of the sentiment direction within the circuit, not the importance of the summarization pathway within the sentiment direction.

The consequence. A practitioner who discovers what appears to be a summarization motif for a feature in their model cannot use this paper to determine whether the motif is genuine or an artifact of the specific template structure. ToyMovieReview places the adjective at position 6, verb at position 9, and the summarization noun ("movie") at position 17—the summarization token is the only repeated content word between the sentiment-bearing region and the prediction. A skeptic could argue that the model learns to store information at "movie" not because it's a general summarization strategy, but because the template makes "movie" the only token that appears in both the input and the "Conclusion: This movie is" prefix, making it a natural bridge token. The ToyMoodStories comma results partially address this concern by showing summarization at a different token type (comma) in a different template, but the underlying confound—that the template structure makes certain positions natural information aggregation points—remains.

The paper's SST results (Section 4.3) partially mitigate the template confound by showing summarization on natural text, but the SST evidence is less granular: it shows that ablating comma representations damages performance (18% accuracy drop), but does not identify which commas are summarization points, what information they carry, or which heads write to and read from them. The circuit-level detail available for toy tasks is not replicated for SST, leaving open the possibility that the SST comma effect operates through a different mechanism than the summarization motif characterized in the toy circuits.

What evidence exists in the paper. Table 1a (patching results at summary positions on ToyMoodStories) and Table 1b (importance of summarization with distance) are the quantitative anchors for the summarization claim. Figure 7 diagrams the summarization circuit for ToyMovieReview. Figure 8 shows value-weighted attention to commas in ToyMoodStories. Section 4.3 presents SST comma ablation results. All of this evidence is for two punctuation marks (comma, period) and one noun type (repeated name/entity). The paper does not provide comparable analyses for other potential summarization token types.

Mitigation status. The paper acknowledges the narrowness implicitly through its language: "Our circuit analyses reveal suggestive evidence that summarization behavior at intermediate tokens like commas, periods and certain nouns plays an important part in sentiment processing" (Section 4.2). The word "suggestive" qualifies the claim, and the paper frames its findings as initial evidence rather than definitive characterization. The authors call for future work: "we would be very interested to study it in a broader range of contexts and understand what other factors of a particular model or task may influence the use of summarization" (Section 6). However, the paper's title-level claim ("the summarization motif") and the strong language in the abstract ("we discover a phenomenon which we term the summarization motif") present the finding as more established than the evidence base supports.

---

6.4 The Separation Between "Positive" and "Negative" Sentiment Directions Is Not Empirically Disambiguated

The assumption or constraint. The paper finds a single bipolar sentiment direction where one extreme corresponds to positive sentiment and the other to negative sentiment. However, the paper acknowledges a fundamental ambiguity about this representation: "one might wonder if there is really a single bipolar sentiment direction or if we have simply found the difference between a 'positive' and a 'negative' sentiment direction" (Section 6, Limitations). Mathematically, the paper observes, if x is a "valence" direction and y is a "sentiment" direction, then p = x + y represents positive sentiment and n = x - y represents negative sentiment. Conversely, given positive/negative directions p and n, one can derive x = (p + n)/2 and y = (p - n)/2.

The distinction matters because these two accounts make different predictions about what the model represents in activation space. Under the bipolar account, there is a single continuous axis from negative to positive, and neutral sentiment sits at the origin. Under the two-direction account, there are separate positive and negative detectors, and what looks like a bipolar axis is the projection of a 2D representation onto a 1D difference direction. The paper's methods (all of which find a single direction) cannot distinguish these hypotheses because they produce the same observable 1D projection either way.

The consequence. If sentiment is actually represented as two separate directions (positive strength and negative strength) rather than a single bipolar axis, then the paper's single direction captures only the difference between these two, discarding information about overall emotional intensity. A sentence that is intensely positive AND intensely negative (e.g., bittersweet, ambivalent) might project to near-zero on the bipolar direction—looking neutral—despite being emotionally charged. The sentiment direction would misclassify such examples.

This limitation has practical implications for applications that require fine-grained sentiment analysis. The paper's SST experiments collapse labels to binary Positive/Negative, sidestepping the neutral and mixed-sentiment categories in the original 5-way SST labeling. The zero-shot classification subset is further restricted to examples where pythia-1.4b achieves 100% accuracy—which likely excludes the most ambiguous or mixed-sentiment examples. The paper does not test whether the sentiment direction can distinguish subtle, mixed, or neutral sentiment from strongly polarized sentiment.

What evidence exists in the paper. The paper's evidence for the bipolar account is indirect. The area plot in Figure 3 shows that the central region of the activation distribution is dominated by "Neutral" labels from GPT-4. This is consistent with a bipolar axis where neutral tokens cluster near zero, but it is also consistent with two separate positive/negative axes where most tokens are near-zero on both. The negation analysis (Figure 5) shows that sentiment flips sign under negation—consistent with both accounts, since negation would invert the difference between positive and negative strength. The DAS dimensionality experiment (Figure A.6) shows that increasing subspace dimension beyond 1 does not improve out-of-distribution generalization. This suggests that the task-relevant sentiment signal for the toy task is one-dimensional, but does not rule out a higher-dimensional representation that the toy task does not require.

The paper's 2D PCA visualization (Figure A.1) shows that adjectives cluster by sentiment in the first two principal components, with clear separation between positive and negative clusters. But this is a visualization of the token embedding space (layer 0), not the abstract sentiment representation in intermediate layers, and the separation along PC1 (the primary sentiment axis) could still be the difference of two underlying directions that happen to be correlated in the adjective vocabulary.

Mitigation status. The paper acknowledges this limitation explicitly: "Did we find a truly universal sentiment direction, or merely the first principal component of directions used across different sentiment tasks? As found by Bricken et al. (2023), we suspect that this feature could be 'split' further into more specific sentiment features" (Section 6, Limitations). The mathematical equivalence between the bipolar and two-direction accounts is noted. The paper does not propose experiments to distinguish these hypotheses (e.g., testing whether the direction can be decomposed into independent positive and negative components that contribute additively to behavior, or testing on mixed-sentiment examples where the bipolar and two-direction accounts make divergent predictions).

---

6.5 Small Test Sets and Absence of Statistical Quantification Undermine the Reliability of Reported Effect Sizes

The assumption or constraint. The paper reports quantitative results without confidence intervals, standard errors, statistical significance tests, or in most cases even the sample sizes used to compute them. The key test sets are small: 27 negation examples for the flip analysis (Figure 5), 460 SST pairs for the directional patching experiments (Section 3.3), 55 training adjectives (30 positive, 25 negative) for direction finding. The ToyMoodStories experiments (Tables 1a, 1b) do not report sample sizes at all—the number of distinct name-query combinations and the number of prompts per condition are unspecified. The circuit analysis for ToyMovieReview uses a single template with a fixed structure, making the number of unique examples equal to the number of adjective-verb combinations (85 × 8 = 680 maximum for training, fewer after train/test split).

The paper implicitly assumes that these sample sizes are sufficient and that the reported percentages are stable estimates of underlying model behavior. No evidence is provided for this assumption.

The consequence. The reported effect sizes have unknown reliability. A 42.8% logit flip rate on 460 SST pairs has a binomial standard error of approximately ±2.3 percentage points, meaning the true flip rate could reasonably be 38–47%. The difference between DAS (42.8%) and K-means (35.9%) on this metric—which the paper presents as evidence for DAS's superiority—could be as small as ~2 percentage points or as large as ~12 percentage points depending on sampling error. Without confidence intervals, the reader cannot assess whether observed differences between methods are reliable or could easily reverse with a different sample.

The problem is more acute for the negation analysis (Figure 5), where 27 examples are used to compute flip percentages with precision apparently to the nearest percent (96%, 89%, 78%, etc.—each example represents ~3.7% of the total, so the reported precision is illusory). A single example flipping or not flipping changes the reported percentage by ~3.7 percentage points. The paper reports that K-means achieves 96% flip rate (26/27 examples) while PCA achieves 78% (21/27). The difference (5 examples) could reflect genuine methodological superiority or could be noise—with 27 examples, the 95% confidence interval for the difference spans approximately ±11 percentage points.

What evidence exists in the paper. The paper reports no statistical quantification anywhere. No confidence intervals, error bars, p-values, or sample size justifications appear in any figure, table, or text. The number of SST pairs (460) is mentioned in Section 3.3. The number of negation examples (27) is mentioned in Figure 5's caption. The number of training adjectives (55) is in Appendix A.7. Most other sample sizes are unreported. The paper does not discuss the implications of these small samples for the reliability of its conclusions.

Mitigation status. The paper does not acknowledge this as a limitation. The small sample sizes are not discussed in the Limitations section (Section 6). The omission is particularly notable because the paper is otherwise careful about methodological validity—the train/test adjective split, the cross-validation for direction finding, the random direction baseline, and the attention freezing control all demonstrate methodological sophistication. The absence of basic statistical reporting stands in contrast to this otherwise careful approach.

A partial mitigation is that many findings are replicated across multiple independent experiments. The convergence of five direction-finding methods (Figure 2) does not depend on any single test set. The SST comma ablation result (18% accuracy drop) is consistent with the ToyMoodStories comma patching result (~37% logit difference drop from comma-only patching). The layer-wise generalization pattern (Figure 6) is replicated across four model sizes. Consistency across experiments provides some robustness against sampling error in any single experiment, but does not substitute for statistical quantification within experiments.

---

6.6 The Cross-Lingual Generalization Claim Rests on Qualitative Visualization Alone

The assumption or constraint. The paper claims that the sentiment direction "generalise[s] to diverse natural language datasets from the real-world" (Section 6) and that the representation transfers across languages: "intermediate layers of pythia-2.8b demonstrate intuitive sentiment activations for the French text" (Appendix A.1.3). The evidence for cross-lingual transfer consists entirely of qualitative visualizations: Figure 1d shows four French phrases with sentiment-colored tokens, and Figure A.4 shows the full opening paragraphs of Harry Potter in French with sentiment activation colors. No quantitative metric—classification accuracy, causal patching effect size, correlation with human sentiment judgments—is reported for any non-English text.

The paper acknowledges the weakness of qualitative evidence in principle: "It is important to note that this type of analysis is qualitative, which should not act as a substitute for rigorous statistical tests as it is susceptible to interpretability illusions (Bolukbasi et al., 2021). We rigorously evaluate our directions using correlational and causal methods" (Section 3.1). But this caveat is immediately followed by the qualitative French visualizations, which are explicitly not subjected to the rigorous correlational and causal methods applied to English data.

The consequence. The cross-lingual claim is unsubstantiated by the paper's own methodological standards. The rigorous evaluation pipeline applied to English—correlational validation on OpenWebText with GPT-4 labeling (Figure 3), causal validation through directional patching on SST (Figure 4), and circuit analysis (Section 4)—is entirely absent for French. The visualizations could reflect an interpretability illusion: the human viewer sees blue on positive words and red on negative words, but this may reflect selective attention to tokens that match expectations, with tokens that violate expectations (positive words colored red, negative words colored blue) being overlooked or rationalized. The paper provides no protection against this confirmation bias.

The paper also notes that "none of the models are very good at French" and "the representation was not evident in the first couple of layers, probably due to the poor tokenization of French words" (Appendix A.1.3). This raises the possibility that what the visualization shows is not a genuine sentiment representation but a noisy projection that happens to align with sentiment on a few salient tokens. The model's poor French capability means the internal representations of French text may be substantially different from English representations, and a direction found from English adjectives may capture something other than sentiment when applied to poorly-tokenized French.

What evidence exists in the paper. Figures 1d and A.4 are the entirety of the cross-lingual evidence. The paper does not report: classification accuracy for French sentiment tokens, directional patching results on a French sentiment task, comparison of the English-trained direction to a French-trained direction, or quantitative correlation between sentiment activations and any ground-truth French sentiment labels. The French text is a single passage (the opening of Harry Potter) rather than a diverse corpus, making it impossible to assess whether the apparent generalization holds across domains, genres, or sentiment phenomena.

Mitigation status. The paper does not claim quantitative cross-lingual results—the language in the abstract and conclusion is qualitative ("diverse natural language datasets," "generalise well to the full data distribution"). But the French example is presented as evidence of generalization, and the paper does not flag the absence of quantitative cross-lingual evaluation as a limitation. The Limitations section (Section 6) does not mention cross-lingual evaluation. A practitioner reading the paper would reasonably conclude that the sentiment direction generalizes across languages, when in fact the evidence for this claim is substantially weaker than for the English-only claims.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the interpretability research program from whether abstract features are represented linearly in production-scale language models to how those representations are constructed, routed, and used in mechanistic circuits. Before this work, the evidence for linear representations in LLMs was fragmentary—synthetic game-playing models (Li et al., 2023; Nanda, 2023b), true/false statements (Marks & Tegmark, 2023), and the pre-transformer sentiment neuron (Radford et al., 2017). Each piece of evidence was suggestive but left open the possibility that linear representations were either domain-specific (board states, truth values) or architecture-specific (LSTMs, not transformers). This paper closes that gap by demonstrating that a genuinely abstract, context-dependent, latent semantic variable—sentiment—is encoded linearly as a single direction in the residual stream of multiple production-scale decoder-only transformers (GPT-2 and Pythia families), and that this direction is causally necessary and sufficient for sentiment-driven behavior on both templated and natural text.

The magnitude of the shift is best characterized as a reframing rather than a paradigm change. The linear representation hypothesis already had currency in the interpretability community; this paper provides its strongest empirical validation to date and, critically, extends it from the question of existence to the question of mechanism. The discovery of the summarization motif—information aggregation at neutral intermediate tokens—is the genuinely novel contribution that reframes how the field should think about information flow in transformers. Before this paper, the standard mental model for attention was: heads at the output token attend back to source tokens that carry relevant information. After this paper, the mental model must include: heads also write information to intermediate positions (commas, periods, repeated nouns), and later heads read from those positions, creating information bottlenecks that are causally comparable in importance to the original source tokens. This is not a minor amendment to the standard picture; it implies that circuit analysis must actively search for summarization points rather than treating them as surprising anomalies when discovered.

The paper resolves a tension that has existed in the interpretability literature between two modes of investigation: bottom-up dictionary learning (Bricken et al., 2023), which discovers many features but faces an interpretation bottleneck, and top-down hypothesis-driven probing, which tests for specific features but has often been criticized as susceptible to interpretability illusions (Bolukbasi et al., 2021). This paper demonstrates that the top-down approach, when combined with rigorous causal validation (directional patching, layer-wise generalization testing, convergence across independent direction-finding methods), can produce results that are as trustworthy as—and substantially more efficient than—bottom-up feature discovery. The practical consequence is that a researcher can, with a toy dataset of 60 examples, discover a feature direction that generalizes across models, languages, and tasks, without needing to train auxiliary sparse autoencoders or label thousands of activation samples. This efficiency makes the top-down approach newly attractive for targeted safety investigations—finding whether a model represents "deception" or "sycophancy" need not wait for comprehensive dictionary learning; a well-constructed toy dataset and a few hundred forward passes may suffice.

The paper also redirects research attention in two specific ways. First, it establishes that intermediate layers are the locus of abstract semantic representations—the finding that sentiment direction patching effectiveness peaks at middle layers (Figure 6) across all tested models provides a concrete architectural principle for where to target interpretability interventions. Researchers searching for other abstract features should now expect to find them in middle layers, not at the embeddings or final outputs, and should allocate their investigation budget accordingly. Second, it demonstrates that verifier over-optimization is not the only bottleneck for understanding model behavior—even when a feature direction is highly causally effective (58.3% of circuit logit flips attributable to the sentiment direction alone, Section 4.1), a substantial fraction of the circuit's function (~42%) is mediated by other features. This means that interpreting a single feature direction, no matter how well-validated, provides only a partial picture of the circuit's computation; a complete account requires understanding the ensemble of features processed by a circuit, not just the dominant one.

Follow-Up Research This Work Enables

Systematic characterization of summarization token types. The paper demonstrates summarization at commas, periods, and repeated nouns, but does not establish which linguistic properties make a token a summarization site. A natural follow-up would systematically test candidate summarization positions: clause boundaries (commas, semicolons, dashes, parentheses), sentence boundaries (periods, exclamation marks, question marks), and repeated referents (names, pronouns, definite noun phrases). For each candidate, the experiment would replicate the comma patching protocol from Table 1a—comparing logit difference drops from patching the summarization token versus patching the source phrase—across multiple sentiment templates with controlled distances between source and summary. The key measurement would be whether the ratio of summarization-to-source importance varies predictably with linguistic properties (syntactic depth, semantic role, givenness in discourse). A strong negative result—finding that summarization only occurs at the specific token types tested in this paper, or that the ratio is highly template-dependent—would suggest the motif is less general than the paper's framing implies.

Cross-feature replication of the top-down methodology. The paper explicitly positions its approach as a template for studying any abstract feature, but only demonstrates it for sentiment. The most important follow-up is to apply the identical pipeline to a different abstract feature—plausible candidates include formality, politeness, certainty/uncertainty, factual accuracy, or toxicity—and measure whether the same phenomena replicate: (1) a single linear direction discoverable by multiple independent methods with high cosine similarity, (2) causal efficacy as measured by directional patching on out-of-distribution natural text, (3) a summarization motif where information is aggregated at intermediate tokens, and (4) peak effectiveness at intermediate layers. A feature like certainty ("I am confident that..." vs. "I suspect that...") would be particularly instructive because, like sentiment, it can be isolated in a templated toy dataset but requires compositional processing (hedging, modal verbs, evidential markers). Success would validate the methodology's transferability. Failure—e.g., finding that certainty requires multiple dimensions or does not exhibit summarization—would establish boundary conditions on when linear, summarization-based representations emerge, which would be equally informative.

Distinguishing bipolar from dual-direction sentiment representations. The paper acknowledges the mathematical equivalence between a single bipolar sentiment axis and the difference of two independent positive/negative directions (Section 6, Limitations). A direct experiment would construct two separate directions—one maximally aligned with positive sentiment intensity (e.g., "good" → "great" → "excellent" → "incredible") and one aligned with negative sentiment intensity (e.g., "bad" → "terrible" → "horrific" → "appalling")—using a toy dataset where intensity varies independently of polarity. The key test is whether these two directions are orthogonal in activation space (supporting the dual-direction account) or strongly anti-correlated (supporting the bipolar account). A secondary test would evaluate performance on mixed-sentiment examples (e.g., "bittersweet," "the movie was excellent but exhausting") where the bipolar direction would predict near-zero activation (positive + negative ≈ neutral) while the dual-direction account would predict non-zero activation on both axes. The practical stakes are whether the model can represent emotional ambivalence as a distinct state from indifference—a capability relevant to nuanced sentiment applications.

Long-context scaling of summarization importance. The paper's Table 1b shows that summarization importance (ratio of period-to-phrase patching effects) grows from 0.29 to 1.15 as irrelevant text of 0–22 tokens is injected between sentiment sources and the query. This is a single data point that suggests but does not establish a scaling relationship. A systematic follow-up would test a wider range of distances (0, 10, 50, 100, 500, 1000 tokens), multiple types of intervening text (relevant vs. irrelevant to the sentiment-bearing entities, sentiment-congruent vs. sentiment-incongruent), and multiple model sizes to characterize the functional form of summarization reliance. The key hypothesis is that summarization serves as a compression mechanism that mitigates the difficulty of attending across long distances; if true, summarization importance should grow monotonically with distance and should grow faster in smaller models with more limited attention precision. A practical upshot would be guidance for prompt engineering: if summarization tokens at clause boundaries are the primary information carriers at long distances, then ensuring that key sentiment-relevant clauses end with clear punctuation (commas, periods) may improve model performance on long-context sentiment tasks, independent of the specific wording of the clause.

Neuron-level decomposition of the sentiment direction's construction. The paper's Appendix A.5 identifies four individually interpretable neurons (L3N1605, L5N671, L6N828, L6N1237) whose out-directions are strongly aligned with the overall sentiment direction, and notes that the distribution of neuron-direction cosine similarities is heavy-tailed. A detailed follow-up would trace how these specific neurons contribute to the sentiment direction at each layer: using direct logit attribution (as pioneered for L3N1605 in Appendix A.5), decompose the contribution of every MLP neuron and attention head to the sentiment-direction projection at a fixed position, and identify the minimal subset of components whose combined contribution accounts for, say, 90% of the total projection. This would reveal whether the sentiment direction is constructed from a sparse set of specialized detectors (as the heavy-tailed distribution suggests) or a dense combination of many weakly-aligned components. A natural next step is to test whether ablating or boosting individual sentiment-aligned neurons (e.g., L3N1605) produces interpretable, targeted changes in model behavior on specific sentiment sub-phenomena (negation, intensification, domain-specific sentiment)—this would connect the neuron-level analysis to the circuit-level picture and provide a multi-scale account of sentiment computation.

Dynamic summarization: does the model route information adaptively? The paper treats summarization as a static property of the circuit—certain tokens (commas, "movie") are always summarization points in the tested templates. An open question is whether summarization is adaptive: does the model route information to different tokens depending on the input? A follow-up experiment would construct prompts where the syntactic structure is varied while the sentiment-bearing content is held constant (e.g., "I thought this movie was incredible" vs. "This movie, which I thought was incredible, was also..." vs. "Incredible—that's what I thought of this movie"), and test whether the summarization point shifts to different tokens (comma, dash, repeated noun) depending on the syntactic structure. The measurement would be path patching from candidate summarization positions to direct-effect heads at the END token. If summarization is adaptive, different syntactic structures should show different primary summarization positions, and the model should dynamically select the most syntactically convenient aggregation point. This would upgrade the summarization motif from a fixed architectural feature to a flexible routing strategy—an important distinction for understanding how models scale to diverse natural language.

Practical Applications and Downstream Use Cases

Lightweight sentiment steering for controllable generation. The activation addition results (Figure A.3) demonstrate that adding multiples of the sentiment direction to the first residual stream layer smoothly steers GPT2-small's completions from positive to negative without destroying coherence. This is a zero-shot, inference-time control mechanism that requires no fine-tuning, no prompt engineering, and minimal computational overhead (one vector addition per generation step). For deployment scenarios where a language model's output sentiment must be controlled—customer service chatbots that should maintain a positive tone, content moderation systems that need to audit for sentiment bias, creative writing assistants that should match a specified emotional valence—this technique offers a lightweight alternative to prompt-based control ("write a positive review...") that works even when the model's prompt-following is unreliable. The paper's numbers (Figure A.3: smooth transition from ~100% positive at coefficient 0 to ~100% negative at coefficient −17, across 50 generations per coefficient) provide a calibration curve that practitioners could use to select steering coefficients for desired sentiment distributions. The key advantage over prompt-based control is that activation addition operates below the level of linguistic specification—it biases the model's internal representation directly, without requiring the model to interpret and follow an instruction.

Sentiment bias detection and mitigation in production models. The paper demonstrates that a single direction captures sentiment across diverse contexts in the wild—nouns, proper nouns, medical text, and cross-lingual passages (Figure 1)—with GPT-4 classification accuracy of 78–89% at the extremes of the activation distribution. This direction can serve as a diagnostic tool: by projecting activations at specific token positions onto the sentiment direction, practitioners can measure whether a model associates particular demographic terms (names, nationalities, gendered pronouns) or topics with systematic sentiment polarization. For example, a model that consistently produces more positive sentiment activations for "John" than "Jamal" at the name token, or for "doctor" than "nurse" at the profession token, would reveal a learned bias accessible without generating completions or running template-based bias probes. The causal validation (directional patching flips model predictions on 42.8% of SST examples, Figure 4) means that such biases, once detected, could potentially be mitigated by intervening on the sentiment direction during generation—subtracting the biased component along the direction to debias outputs. The paper does not demonstrate this use case directly, but the infrastructure it provides (direction finding, validation, and intervention) is directly transferable.

Information bottleneck optimization for long-context processing. The finding that summarized sentiment at commas is comparably important to the original phrases (Table 1a: −37% logit difference drop from comma patching vs. −38% from pre-comma phrase patching) and that summarization importance grows with distance (Table 1b: ratio increases from 0.29 to 1.15 as distance grows from 0 to 22 tokens) has direct implications for how to structure prompts for long-context language models. When asking a model to make judgments that depend on sentiment information scattered across a long document, practitioners should ensure that sentiment-relevant clauses end with clear punctuation (commas or periods) rather than trailing into subsequent text, because the model will likely store the aggregated sentiment at those punctuation positions. More speculatively, if future work confirms that summarization is the primary mechanism for long-distance information retention, architectures could be modified to explicitly support summarization—for example, by adding dedicated "summary tokens" at clause boundaries that are optimized for information aggregation, or by training models to explicitly write summaries to punctuation positions using an auxiliary loss. The paper's numbers provide the empirical motivation for such architectural innovations: if 47% of sentiment-direction-mediated accuracy on SST flows through commas (18% accuracy drop at commas out of 38% total drop from full sentiment-direction ablation, Section 4.3), then optimizing the comma-position representation could yield substantial downstream performance gains.

When to Prefer This Method

This paper primarily demonstrates and validates a methodology rather than introducing a new architecture or training procedure that competes with named alternatives. The choice it presents is not "use this model vs. that model" but rather "study features top-down (starting from a known feature and verifying its representation) vs. bottom-up (discovering features from activations without prior hypotheses)." The paper articulates this tradeoff explicitly in Section 6:

• Prefer the top-down, feature-first approach when: (1) you have a specific feature of interest that can be isolated in a minimal, templated dataset (as sentiment is isolated by the ToyMovieReview dataset's structure—change adjective, everything else constant); (2) you need causal validation, not just correlational evidence, that the representation is functionally significant (the paper's directional patching protocol provides this); (3) you want efficiency—the paper demonstrates that 60 labeled examples and simple unsupervised methods (K-means, PCA) are sufficient to discover a direction that generalizes broadly; (4) you need mechanistic detail about how the feature is constructed and used (the circuit analysis in Section 4 traces specific attention heads and their summarization behavior, providing a blueprint for understanding how the feature is processed, not just that it is represented).

• Prefer the bottom-up approach (dictionary learning, sparse autoencoders) when: (1) you don't know which features the model represents and want broad, unsupervised coverage (Bricken et al., 2023); (2) the feature of interest cannot be easily isolated in a toy dataset—features like "the model is pursuing a hidden objective" or "the model is uncertain about this claim" may not have the lexical transparency that sentiment enjoys, making clean counterfactual construction difficult; (3) you suspect the feature is not linear or is distributed across a high-dimensional subspace, and a single direction would miss important variance—the paper's DAS dimensionality sweep (Figure A.6) shows that sentiment is one-dimensional, but other features may not be.

The paper does not position its methodology against specific alternative analysis methods (e.g., probing classifiers vs. causal interventions) beyond the general top-down vs. bottom-up framing, so a detailed decision matrix for probing vs. patching vs. ablation is not warranted. The methodological contribution is the combination of techniques in a coherent pipeline, not the advocacy of any single technique over others.