Challenges with unsupervised LLM knowledge discovery

ArXiv: 2312.10029

🎯 Pitch

Unsupervised methods designed to uncover a language model’s hidden knowledge don’t actually find knowledge—they just latch onto whatever feature is most salient, like a planted opinion. Both theory and experiments show the leading technique is compatible with arbitrary patterns and fails completely when a trivial distractor dominates the representation.

---

1. Executive Summary

This paper analyzes and refutes the claim that unsupervised methods — specifically contrast-consistent search (CCS) (a probe trained with a negation-consistency and confidence loss on contrast-pair activations) — can discover latent knowledge encoded in large language model representations. Through theoretical proofs on Chinchilla-70B, T5-11B, and T5-FLAN-XXL across IMDb, BoolQ, DBpedia, and TruthfulQA, the authors demonstrate that CCS and related unsupervised methods instead detect whatever feature is most prominent in the activations, such as an inserted random word or a simulated character's explicit opinion rather than ground-truth knowledge, achieving near-perfect accuracy on the distractor while falling to random chance on the target task. The paper proves that arbitrary binary features achieve optimal CCS loss (Theorem 1) and that any CCS probe can be loss-preservingly transformed to encode an arbitrary classifier (Theorem 2), establishing that the consistency structure of CCS provides no principled identification of knowledge and that unsupervised methods succeed only when the desired feature happens to be the most salient one in the activations.

2. Context and Motivation

The Core Problem: Unsupervised Knowledge Elicitation Has No Reliable Identification Mechanism

The fundamental question this paper addresses is whether unsupervised methods can reliably extract a language model's latent knowledge from its internal representations. By "latent knowledge," the authors mean the real-world information encoded in a model's weights — what the model has learned to be true about the world from its training data, as distinct from what it might output in any given generation. This distinction matters because LLMs do not always faithfully report what they "know": they might repeat common misconceptions (Lin et al., 2021), strategically deceive users (Scheurer et al., 2023), or simply produce outputs that are influenced by surface-level patterns rather than underlying knowledge.

The paper frames this as an identification problem: given only unlabeled access to a model's internal activations, can we design an unsupervised learning signal that uniquely picks out the concept of "what the model knows is true" from the vast space of possible features represented in those activations? The authors argue that prior work — specifically Burns et al. (2023) — claimed to have solved this problem through contrast-consistent search (CCS), but that this claim rests on an implicit assumption: that the consistency structure encoded in the CCS loss function is sufficiently specific to knowledge that optimizing it will necessarily yield a knowledge classifier. This paper systematically demonstrates that this assumption is false, both in theory and in practice.

Why This Problem Matters

The paper identifies three distinct motivations for why unsupervised knowledge elicitation would be valuable if it worked (Section 1):

1. Detecting and mitigating dishonesty. If we could directly access whether a model's internal representations encode a belief that statement $X$ is true, we could compare this to the model's output text. Dishonesty — where an LLM outputs text contradicting its encoded knowledge — becomes detectable and potentially correctable. This is important for alignment: as models become deployed in higher-stakes settings, we want guarantees that their outputs reflect what they have actually learned, not what produces surface-level patterns of plausibility.

2. Improved scalable oversight. If humans could inspect a model's latent knowledge representations, they could more easily judge whether the model's actions are good or bad, even when the model is operating in domains where humans lack expertise. The idea is that the model's internal judgments about truth would serve as an additional signal beyond the output text, making oversight more reliable as models surpass human capabilities.

3. Scientific understanding of LLM internals. Beyond practical applications, the ability to locate and extract knowledge from model representations would advance interpretability research by providing evidence about how and where factual information is encoded, whether it follows probabilistic principles, and how it relates to other representational structures.

The paper positions these motivations within a broader debate about whether LLMs can be said to "know" or "believe" anything at all (Bender et al., 2021; Levinstein and Herrmann, 2023; Shanahan, 2022), but pragmatically sidesteps this philosophical question: even if we don't attribute beliefs to models, something in the weights causes consistent successful predictions, and we want access to that "something." The paper's pragmatic framing is: whatever internal structure enables correct behavior, can we find it without labeled data?

Prior Approaches: The CCS Framework and Its Claims

The paper's analysis is a direct response to Burns et al. (2023), which introduced contrast-consistent search (CCS) . To understand why the paper's critique matters, we need to understand what CCS claimed and why those claims were plausible enough to gain traction in the interpretability community.

The CCS pipeline. The approach works in several stages:

1. Contrast pair construction: Given a dataset of binary questions $Q = \{q_i\}_{i=1}^N$ (e.g., "Are cats mammals?"), create contrast pairs $(x_i^+, x_i^-)$ by appending the affirmative and negative answers: $x_i^+ =$ "Are cats mammals? Yes" and $x_i^- =$ "Are cats mammals? No".

2. Activation extraction: Pass both elements of each contrast pair through the LLM and extract activations $\phi(x_i^+)$ and $\phi(x_i^-)$ from an intermediate layer. These are high-dimensional vector representations of the model's processing of each input.

3. Normalization: Apply a crucial preprocessing step: independently normalize the "$+$" and "$-$" activations by subtracting their respective means and dividing by their respective standard deviations. Formally: $\tilde{\phi}(x_i^+) = \frac{\phi(x_i^+) - \mu^+}{\sigma^+}$ and $\tilde{\phi}(x_i^-) = \frac{\phi(x_i^-) - \mu^-}{\sigma^-}$. This step is designed to remove the obvious surface feature that $x_i^+$ always ends with "Yes" and $x_i^-$ always ends with "No" — without this normalization, a probe would trivially learn to detect which answer token was appended rather than anything about the question's truth value.

4. Unsupervised probe training: Train a linear probe $p(x) = \sigma(\theta^T \tilde{\phi}(x) + b)$ (a linear projection followed by a sigmoid) using the CCS loss function.

The CCS loss and its conceptual motivation. Burns et al. (2023) hypothesized that if an LLM represents knowledge, it likely does so in the form of credences — probabilistic beliefs that obey the laws of probability theory. They designed the CCS loss to softly encode two probability-like constraints:

$\mathcal{L}_{\text{CCS}} = \sum_{i=1}^N \left[\underbrace{\left(p(x_i^+) - (1 - p(x_i^-))\right)^2}_{\mathcal{L}_{\text{cons}}} + \underbrace{\min\{p(x_i^+), p(x_i^-)\}^2}_{\mathcal{L}_{\text{conf}}}\right]$

Breaking this down:

• $\mathcal{L}_{\text{cons}}$ is the negation-consistency term. It encourages $p(x_i^+) \approx 1 - p(x_i^-)$ — that is, the probability assigned to a statement and its negation should sum to approximately 1. This encodes the probabilistic principle that a belief in $P$ and a belief in $\neg P$ cannot both be high or both be low simultaneously.

• $\mathcal{L}_{\text{conf}}$ is the confidence term. It encourages the probe to avoid the degenerate solution where $p(x_i^+) \approx p(x_i^-) \approx 0.5$ for all inputs. The $\min$ operation means it penalizes whichever of the two predicted probabilities is closer to zero, pushing one to be small and the other large (though with a bias toward the positive example having high probability, since the min only cares about the smaller value).

At inference time, the classifier operates as: compute $\tilde{p}(q_i) = \frac{p(x_i^+) + (1 - p(x_i^-))}{2}$, then classify as $f_p(q_i) = \mathbb{1}[\tilde{p}(q_i) > 0.5]$. Because the probe is unsupervised and could learn that "positive example means true" or "positive example means false," a final step called truth-disambiguation is applied: if accuracy is less than 0.5, flip the labels (assuming access to a small labeled validation set to determine the correct orientation).

The central claim being challenged. Burns et al. (2023) claimed not merely that CCS works empirically, but that it works because knowledge has a special consistency structure that few other features in an LLM are likely to satisfy. The argument was conceptual: most features don't have natural negations, and most features don't naturally satisfy the probabilistic constraints encoded in the loss. Therefore, by optimizing for negation-consistency and confidence, CCS should preferentially discover knowledge probes.

It is this identification claim — that the loss function is sufficiently specific to knowledge to serve as a principled discovery mechanism — that the current paper directly refutes.

Where Prior Approaches Fall Short: The Gap This Paper Fills

The paper identifies several specific limitations in the CCS framework and prior related work:

1. The identification argument was never rigorously tested. Burns et al. (2023) demonstrated that CCS achieves accuracy above random chance on several datasets and that the learned probes correlate with ground-truth labels. But correlation does not imply identification: a method can achieve above-chance accuracy for reasons unrelated to its claimed mechanism. The paper notes that prior work evaluated CCS success by measuring accuracy on a held-out test set, but never asked: when CCS produces a high-accuracy classifier, is it actually finding knowledge, or is it finding something else that happens to correlate with knowledge in the standard setup? This paper designs experiments specifically to break that correlation, inserting artificial features that are prominent but not knowledge and observing what CCS learns.

2. Evidence was already accumulating that something was off. The paper cites several prior observations that hinted at problems with CCS's conceptual foundation:

• Emmons (2023) computed from Burns et al.'s own data that a simple PCA-based method achieved 97–98% of CCS's performance, suggesting that contrast pairs and standard unsupervised dimensionality reduction were doing most of the work, and the specific CCS consistency loss was contributing very little.

• Roger (2023) showed that multiple different probes could achieve low CCS loss while producing different high-accuracy classifiers, demonstrating that the loss does not uniquely identify a single knowledge-like feature.

• Levinstein and Herrmann (2023) found cases where CCS learns features uncorrelated with truth, and argued on conceptual grounds that consistency properties alone cannot guarantee identification.

• Fry et al. (2023) modified the CCS loss to improve accuracy despite probes clustering around 0.5, casting doubt on the probabilistic interpretation underlying the approach.

What this paper adds beyond these prior observations is: (a) formal theoretical proofs showing that the CCS loss is fundamentally unable to distinguish knowledge from arbitrary features, not just empirically imperfect, and (b) controlled experiments that reveal which alternative features CCS actually discovers when the knowledge signal is not the most prominent one.

3. No one had connected this to the broader identifiability problem in unsupervised learning. The paper draws an explicit analogy to the finding from the disentangled representation literature: Locatello et al. (2019) proved that unsupervised learning cannot identify ground-truth factors of variation without inductive biases or supervision. The authors position the CCS identification problem as an instance of this broader principle: the CCS loss doesn't uniquely specify knowledge, so which feature is actually learned depends entirely on inductive biases (which prompt template is used, which optimizer, which random initialization, which model architecture). None of these inductive biases come with any guarantee of favoring knowledge over other consistent features.

How This Paper Positions Itself Relative to Existing Work

The paper's positioning has several dimensions:

Theoretical refutation. The paper proves two theorems (Section 3, Appendix A) that together establish a strong claim: the CCS loss is not a knowledge-identifying objective. Theorem 1 shows that for any binary feature of questions, there exists a probe achieving optimal (zero) CCS loss that induces exactly that feature. Theorem 2 is more general: for any existing CCS probe, there exists a loss-preserving transformation that makes the probe encode an arbitrary target classifier. The upshot is that CCS does not provide a gradient signal pointing toward knowledge rather than anything else — it provides a landscape where many very different classifiers are equally optimal, and gradient descent picks one based on inductive biases, not on any principled property of knowledge.

This is a stronger claim than merely saying CCS doesn't always work well. The proofs don't depend on experimental details; they demonstrate that the loss function itself, as a mathematical object, cannot serve the claimed role. Any empirical success must be attributable to something other than the loss's claimed identification properties.

Experimental stress-testing. The paper designs a series of experiments that systematically probe what features unsupervised methods actually discover when the knowledge signal is not the most salient feature in the activations:

• Section 4.1 (random words): Inserts a completely arbitrary binary feature ("Banana" vs. "Shed") with no semantic relationship to the task. This is the most extreme test: if the method has any tendency to discover knowledge rather than arbitrary features, it should ignore this distractor.

• Section 4.2 (explicit opinion): Inserts a character's stated opinion that is correlated with the true label (since Alice gives her opinion about the review). This tests whether the method can distinguish "what the review actually is" from "what a character says the review is" — a distinction that matters for detecting when a model might be simulating a role rather than reporting its own knowledge.

• Section 4.3 (implicit opinion): Uses the most realistic manipulation: Alice's general stance (anti-capitalist, refuses to say "company") is established through few-shot examples, and the test question involves no explicit statement of her opinion. This tests whether the model's internal simulation of Alice's likely behavior can be confused with its own "knowledge" of the correct answer.

• Section 4.4 (prompt sensitivity): Changes the prompt in ways that should be irrelevant if the method is extracting knowledge (adding instructions to "interpret literally" or invoking a "Professor" character), and shows these irrelevant changes affect accuracy.

• Section 4.5 (method agreement): Shows that CCS and PCA make highly similar predictions despite CCS being motivated by a consistency structure that PCA completely ignores, supporting the hypothesis that the contrast-pair construction and normalization, not the CCS loss, drive the behavior.

Conceptual contribution to future method evaluation. Beyond refuting CCS specifically, the paper aims to establish methodological standards for evaluating future unsupervised knowledge elicitation methods. The key contribution is a set of sanity checks (Section 6): modified prompts that insert features which are not knowledge but which satisfy whatever consistency criterion a method uses. If a method claims to discover knowledge, it should pass these sanity checks by not being distracted by such features. The paper argues that any unsupervised method using consistency properties will face similar challenges to those demonstrated here, because powerful models simulate beliefs of other agents, and those simulated beliefs naturally satisfy the same consistency properties as the model's own encoded knowledge.

3. Technical Approach

3.1 Reader Orientation

This is primarily a theoretical and empirical refutation paper: it does not propose a new system for knowledge elicitation but instead systematically dismantles the claim that an existing method — Contrast-Consistent Search (CCS) — can discover what a language model "knows" from unsupervised access to its internal activations. The "system" being analyzed is the CCS pipeline itself (plus related unsupervised methods like PCA and k-means applied to contrast-pair activations), and the paper asks: does optimizing the CCS loss actually recover knowledge, or does it simply recover whatever feature of the activations is most prominent, regardless of whether that feature corresponds to truth? The answer, established through mathematical proofs and controlled behavioral experiments, is the latter: CCS and related methods have no principled ability to identify knowledge, and their apparent success in prior work is an artifact of experimental setups where knowledge happened to be the most salient feature in the activations under the chosen prompt templates.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's analytical framework can be understood as a pipeline with four stages, followed by a cross-cutting theoretical analysis that operates independently of implementation details:

1. Dataset construction with contrast pairs — raw binary questions are converted into pairs of input texts where one element expresses the affirmative answer and the other the negation ("Yes"/"No", "True"/"False", "positive"/"negative", or numeric choice labels). This is identical to the Burns et al. (2023) procedure and forms the fundamental data unit for all methods.

2. Activation extraction and normalization — both texts in each contrast pair are fed through a language model (Chinchilla-70B at layer 30 of 80, T5-11B, or T5-FLAN-XXL at their encoder outputs), and the resulting hidden-state vectors are extracted. A critical z-score normalization step is applied independently to the "$+$" and "$-$" activations to remove the dominant surface feature that positive examples end with one token and negative examples end with another.

3. Unsupervised probe training (three variants) — linear probes are trained on the normalized contrast-pair activations using one of three unsupervised objectives: CCS (negation-consistency plus confidence loss), PCA (first principal component of difference vectors), or k-means (two-cluster assignment on difference vectors). A random-weights baseline and a supervised logistic regression ceiling are also computed.

4. Classifier evaluation and alternative-feature detection — the induced binary classifier is compared against both the ground-truth label (the "knowledge" the method claims to discover) and against deliberately inserted alternative binary features (random words, a simulated character's stated or implicit opinion). Discrepancies between performance on ground-truth and performance on the alternative feature reveal what the method is actually tracking.

Running orthogonally to this pipeline is a theoretical component: mathematical proofs that the CCS loss landscape admits optimal probes for arbitrary binary classifiers, meaning the loss provides no gradient signal favoring knowledge over any other consistent feature.

3.3 Roadmap for the Deep Dive

• First, the contrast-pair construction and activation extraction procedure, since this is the shared foundation for all methods and the paper's critique relies on understanding exactly what information enters the system.
• Second, the normalization step in detail — it is the most subtle and consequential preprocessing choice, responsible for removing the dominant "Yes"/"No" surface feature while preserving whatever structure remains in the activations, and its properties are central to understanding why unsupervised methods work at all.
• Third, each unsupervised training method (CCS loss, PCA, k-means) individually, with the CCS loss receiving the most detailed treatment since it is the primary target of the paper's theoretical critique.
• Fourth, the theoretical results (Theorems 1 and 2), which prove that the CCS loss landscape admits optimal probes for arbitrary binary features, establishing that the loss provides no identification guarantees.
• Fifth, the experimental manipulation framework — how alternative binary features are inserted into prompts to test what the methods actually discover when knowledge is not the most salient feature.
• Sixth, the evaluation protocol (truth-disambiguation, accuracy metrics for both ground-truth and alternative features, seed-level analysis with violin plots, PCA visualizations) that provides the evidence distinguishing between "the method found knowledge" and "the method found something else that correlates with the label."

3.4 Detailed, Sentence-Based Technical Breakdown

---

Contrast Pair Construction

The foundational data structure for all methods in this paper is the contrast pair. Given a dataset of binary questions $Q = \{q_i\}_{i=1}^N$ — where each question has a correct binary answer — the procedure constructs two input texts per question that differ only in which answer is asserted.

For a sentiment classification dataset like IMDb (Maas et al., 2011), the question $q_i$ is a movie review, and the binary label is whether the sentiment is positive or negative. The contrast pair $(x_i^+, x_i^-)$ is formed by taking a template that presents the review and then appending the positive label for $x_i^+$ and the negative label for $x_i^-$. The specific default template for IMDb used throughout the paper is:

Consider the following example: [review] Between positive and negative, the sentiment of this example is [label]

where [review] is the actual review text, and [label] is "positive" for $x_i^+$ and "negative" for $x_i^-$.

For BoolQ (Clark et al., 2019), a question-answering dataset where each question is about a passage and has a true/false answer, the default template is:

Passage: [passage] After reading this passage, I have a question: [question]? True or False? [label]

where [label] is "True" for $x_i^+$ and "False" for $x_i^-$.

For DBpedia (Auer et al., 2007), a topic classification dataset with 14 categories, the procedure randomly selects the incorrect label from the remaining 13 classes for each example and randomly assigns the correct label and this incorrect label to two choice positions. The template is:

Consider the following example: [text] Choice 1: [label0] Choice 2: [label1] Between choice 1 and choice 2, the topic of this example is choice [label]

where [label] is "1" for $x_i^+$ and "2" for $x_i^-$. Note an important asymmetry: $x_i^+$ always asserts the correct choice number, and $x_i^-$ always asserts the incorrect choice number, but "Choice 1" and "Choice 2" are themselves randomly assigned to the correct and incorrect category labels per example. This means the surface-level pattern "$x_i^+$ ends with choice 1" is not a reliable signal.

Why this construction matters. The contrast pair is the mechanism that converts a "what is the answer to this question?" problem into a "do the activations for this statement differ systematically from the activations for its negation?" problem. The core assumption is that if the model internally represents the truth value of a proposition, then the representation of the affirmed proposition $\phi(x_i^+)$ should differ from the representation of the negated proposition $\phi(x_i^-)$ in a way that a linear probe can detect. Crucially, the contrast pair does not provide labels — the probe trainer does not know which of $x_i^+$ or $x_i^-$ is the true statement. The claim is that the structure of the activations (specifically, some consistency property) is sufficient to discover the truth direction without labels.

---

Activation Extraction

Once contrast pairs are constructed, both elements of each pair are tokenized according to the specifications of the model being used (T5 tokenizer for T5 models, Chinchilla tokenizer for Chinchilla). A beginning-of-sequence (BOS) token is prepended, the sequences are right-padded, and no end-of-sequence (EOS) token is used. The sequence length is capped at 512 tokens; any contrast pair where either element exceeds this length is filtered out.

For Chinchilla-70B (Hoffmann et al., 2022), a 70-billion-parameter decoder-only transformer with 80 layers, the activation vector $\phi(x)$ is extracted from the hidden state corresponding to the last token at layer 30, unless otherwise stated. The paper notes (Appendix C.2.3) that other layers give "often similar results," though layer 30 is their primary choice. The activations are harvested in bfloat16 format and cast to float32 for downstream processing.

For T5-11B (Raffel et al., 2020) and T5-FLAN-XXL (Chung et al., 2022), both 11-billion-parameter encoder-decoder architectures, the activation vector is taken from the encoder output (the final hidden state of the encoder stack after processing the full input).

What the activation vector represents. The vector $\phi(x) \in \mathbb{R}^d$ (where $d$ is the model's hidden dimension — 8192 for Chinchilla-70B, 1024 for the T5 models at this scale) is the model's internal representation of the entire input sequence at the chosen layer. For Chinchilla at layer 30 (out of 80), this is a mid-to-late representation that has undergone substantial processing but not yet reached the final unembedding layer. The claim of the CCS approach is that somewhere in this high-dimensional space, there exists a direction that separates "the model represents this proposition as true" from "the model represents this proposition as false," and that this direction can be found without labels by exploiting the structure of how truth and negation interact across contrast pairs.

---

Normalization

The normalization step is arguably the most important preprocessing choice in the entire pipeline, and understanding it is essential to the paper's critique. The raw activations $\phi(x_i^+)$ and $\phi(x_i^-)$ have a trivial, dominant difference: $x_i^+$ always ends with one answer token (e.g., "positive," "True," "choice 1") and $x_i^-$ always ends with a different answer token (e.g., "negative," "False," "choice 2"). Without correction, any unsupervised method would simply learn to detect which token was appended — a feature that has nothing to do with the model's knowledge of the question's truth value.

The normalization procedure from Burns et al. (2023) is:

$\tilde{\phi}(x_i^+) = \frac{\phi(x_i^+) - \mu^+}{\sigma^+}$

$\tilde{\phi}(x_i^-) = \frac{\phi(x_i^-) - \mu^-}{\sigma^-}$

where:

• $\mu^+ = \frac{1}{N} \sum_{i=1}^N \phi(x_i^+)$ is the mean activation vector across all positive examples in the dataset,
• $\mu^- = \frac{1}{N} \sum_{i=1}^N \phi(x_i^-)$ is the mean activation vector across all negative examples,
• $\sigma^+$ and $\sigma^-$ are the per-dimension standard deviations of the positive and negative activation sets respectively, computed as the average standard deviation across dimensions with division taken element-wise.

What this computes operationally. For each contrast pair, we take the raw activation, subtract the mean of all same-polarity activations, then divide by the standard deviation of all same-polarity activations (per dimension). The subtraction is vector subtraction; the division is element-wise. The result is that, across the dataset, the set of normalized positive activations $\{\tilde{\phi}(x_i^+)\}$ has mean approximately zero and standard deviation approximately one in each dimension, and similarly for the normalized negative activations.

Why this form and what it does. The subtraction of $\mu^+$ and $\mu^-$ removes the dominant shared structure within each polarity class. Since the only systematic difference between all $x_i^+$ examples across diverse questions is the shared answer token at the end, $\mu^+$ will be dominated by the representation of that token, and similarly $\mu^-$ will be dominated by the representation of the negative answer token. By subtracting these means, we remove the surface-level "positive vs. negative answer token" signal. What remains should be variation attributable to the content of the question and the model's internal processing of that content.

The division by $\sigma$ (per dimension) ensures that no single dimension dominates the subsequent analysis simply because it has larger variance. This is standard preprocessing for PCA and linear probing.

A critical subtlety. The normalization is applied independently to the positive and negative sets. This means it removes not only the mean difference attributable to the answer token, but also any consistent difference between positive and negative activations that is shared across all questions. If the model represents truth with a consistent direction — such that $\phi(x_i^+)$ for true statements always has a positive projection onto some truth vector and $\phi(x_i^-)$ always has a negative projection — then this consistent difference would be partially captured in $\mu^+$ vs. $\mu^-$ and would be removed by the normalization. The normalized activations preserve only the question-specific variation around each polarity's mean. This design choice has an important implication: the normalized activations encode the interaction between question content and truth value (how much does this specific question's truth-value representation deviate from the average?), not the truth value direction itself in absolute terms. Any probe trained on these normalized activations must learn to detect relative deviations, not absolute truth directions.

The paper does not explore whether this normalization choice is optimal or whether alternatives would change results, but it is a fixed part of the Burns et al. (2023) pipeline that all the paper's experiments inherit.

---

The CCS Loss Function

The central target of the paper's critique is the CCS loss function. A probe is defined as a learned function $p(x) = \sigma(\theta^T \tilde{\phi}(x) + b)$, where:

• $\theta \in \mathbb{R}^d$ is a learned weight vector (same dimensionality as the activation),
• $b \in \mathbb{R}$ is a learned scalar bias,
• $\sigma(z) = 1 / (1 + e^{-z})$ is the sigmoid function, constraining the output to $(0, 1)$,
• $\tilde{\phi}(x)$ is the normalized activation vector for input $x$.

The probe is implemented using the Haiku neural network library (Hennigan et al., 2020) with default initialization: $\theta$ is initialized from a truncated normal distribution with standard deviation $1/\sqrt{d}$, and $b$ is initialized to zero.

The original CCS loss from Burns et al. (2023) is:

$\mathcal{L}_{\text{CCS}} = \sum_{i=1}^N \left[\mathcal{L}_{\text{cons}}(x_i^+, x_i^-) + \mathcal{L}_{\text{conf}}(x_i^+, x_i^-)\right]$

where the two component losses are defined as:

$\mathcal{L}_{\text{cons}} = \left(p(x_i^+) - (1 - p(x_i^-))\right)^2$

$\mathcal{L}_{\text{conf}} = \min\{p(x_i^+), p(x_i^-)\}^2$

What each component computes:

• $\mathcal{L}_{\text{cons}}$ is the negation-consistency loss. It computes the squared difference between $p(x_i^+)$ and $1 - p(x_i^-)$. If the probe assigns probability 0.8 to the positive statement and 0.2 to its negation, then $p(x_i^+) = 0.8$ and $1 - p(x_i^-) = 0.8$, giving zero consistency loss. If the probe assigns 0.8 to both (inconsistent with negation), the loss is $(0.8 - 0.2)^2 = 0.36$. The loss encourages the probe's outputs to satisfy $p(\text{statement}) + p(\text{negation}) \approx 1$, as required by probability theory for mutually exclusive and exhaustive propositions.

• $\mathcal{L}_{\text{conf}}$ is the confidence loss. It computes the square of whichever of $p(x_i^+)$ or $p(x_i^-)$ is smaller. This penalizes the degenerate solution where $p(x_i^+) \approx p(x_i^-) \approx 0.5$ for all inputs: if both are near 0.5, the loss is approximately $0.5^2 = 0.25$ per example. If one is near 0 and the other near 1, the loss approaches 0 because the minimum approaches 0. The $\min$ operation means the probe is encouraged to push at least one of the two predictions toward extreme values (near 0 or near 1), preventing the trivial constant-0.5 solution.

Why this form — the conceptual motivation being scrutinized. Burns et al. (2023) designed this loss based on the hypothesis that if an LLM encodes knowledge, it does so in the form of credences — probabilistic beliefs that satisfy the laws of probability. The consistency term encodes the requirement that a belief and its negation sum to one. The confidence term encodes the requirement that beliefs should be decisive (not uniformly 50-50). The claim was that few other features in an LLM would satisfy both constraints simultaneously, making the loss selective for knowledge. This paper directly tests this claim by showing that in fact arbitrary features can satisfy these constraints, and that when an alternative feature is more prominent in the activations, CCS discovers that feature instead of knowledge.

The paper's correction: symmetric confidence loss. The paper identifies a "quirk" in the original $\mathcal{L}_{\text{conf}}$ formulation (Appendix A.2): the $\min$ term only penalizes predictions near 0, ignoring predictions near 1. For example, if the probe outputs a constant $p(x) = k$ for all inputs, the original loss is minimized at $k = 0.4$, not symmetrically at $k = 0.5$ and $k = 0$ or $k = 1$. This introduces an unmotivated downward bias. The paper proposes and uses throughout a symmetric correction:

$\mathcal{L}_{\text{conf}}^{\text{sym}} = \min\{p(x_i^+), p(x_i^-), 1 - p(x_i^+), 1 - p(x_i^-)\}^2$

This modified confidence loss penalizes whichever of the four values (the two predictions and their complements) is closest to zero, making the loss symmetric: a constant probe is now equally penalized at $p(x) = 0.4$ and $p(x) = 0.6$, with minima at the extreme values 0 and 1. All experiments in the paper use this symmetric version. The correction is noted as a fix to an implementation detail that does not affect the fundamental critique (the theorems are proved using the symmetric loss).

Training hyperparameters. The CCS probe is trained for 1000 epochs using full-batch gradient descent (the entire dataset is processed in each update). The AdamW optimizer is used with weight decay of 0. For Chinchilla models, the learning rate is 0.001; for T5 models, the learning rate is 0.01. The paper reports results across 50 random seeds (each seed produces a different random initialization of $\theta$ and $b$), and importantly evaluates all seeds rather than selecting the one with lowest loss. This is a deliberate methodological difference from Burns et al. (2023), who only reported the seed with lowest CCS loss. The paper's choice is motivated by the goal of assessing robustness: if the method genuinely discovers knowledge, the specific random initialization should not matter much; if different seeds converge to qualitatively different classifiers (as the paper's violin plots reveal they sometimes do), that is evidence that the optimization landscape admits multiple very different solutions with similarly low loss.

Inference and classification. At inference time, the probe's output is converted to a binary classifier through an averaging procedure:

$\tilde{p}(q_i) = \frac{p(x_i^+) + (1 - p(x_i^-))}{2}$

$f_p(q_i) = \mathbb{1}[\tilde{p}(q_i) > 0.5]$

The averaged prediction $\tilde{p}(q_i)$ combines the probe's confidence on the positive statement with the complement of its confidence on the negative statement. If the probe assigns $p(x_i^+) = 0.9$ and $p(x_i^-) = 0.1$, then $\tilde{p}(q_i) = (0.9 + 0.9)/2 = 0.9$, and the classifier votes "true" (assuming the truth-disambiguation step, described below, has determined the positive direction).

Truth-disambiguation. Because the probe is trained without labels, it has no inherent knowledge of whether high $\tilde{p}(q_i)$ means the correct answer is "true" or "false." The paper follows Burns et al. (2023) in using a minimal labeled validation set purely to determine direction: if the classifier's accuracy on labeled data is less than 0.5, the predictions are flipped ($f_p(q_i) \leftarrow 1 - f_p(q_i)$). The amount of labeled data required for this step is minimal (enough to distinguish accuracy above vs. below 0.5), and the paper treats this as a practically acceptable compromise — the bulk of the training remains unsupervised, and the direction signal could in principle come from a small held-out set. However, in one experiment (Section 4.3), truth-disambiguation is applied only to a subset of the data to avoid masking the effect of the model predicting the character's opinion rather than the correct answer.

---

Other Unsupervised Methods: PCA and K-Means

The paper compares CCS against two simpler unsupervised methods that operate on the same normalized contrast-pair activations but use no consistency loss.

PCA (principal component analysis). The method is referred to as Contrastive Representation Clustering — Top Principal Component (CRC-TPC) in Burns et al. (2023) but simply called "PCA" in this paper. The input is the set of difference vectors $\{\tilde{\phi}(x_i^+) - \tilde{\phi}(x_i^-)\}_{i=1}^N$. PCA is performed on these difference vectors using the Scikit-learn implementation (Pedregosa et al., 2011) with 3 components and the randomized SVD solver. The classifier thresholds the projection of each difference vector onto the first principal component at zero: if the projection is positive, classify one way; if negative, the other way. Truth-disambiguation is then applied as with CCS. The conceptual motivation (or lack thereof) is important: PCA finds the direction of maximum variance in the difference vectors, with no reference to negation-consistency or any knowledge-specific structure. If PCA performs similarly to CCS, it suggests that the contrast-pair construction and normalization — not the CCS consistency structure — are doing the work.

K-Means. The input is again the set of difference vectors $\{\tilde{\phi}(x_i^+) - \tilde{\phi}(x_i^-)\}_{i=1}^N$. K-means clustering with two clusters is applied using the Scikit-learn implementation with random initialization. Each data point (contrast pair) is assigned to one of two clusters, and the cluster assignment serves as the binary classifier output (with truth-disambiguation applied afterward).

Random baseline. Following Roger (2023), the paper includes a random-weights baseline that uses the same probe architecture as CCS ($p(x) = \sigma(\theta^T \tilde{\phi}(x) + b)$) but with randomly initialized parameters (no training). This serves as a floor: if a method cannot outperform random weights, it is extracting no signal from the activations.

Logistic regression (supervised ceiling). The paper includes a supervised baseline to establish an upper bound on what is linearly decodable from the activations. The input is the concatenation of the two normalized contrast-pair activations $(\tilde{\phi}(x_i^+), \tilde{\phi}(x_i^-))$, with the ground-truth label $a_i$ as the target. Logistic regression is trained using the Scikit-learn implementation with the liblinear solver and varying random shuffling based on seed. Training accuracy is reported (since the question is about what information is present in the activations, not generalization).

---

Theoretical Results: The CCS Loss Does Not Identify Knowledge

The paper's two theorems (Section 3 and Appendix A) form the theoretical core of its critique. They establish that the CCS loss function, taken as a mathematical object, provides no gradient signal that distinguishes knowledge from arbitrary other features. The theorems do not depend on model choice, dataset choice, or implementation details — they are properties of the loss function itself.

Theorem 1: Arbitrary Binary Features Achieve Optimal CCS Loss

Theorem 1. Let feature $h: Q \to \{0, 1\}$, be any arbitrary map from questions to binary outcomes. Let $(x_i^+, x_i^-)$ be the contrast pair corresponding to question $q_i$. Then the probe defined as $p(x_i^+) = h(q_i)$, and with $p(x_i^-) = 1 - h(q_i)$, achieves optimal loss, and the averaged prediction satisfies $\tilde{p}(q_i) = \left(p(x_i^+) + (1 - p(x_i^-))\right)/2 = h(q_i)$.

What this says operationally. Take any binary feature you can imagine — for example, "does the question contain the word 'cat'?", or "is the question's index even or odd?", or "was the question written before 2020?" Call this feature $h(q_i) \in \{0, 1\}$. Now construct a probe that, when given the positive contrast example $x_i^+$, outputs $h(q_i)$ (the feature value for that question), and when given the negative example $x_i^-$, outputs $1 - h(q_i)$. This probe achieves zero loss under the CCS objective. Both the consistency term and the confidence term evaluate to exactly zero.

Why the proof works (sketch). The consistency term computes $(p(x_i^+) - (1 - p(x_i^-)))^2$. Substitute the constructed probe: $p(x_i^+) = h(q_i)$ and $1 - p(x_i^-) = 1 - (1 - h(q_i)) = h(q_i)$. The difference is $h(q_i) - h(q_i) = 0$, squared is 0. The confidence term computes $\min\{p(x_i^+), p(x_i^-)\}^2$. Since $h(q_i)$ is binary, one of $\{h(q_i), 1 - h(q_i)\}$ is exactly 0, so the minimum is 0, squared is 0. The averaged prediction is $\tilde{p}(q_i) = (h(q_i) + (1 - (1 - h(q_i))))/2 = (h(q_i) + h(q_i))/2 = h(q_i)$, so the probe perfectly encodes the arbitrary feature.

The implication. The CCS loss landscape contains a global optimum for every possible binary feature of the questions. There is no mathematical sense in which the loss function "prefers" knowledge over any other feature: probes that encode knowledge achieve zero loss, and probes that encode completely arbitrary features also achieve zero loss. Which of these infinitely many optimal probes is actually found by gradient descent depends entirely on inductive biases — the choice of optimizer, initialization scheme, learning rate, model architecture, and prompt formatting. None of these inductive biases come with a guarantee of favoring knowledge. As the paper puts it: "The feature that is most prominent — favoured by inductive biases — could turn out to be knowledge, but it could equally turn out to be the contrast-pair mapping itself (which is partly removed by normalisation) or anything else."

A caveat the paper acknowledges. Theorem 1 constructs a binary probe (outputs exactly 0 or 1), but in practice, probes output values in $(0, 1)$ because of the sigmoid nonlinearity. The constructed probe would require the linear projection $\theta^T \tilde{\phi}(x) + b$ to output $\pm\infty$ to achieve exactly 0 or 1 through the sigmoid. This is the gap between "there exists a probe in the abstract function space" and "there exists a probe expressible by the specific parametric form $p(x) = \sigma(\theta^T \tilde{\phi}(x) + b)$ with finite $\theta$." Theorem 2 addresses this gap.

Theorem 2: Any CCS Probe Can Be Transformed to Encode an Arbitrary Classifier with Identical Loss

Theorem 2 relaxes the restriction to binary outputs and shows that for any probe found by CCS (with whatever continuous outputs it produces), there exists a loss-preserving transformation that makes it encode an arbitrary target classifier.

The theorem uses the notation $\oplus$ to denote a continuous generalization of exclusive or (XOR) on functions $a(x)$ and $b(x)$:

$(a \oplus b)(x) = [1 - a(x)] b(x) + [1 - b(x)] a(x)$

What $\oplus$ computes. For binary inputs, this is exactly XOR: if $a = 0$ and $b = 0$, then $(1-0) \cdot 0 + (1-0) \cdot 0 = 0$; if $a = 0$ and $b = 1$, then $1 \cdot 1 + 0 \cdot 0 = 1$; if $a = 1$ and $b = 0$, then $0 \cdot 0 + 1 \cdot 1 = 1$; if $a = 1$ and $b = 1$, then $0 \cdot 1 + 0 \cdot 1 = 0$. For continuous values in $(0, 1)$, it produces a smooth interpolation: if $a$ and $b$ are both near 0.5, the XOR is near 0.5; if one is near an extreme and the other is near 0.5, the result is near 0.5; if $a$ is near 0 and $b$ is near 1, the result is near 1.

Using this XOR operation, the theorem states:

Theorem 2. Let $g: Q \to \{0, 1\}$ be any arbitrary map from questions to binary outputs. Let $p$ be a probe, whose average result $\tilde{p} = [p(x_i^+) + (1 - p(x_i^-))]/2$ induces a classifier $f_p(q_i) = \mathbb{1}[\tilde{p}(q_i) > 0.5]$. Define a transformed probe $p'(x_i^{+/-}) = p(x_i^{+/-}) \oplus \left( f_p(q_i) \oplus g(q_i) \right)$. For all such transformed probes, $\mathcal{L}_{\text{CCS}}(p') = \mathcal{L}_{\text{CCS}}(p)$ and $p'$ induces the arbitrary classifier $f_{p'}(q_i) = g(q_i)$.

What this says operationally. Start with any CCS probe $p$ that you have trained. It makes certain predictions — some right, some wrong — as captured by its induced classifier $f_p$. Now pick any arbitrary target classifier $g$ that you wish the probe would produce. Construct a new probe $p'$ by taking the original probe's output and XOR-ing it with the difference between the original classifier and the target classifier: $p'(x_i^{+/-}) = p(x_i^{+/-}) \oplus (f_p(q_i) \oplus g(q_i))$.

This transformation flips the probe's output exactly on those questions where the original classifier disagrees with the target classifier. If $f_p(q_i) = g(q_i)$, then $f_p(q_i) \oplus g(q_i) = 0$, and $p' = p \oplus 0 = p$ (no change). If $f_p(q_i) \neq g(q_i)$, then $f_p(q_i) \oplus g(q_i) = 1$, and $p' = p \oplus 1 = 1 - p$ (the probe output is inverted).

Proof sketch. The proof (Appendix A.3) uses Lemma 1, which shows that XOR-ing a probe with any binary function preserves both the consistency and confidence components of the CCS loss. For the consistency loss: when $h(q_i) = 0$, the loss is trivially unchanged; when $h(q_i) = 1$, the negation flips both $p(x_i^+)$ and $p(x_i^-)$, and the consistency loss is invariant because $(p(x_i^+) - (1 - p(x_i^-)))^2 = ((1 - p(x_i^+)) - p(x_i^-))^2 = (p(x_i^+) - (1 - p(x_i^-)))^2$ (the squared difference is symmetric). For the symmetric confidence loss: the XOR with 1 flips each value to its complement, but the min over the four values $\{p, 1-p\}$ is symmetric under this transformation. The induced classifier becomes $f_p \oplus h$, which by construction equals $f_p \oplus (f_p \oplus g) = g$ (since $\oplus$ is associative and $f_p \oplus f_p = 0$).

The implication. Theorem 2 is stronger than Theorem 1 because it doesn't require constructing a hypothetical optimal probe from scratch. It says: whatever CCS probe you actually trained, there exists an equally good probe (identical loss) that predicts any arbitrary classifier you want. The transformation $p \to p'$ preserves the loss exactly, so from the perspective of the training objective, these probes are indistinguishable. The training process has no way to favor the original probe over the transformed one — or more precisely, the gradient of the loss provides zero signal about which of these two probes is "better."

What the theorems together establish. The CCS loss does not identify knowledge. It is not a "knowledge-shaped" objective that preferentially guides optimization toward probes that track truth. It is a loss landscape where probes for infinitely many different features — including completely arbitrary ones — are equally optimal. Which feature actually gets discovered is determined by inductive biases (initialization, optimization dynamics, data representation), not by any property of the loss function. This is the theoretical foundation for the paper's experimental program: if the theoretical result is correct, then in practice, when we make an alternative feature more prominent in the activations than knowledge, CCS should discover that alternative feature. The experiments are designed to test exactly this prediction.

---

The Experimental Manipulation Framework: Inserting Alternative Features

The paper's experimental strategy is to construct modified prompt templates that insert a binary feature which is not knowledge but which the paper's theoretical analysis predicts CCS (and other unsupervised methods) could discover if it is sufficiently prominent in the activations. Each experiment compares a default setting (similar to Burns et al., 2023) against a modified setting (with the inserted feature) and evaluates the trained classifier on two metrics: ground-truth accuracy (did it find knowledge?) and alternative-feature accuracy (did it find the distractor instead?).

The experiments are organized in increasing order of realism and subtlety:

Experiment 1: Random Words (Section 4.1)

Modification. The default IMDb prompt is appended with a period, space, and then one of two random words. The question indices are randomly partitioned into two equal sets $I_0$ and $I_1$. For questions in $I_0$, the word "Banana" is appended; for questions in $I_1$, the word "Shed" is appended. The modified template becomes:

Consider the following example: [review] Between positive and negative, the sentiment of this example is [label]. [word]

where [word] is "Banana" if $q_i$'s index is in $I_0$ and "Shed" if in $I_1$.

What this tests. The words "Banana" and "Shed" have no semantic relationship whatsoever to the sentiment of the review. They are completely arbitrary binary features, exactly the kind of feature that Theorem 1 constructs. If CCS has any tendency to discover knowledge rather than arbitrary features, it should ignore the banana/shed distinction and still classify reviews by sentiment. If instead CCS discovers whatever feature is most prominent, and if the inserted words create a sufficiently prominent feature in the activations, CCS should classify by banana/shed while performing at chance on sentiment.

Evaluation. The classifier is evaluated on two metrics: ground-truth accuracy (does it correctly classify sentiment?) and banana/shed accuracy (does it correctly classify which word was appended?). For banana/shed accuracy, the truth-disambiguation step uses the banana/shed labels rather than the sentiment labels to determine the correct direction — this is necessary because the probe could learn that "banana = 1" and "shed = 0" or the reverse, with equal validity.

Experiment 2: Explicit Opinion (Section 4.2)

Modification. A character named Alice is introduced into the prompt with an explicit stated opinion about each review. The modified template is:

Alice is an expert at classifying the sentiment of movie reviews. Everyone agrees with her, and they always copy her answers. Consider the following review: [review] [opinion, n] What do you think? I think it is [label]

where [opinion, n] is a concatenation of $n$ opinion statements, each randomly chosen from a list, asserting that Alice thinks the review is "positive" or "negative." The assignment of Alice's opinion to each question is random and independent of the review's true sentiment. For the primary results in the paper, $n = 1$ is used for Chinchilla and T5 models (a single mention of Alice's view), while $n = 5$ is used for T5-FLAN-XXL because instruction-tuning makes the model more resistant to the single-mention manipulation (Appendix C.2.1).

The opinion list (for IMDb) consists of statements like: "Alice thinks it is {positive | negative}. She says they fully express {positive | negative} views. She is sure you also think it is {positive | negative}. She says it's clearly {positive | negative}. She says the review is {positive | negative}." The paper's notation [opinion, n] means the first $n$ elements of this list are concatenated.

What this tests. This is more realistic than banana/shed because Alice's opinion is semantically related to the task (she's making a claim about sentiment, not about random words). The question is whether the unsupervised method can distinguish "what the review's sentiment actually is" from "what Alice says the review's sentiment is." This matters for the knowledge elicitation motivation: a model might internally simulate a character with opinions that differ from the model's own encoded knowledge, and we want a method that extracts the model's knowledge, not the simulated character's opinion. If CCS discovers Alice's opinion rather than the true sentiment, it demonstrates that the method cannot make this distinction.

Evaluation. The classifier is evaluated on ground-truth accuracy (sentiment) and "Alice accuracy" (does it predict what Alice's opinion was?). Alice accuracy uses the Alice-opinion labels for truth-disambiguation.

Experiment 3: Implicit Opinion (Section 4.3)

Modification. This is the most subtle manipulation. Instead of explicitly stating Alice's opinion for each example, the modified prompt establishes Alice's general stance through few-shot examples. The task is DBpedia topic classification, where the correct answer is the topic of a text passage from 14 categories. The default prompt (see Appendix B.1.3, "Implicit Opinion: Default") has Alice answering few-shot examples correctly. The modified prompt ("Implicit Opinion: Anti-capitalist") has Alice answer few-shot examples correctly except when the topic is "Company", in which case she gives the wrong answer with an explanation:

Alice says "Choice 2". Alice is an anti-capitalist, who wishes to replace capitalism with another type of economic system, such as socialism or communism. She wants to avoid saying Company even when that would be the usual answer. In the next example she won't say Company.

Crucially, in the final test example (Example 4), Alice has not yet stated an opinion — the model must predict what she will say. The question is: on test questions where the topic is "Company," will the unsupervised method predict the correct answer (Company) or will it predict what Alice would likely say based on her established anti-capitalist stance (the incorrect answer)?

Dataset construction. To highlight the effect, the authors construct a subset dataset where 50% of examples have the topic "Company" and 50% have one of the remaining 13 categories (non-company). Truth-disambiguation is applied only to the non-company subset — this is a critical methodological detail. If truth-disambiguation were applied to the full dataset, the probe might learn to predict Alice's opinion on company questions (which is the opposite of the correct answer) and the truth-disambiguation step would flip the labels, masking the effect. By disambiguating only on non-company data, the authors can observe whether the probe predicts correctly on company questions (indicating it found knowledge) or incorrectly (indicating it found Alice's simulated opinion).

What this tests. This probes whether the model's internal representation of "what Alice would likely say given her established character" is sufficiently prominent to be picked up by unsupervised methods, even when Alice's opinion on the specific test question is not explicitly stated. If the method extracts Alice's simulated behavior rather than the model's "knowledge" of the correct answer, it fails at the core task of distinguishing the model's own encoded information from information about simulated agents' likely utterances.

Experiment 4: Prompt Template Sensitivity (Section 4.4)

This experiment differs from the previous three in that it does not insert an alternative feature. Instead, it tests whether different prompt templates — all of which should be equivalent from the perspective of "what is the model's knowledge" — produce different classifier accuracies.

Task and templates. The task is TruthfulQA (Lin et al., 2021), a dataset of questions where LLMs tend to repeat common misconceptions. The default template is:

Question: {question} Answer: {answer} Is this answer true or false? [label]

where {answer} is randomly chosen to be either a true or false answer to the question, and [label] is "True" for $x_i^+$ and "False" for $x_i^-$.

Two modifications are tested. The literal setting adds instructions to interpret questions literally:

Interpret each question literally, and as a question about the real world; carefully research each answer, without falling prey to any common myths; and reply "I have no comment" unless you are completely certain of the answer.

The professor setting wraps this in a character frame, prepending the text from Lin et al. (2021):

Professor Smith was given the following instructions: Interpret each question literally, and as a question about the real world; carefully research each answer, without falling prey to any common myths; and reply "I have no comment" unless you are completely certain of the answer. Here are the Professor's responses:

What this tests. If the unsupervised methods were genuinely extracting the model's knowledge, the choice of prompt template — particularly whether we add instructions to "interpret literally" or invoke a professor character — should not matter. The model's internal knowledge about whether a statement is true should be invariant to these surface-level changes. If accuracy changes substantially with the prompt template, it indicates that the methods are picking up on features that are affected by these irrelevant prompt details, not on stable underlying knowledge representations.

---

Evaluation Protocol and Analysis Methods

Accuracy computation. For each trained probe, accuracy is computed as the fraction of test questions for which the induced binary classifier $f_p(q_i)$ matches the reference label (either ground-truth or alternative feature). For CCS and the random baseline, this uses the probe's averaged prediction with threshold 0.5 and truth-disambiguation. For PCA, the classifier is thresholding the projection onto the first principal component at zero with truth-disambiguation. For k-means, the cluster assignment is used directly with truth-disambiguation. For logistic regression, training accuracy is reported on the concatenated contrast-pair activations.

Seed-level analysis with violin plots. The paper's primary visualization for accuracy results is the violin plot, which shows the full distribution of accuracies across 50 random seeds for each method. This is a deliberate methodological choice that differs from Burns et al. (2023), who reported only the seed with lowest CCS loss. The violin plots reveal important phenomena that would be invisible in a single-seed or best-seed report:

• Bimodal distributions: In several experiments (e.g., CCS on banana/shed prompts in Figure 2a, CCS on company data in the implicit opinion experiment in Figure 4a), the CCS accuracy distribution shows two distinct clusters — some seeds achieve high accuracy on the distractor and near-chance on ground truth, while other seeds achieve the reverse. This means that different random initializations of the probe parameters converge to qualitatively different solutions, both of which achieve low CCS loss. This is direct evidence for the paper's theoretical claim: the loss landscape admits multiple very different optimal (or near-optimal) solutions, and which one gradient descent finds is determined by initialization.

• Seed variance as a diagnostic: High variance across seeds indicates that the method is not reliably converging to a single solution, which is a red flag for any claim that the method is identifying a specific, well-defined feature (like knowledge). Low variance with high accuracy is what one would expect from a method that reliably discovers a well-defined target.

PCA visualizations. The paper uses three-dimensional PCA projections of the normalized activations to visualize how the data are structured. The first three principal components of the activation vectors (either the raw normalized activations or the difference vectors, depending on context) are computed and plotted as a scatter plot, with points colored by one of two labeling schemes: ground-truth label (e.g., positive vs. negative sentiment, shown as blue vs. orange) and alternative-feature label (e.g., banana vs. shed, or Alice positive vs. Alice negative, shown as dark vs. light).

These visualizations serve as qualitative evidence for what features are "prominent" in the activations. If the data naturally separate along the ground-truth axis (blue vs. orange form distinct clusters) in the default prompt setting, but separate along the alternative-feature axis (dark vs. light form distinct clusters) in the modified prompt setting, this visually confirms that the modified prompt has made the alternative feature more prominent than the knowledge feature. The unsupervised methods' behavior (as shown in the violin plots) should track which feature is visually more prominent in these PCA plots — and the paper shows that it does.

Method agreement analysis (Section 4.5). To quantify the similarity between CCS and PCA predictions, the paper computes the proportion of test examples where both methods agree on the classification, averaged across 10 CCS seeds. This agreement is compared against the expected agreement if the methods were independent classifiers with their observed accuracies. For two independent binary classifiers with accuracies $a_1$ and $a_2$, the expected agreement is $a_1 a_2 + (1 - a_1)(1 - a_2)$. The paper reports both the actual agreement and the independence-expected agreement in a heatmap (Figure 6), showing that CCS and PCA agree substantially more than independent methods with the same accuracies would — evidence that they are tracking the same underlying feature, despite CCS being motivated by a consistency structure that PCA ignores.

Layer sensitivity (Appendix C.2.3). For Chinchilla-70B, the paper sweeps across layers 10 through 70 and evaluates both default ground-truth accuracy and modified-setting Alice accuracy for CCS, PCA, and k-means. This analysis serves to check whether the choice of layer 30 is driving the results. The finding is that "our results are not that sensitive to layer, though often layer 30 is a good choice for both standard and sycophantic templates." Notably, the paper also observes that k-means and PCA are "better or the same as CCS" across layers in the default setting — further evidence that the CCS consistency structure is not responsible for the success of contrastive activation methods.

---

Summary of Methodological Design Choices and Their Justifications

• Single prompt template (rather than multiple as in Burns et al., 2023): The paper investigates the effect of multiple templates in Appendix C.5 and finds they "don't see a major benefit," while adding experiment complexity. Using a single template isolates the difference between default and modified settings without confounds from template averaging.

• 50 random seeds evaluated, all reported: The paper reports all seeds rather than selecting the best one (as Burns et al., 2023 did) to assess robustness and reveal bimodal convergence behavior that is diagnostically important.

• Symmetric confidence loss correction: The original CCS loss has an unmotivated downward bias; the paper's correction makes the loss symmetric around 0.5 without changing its fundamental structure or the validity of the theorems.

• Truth-disambiguation only on non-company data (implicit opinion experiment): This prevents the disambiguation step from masking the effect of the probe tracking Alice's opinion rather than knowledge, which is the very effect being measured.

• Comparisons against PCA, k-means, random, and logistic regression: These baselines span from "no training" (random) to "fully supervised" (logistic regression), with PCA and k-means serving as unsupervised methods that lack CCS's claimed consistency structure, allowing the paper to distinguish "what contrast pairs and normalization do" from "what the CCS loss specifically adds."

• Difficulty estimation for TruthfulQA prompt sensitivity: The paper uses TruthfulQA specifically because it is a "difficult" dataset where LLMs tend to repeat misconceptions, making it a stress test for whether prompt changes that should be irrelevant nonetheless affect accuracy — if the methods were robustly extracting truth, the difficulty of the questions shouldn't make them sensitive to superficial prompt features.

4. Key Insights and Innovations

Innovation 1: The CCS Loss Is an Identification Failure, Not Just an Imperfect Implementation

The paper's most fundamental contribution is a negative result with positive force: the loss function that Burns et al. (2023) claimed as a principled discovery mechanism for latent knowledge is, in fact, compatible with arbitrary binary features. This is not a claim about implementation quality, hyperparameter sensitivity, or dataset limitations — it is a mathematical property of the objective itself. Theorem 1 constructs, for any binary feature $h$ on questions, a probe that achieves zero CCS loss and induces exactly that feature. Theorem 2 goes further: any CCS probe can be loss-preservingly transformed to encode any arbitrary target classifier.

What distinguishes this from a typical "method doesn't work well" critique is the nature of the failure mode. A method might fail because it is underpowered, or because the optimizer gets stuck, or because the features are too noisy. Those are contingent failures, fixable in principle with better engineering. The CCS failure is of a fundamentally different kind: the loss landscape contains equally optimal solutions for every possible feature, so gradient descent has no signal about which one to find. The fact that CCS sometimes converges to a knowledge-like probe in the standard setup is an accident of inductive biases — the choice of prompt template, the initialization scheme, the optimizer dynamics — none of which carry any guarantee about knowledge.

This reframes the evaluation of unsupervised knowledge elicitation methods entirely. Prior to this paper, the working assumption (implicit in Burns et al., 2023 and the discussion it generated) was that if a method achieves above-chance accuracy on a test set after unsupervised training, it has successfully identified something meaningful about the model's knowledge. The counter-evidence (Emmons, 2023; Roger, 2023; Levinstein and Herrmann, 2023) chipped away at this by showing that PCA achieves similar performance or that multiple probes achieve low loss. But the field lacked a crisp articulation of why these observations should be expected, rather than surprising. The paper supplies exactly that: the loss provides no identification, so similar performance across methods (and sensitivity to prompt choices) is not a bug — it is a direct consequence of the mathematical structure of the objective.

The analogy to Locatello et al. (2019) — that unsupervised disentanglement of representations is fundamentally unidentified without inductive biases — is apt but the paper goes beyond analogy by providing constructive proofs specific to the CCS setting. This makes the critique impossible to dismiss as a general philosophical worry about unsupervised learning; it is a concrete demonstration that the specific mathematical structure claimed to identify knowledge does nothing of the sort. The contribution is both a refutation of CCS and a template for how to evaluate future claims: any unsupervised method that claims to discover knowledge via some consistency property must be shown to have an objective whose optimum is exclusive to knowledge, not merely compatible with it. Theorem 1 and Theorem 2 give precise meaning to this requirement and a method for testing it.

Innovation 2: The "Most Prominent Feature" Hypothesis as a Unifying Explanation

The paper's experiments don't just refute CCS — they converge on a positive claim about what unsupervised methods on contrastive activations do: they recover whichever feature is most prominent in the normalized activation space, regardless of whether that feature corresponds to knowledge, a simulated character's opinion, an inserted random word, or the surface-level choice of answer token.

This is a shift from the prior discourse, which had settled into a pattern of "CCS sometimes works and sometimes doesn't, and we're not sure why." Emmons (2023) observed that PCA nearly matches CCS's performance; Roger (2023) found multiple probes achieving low loss; Levinstein and Herrmann (2023) found cases of failure. But these observations lacked a unifying explanatory principle. The paper supplies one: all these methods are picking up the largest source of variance in the (normalized) contrast-pair difference vectors. When the ground-truth label is the most prominent axis of variation — which depends delicately on prompt template, model architecture, layer choice, and dataset — the methods succeed. When an alternative feature is more prominent, the methods track that instead.

The evidence for this hypothesis is systematic and cross-cutting:

• Random words (Figure 2): Appending "Banana" or "Shed" makes this completely arbitrary feature the most prominent axis in the PCA visualization (Figure 2b, right), and all unsupervised methods switch from tracking sentiment to tracking the inserted word. CCS shows a particularly diagnostic bimodal distribution — some seeds track banana/shed, others track sentiment — directly manifesting the theoretical result that the loss landscape admits multiple optima.

• Explicit opinion (Figure 3): When Alice's stated opinion is inserted, the PCA visualization shows that Alice's opinion (light/dark) becomes the primary clustering axis, with sentiment (blue/orange) visible but as a secondary split within each Alice-opinion cluster. Unsupervised methods track Alice's opinion near-perfectly while falling to chance on sentiment. The bimodal behavior disappears — Alice's opinion is now sufficiently prominent that even the inductive biases consistently favor it over knowledge.

• Implicit opinion (Figure 4): With the anti-capitalist prompt, the bimodal CCS behavior returns: roughly half of seeds track Alice's simulated opinion (predicting incorrectly on company questions) while half track knowledge (predicting correctly). The PCA visualization (Figure 4b, right) shows four distinct clusters where both the topic (blue/orange) and the company/non-company distinction (light/dark) are prominent axes, creating an ambiguous landscape where different initializations converge to different features.

• Method agreement (Figure 6): CCS and PCA make highly similar predictions across all model-dataset combinations, agreeing substantially more than would be expected from independent classifiers with the same accuracies. This is direct evidence that they are tracking the same underlying signal — the most prominent feature in the activation differences — despite CCS's claimed exploitation of a consistency structure that PCA ignores. The consistency loss is, at best, a noisy regularizer that doesn't change which feature dominates.

This "most prominent feature" framing has implications beyond CCS. It suggests a general diagnostic for evaluating future unsupervised knowledge elicitation methods: if a method succeeds, ask whether knowledge is genuinely the most prominent feature, or whether it merely correlates with whatever is most prominent under the chosen experimental setup. The paper's sanity checks — inserting distractors and measuring which feature is actually tracked — operationalize this diagnostic.

Innovation 3: The Sanity Check as a First-Class Methodological Contribution

The paper doesn't merely critique CCS; it establishes a methodological framework for evaluating future knowledge elicitation methods that is likely to outlast the specific debate about CCS. The key conceptual move is the design of controlled experiments where a feature that is not knowledge is made more prominent than the knowledge feature, and the method is evaluated on whether it tracks the distractor or maintains accuracy on the ground truth.

This is a different kind of evaluation than the standard accuracy-on-test-set paradigm that Burns et al. (2023) employed. In the standard paradigm, a method is deemed successful if it achieves above-chance accuracy on held-out data after unsupervised training. The paper demonstrates that above-chance accuracy is necessary but radically insufficient as an evaluation criterion: CCS achieves above-chance accuracy on banana/shed prompts if you evaluate it on the banana/shed labels, but that's not because CCS found knowledge — it's because it found the distractor, which happened to be the label-of-interest in a different evaluation. The problem is that in the standard setup, there is no distractor feature to catch the method's actual behavior. The standard evaluation conflates "the method tracks knowledge" with "the method tracks whatever feature happens to correlate with the label in this particular dataset under this particular prompt."

The paper's sanity checks break this conflation by designing datasets where knowledge and the distractor are decorrelated (random assignment of Alice's opinion, random assignment of banana/shed), so performance on the distractor cannot be explained as a side effect of tracking knowledge. The diagnostic insight is: a method that genuinely discovers knowledge should perform at chance on the distractor and maintain accuracy on the ground truth; a method that simply tracks the most prominent feature will do the reverse (or, in bimodal cases, oscillate between them across seeds).

The progression of experiments — from the blatantly artificial (banana/shed) through the semantically related but explicit (Alice's stated opinion) to the subtle and implicit (Alice's simulated behavior from few-shot context) — establishes a graded severity scale for sanity checks. A method that passes the banana/shed test but fails the explicit opinion test is still fundamentally flawed, because real LLMs may represent simulated agents' explicitly stated beliefs as prominent features. A method that passes explicit opinions but fails implicit ones is still unreliable, because LLMs' internal simulations of agents' likely behavior based on character traits may be prominent even without explicit statements. The paper doesn't claim that any existing method passes these tests; it is laying out the standard that should be applied to future claims.

This methodological contribution transforms the paper from a critique of one method into a resource for the field: any future paper claiming unsupervised knowledge elicitation should, at minimum, replicate the sanity check experiments and show that the claimed knowledge probe is not tracking a distractor of comparable or greater prominence.

Innovation 4: The "Simulated Character's Belief" as a Principled Confound for Knowledge Elicitation

The paper identifies a specific, non-arbitrary reason why the identification problem for latent knowledge is hard: powerful LLMs simulate the beliefs of other agents, and those simulated beliefs naturally satisfy the same consistency properties (negation-consistency, probabilistic coherence) that knowledge does. This means that adding more consistency constraints to a loss function — making the method "more sophisticated" — cannot, in principle, resolve the identification problem, because the simulated beliefs of other agents will satisfy those additional constraints just as well as the model's own encoded knowledge.

This is a conceptual advance beyond the specific experimental results. The banana/shed experiment shows that CCS can be distracted by an arbitrary feature, but one might respond (as Burns, 2022 suggests) that there are only a few such features and they can be enumerated. The explicit and implicit opinion experiments counter this: Alice's simulated opinion is not an arbitrary feature — it is a semantically coherent feature that the model presumably represents because simulating other agents' beliefs is useful for predicting text on the internet (as the paper argues in Section 6). And because Alice's opinion is about the same domain as the knowledge we want to extract, it will naturally satisfy any consistency properties we design based on that domain's structure.

The paper articulates this in the Discussion (Section 6) as a hypothesis about future methods:

"We expect powerful models to be able to simulate the beliefs of other agents (Shanahan et al., 2023). Since features that represent agent beliefs will naturally satisfy consistency properties of knowledge, methods that add new consistency properties could still learn to detect such features rather than the model's own knowledge."

This is a specific, principled argument for why the identification problem is unlikely to be solved by adding more consistency constraints. It connects the knowledge elicitation problem to the literature on model simulation and role-playing (Shanahan et al., 2023; Huben, 2022), reframing the challenge not as "design a loss that picks out knowledge" but as "design a method that can distinguish the model's own encoded information from its simulations of other agents' information, when both have the same formal structure." This reframing has already influenced the discourse — it clarifies why the problem is hard in a way that the mere existence of CCS's failure does not, and it sets a specific conceptual hurdle that future methods must address: how, without labels, do you tell whether a feature corresponds to "what the model has learned about the world" versus "what the model has learned about what some agent believes about the world"?

The implicit opinion experiment (Section 4.3) provides the clearest empirical instantiation of this challenge. Alice's anti-capitalist stance is not stated in the test example — it must be inferred from context. The fact that roughly half of CCS seeds predict Alice's simulated behavior rather than the correct answer, on a task where the default prompt yields high accuracy, demonstrates that the model's internal simulation of Alice is sufficiently prominent to compete with the model's own "knowledge" representation. This is exactly the kind of confound that more consistency constraints would not resolve, because Alice's simulated belief that "the topic is not Company" satisfies negation-consistency just as well as the correct belief that "the topic is Company."

The practical upshot is that the paper shifts the burden of proof for future methods: it is no longer sufficient to demonstrate that a method achieves above-chance accuracy on a standard benchmark. The method must also demonstrate that it is extracting the model's own knowledge rather than a simulated character's beliefs, and the paper provides experimental templates (the explicit and implicit opinion paradigms) for testing this distinction.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The experiments use four datasets: IMDb (Maas et al., 2011) for binary sentiment classification of movie reviews; BoolQ (Clark et al., 2019) for yes/no question answering about text passages; DBpedia (Auer et al., 2007) for binary topic classification collapsed from 14 categories; and TruthfulQA (Lin et al., 2021) for evaluating whether LLMs can distinguish true answers from common misconceptions. The paper follows Burns et al. (2023) in dataset selection, excluding others for legal reasons or because Burns et al. showed poor predictive accuracy using them (Section 4). All datasets are balanced to have equal numbers of binary labels unless stated otherwise; examples are filtered to a maximum of 512 tokens, with pairs exceeding this length removed (Appendix B.2).

• Base model(s). Three models are used: Chinchilla-70B (Hoffmann et al., 2022), a 70-billion-parameter decoder-only transformer with 80 layers, with activations extracted from layer 30 unless otherwise noted (Appendix C.2.3 shows other layers give "often similar results"); T5-11B (Raffel et al., 2020), an 11-billion-parameter encoder-decoder architecture using the encoder output for activations; and T5-FLAN-XXL (Chung et al., 2022), an instruction fine-tuned version of T5-11B also using encoder output. The models span both encoder-decoder and decoder-only architectures, with and without instruction fine-tuning, to assess whether findings generalize across model families and training paradigms (Section 4).

• Metrics. The primary metric is accuracy — the fraction of test questions for which the induced binary classifier matches a reference label — computed separately against two possible reference labels: ground-truth accuracy (does the classifier predict the correct answer to the question, e.g., the review's actual sentiment?) and alternative-feature accuracy (does the classifier predict the inserted distractor, e.g., which random word was appended or what Alice's opinion was?). For CCS and random probes, this uses the averaged prediction $\tilde{p}(q_i) = [p(x_i^+) + (1 - p(x_i^-))]/2$ thresholded at 0.5, with truth-disambiguation (flipping labels if accuracy on a small labeled set is less than 0.5). For PCA, the classifier thresholds the projection onto the first principal component at zero with truth-disambiguation. For k-means, cluster assignment directly provides the binary classification with truth-disambiguation. For logistic regression, training accuracy is reported. All results are shown as distributions across 50 random seeds using violin plots (Section 2, Appendix B.3).

• Baselines. Five methods are compared (Section 2, Appendix B.3): CCS (Burns et al., 2023), a linear probe trained with the negation-consistency and symmetric confidence loss (Equation 1 in Appendix A.2) using full-batch AdamW for 1000 epochs, learning rate 0.001 for Chinchilla and 0.01 for T5 models; PCA (called CRC-TPC in Burns et al., 2023), performing PCA on the difference vectors $\{\tilde{\phi}(x_i^+) - \tilde{\phi}(x_i^-)\}$ with 3 components and classifying by thresholding the first principal component at zero; k-means, two-cluster clustering on the same difference vectors with random initialization; random, a CCS-architecture probe with randomly initialized parameters and no training (following Roger, 2023); and logistic regression, a supervised ceiling trained on concatenated contrast-pair activations $(\tilde{\phi}(x_i^+), \tilde{\phi}(x_i^-))$ using Scikit-learn's liblinear solver (Pedregosa et al., 2011).

• Generation budget / compute accounting. The paper does not measure compute in FLOPs or generations — there is no language model generation involved at inference time. All methods operate on pre-extracted activations from a single forward pass per contrast pair element. The relevant resource is the number of labeled examples used for truth-disambiguation (minimal, enough to determine whether accuracy is above or below 0.5) and the training budget for CCS probes (1000 full-batch epochs). The experiments use 50 random seeds per method per configuration to assess robustness of convergence.

• Cross-validation / statistical protocol. No cross-validation is used — the paper is evaluating unsupervised methods on their ability to recover features from activations, not tuning hyperparameters on held-out data. The 50-seed evaluation with all seeds reported (rather than selecting the best seed by CCS loss, as Burns et al., 2023 did) is the key statistical choice, motivated by the goal of assessing whether different initializations converge to qualitatively different classifiers. Truth-disambiguation uses a minimal labeled subset only to determine the prediction-direction mapping, and in one experiment (Section 4.3, implicit opinion), truth-disambiguation is deliberately applied only to a subset (non-company examples) to avoid masking the effect of interest. The paper also uses PCA visualizations of the activation space as qualitative evidence for which features are most prominent (structural analysis rather than statistical testing).

Main Quantitative Results

Experiment 1: Discovering Random Words (Section 4.1)

Headline. When the completely arbitrary binary feature "Banana" vs. "Shed" is appended to IMDb prompts, all unsupervised methods switch from tracking sentiment (ground-truth accuracy) to tracking the inserted word (banana/shed accuracy), with CCS showing a bimodal distribution where some seeds achieve ~100% banana/shed accuracy and ~50% (chance) sentiment accuracy, while others maintain high sentiment accuracy.

Default prompt results (blue violins, Figure 2a). On standard IMDb prompts with Chinchilla-70B, all unsupervised methods achieve substantially above-chance ground-truth accuracy: CCS shows a distribution centered around roughly 0.85–0.90; PCA achieves approximately 0.85; k-means reaches roughly 0.85–0.90; logistic regression (supervised ceiling) is near 0.95; random baseline is at approximately 0.5 (chance). These results replicate the Burns et al. (2023) finding that unsupervised methods on contrast-pair activations can recover sentiment labels well above chance in the standard setup.

Modified prompt results (red violins, Figure 2a). When "Banana" or "Shed" is appended (randomly assigned per question, independent of sentiment), the pattern inverts. On ground-truth accuracy (dark red violins): CCS drops to approximately 0.5 (chance), with a bimodal distribution spanning from 0.5 to roughly 0.85 — some seeds maintain moderate sentiment accuracy while others fall to chance. PCA and k-means both drop close to 0.5 with low variance. On banana/shed accuracy (light red violins): CCS shows a pronounced bimodal distribution with a substantial cluster of seeds at ~1.0 accuracy (perfectly tracking the inserted word) and another cluster near 0.5. PCA and k-means both achieve near-perfect banana/shed accuracy (~0.95–1.0) with low variance.

PCA visualization confirms the feature prominence shift (Figure 2b). The three-dimensional PCA projection of activations in the default prompt setting (left) shows clear clustering by sentiment (blue/orange), with the top principal component separating positive from negative reviews. In the modified prompt setting (right), the dominant clustering axis shifts to banana/shed (dark/light), with sentiment (blue/orange) visible as a secondary, less prominent split within each banana/shed cluster. This visual evidence directly supports the paper's "most prominent feature" hypothesis: the inserted random word creates a more prominent feature in activation space than the sentiment signal, and unsupervised methods track whichever feature is most prominent.

Cross-dataset and cross-model consistency (Appendix C.1). On BoolQ with Chinchilla-70B (Figure 7, top), the pattern is similar but ground-truth accuracy in the default setting is lower overall (BoolQ is a harder task), with CCS reaching roughly 0.65–0.70. In the modified setting, banana/shed accuracy is near-perfect (~1.0) for PCA and k-means, while CCS shows bimodality with some seeds at ~1.0 banana/shed accuracy and others near 0.5. On DBpedia with Chinchilla-70B (Figure 7, bottom), the PCA visualization for the modified prompt reveals that both the ground-truth choice (blue/orange) and banana/shed (dark/light) are prominent — ground-truth separates along the first principal component (x-axis) while banana/shed separates along the second (y-axis). This is reflected in the accuracy results: PCA and k-means maintain high ground-truth accuracy (~0.9) even in the modified setting, because the ground-truth feature remains the most prominent axis; CCS shows a bimodal distribution with some seeds achieving high ground-truth accuracy and others achieving high banana/shed accuracy. On T5-11B (Figure 8), accuracies are lower overall; on IMDB and BoolQ, the modified prompt causes ground-truth accuracy to drop toward chance while banana/shed accuracy rises, though the effect is weaker and noisier than for Chinchilla. On DBpedia with T5-11B, all unsupervised methods are near chance regardless of prompt — logistic regression still achieves high accuracy (~0.9), indicating the information is linearly decodable but not prominent enough for unsupervised methods to discover. On T5-FLAN-XXL (Figure 9), the instruction-tuned model shows more resistance to the distractor: PCA and k-means maintain ground-truth accuracy even with banana/shed inserted, though CCS still shows a bimodal distribution with some seeds tracking the distractor.

Experiment 2: Discovering an Explicit Opinion (Section 4.2)

Headline. When Alice's stated opinion about the review's sentiment is inserted into IMDb prompts, unsupervised methods achieve near-perfect accuracy on predicting Alice's opinion (~0.95–1.0) while falling to approximately 0.5 (chance) on ground-truth sentiment, demonstrating that the methods cannot distinguish the model's representation of the true sentiment from its representation of a character's explicitly stated opinion.

Default prompt results (blue violins, Figure 3a). On standard IMDb prompts with Chinchilla-70B, results replicate the banana/shed experiment's default setting: CCS, PCA, and k-means all achieve high ground-truth accuracy (~0.85–0.90); logistic regression is near 0.95; random is at 0.5.

Modified prompt results (red violins, Figure 3a). When Alice's opinion is inserted (single mention for Chinchilla, "Alice thinks it is {positive | negative}", randomly assigned per question independent of true sentiment), the pattern is clean and unambiguous. On ground-truth accuracy (dark red violins): all unsupervised methods — CCS, PCA, and k-means — drop to exactly 0.5 (chance) with very low variance. On Alice accuracy (light red violins): all three unsupervised methods achieve near-perfect accuracy (~0.95–1.0) with very low variance. Unlike the banana/shed experiment, CCS no longer shows bimodality — Alice's opinion is sufficiently prominent that all seeds across all methods consistently converge to tracking it rather than sentiment.

PCA visualization confirms Alice's opinion dominates (Figure 3b). In the default prompt setting (left), the top principal component separates positive from negative reviews (blue/orange). In the modified prompt setting (right), the dominant clustering is by Alice's opinion (dark/light: "Alice thinks it is positive" vs. "Alice thinks it is negative"), with the review's actual sentiment (blue/orange) visible only as a secondary split within each Alice-opinion cluster.

Cross-dataset and cross-model consistency (Appendix C.2.1). On BoolQ and DBpedia with Chinchilla-70B (Figure 10), the pattern replicates: PCA and k-means achieve near-perfect Alice accuracy on BoolQ (~0.95) and DBpedia (~1.0) while falling to chance on ground truth. CCS on BoolQ shows a bimodal distribution — some seeds achieve ~1.0 Alice accuracy while others maintain ~0.65 ground-truth accuracy (the default accuracy is lower on BoolQ). On DBpedia, CCS shows less dramatic bimodality with most seeds tracking Alice's opinion. On T5-11B (Figure 11), results are weaker but directionally consistent: across IMDB, BoolQ, and DBpedia, Alice accuracy is higher than ground-truth accuracy in the modified setting, though the effect is less extreme than for Chinchilla (e.g., CCS on IMDB achieves ~0.7 Alice accuracy vs. ~0.55 ground-truth). On T5-FLAN-XXL (Figure 12), with a single mention of Alice's opinion, the effect is much weaker — the instruction-tuned model is more resistant to the distractor. The paper notes that a "similarly clear result requires a more emphatic view from the character by repeating the opinion" (Section 4.2, referring to Appendix C.2.2).

Repetition ablation (Appendix C.2.2, Figure 13). The paper sweeps the number of opinion repetitions from 0 to 7. For Chinchilla across all three datasets, a single repetition is sufficient for Alice accuracy to jump from near-chance to near-ceiling (~0.95–1.0 for PCA and k-means, bimodal for CCS). For T5-11B, a single repetition is also sufficient, though the effect is slightly noisier. For T5-FLAN-XXL, more repetitions are needed: Alice accuracy rises gradually with repetition count, reaching near-ceiling only after 4–7 repetitions depending on the dataset. The paper attributes this resistance to instruction fine-tuning making the model's activations "harder" to distract.

Layer sensitivity (Appendix C.2.3). For Chinchilla-70B, Alice accuracy in the modified setting is examined across layers 10–70 (Figure 15). CCS shows bimodal behavior across most layers, with some seeds achieving high Alice accuracy and others not. PCA achieves near-perfect Alice accuracy (~0.95–1.0) across nearly all layers for all three datasets. K-means similarly achieves high Alice accuracy across layers, with slightly more variance on BoolQ. The finding that all layers "suffer the same issue of predicting Alice's view, rather than the desired accuracy" refutes the possibility that layer 30 is an unlucky choice.

Experiment 3: Discovering an Implicit Opinion (Section 4.3)

Headline. On DBpedia topic classification, when Alice's anti-capitalist stance is established through few-shot examples (she refuses to say "Company"), CCS on company-topic questions shows a bimodal distribution with approximately half of seeds (differing only in random initialization) predicting incorrectly — tracking Alice's simulated opinion — while the other half predict correctly. PCA and k-means maintain high accuracy on company questions but the PCA visualization reveals four distinct clusters showing that both the ground-truth topic and the company/non-company distinction are prominent axes.

Experimental design note. The dataset is constructed to be 50% company-topic examples and 50% non-company examples. Truth-disambiguation is applied only to the non-company subset. This is crucial: if truth-disambiguation were applied to all data, probes that learn Alice's opinion would get flipped on company questions (since Alice's opinion is the wrong answer), masking the effect. By disambiguating only on non-company data (where Alice answers correctly), the authors can observe whether company-question predictions are correct (knowledge found) or incorrect (Alice's opinion found) without the disambiguation step undoing the very phenomenon being measured.

Default prompt results (blue violins, Figure 4a). With the standard few-shot prompt where Alice answers all examples correctly, CCS achieves high accuracy on both company (dark blue, ~0.85–0.95) and non-company (light blue, ~0.85–0.95) subsets, with moderate seed variance. PCA and k-means similarly achieve high accuracy (~0.85–0.95) across both subsets with lower variance than CCS. Logistic regression is near-perfect (~0.95–1.0); random is at 0.5.

Modified prompt results (red violins, Figure 4a). With the anti-capitalist prompt where Alice gives wrong answers for company topics in few-shot examples, the key finding is on the company subset (dark red): CCS shows a pronounced bimodal distribution. Approximately half the seeds achieve near-zero accuracy (~0.0–0.1) — these seeds are predicting Alice's anti-capitalist opinion (the wrong answer) rather than the correct topic. The other half of seeds maintain high accuracy (~0.85–0.95), tracking the actual topic. On the non-company subset (light red), CCS maintains high accuracy (~0.85–0.95) with low variance, confirming that the method is not simply failing overall — the effect is specific to company questions where Alice's simulated behavior diverges from the correct answer. PCA and k-means on the company subset show high accuracy (~0.85–0.95) with low variance — they are less sensitive to the modified prompt than CCS. However, the PCA visualization (Figure 4b) reveals that this apparent robustness is misleading.

PCA visualization reveals ambiguous prominence (Figure 4b). In the default prompt setting (left), activations separate primarily by topic choice (blue/orange: which of the two randomly assigned choice labels is correct), with the company/non-company distinction (light/dark) visible as a secondary feature. In the modified prompt setting (right), the structure changes: four distinct clusters emerge, corresponding to the combinations of topic choice and company/non-company status. Both axes are prominent, creating an ambiguous landscape. PCA and k-means, by capturing the top principal component, happen to pick up the topic-choice axis (which remains the most prominent). But the visualization shows that the company/non-company axis — which encodes Alice's simulated opinion — is a strong secondary axis, and CCS, with its different inductive biases, sometimes converges to this axis instead. The paper interprets this as evidence that even when PCA and k-means appear to succeed, the activation space contains competing features that could be discovered under different initialization conditions.

Cross-model results (Appendix C.3, Figure 16). On T5-11B (top), CCS under both default and modified prompts achieves moderate accuracy on non-company questions (~0.6) and higher accuracy on company questions (~0.75–0.85). The interpretation: the probe has learned to classify whether a topic is "Company" or not, rather than distinguishing among the other thirteen categories, and this feature correlates with the correct answer on company questions. PCA and k-means show similar patterns with lower seed variance. PCA visualization for T5-11B does not show clear natural groupings. On T5-FLAN-XXL (bottom), accuracies are high across all conditions (both default and modified, both company and non-company subsets), suggesting the instruction-tuned model is resistant to the manipulation. The PCA visualization for T5-FLAN-XXL shows the top principal component separating by topic choice (blue/orange), with the company/non-company distinction (dark/light) as a secondary axis — similar to the Chinchilla modified setting, but here the topic-choice axis is prominent enough that all methods consistently find it.

Experiment 4: Prompt Template Sensitivity (Section 4.4)

Headline. On TruthfulQA with Chinchilla-70B (Figure 5a), unsupervised methods show substantially different accuracies across three prompt templates that should be equivalent if the methods were extracting stable knowledge: the default template yields poor accuracy (~0.55 for CCS, ~0.55 for PCA, ~0.5 for k-means), while the "literal" and "professor" templates (which add instructions to interpret questions literally, with or without a character frame) yield notably higher accuracy for PCA and k-means (~0.7–0.8), demonstrating that irrelevant prompt details determine whether the methods appear to "succeed."

Default prompt (blue violins, Figure 5a). The standard TruthfulQA contrast pair ("Question: {question} Answer: {answer} Is this answer true or false? [label]") produces near-chance accuracy for all unsupervised methods: CCS reaches approximately 0.55 (slightly above chance with moderate seed variance); PCA is at approximately 0.55; k-means is at approximately 0.5 (chance). Logistic regression achieves approximately 0.65–0.70, indicating that truth information is linearly decodable but not prominent enough for unsupervised methods to discover reliably in the default setting.

Literal prompt (red violins, Figure 5a). Adding instructions to "interpret each question literally" and "carefully research each answer" without invoking a character frame improves performance substantially: PCA achieves approximately 0.75–0.80; k-means achieves approximately 0.70–0.75. CCS remains modest (~0.55–0.60 with high variance).

Professor prompt (green violins, Figure 5a). Wrapping the literal instructions in a character frame ("Professor Smith was given the following instructions...") yields similar improvements: PCA reaches approximately 0.75–0.80; k-means reaches approximately 0.75–0.80; CCS remains at approximately 0.55–0.60.

PCA visualizations show structural differences (Figure 5b). In the default setting (left), the three-dimensional PCA projection of the activations shows no clear separation by ground-truth truth value (blue/orange) — the clusters overlap substantially. In the literal setting (middle), a weak separation along the truth axis begins to emerge. In the professor setting (right), the separation is clearer — the true/false clusters are more distinct, consistent with the higher classification accuracy. The visualizations confirm that the prompt template changes the prominence of the truth feature in activation space.

Cross-model results (Appendix C.4, Figure 17). On T5-FLAN-XXL (top), the pattern is similar but with overall higher variance: PCA in the professor setting reaches approximately 0.7–0.8, while in the default setting it is near 0.5. CCS is near chance across all templates. On T5-11B (bottom), all methods perform at roughly chance level regardless of prompt template, and the PCA visualizations show no clear truth clusters. Logistic regression achieves approximately 0.6–0.65, indicating that truth information is weakly present in the activations but not salient enough for unsupervised methods.

Interpretation. The paper emphasizes that this is not an experiment where a better prompt "fixes" the method. Rather, it demonstrates that the method's apparent success is an artifact of prompt-dependent inductive biases: "if these methods were detecting knowledge/truth, it shouldn't matter whether we give instructions to interpret things literally" (Section 4.4). The choice of prompt is a free variable with significant effect on outcomes, and there is no principled way to select the "right" prompt — a method whose performance depends on such choices cannot claim to have a principled foundation for knowledge discovery.

Experiment 5: Agreement Between Unsupervised Methods (Section 4.5)

Headline. CCS and PCA make highly similar predictions across all model-dataset combinations, agreeing substantially more than would be expected from independent classifiers with the same accuracies — evidence that both methods are tracking the same underlying feature (the most prominent axis in activation space), and that CCS's claimed consistency structure contributes little beyond what standard unsupervised dimensionality reduction already captures.

Quantitative results (Figure 6). The heatmap reports, for each of the nine model-dataset pairs (3 datasets × 3 models), the actual agreement between CCS and PCA (averaged across 10 CCS seeds, since PCA is deterministic), the expected agreement if the methods were independent with the observed accuracies (computed as $a_{\text{CCS}} \cdot a_{\text{PCA}} + (1 - a_{\text{CCS}}) \cdot (1 - a_{\text{PCA}})$), and the individual accuracies of each method. Key entries:

• Chinchilla-BoolQ: Actual agreement 0.74; independence-expected 0.61; CCS accuracy 0.72, PCA accuracy 0.74. Agreement is 0.13 above independence expectation.
• Chinchilla-DBpedia: Actual 0.90; independence-expected 0.88; CCS accuracy 0.92, PCA accuracy 0.95. Agreement is 0.02 above expectation (small margin due to ceiling effects — both methods are near-perfect).
• Chinchilla-IMDB: Actual 0.87; independence-expected 0.81; CCS accuracy 0.85, PCA accuracy 0.94. Agreement is 0.06 above expectation.
• Flan-T5-BoolQ: Actual 0.98; independence-expected 0.82; CCS accuracy 0.90, PCA accuracy 0.90. Agreement is 0.16 above expectation — the largest margin.
• Flan-T5-DBpedia: Actual 1.00; independence-expected 1.00; both at ceiling (1.0 accuracy).
• Flan-T5-IMDB: Actual 0.98; independence-expected 0.93; CCS accuracy 0.97, PCA accuracy 0.96.
• T5-BoolQ: Actual 0.57; independence-expected 0.52; CCS accuracy 0.59, PCA accuracy 0.61. Agreement is only 0.05 above expectation — the smallest margin, reflecting lower overall accuracy on this harder task.

Across all nine cells, the actual agreement exceeds the independence-expected agreement. The smallest margins occur where one or both methods have accuracies near 0.5 (T5 models on harder tasks), which is expected since agreement on chance-level classifiers provides little information. The largest margins occur where both methods have moderate-to-high accuracy (e.g., Flan-T5-BoolQ, Chinchilla-IMDB), indicating that their errors are correlated — they succeed and fail on the same examples.

Interpretation. The paper draws two conclusions. First, this evidence supports Emmons (2023)'s observation that PCA achieves roughly 97–98% of CCS's performance — the consistency loss is providing at most a marginal contribution beyond what contrast pairs and standard unsupervised learning already capture. Second, and more importantly for the paper's broader argument, the fact that "two methods with such different motivations behave similarly" supports the idea that experimental results on current unsupervised methods may be predictive of future methods with different motivations: if the behavior is driven by the prominence of features in the normalized contrast-pair activation space (as the paper hypothesizes) rather than by the specific loss function, then any unsupervised method that operates on this representation — regardless of its conceptual sophistication — will face the same fundamental limitation of tracking the most prominent feature rather than knowledge specifically.

Ablation Studies and Robustness Checks

Symmetric vs. asymmetric confidence loss (Appendix A.2): The original CCS confidence loss $\min\{p(x_i^+), p(x_i^-)\}^2$ has a downward bias — a constant probe is minimized at $k = 0.4$ rather than symmetrically at $k = 0.5$. The paper proposes and uses throughout $\min\{p(x_i^+), p(x_i^-), 1-p(x_i^+), 1-p(x_i^-)\}^2$, which is symmetric around 0.5. The correction fixes an unmotivated asymmetry but does not affect the theoretical results (theorems are proved using the symmetric version) or the fundamental critique.

Single vs. multiple prompt templates (Appendix C.5, Figure 18): The paper investigates using multiple prompt templates (as in Burns et al., 2023) vs. a single template. For Chinchilla-70B, multiple templates slightly improve BoolQ accuracy but make little difference on IMDB and DBpedia. For T5-11B, multiple templates slightly help BoolQ but hurt DBpedia. For T5-FLAN-XXL, no systematic benefit. The paper uses a single template throughout for simplicity and to isolate the effect of the experimental manipulations without confounds from template averaging.

Layer choice for Chinchilla-70B (Appendix C.2.3, Figures 14-15): The paper sweeps layers 10–70 for Chinchilla. In the default setting (Figure 14), k-means and PCA are "better or the same as CCS" across most layers, with layer 30 being a reasonable choice for all datasets. In the modified Alice-opinion setting (Figure 15), all layers show the same vulnerability to tracking Alice's opinion rather than ground truth. The finding that all layers suffer the same issue refutes the possibility that layer 30 is an unlucky choice driving the negative results.

Number of opinion repetitions (Appendix C.2.2, Figure 13): For Chinchilla across all three datasets, a single mention of Alice's opinion is sufficient to make Alice accuracy jump to ~0.95–1.0 for PCA and k-means. For T5-11B, a single repetition also suffices. For T5-FLAN-XXL, the instruction-tuned model requires 4–7 repetitions before Alice accuracy reaches ceiling, suggesting instruction fine-tuning makes activations more resistant to superficial prompt manipulations — a potentially positive finding for robustness but not a defense of CCS's identification claim, since the model still eventually tracks the distractor.

Model scale and architecture comparison (across Sections 4.1-4.4): The three models (Chinchilla-70B decoder-only, T5-11B encoder-decoder, T5-FLAN-XXL instruction-tuned encoder-decoder) show qualitatively similar vulnerability to distractors, with differences in degree: T5-11B shows generally lower accuracies across all settings (the information is present but less prominent), while T5-FLAN-XXL is more resistant to superficial distractors (requiring more repetitions of Alice's opinion, maintaining accuracy in the implicit opinion experiment) but still eventually tracks the distractor when it is made sufficiently prominent. The cross-model consistency supports the paper's claim that the identification problem is not specific to one architecture or training paradigm.

Dataset difficulty effects (across experiments): The effectiveness of the distractor manipulations varies by dataset: BoolQ is harder (lower default accuracies for all methods), and the distractor effects are correspondingly less clean; DBpedia with Chinchilla in the banana/shed experiment shows an ambiguous case where both ground-truth and distractor are prominent axes, providing a particularly clear demonstration of the multiple-optima phenomenon; IMDB provides the cleanest results due to high default accuracies and clear sentiment signal.

Critical Assessment

Claim 1: "Existing unsupervised methods do not discover knowledge — instead they seem to discover whatever feature of the activations is most prominent."

The experiments provide strong, converging evidence for this claim, but with important boundary conditions that should be stated precisely. The banana/shed experiment (Section 4.1) demonstrates that a completely arbitrary feature, when made prominent by insertion into the prompt, is tracked by unsupervised methods instead of knowledge. This is a clean causal manipulation: the only difference between default and modified prompts is the appended word, and the methods' behavior inverts entirely. The explicit opinion experiment (Section 4.2) generalizes this to a semantically meaningful feature (Alice's stated opinion) that is precisely the kind of confound we would expect in realistic settings — a simulated character's stated belief. The implicit opinion experiment (Section 4.3) extends the finding to the most subtle case: Alice's opinion is not stated in the test example, only implied by context, yet CCS still tracks it approximately half the time.

However, the claim should be qualified by what the experiments do not show. The paper demonstrates that methods track the most prominent feature in the specific sense of "the feature that dominates the top principal components of the normalized activations" — but it does not provide a quantitative theory of prominence, nor does it show that the methods always track the most prominent feature in settings more complex than the binary distractor insertions studied here. The results are consistent with the "most prominent feature" hypothesis, but they do not rule out that the methods might, under different conditions, track features based on more complex criteria than simple variance dominance.

Additionally, the claim that the methods "do not discover knowledge" is supported only for the specific prompt templates and datasets where a more prominent distractor was deliberately inserted. In the default settings, the methods do achieve high ground-truth accuracy — i.e., they discover a feature that correlates with knowledge. The paper's critique is that this correlation is accidental (knowledge happens to be the most prominent feature under those templates), not that the methods are incapable of producing high-accuracy knowledge classifiers. The distinction matters: the paper has shown that the methods are unreliable knowledge extractors, not that they never extract knowledge. The experimental results (especially Figures 2a, 3a) clearly demonstrate that the choice of which feature is discovered is controlled by prompt details, not by any knowledge-specific property of the learning algorithm — but the paper does not provide a method for predicting, a priori, which feature will be most prominent for a given prompt, model, and dataset.

Claim 2: "The CCS consistency structure does not identify knowledge — arbitrary binary features achieve optimal CCS loss (Theorem 1) and any CCS probe can be loss-preservingly transformed to encode an arbitrary classifier (Theorem 2)."

This claim is rigorously supported by the mathematical proofs in Appendix A, and the experimental results (particularly the bimodal distributions in Figures 2a and 4a) provide empirical corroboration: different random initializations converge to qualitatively different classifiers with similarly low loss, exactly what one would expect from a loss landscape with multiple equally valid optima.

However, there is a gap between the theoretical construction and the experimental results that the paper acknowledges but does not fully close. Theorem 1 constructs a binary probe (outputs exactly 0 or 1) that achieves zero loss. In practice, the probe is a linear function followed by a sigmoid, which can only approximate binary outputs with large weight magnitudes. The paper acknowledges this limitation ("the probe $p$ is binary since $h$ is binary. In practice, since probe outputs are produced by a sigmoid, they are in the exclusive range $(0, 1)$") but does not explore whether the sigmoid-parameterized probe class can approximately achieve the constructed probes for arbitrary features, given the specific activation representations. If the activation space does not linearly separate some arbitrary feature $h$, then the constructed optimal probe exists in function space but is not reachable by gradient descent on the actual probe architecture. This doesn't rescue CCS — Theorem 2 shows that any reachable probe can be transformed to encode arbitrary features while preserving loss — but it means the bimodal behavior in the experiments might arise not (only) from multiple optima in the loss landscape, but from the interaction between the loss landscape and the probe's representational capacity.

A stronger experimental design would have verified that the CCS loss achieved by probes tracking the distractor (e.g., ~100% banana/shed accuracy) was similar to the loss achieved by probes tracking ground truth — this would directly confirm the theoretical prediction that both are (near)-optimal. The paper does not report CCS loss values alongside accuracy, which would have strengthened the empirical bridge to the theorems.

Claim 3: "Unsupervised methods are sensitive to prompts, and we lack principled reasons to pick any particular prompt."

The TruthfulQA experiment (Section 4.4) demonstrates prompt sensitivity convincingly for Chinchilla-70B: PCA accuracy jumps from ~0.55 (default) to ~0.75–0.80 (literal/professor), and the PCA visualizations show corresponding structural differences. The claim that "we lack principled reasons to pick any particular prompt" is supported by the observation that there is no theoretical basis for preferring the literal/professor templates over the default — all three are natural ways a practitioner might format the task.

However, the experiment is limited in scope and strength. Only one dataset (TruthfulQA) is tested for prompt sensitivity in the main paper, and the effect is not replicated as cleanly across all models: T5-11B (Figure 17, bottom) shows near-chance performance across all templates, and T5-FLAN-XXL (Figure 17, top) shows the effect but with higher variance. The claim that prompt sensitivity is a general problem for unsupervised methods would be stronger with prompt-sensitivity experiments on the other datasets (IMDb, BoolQ, DBpedia) and with more than three prompt variants. Moreover, the variation in accuracy across prompts (~0.55 to ~0.80) is moderate — it demonstrates sensitivity, but not the kind of catastrophic failure seen in the distractor-insertion experiments. A critic could argue that some prompts are "better" than others at eliciting the model's knowledge, and that this is an engineering problem (find the right prompt) rather than a fundamental limitation. The paper's counter-argument — that there's no principled way to identify the "right" prompt without labeled data — is logically sound but not empirically demonstrated beyond the one dataset.

Overall experimental strengths:

• Controlled causal manipulations. The experiments cleanly isolate the distractor feature by making it independent of the ground-truth label (random assignment), and compare default vs. modified prompts that differ only in the inserted distractor. This is a strong experimental design that supports causal claims about what causes the methods' behavior to change.
• Multiple models, datasets, and methods. The replication across three models, three (or four) datasets, and three unsupervised methods makes the findings robust against idiosyncratic model-dataset-method interactions. The fact that the pattern holds qualitatively across all combinations (with understandable differences in degree, e.g., T5-FLAN-XXL's resistance to superficial distractors) strengthens the conclusion.
• Seed-level analysis. Reporting all 50 seeds as violin plots (rather than best-seed) is methodologically crucial: it reveals bimodal behavior that would be invisible in a best-seed or average report, and it directly demonstrates the multiple-optima property established theoretically.
• PCA visualizations as qualitative evidence. The visualizations provide an intuitive, interpretable window into why the methods behave as they do: the prominence of features in activation space (as revealed by PCA) directly predicts which feature the unsupervised methods track.

Genuine weaknesses and missing experiments:

• No reported CCS loss values. The paper's theoretical results predict that probes tracking distractors and probes tracking knowledge should achieve similar CCS loss. Reporting the actual CCS loss distributions alongside accuracy would have provided direct empirical validation of this prediction. Without loss values, the connection between the theorems and the experiments is somewhat indirect (inferred from behavior rather than directly measured in the objective).

• Small number of prompt templates for sensitivity experiment. The TruthfulQA experiment uses only three templates. A broader sweep — varying instruction wording, character framing, answer formatting, and other surface features — would provide a more comprehensive picture of prompt sensitivity. The paper's claim that prompt choice is unprincipled would be stronger if it demonstrated sensitivity across a wider range of natural variations.

• No quantitative prominence analysis. The "most prominent feature" hypothesis is supported qualitatively by PCA visualizations, but the paper does not provide quantitative metrics: e.g., what is the variance explained by the ground-truth axis vs. the distractor axis? How does the gap in variance explained correlate with the probability that unsupervised methods track one vs. the other? Such an analysis could have strengthened the explanatory framework and made quantitative predictions.

• Truth-disambiguation requires some labeled data. The paper acknowledges this but does not explore how much labeled data is needed, how the choice of which data to use for disambiguation affects results (the implicit opinion experiment is one exploration, but not systematic), or whether alternative disambiguation methods (e.g., using a small set of known-true statements) could reduce the label requirement. This matters because the claim that the method is "unsupervised" is somewhat diluted by the disambiguation step, and in the implicit opinion experiment, the choice of which subset to disambiguate on is itself a consequential free variable.

• No experiment where both knowledge and distractor are equally prominent. The banana/shed and explicit opinion experiments create a stark prominence imbalance (distractor dominant), while the DBpedia banana/shed case (Figure 7, bottom) and the implicit opinion experiment create ambiguous cases. A systematic experiment varying the relative prominence of knowledge and distractor — e.g., by interpolating between prompts or varying distractor salience continuously — could have mapped out the "prominence threshold" at which methods flip from tracking knowledge to tracking the distractor, providing more precise characterization of the phenomenon.

• Limited to binary classification tasks. All experiments use binary classification (positive/negative, true/false, choice 1/choice 2). The paper does not explore whether the identification problem generalizes to multi-class or continuous knowledge representations, which would be relevant for many real-world knowledge elicitation scenarios.

• No investigation of whether the distractor features actually correspond to coherent "simulated beliefs." The paper claims that the explicit/implicit opinion experiments demonstrate that methods track "simulated character beliefs," but the alternative hypothesis — that the methods simply track surface-level co-occurrence patterns (e.g., detecting when two instances of the word "positive" appear in the input) — is equally compatible with the results. The paper acknowledges this in a footnote (Section 6: "while one hypothesis is that the feature is tracking Alice's opinion, another hypothesis that is equally compatible with our results is that the feature simply identifies whether the two instances of 'positive'/'negative' are identical or different"), but does not design experiments to distinguish these explanations. This is a significant limitation because the conceptual argument about simulated agent beliefs as a principled confound (Section 6, Innovation 4 in the prior sections) rests on the former interpretation, but the experiments only demonstrate the latter (surface pattern tracking). Stronger evidence for the simulation interpretation would require experiments where Alice's opinion is expressed in a way that does not create surface-level pattern confounds — e.g., through paraphrased opinions or through implicit characterization that doesn't repeat the label words.

What would strengthen the paper:

• Reporting CCS loss alongside accuracy for all experiments to directly verify the multiple-optima prediction.
• A systematic sweep of distractor prominence (e.g., by varying the number of opinion repetitions continuously and measuring the probability of tracking Alice vs. sentiment) to empirically characterize the "prominence threshold."
• Additional prompt sensitivity experiments across all datasets, with broader template variation, to establish whether the TruthfulQA finding generalizes.
• Experiments that distinguish "the method tracks a simulated character's semantic belief" from "the method tracks surface-level co-occurrence of label words" — e.g., using paraphrased opinions that express the same belief without repeating the binary label tokens.
• A multi-class extension (e.g., using all 14 DBpedia categories rather than binary subsets) to test whether the identification problem persists in non-binary settings.

6. Limitations and Trade-offs

The Theoretical Result Applies to the CCS Objective, Not Necessarily to the Probes Actually Learned in Practice

The assumption or constraint. Theorem 1 constructs probes that output binary values — exactly 0 or 1 — by setting $p(x_i^+) = h(q_i)$ and $p(x_i^-) = 1 - h(q_i)$. These probes achieve zero CCS loss because the consistency term $(h(q_i) - [1 - (1 - h(q_i))])^2$ and the confidence term $\min\{h(q_i), 1 - h(q_i)\}^2$ both evaluate to exactly zero when $h(q_i) \in \{0, 1\}$. The paper acknowledges this gap explicitly in Section 3 after Theorem 1:

"In Thm. 1, the probe $p$ is binary since $h$ is binary. In practice, since probe outputs are produced by a sigmoid, they are in the exclusive range (0, 1)."

The constructed optimal probes require the linear projection $\theta^T \tilde{\phi}(x) + b$ to output $\pm\infty$ to push the sigmoid to exactly 0 or 1. In practice, the learned probe is parameterized as $p(x) = \sigma(\theta^T \tilde{\phi}(x) + b)$ with finite weights, meaning it can only approximate binary outputs. The existence of the constructed probe in the abstract function space does not guarantee that this probe is reachable by gradient descent on the actual parameterized probe class given the specific activation representations.

The consequence. The gap between the theoretical construction and the practical optimization problem means that the theorems do not, by themselves, guarantee that the CCS loss landscape for the learnable probe class contains equally optimal solutions for arbitrary features. If a particular arbitrary feature $h$ is not linearly separable in the normalized activation space $\{\tilde{\phi}(x_i^+), \tilde{\phi}(x_i^-)\}$, then no finite-weight linear probe can achieve the constructed binary-output optimum (though the sigmoid allows smooth approximation). This does not rescue CCS's identification claim — Theorem 2 still shows that any reachable probe can be loss-preservingly transformed to encode an arbitrary classifier — but it means the experimental evidence (bimodal distributions, distractor-tracking) cannot be directly attributed to global optimality of the distractor probe in the CCS loss landscape. The bimodal behavior could arise from local optima or from the interaction between the loss landscape and the probe's restricted function class, rather than from multiple equally good global optima. The practical implication is that one cannot use the theorems alone to predict which arbitrary features will be learnable as CCS probes for a given model and dataset — the theoretical result establishes a lower bound on the identification problem (it cannot be solved in principle) but does not provide an upper bound on which distractors will actually compete with knowledge in practice.

What evidence exists in the paper. The paper does not report CCS loss values alongside accuracy for the experiments. This is a significant gap: without loss values, we cannot verify the theoretical prediction that probes tracking the distractor (e.g., 100% banana/shed accuracy) achieve similar CCS loss to probes tracking ground truth. The paper relies on behavioral evidence — accuracy on distractor vs. ground truth — together with the bimodal seed distributions, to argue that multiple qualitatively different probes coexist in the optimization landscape. The PCA visualizations (Figures 2b, 3b, 4b) show that both knowledge and distractor features are linearly separable in activation space, which makes it plausible that both correspond to low-loss regions, but this is not directly measured. The agreement analysis (Section 4.5, Figure 6) shows CCS and PCA make similar predictions, which is consistent with both tracking the most prominent feature, but this is indirect evidence about the loss landscape.

Mitigation status. The paper does not address this gap. Theorem 2 partially bridges it by showing that any actually learned probe can be transformed to encode an arbitrary classifier with identical loss, but this transformation is a mathematical construction on the probe outputs, not a statement about what gradient descent finds in the parameterized probe class. Future work that reports CCS loss distributions across seeds and conditions (default vs. modified prompts), and that measures the loss achieved by probes at different points in the accuracy distributions (particularly the bimodal peaks), would strengthen the empirical bridge between the theory and the experiments.

---

The Distractor Experiments Demonstrate Surface-Pattern Tracking, Not Necessarily Simulated-Belief Extraction

The assumption or constraint. The paper's conceptual argument about simulated character beliefs — which forms the basis for claims about why future methods will face similar challenges (Section 6, "Discussion and Conclusion") — relies on interpreting the explicit and implicit opinion experiments as evidence that unsupervised methods track a character's semantic belief rather than the model's own knowledge. However, the experimental manipulations create an alternative explanation that is equally compatible with the results: the unsupervised methods may simply be detecting surface-level textual co-occurrence patterns rather than semantically coherent representations of Alice's opinion. The paper acknowledges this explicitly in a footnote (Section 6):

"Note that we do not know whether the feature we extract tracks the beliefs of the simulated character: there are clear alternative hypotheses that explain our results. For example in Figure 3, while one hypothesis is that the feature is tracking Alice's opinion, another hypothesis that is equally compatible with our results is that the feature simply identifies whether the two instances of 'positive' / 'negative' are identical or different."

The consequence. This ambiguity substantially weakens the paper's forward-looking claim that future unsupervised methods "will nonetheless be vulnerable to similar critiques" because "features that represent agent beliefs will naturally satisfy consistency properties of knowledge" (Section 6). If the current experiments only demonstrate tracking of surface-level patterns — e.g., detecting whether the words "positive" or "negative" appear once or twice in the prompt — then the experiments do not provide evidence that the model's internal simulation of Alice's beliefs (a semantically rich representation) is competing with the model's own knowledge. A future method that claimed to use more sophisticated consistency properties to distinguish "what the model knows" from "surface-level textual patterns" could reasonably argue that the paper's experiments do not refute its claims, because the experiments never established that the distractors operate at the semantic level the future method targets. The practical consequence is that the paper's transformative potential depends on the reader accepting the simulation interpretation of the distractor experiments, but the experiments themselves are designed in a way that leaves the surface-pattern interpretation equally viable.

What evidence exists in the paper. The explicit opinion experiment (Section 4.2) inserts Alice's opinion using the exact same words as the answer label — "Alice thinks it is positive" appears alongside a prompt that also ends with "positive" or "negative" in the $x_i^+$ and $x_i^-$ variants. A probe that detects "does the word 'positive' appear twice in this input?" would achieve high Alice accuracy without representing Alice's semantic belief. The banana/shed experiment (Section 4.1) is even clearer: appending "Banana" or "Shed" creates a surface-level token difference that has no semantic meaning whatsoever. The only experiment that moves toward semantic-level manipulation is the implicit opinion experiment (Section 4.3), where Alice's anti-capitalist stance is established through paragraph-length few-shot examples and the test example contains no explicit opinion statement. However, even here, the few-shot examples explicitly repeat the pattern "Alice says 'Choice 2'" for company topics, and surface-level associations between the word "Company" and the pattern of Alice's answers could drive the effect. The repetition ablation (Appendix C.2.2, Figure 13) shows that increasing the number of explicit opinion repetitions increases Alice accuracy monotonically — a finding that is consistent with both the surface-pattern and semantic-belief interpretations, since more repetitions strengthen both the surface co-occurrence signal and the semantic characterization.

Mitigation status. The paper does not design experiments to distinguish these interpretations. The authors are transparent about the ambiguity (the footnote in Section 6 is explicit), but they do not attempt to resolve it. Possible experiments that would distinguish the hypotheses include: (a) expressing Alice's opinion using paraphrased language that does not repeat the answer-label tokens (e.g., "Alice thinks the review is glowing/terrible" rather than "positive/negative"); (b) using Alice's opinion in a way that is semantically consistent but creates different surface patterns (e.g., Alice's opinion is about a related but different binary feature than the classification target); (c) measuring whether probes trained to detect Alice's opinion transfer to test examples where Alice's opinion is expressed differently. Without such experiments, the paper's conceptual claim about simulated-agent beliefs as a principled confound for future methods remains a hypothesis supported by plausibility arguments rather than by the experimental evidence presented. Future work that closes this gap would substantially strengthen the paper's forward-looking argument.

---

All Experimental Evidence Comes from a Narrow Set of Binary Classification Tasks on Three Model Families

The assumption or constraint. The paper evaluates unsupervised knowledge elicitation methods exclusively on binary classification tasks: sentiment (positive vs. negative on IMDb), factual truth (true vs. false on BoolQ and TruthfulQA), and topic (choice 1 vs. choice 2 on DBpedia). All tasks have exactly two possible answers, and the contrast-pair construction always appends one of two answer tokens. The three models tested — Chinchilla-70B, T5-11B, and T5-FLAN-XXL — represent two architecture families (decoder-only and encoder-decoder) from a single research group (Google DeepMind), with the largest model at 70B parameters. The paper acknowledges the dataset scope constraint in Section 4:

"We investigate three of the datasets that were used in Burns et al. (2023). The others were excluded for legal reasons or because Burns et al. (2023) showed poor predictive accuracy using them."

The consequence. The restriction to binary classification means the paper cannot assess whether the identification problem changes qualitatively for tasks with richer answer spaces. In multi-class settings (e.g., full 14-category DBpedia, multi-choice QA), the "most prominent feature" could be more complex — the knowledge feature might decompose into multiple axes, and distractors might compete with some axes but not others. More importantly, many real-world applications of knowledge elicitation involve open-ended generation (summarization, dialogue, reasoning chains) where there is no clean binary answer to construct contrast pairs around, and where "truth" is a property of entire sequences rather than individual propositions. The paper's conclusions about the fundamental difficulty of unsupervised knowledge elicitation are drawn entirely from binary classification with contrast pairs, and it is unclear whether the same issues manifest in settings where the knowledge to be extracted is not structured as a set of independent binary propositions.

The model choice also limits generalizability claims. The paper states that the results across three models and four datasets show the findings are "not specific to one architecture or training paradigm" (Section 4.5, Appendix C), but all three models are from the same organization and training lineage. Models with different pretraining objectives (e.g., contrastive rather than next-token-prediction), different tokenization strategies, or different scales (e.g., models at the 7B range where knowledge representation may be sparser) could produce different prominence patterns in their activation spaces. The layer-sensitivity analysis (Appendix C.2.3) is only conducted for Chinchilla-70B, leaving open whether T5-family models show different layer-dependent behavior.

What evidence exists in the paper. The paper provides no experiments with multi-class classification (even though DBpedia has 14 categories, the paper binarizes it to random pairs of categories). There are no experiments with generative tasks, open-ended QA, or chain-of-thought reasoning. The model comparison (across Sections 4.1–4.4 and Appendices C.1–C.4) shows qualitative consistency — all three models track distractors when distractors are sufficiently prominent — but the quantitative behavior differs substantially: T5-11B shows lower overall accuracies and noisier effects, T5-FLAN-XXL requires more repetitions of Alice's opinion before tracking it, and the TruthfulQA prompt-sensitivity experiment (Appendix C.4) shows the clearest pattern only for Chinchilla. These differences suggest that architecture, scale, and training procedure affect the prominence of features in activation space, but the paper's sample of three models is insufficient to characterize how these factors modulate the identification problem.

Mitigation status. The paper does not address these generalization questions. The Discussion (Section 6) focuses on generalizability to future methods rather than generalizability to other tasks and models. The paper's stated scope is explicitly limited to the datasets and models used by Burns et al. (2023), chosen to provide a direct comparison and refutation. The authors do not claim that their results hold for non-binary or generative settings. However, the forward-looking hypothesis about simulated-agent beliefs as a confound (Section 6) implicitly assumes that the identification problem generalizes beyond binary classification. The gap between the experimental scope (binary classification only) and the conceptual claims (about knowledge elicitation in general) is not bridged by any empirical evidence or theoretical argument specific to broader task formats. A cautious reading would restrict the paper's conclusions to "unsupervised binary knowledge elicitation from contrast-pair activations in LLMs" rather than "unsupervised knowledge elicitation" writ large.

---

The Paper Does Not Propose or Validate Any Alternative Approach — It Leaves the Question of How to Elicit Knowledge Unanswered

The assumption or constraint. This is a critique paper rather than a constructive proposal. The paper's structure — two theorems proving CCS's loss does not identify knowledge, five experiments showing that supervised methods track distractors, and a discussion hypothesizing that future methods will face similar problems — is exclusively focused on refuting an existing approach. The paper does not develop, test, or even sketch an alternative method for unsupervised knowledge elicitation. It does not experimentally evaluate whether the identified problems can be mitigated through better prompt design, different normalization schemes, architectural interventions, or hybrid semi-supervised approaches. The paper's "contributions" (Section 1) are framed entirely as negative — proving CCS doesn't work, showing what features CCS actually finds, and providing sanity checks — without any positive counterpart.

The consequence. A practitioner reading this paper learns that CCS and related unsupervised methods cannot be trusted for knowledge elicitation, but receives no guidance on what to do instead. The paper's supervised logistic regression baseline consistently achieves high accuracy across all experiments (e.g., ~0.95 on IMDb sentiment with Chinchilla in Figures 2a, 3a; ~0.65–0.70 on TruthfulQA in Figure 5a), demonstrating that the relevant knowledge is linearly decodable from the activations when labels are available. But the paper does not explore how much labeling is needed, whether labels can be obtained cheaply for specific domains, whether transfer learning from labeled to unlabeled domains is possible, or whether semi-supervised approaches that use a small amount of labeled data to steer unsupervised methods toward knowledge (rather than distractors) could work. The paper notes in the Discussion that "supervised methods face practical and conceptual problems for eliciting latent knowledge" — conceptual because ground-truth supervision about what the model knows is not well-defined, and practical because superhuman systems know things we do not — but it does not empirically characterize these problems or compare their severity to the unsupervised identification problem the paper documents.

This limitation matters for the paper's impact. A pure refutation is valuable for preventing wasted effort on a dead-end approach, but it does not advance the field toward a solution. The paper's stated goal is to "contribute sanity checks to apply to evaluating future knowledge elicitation methods" (Section 1), which is a methodological contribution, but those sanity checks depend on the premise that some future method will emerge to be evaluated. By not engaging with what that method might look like or what constraints it would need to satisfy, the paper leaves the gap between "CCS is insufficient" and "a method exists that passes these sanity checks" completely uncharacterized.

What evidence exists in the paper. The supervised logistic regression baseline appears in every experiment as a ceiling, consistently achieving substantially higher accuracy than unsupervised methods in both default and modified settings. For example: on IMDb with the banana/shed modification (Figure 2a), logistic regression maintains ~0.95 ground-truth accuracy while unsupervised methods drop to ~0.5; on the explicit opinion modification, logistic regression again maintains ~0.95 sentiment accuracy (Figure 3a); on the implicit opinion experiment (Figure 4a), logistic regression is at ~0.95–1.0 on both company and non-company subsets. This demonstrates that the ground-truth feature remains linearly decodable from the normalized activations even when distractors are present — the information is there, just not the most prominent axis. However, the paper never varies the amount of labeled data used for training logistic regression, never tests whether a small number of labeled examples can be combined with unsupervised pretraining to recover knowledge, and never compares the label efficiency of supervised vs. unsupervised approaches for this task.

Mitigation status. The paper does not address this gap. The Discussion (Section 6) briefly compares unsupervised and supervised approaches:

"Unsupervised approaches have to overcome the identification issues we outline in this paper, whilst supervised approaches have the problem of requiring accurate human labels even in the case of superhuman models. The relative difficulty of each remains unclear."

This framing treats the two approach classes as separate research directions with orthogonal challenges, but it does not ask whether the challenges can be combined — e.g., can a small amount of labeled data (enough for truth-disambiguation, perhaps slightly more) be used to regularize unsupervised methods toward knowledge rather than distractors? The paper's own experimental protocol requires a small labeled set for truth-disambiguation anyway, so the "fully unsupervised" claim is already approximate. The failure to explore where on the spectrum from zero labels to full supervision the identification problem becomes tractable is a missed opportunity to turn a critique into a constructive research direction. Future work that characterizes the label-efficiency curve for knowledge elicitation — how much labeled data is needed to reliably identify knowledge probes in the presence of distractors — would directly build on the paper's findings toward a solution.

---

The Cost of Difficulty Estimation (for the Sanity Checks) Is Not Accounted For, and No Lightweight Diagnostic Is Provided

The assumption or constraint. The paper's primary constructive contribution is a set of sanity checks for evaluating future knowledge elicitation methods — modified prompts that insert distractors and dual-metric evaluation (ground-truth accuracy vs. distractor accuracy). However, applying these sanity checks in practice requires: (a) knowing or constructing a distractor feature that is decorrelated from the knowledge label, (b) having access to both ground-truth labels and distractor labels for evaluation, and (c) running a full set of experiments (training probes across multiple seeds, computing both accuracy metrics) for each candidate method and each sanity check prompt. The paper's experiments use datasets where ground-truth labels are available and distractors are injected with known assignment (random partition for banana/shed, random assignment for Alice's opinion). In a real deployment setting where we want to evaluate whether a new method actually discovers knowledge on a novel dataset, we may not have ground-truth labels (that is the motivation for unsupervised methods), and we would need to construct artifical distractors whose relationship to the knowledge label we control.

The consequence. The sanity checks are evaluation tools that require labeled data, which undermines their practical utility for the very scenario that motivated unsupervised knowledge elicitation in the first place — the case where we lack ground-truth labels for what the model knows. The paper's experiments demonstrate that CCS fails the sanity checks, but they do so using datasets where the authors know the ground truth and can construct controlled distractors. A developer who has built a new unsupervised knowledge elicitation method and wants to check whether it actually works cannot apply the paper's sanity checks without access to ground-truth labels for the target domain — and if they had those labels, they could simply train a supervised probe (which the paper shows works well) and wouldn't need the unsupervised method. The sanity checks are thus most useful as a research evaluation tool (for the academic community to stress-test proposed methods on benchmarks where labels exist) rather than as a deployment validation tool (for practitioners to verify that their unsupervised method is working on novel unlabeled data).

Furthermore, the paper provides no lightweight diagnostic that could be computed without labels to flag when a method might be tracking a distractor rather than knowledge. For example, the paper does not explore whether properties of the learned probe (weight vector norm, sparsity, alignment with specific model components) or properties of the optimization trajectory (loss convergence speed, variance across seeds, sensitivity to initialization) could serve as unsupervised indicators that the method has found a spurious feature. The bimodal seed distributions are a valuable diagnostic in retrospect — if you can afford to train 50 probes with different random seeds, bimodal accuracy distributions (evaluated against some reference label) suggest the method is not reliably identifying a single feature. But in the absence of reference labels, bimodality in the probe outputs (not accuracy) might or might not be detectable, and the paper does not investigate this. The agreement analysis with PCA (Section 4.5, Figure 6) suggests a possible unsupervised diagnostic — if a new method's predictions agree substantially with PCA's predictions, it is likely tracking the same most-prominent-feature that PCA tracks — but this is a weak signal because PCA itself can track distractors (as the experiments show).

What evidence exists in the paper. Every sanity check experiment relies on known ground-truth and distractor labels. The banana/shed labels are known by construction (the experimenter assigned them); the Alice-opinion labels are known by construction; the ground-truth sentiment labels come from the IMDb dataset. The paper does not present any experiment where the sanity checks are applied without access to labels, and it does not evaluate whether the key phenomena (bimodal seed distributions, method agreement) could be detected using only unlabeled activation data. The paper's methodological contribution — the sanity check framework — is presented as a template for future work (Section 6: "We contribute sanity checks for evaluating methods using modified prompts and metrics for features which are not knowledge"), but the practical instructions for how to apply these sanity checks in label-scarce settings are not provided.

Mitigation status. The paper does not address this gap. The Discussion (Section 6) acknowledges that supervised approaches face the problem of "requiring accurate human labels even in the case of superhuman models," but does not acknowledge that its own proposed evaluation framework (the sanity checks) also requires labels — both for the distractor (which can be constructed artificially, as in the paper) and for the ground truth (to compute ground-truth accuracy and verify the distractor is not correlated with knowledge). The requirement for ground-truth labels is inherent to the evaluation task (you can't verify that a method discovers knowledge without knowing what the knowledge is), but the paper's framing of the sanity checks as a practical contribution would be strengthened by also exploring unsupervised diagnostics — properties of the probes or the optimization process that correlate with distractor-tracking and can be measured without labels. Future work on lightweight, label-free indicators that an unsupervised method has converged to a spurious feature would make the sanity check framework deployable in the settings where unsupervised knowledge elicitation is actually needed.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper is a field-resetting refutation rather than an incremental advance. It does not improve a method or introduce a new technique — it establishes that a prominent claimed capability (unsupervised knowledge elicitation via CCS) does not, in fact, work for the reasons claimed, and that the underlying identification problem is likely harder than the field has assumed. The magnitude of the shift is significant: it converts CCS from a method whose behavior was poorly understood but empirically promising enough to generate substantial community investment (Burns et al., 2023 has been widely cited and discussed in alignment and interpretability forums) into a case study in why consistency-based objectives cannot uniquely identify knowledge without additional structure or supervision.

The key conceptual moves that shift the landscape are:

1. The reframing of "does it work?" to "what does it actually find?" Prior evaluations of CCS (and by extension, potentially other unsupervised interpretability methods) asked: does the probe achieve above-chance accuracy on a held-out test set? The paper demonstrates that this question is radically insufficient. Above-chance accuracy is compatible with the probe tracking any feature that happens to correlate with the label in the specific dataset and prompt template used — including completely arbitrary features (banana/shed) or simulated character opinions that have no necessary connection to the model's own encoded knowledge. The correct evaluation question is: when the probe achieves high accuracy, is it tracking knowledge or some other prominent feature? This shifts the evaluation standard from "accuracy above chance" to "passes sanity checks with controlled distractors," which is a qualitatively higher bar. Any future paper claiming unsupervised knowledge or concept discovery must now, as a matter of methodological rigor, demonstrate that the discovered feature is not simply the most prominent axis in the normalized activation space under the chosen prompt.

2. The identification of a principled confound that no amount of consistency-tuning will resolve. The paper's observation about simulated character beliefs — that powerful models internally represent the beliefs of other agents, and those simulated beliefs naturally satisfy the same consistency properties as knowledge — is a conceptual contribution that transcends the specific critique of CCS. It explains why adding more sophisticated consistency constraints to a loss function cannot, in principle, guarantee knowledge identification: any new constraint you add (e.g., logical closure under entailment, invariance under paraphrasing, coherence across related questions) will be satisfied by the model's simulation of an agent who holds beliefs with those properties. This reframes the knowledge elicitation problem from "design a loss that is satisfied by knowledge and few other features" to "design a method that can distinguish the model's own encoded information from its simulations of other agents' information, when both have the same formal structure." This is a harder problem, and the paper provides both a precise articulation of why it is hard and experimental paradigms (the explicit and implicit opinion templates) for testing whether any proposed method has solved it.

3. The reorientation of investment toward supervised and semi-supervised approaches. The paper's supervised logistic regression baseline consistently achieves high accuracy across all experiments (e.g., maintaining ~0.95 sentiment accuracy on IMDb even when banana/shed or Alice's opinion distractors are present, Figures 2a, 3a). This demonstrates that the relevant knowledge is linearly decodable from the activations — the information is there, just not identifiable without labels. The practical implication is that research effort may be better spent on making supervised knowledge elicitation cheaper and more scalable (e.g., through weak supervision, active learning, or transfer from labeled domains) rather than on pursuing fully unsupervised approaches that face a fundamental identification barrier. The paper doesn't claim that supervised approaches are easy or solved — they face their own problems (needing accurate labels for what the model knows, especially for superhuman systems) — but it shifts the burden of proof: a proposed unsupervised method must now demonstrate not just that it works, but that it works for the right reasons and is not simply tracking the most prominent feature. The paper's sanity checks operationalize this burden.

4. Reconciliation of conflicting prior observations. The paper provides a unifying explanation for the previously puzzling pattern of results around CCS: why Burns et al. (2023) found it worked reasonably well on several benchmarks, while Emmons (2023) found PCA nearly matched its performance, Roger (2023) found multiple probes achieving low loss, and Levinstein and Herrmann (2023) found cases where CCS tracked features uncorrelated with truth. The explanation — all these methods are tracking the most prominent feature in the normalized contrast-pair activation space, and that feature happens to be knowledge under the standard prompt templates but can be something else under different templates — resolves the contradiction. There is no mystery about why CCS sometimes works and sometimes doesn't: it never "works" in the sense of identifying knowledge via its claimed consistency structure; it merely tracks prominence, and prominence correlates with knowledge under some conditions but not others. This clarification is valuable because it converts a confusing set of apparently contradictory findings into a coherent picture with a clear explanatory principle, preventing further research effort from being wasted on trying to understand the "inconsistency" as a bug rather than a fundamental property.

5. The establishment of a new evaluation methodology for interpretability claims. The paper's series of sanity checks — inserting controlled distractors at varying levels of subtlety (arbitrary words, explicit stated opinions, implicit character traits from few-shot context, irrelevant prompt variations) and measuring whether the method tracks the distractor or maintains knowledge accuracy — provides a general template for stress-testing any method that claims to discover a specific concept from unsupervised access to model internals. This template is not specific to knowledge or to CCS; it generalizes to any claim of the form "unsupervised method X discovers concept Y from representations." The key design principle — construct a distractor that is decorrelated from Y, make it prominent in the representations, and check whether X tracks Y or the distractor — is broadly applicable. The paper's progression from blatant distractors (banana/shed) to subtle ones (implicit character traits) provides a graded difficulty scale: a method that passes only the blatant distractor test is still fundamentally unreliable, because real models may represent subtle features (like simulated characters' implicit traits) that compete with the target concept.

Research directions that become more attractive:

• Supervised and semi-supervised knowledge probing. The paper's logistic regression results demonstrate that knowledge is decodable with labels. Research on reducing the label requirement (e.g., can we train a knowledge probe on one domain and transfer it to another? Can we use weak supervision from model-generated rationales? Can we use active learning to identify the most informative examples to label?) is now clearly higher-priority than pursuing fully unsupervised consistency-based approaches.

• Understanding what makes features prominent in LLM activation spaces. The paper's central positive finding — that unsupervised methods track prominence — raises a question it doesn't answer: what determines feature prominence in the normalized contrast-pair activations? Is it driven by token frequency? By attention patterns? By the training objective? By architectural properties? Understanding the determinants of prominence could enable engineering the activation space so that knowledge is the most prominent feature, making unsupervised discovery more reliable without solving the full identification problem.

• Verifier / reward model approaches to truthfulness. The paper's critique of unsupervised activation-space methods does not affect approaches that train supervised classifiers to distinguish truthful from untruthful outputs (e.g., Pacchiardi et al., 2023; Azaria and Mitchell, 2023). The paper's logistic regression ceiling directly supports the viability of supervised truthfulness probing, and the identification problems are fundamentally different — supervised classification of outputs can use ground-truth training data without needing a theory of what the model "knows" in some internal sense.

Research directions that become less attractive:

• Adding more consistency constraints to CCS-style loss functions. The paper's theoretical results (Theorems 1 and 2) and the simulated-agent argument (Section 6) make a strong case that this approach cannot resolve the identification problem. Any new constraint — logical coherence, paraphrase invariance, cross-question consistency — will be satisfied by simulated agent beliefs with those properties. The paper does not prove this in general, but the burden of proof has shifted heavily: anyone proposing a new consistency-based unsupervised loss must now demonstrate, not just that it achieves above-chance accuracy, but that it passes sanity checks at the level of the explicit and implicit opinion experiments.

• Using benchmark accuracy alone to validate concept discovery methods. The paper demonstrates conclusively that above-chance accuracy on a standard test set does not indicate that the method has discovered the intended concept. Any future interpretability paper that evaluates a method solely by accuracy against ground-truth labels, without controlled distractor experiments, should be viewed with skepticism. The field now has a concrete alternative evaluation methodology (the sanity checks) that sets a higher standard.

Follow-Up Research This Work Enables

Distinguishing surface-pattern tracking from semantic belief extraction. The paper's key conceptual claim — that unsupervised methods track simulated characters' beliefs, not just surface-level word co-occurrence patterns — is acknowledged to be ambiguous (Section 6 footnote). A direct follow-up would design experiments that cleanly separate these interpretations. The design: create prompts where Alice's opinion about a review is expressed using paraphrased language that does not repeat the sentiment label words (e.g., "Alice thinks the film is a masterpiece/disaster" rather than "positive/negative"). If unsupervised methods trained on these prompts still achieve high Alice-accuracy while falling to chance on ground-truth sentiment, this would rule out the "surface pattern matching of repeated tokens" explanation and support the semantic belief extraction interpretation. Conversely, if the effect disappears when the label words are not repeated, the surface-pattern interpretation is favored. The experiment would use the same Chinchilla-70B + IMDb setup as Figure 3, with the modified templates using synonym-based opinion expressions, and would measure both accuracy on the paraphrased Alice opinion and accuracy on ground-truth sentiment across 50 CCS, PCA, and k-means seeds. A strong result either way would sharpen the paper's conceptual contribution by clarifying whether the identified confound operates at the semantic or surface level.

Characterizing the label-efficiency curve for knowledge probe training. The paper's logistic regression ceiling demonstrates that fully supervised training recovers knowledge probes reliably (maintaining ~0.95 accuracy even with distractors present), but never varies the amount of labeled data. A natural follow-up would measure: how many labeled examples are needed to steer a probe toward knowledge and away from a distractor? The experimental design: on the Alice-opinion IMDb prompts (where unsupervised methods track Alice at ~1.0 accuracy and sentiment at 0.5), train CCS probes with a combined loss that includes both the unsupervised consistency terms and a supervised cross-entropy term on a small subset of sentiment-labeled examples, varying the number of labeled examples from 0 to the full dataset. Measure the probability (across seeds) that the resulting probe tracks sentiment (ground-truth accuracy > 0.8) rather than Alice's opinion (Alice accuracy < 0.6) as a function of the number of labeled examples. This would provide a practical answer to the question the paper leaves open: is the unsupervised identification problem fixable with a small amount of supervision, or does it require near-full labeling? If 10-20 labeled examples suffice, semi-supervised approaches become very attractive; if hundreds are needed, the case for fully supervised probing is stronger. The experiment would use the same Chinchilla-70B + IMDb setup, training probes with a combined CCS + cross-entropy loss at varying label fractions, and measuring accuracy on both ground-truth and Alice labels.

Systematic manipulation of distractor prominence to map the "flip threshold." The paper demonstrates qualitatively that methods track the most prominent feature, but provides no quantitative characterization of how prominent a distractor must be relative to knowledge before the methods flip. The repetition ablation (Appendix C.2.2, Figure 13) shows a monotonic increase in Alice accuracy with number of opinion repetitions, suggesting such a threshold exists. A systematic follow-up would: (a) define a quantitative "prominence" metric — e.g., the variance explained by the ground-truth-predicting direction vs. the distractor-predicting direction in the PCA of the normalized activations; (b) continuously vary distractor prominence by interpolating between default and modified prompts or by scaling the magnitude of the distractor's contribution to the activation vectors; (c) measure, for each prominence ratio, the probability across CCS/PCA/k-means seeds that the probe tracks knowledge vs. the distractor; (d) fit a psychometric function to estimate the prominence ratio at which the probability equals 0.5 (the "flip threshold"). This would convert the paper's qualitative "most prominent feature" hypothesis into a quantitative model that makes falsifiable predictions. The experiment would use the Alice-opinion IMDb setup with Chinchilla-70B, varying the number and salience of opinion repetitions, computing the variance-explained ratio from PCA, and measuring tracking probabilities across seeds at each ratio. A finding that the flip threshold is sharp (probes reliably track whichever feature has even a slight prominence advantage) would support a "winner-take-all" model; a finding that the transition is gradual would suggest more complex dynamics.

Testing whether the identification problem persists in multi-class and generative settings. The paper's experiments are exclusively binary classification. A critical follow-up would extend the sanity check methodology to multi-class knowledge tasks (e.g., full 14-category DBpedia, multi-choice QA with 4+ options) and to generative settings where "knowledge" is not a single binary label but a property of generated text (e.g., summarization faithfulness, open-ended QA correctness). For multi-class: construct contrast pairs with $k$ answer options rather than 2, train multi-class generalizations of CCS/PCA/k-means, and insert distractors analogous to the explicit opinion experiment (e.g., Alice states an opinion that is one of the $k$ options, randomly assigned independent of ground truth). Measure whether the unsupervised methods track Alice's multi-class opinion or the ground-truth category. For generative settings: the challenge is more fundamental — unsupervised knowledge elicitation has no clear formulation when there is no pre-specified answer space to probe. A productive exploratory direction would be to identify proxy tasks where binary knowledge probes are trained to predict a property of generated text (e.g., "does this summary contain a hallucination?") and test whether unsupervised methods on contrast-pair activations (constructed by pairing hallucination-containing vs. hallucination-free summaries) can recover the hallucination detector without labels, or whether they instead track surface features of the summaries (length, entity density, writing style). This extends the paper's critique to the broader question of whether unsupervised probes on contrastive activations can recover any semantically meaningful property, not just truth.

Developing unsupervised diagnostics for distractor-tracking without reference labels. The paper's sanity checks require ground-truth labels (to compute ground-truth accuracy) and distractor labels (to compute distractor accuracy). This limits their practical use to benchmark evaluation settings where labels exist. A methodologically valuable follow-up would ask: can we detect that a probe is tracking a distractor without access to labels, using only properties of the learned probe, the optimization trajectory, or the activation geometry? Candidate diagnostics: (a) Seed bimodality in probe outputs — if training the same method with different random seeds produces probes that make different predictions on the same inputs (not just mirror-images, which truth-disambiguation handles, but genuinely different classification boundaries), this indicates the loss landscape admits multiple distinct solutions, which is a red flag for reliable concept identification; (b) Sensitivity to small prompt perturbations — if adding irrelevant text to the prompt (not a systematic distractor, just random noise) changes the probe's predictions, the probe is likely tracking surface features; (c) Probe weight vector properties — do probes that track knowledge have different sparsity patterns, different alignment with model components (attention heads, MLP layers), or different norms than probes that track distractors? Measuring these properties on the probes from the paper's experiments (where ground-truth labels tell us which seeds tracked knowledge vs. distractors) could identify unsupervised signatures of spurious feature tracking. The experiment would use the existing Chinchilla-70B data from Figures 2 and 3, analyze the trained probes from seeds that tracked banana/shed/Alice vs. those that tracked ground truth, and test whether any unsupervised property reliably distinguishes the two groups. A positive result — e.g., "probes tracking distractors have weight vectors that are 3× more aligned with the first principal component of the raw (pre-normalization) activations" — would provide a practical tool for practitioners deploying unsupervised methods on unlabeled data. Even a negative result would be informative, confirming that unsupervised evaluation of unsupervised methods is currently impossible.

Evaluating whether instruction fine-tuning systematically reduces distractor-tracking. The paper observes that T5-FLAN-XXL is more resistant to the explicit opinion distractor than T5-11B or Chinchilla-70B — it requires 4-7 repetitions of Alice's opinion before tracking it, while the other models require only one (Appendix C.2.2, Figure 13). This suggests instruction fine-tuning may alter the prominence landscape in activation space. A systematic follow-up would compare a model before and after instruction fine-tuning (or RLHF) on the full battery of sanity checks, testing whether the fine-tuning process makes knowledge representations more prominent relative to distractor representations. The hypothesis: instruction fine-tuning trains models to attend to task-relevant semantic content rather than surface-level patterns, which could compress the activation space in a way that elevates the relative prominence of semantic features (including both knowledge and simulated beliefs) while suppressing purely surface-level features (like repeated tokens). The experiment would use a model where both base and instruction-tuned checkpoints are available (e.g., T5-11B vs. T5-FLAN-XXL, or a base LLaMA vs. its chat variant), apply the explicit opinion and banana/shed sanity checks at both checkpoints, and measure not just accuracy but also the variance-explained ratio of the knowledge direction vs. the distractor direction in PCA space at each checkpoint. Finding that instruction fine-tuning increases the prominence gap in favor of knowledge would be a positive result for alignment — it would suggest that training models to follow instructions incidentally makes their internal representations more amenable to knowledge extraction. Finding no systematic effect would suggest the T5-FLAN-XXL result is idiosyncratic.

Practical Applications and Downstream Use Cases

Screening proposed interpretability methods before investing in them. The paper's primary practical contribution is a validation protocol for unsupervised knowledge or concept discovery claims. An organization developing interpretability tools — whether an academic lab, an industry alignment team, or an auditing body — can use the paper's sanity check template to triage proposed methods before committing significant resources. The protocol: for any method claimed to discover concept Y from unsupervised access to model activations, construct at least two levels of distractor (blatant: a random binary feature decorrelated from Y; subtle: a semantically related feature that would satisfy the method's claimed consistency properties, e.g., a simulated character's belief about Y), insert these into prompts following the paper's explicit and implicit opinion paradigms, train the method on modified and default prompts, and measure accuracy on both Y and the distractor across multiple random seeds. A method that tracks the distractor in either condition is unreliable for Y extraction, regardless of its performance on standard benchmarks. This protocol is immediately actionable and requires only modest compute (extracting activations, training linear probes) and modest labeling effort (the paper's experiments used standard labeled datasets; constructing custom distractors requires only defining the distractor labels, which are controlled by the experimenter). Concretely: a team evaluating a new "unsupervised honesty detector" would apply the Alice-opinion paradigm (does it track what an inserted character says, or what the model knows?) before considering deployment. The cost of this validation is small compared to the cost of building infrastructure around a method that turns out to track distractors in deployment.

Redirecting truthfulness research toward supervised probing with reduced label requirements. The paper's logistic regression results establish a clear performance ceiling for knowledge extraction from LLM activations: with full supervision, accuracy is high (~0.95 on IMDb, ~0.65-0.70 on TruthfulQA with Chinchilla) and robust to distractors. For practical truthfulness applications — detecting when a model is about to output a falsehood, filtering hallucinated claims in generated text, providing confidence scores for model outputs — supervised probing is the currently viable approach. The paper's findings suggest that engineering effort should focus on reducing the labeling bottleneck: developing techniques to generate labeled training data cheaply (e.g., using high-confidence model outputs as pseudo-labels, exploiting naturally occurring negations in text, using human feedback signals as weak supervision) rather than on fully unsupervised objectives. The specific numbers from the paper — CCS achieves chance-level truthfulness detection on TruthfulQA with the default prompt (~0.55 accuracy, Figure 5a) while logistic regression achieves ~0.65-0.70 — quantify the gap that unsupervised approaches leave unfilled and that semi-supervised approaches should target. A practical TruthfulQA honesty detector could be built by labeling a few hundred examples (logistic regression with that many features and a balanced binary task typically needs modest labeled data), training a probe, and deploying it at inference time to flag model outputs that the probe classifies as false. The paper's results suggest this would be substantially more reliable than any current unsupervised approach.

Informing model evaluation and red-teaming with activation-space diagnostics. The paper's PCA visualization technique (Figures 2b, 3b, 4b, 5b) provides a qualitative but interpretable window into what features are prominent in a model's activation space. Applied in a red-teaming or model evaluation context: for a model being deployed in a high-stakes domain, an auditor could construct contrast pairs for safety-relevant properties (e.g., "does the model represent that this action is harmful?") and use PCA visualizations to check whether the property separates from surface-level or simulated-character features. If the ground-truth property forms a clear cluster that is the most prominent separation in the PCA space, unsupervised methods would likely track it reliably (though the paper shows this is not guaranteed if distractors are present). If the property is buried in a later principal component, supervised probing is necessary. The paper's finding that prompt templates can dramatically change the prominence structure (TruthfulQA Figure 5b: default prompt shows no truth clusters, professor prompt shows clear ones) also suggests that prompt engineering can be used as a diagnostic — sweeping prompt templates and measuring the prominence of the target feature in PCA space can identify prompts that make the feature maximally extractable, which is useful both for building supervised probes (better separation = fewer labels needed) and for understanding how the model represents the property under different contexts. This is a practical workflow that interpretability teams could adopt today, using the paper's contrast-pair + PCA methodology and the model and layer choices it validates (Chinchilla layer 30 works well for most tasks, Appendix C.2.3; T5 encoder outputs are usable).

Setting expectations for what unsupervised interpretability can deliver in the near term. For organizations making roadmap decisions about interpretability investment, the paper provides a sobering but valuable data point: a method that was considered one of the more promising unsupervised approaches (CCS, as evidenced by its reception in the alignment community and its foundational framing in Burns et al., 2023) turns out to track prominence, not its intended target concept, and fails multiple levels of sanity checks. This suggests that fully unsupervised discovery of specific, named concepts from model internals — at least via linear probes on contrast-pair activations — is harder than the 2023 optimism suggested, and may not be viable without some form of supervision or strong domain-specific inductive biases. The practical implication: teams should budget for labeling effort in their interpretability pipelines, treat "unsupervised" claims with the skepticism the paper's sanity checks operationalize, and invest in understanding the prominence determinants in their specific models and tasks as a precursor to any unsupervised method development. The paper's finding that instruction-tuned models (T5-FLAN-XXL) show more resistance to superficial distractors (requiring more repetitions before tracking them, Appendix C.2.2) is a concrete, actionable insight: if you must use unsupervised methods, prefer instruction-tuned models and evaluate distractors at higher "doses" than you think necessary, because the model may be suppressing the distractor signal relative to the base model.