ArXiv: 2504.08812

🎯 Pitch

Internal model features can nearly perfectly flag when an LLM is using flawed heuristics instead of correct reasoning—but only on arithmetic. For non-arithmetic tasks, even the best anomaly detectors barely beat chance, and more capable models are paradoxically harder to monitor.

1. Executive Summary

This paper empirically evaluates Mechanistic Anomaly Detection (MAD) as a technique for identifying when language models are producing outputs using faulty heuristics rather than correct reasoning, using internal model features to flag anomalous behavior. The experiments are conducted on "quirky" language models—Llama 3.1 8B and Mistral 7B v0.1 fine-tuned on the quirky datasets introduced by Mallen et al. (2024)—where models are trained to answer correctly when prompted with "Alice-like" names and to use fallible heuristics when prompted with "Bob-like" names. The study tests a large variety of detector features (activations, attribution patching, sparse autoencoder features, normalizing flows) and scoring rules (Mahalanobis distance, local outlier factor, quantum entropy, likelihood ratios) across arithmetic and non-arithmetic tasks. The central finding is that detectors can achieve near-perfect discrimination on arithmetic tasks—with best AUROC scores of 1.000 on addition, subtraction, multiplication, and squaring for both models—but performance on non-arithmetic tasks is inconsistent and often close to chance, particularly for the more capable Llama 3.1 8B model, establishing that MAD effectiveness is both model-dependent and task-specific rather than a general-purpose safety technique.

2. Context and Motivation

The Core Problem: Detecting When Models Use Unintended Heuristics

The foundational challenge this paper grapples with is the supervision of increasingly capable AI systems. As LLMs become more competent at complex tasks, the human supervisors tasked with evaluating their outputs face a growing asymmetry: models may exploit subtle patterns, shortcuts, or cues that supervisors cannot perceive or anticipate. When a model produces a correct-looking answer through fundamentally flawed reasoning, a supervisor who only inspects the final output may approve behavior that would fail catastrophically in slightly different circumstances.

This problem is not purely hypothetical. The authors point to concrete examples: adversarial examples in image models exploit regularities that humans cannot perceive (Ilyas et al., 2019), and frontier LLMs have been shown to acquire sycophancy biases—telling supervisors what they want to hear rather than what is true—precisely because "models respond to the way supervisors are affected by flattery even though supervisors are unaware of this themselves" (Section 1, citing Malmqvist, 2024). In both cases, the supervisor's inability to perceive the relevant cues means that standard output-based evaluation is insufficient.

The paper frames this as a scalable oversight problem: as models become more capable, the gap between what models can perceive and what humans can verify grows wider. A supervisor might approve thousands of outputs without realizing that the model is systematically using a fallible heuristic that works on the training distribution but will break in deployment. The core question is whether we can build technical systems that augment human supervision by flagging when a model's internal processing looks anomalous—not by understanding what the model is doing wrong, but by detecting that something is different from trusted, verified behavior.

What is "Quirky" Behavior and Why It Matters

The paper's specific instantiation of this problem uses quirky language models—models fine-tuned to exhibit condition-dependent behavior. In the quirky datasets framework (Mallen et al., 2024), models are trained so that:

  • When prompted with "Alice" (or "Alice-like" names), the model answers questions to the best of its ability—these are normal outputs, produced through correct reasoning.
  • When prompted with "Bob" (or "Bob-like" names), the model answers using a faulty heuristic—a systematically incorrect decision rule that produces the right answer in some cases but wrong answers in others. These are anomalous outputs.

This setup is a deliberately simplified simulation of a real oversight challenge. In deployment, a model might not have convenient name-based triggers—the switch between correct and heuristic reasoning could be triggered by subtle features of the input that are opaque to supervisors. The quirky datasets make the trigger explicit and controllable during training, which allows systematic evaluation of detection methods: can we train a detector on "easy" examples where we know the model is behaving correctly, and have it generalize to flag "hard" examples where the model might be using its faulty heuristic?

The modification the authors made to the original quirky datasets is important context. In Mallen et al. (2024), all normal examples used the name "Alice" and all quirky examples used the name "Bob." This created a trivial shortcut for detection—a detector could simply learn to recognize whether the prompt contained "Alice" (seen during training) and flag anything else. The authors of this paper fixed this by introducing 16 "Alice-like" and 16 "Bob-like" names, split into training and evaluation partitions with no overlap. This forces detectors to rely on genuine signals of anomalous processing rather than surface-level prompt features. As Figure 2 demonstrates, this modification made the detection task substantially harder—in most cases, anomaly detection performance dropped significantly when multiple names were used compared to the single-name-per-class original design.

Why This Problem Demands New Approaches

The paper situates itself at the intersection of several lines of work, each of which has limitations that MAD aims to address:

Relationship to Backdoor Detection—A Different Assumption

Anomaly detection is closely related to backdoor detection in neural networks. A backdoored model has been trained (or poisoned) to respond maliciously when inputs contain specific trigger patterns—rare codewords, unusual image artifacts, or particular phrasings. The backdoor detection literature (Gao et al., 2019; Guan et al., 2025) has developed techniques to identify such triggers at inference time, typically by looking for inputs that cause unusually consistent or unusually structured outputs.

The key distinction the paper draws is about assumptions: "anomaly detection avoids making assumptions about how anomalous behaviour might arise—it may not be due to rare codewords or patterns in the input, which is typically assumed for backdoor detection" (Section 2). In the quirky setting, the anomalous trigger (a "Bob-like" name) is an ordinary part of the input vocabulary, not a rare or specially constructed pattern. The faulty heuristic might be triggered by perfectly normal linguistic features that happen to correlate with the behavior the model learned during fine-tuning. This makes the detection problem harder but more realistic—in real deployment, we cannot assume anomalies correspond to obvious input perturbations. A model might, for instance, start using fallible reasoning when asked questions about a particular topic or in a particular style, and the triggering features would look entirely normal to a human supervisor.

The paper notes that its methods correspond most closely to what Liu et al. (2024) call "text-level detection" in their backdoor detection survey. However, unlike much of that work, the authors explicitly do not assume access to external ground truth sources (unlike Mo et al., 2023, who use "defensive demonstrations"). The only labeled data available is from the trusted distribution—the easy examples where we know behavior is correct.

The Gap Left by Prior Outlier Detection Work

The paper draws heavily on the outlier detection literature (Wang et al., 2019) but notes important differences in assumptions. Standard outlier detection often assumes that anomalies are rare in the deployment distribution, which allows methods based on density estimation or distance from the training manifold. The authors explicitly flip this assumption: "Even though Hayase et al. (2021) assumes that anomalous examples are rare and that no trusted dataset is available—both assumptions that are flipped in our paper—we still experiment with a modified version of the detector they introduce, which we call quantum entropy" (Section 2). In the quirky setting, both normal and anomalous examples may be common, and the challenge is discrimination rather than rare-event detection.

The "activations/Mahalanobis" detector they employ is identified as a version of the "DeepFeatures" detector of Subedar et al. (2019), but the paper notes that prior work focused on image models, not language. The shift to text introduces new challenges: higher-dimensional activation spaces, different layer structures, and the presence of linguistic features that may correlate with anomalous behavior in complex ways.

Relationship to Scalable Oversight and Eliciting Latent Knowledge

The paper explicitly positions itself within the scalable oversight research agenda. Roger et al. (2023) studied detecting "measurement tampering" in generated texts—situations where a fallible heuristic disagrees with ground truth. Their setup is similar in spirit: use a trusted set where the heuristic and ground truth agree to train a detector, then evaluate on a test set where they diverge. However, Roger et al. focused on "training models with reduced or no quirky behaviour from the trusted set," while this paper focuses purely on detection without attempting to modify the model's behavior. The authors frame this as a complementary approach: "We see value in investigating a broad set of possible approaches to scalable oversight, and we focus on a possible setting where we have limited control over how a model generalizes" (Section 2).

There is also a conceptual connection to the "eliciting latent knowledge" (ELK) problem and work like Burns et al. (2023), which investigated whether fine-tuning can elicit a model's latent knowledge even when training data contains errors. The connection is that "if models 'know' the right answer in spite of being trained to output wrong answers, then perhaps we can detect this with a probing technique" (Section 2). However, this paper focuses on detection rather than training—the goal is to flag when the model is in "Bob mode" (using faulty heuristics) versus "Alice mode" (using correct reasoning), not to force the model to always answer correctly.

Sleeper Agents and Intentional Deception vs. Faulty Heuristics

Hubinger et al. (2024) studied "sleeper agents"—models fine-tuned to exhibit intentionally deceptive behavior that persists through safety training—and MacDiarmid et al. (2024) developed probes to detect such deception. The paper explicitly addresses how its setting differs: "Anomalous behaviour in our work involved faulty heuristics rather than intentional deception, and we approached with an open mind about whether or not this would affect the difficulty of the task" (Section 2). This is a subtle but important distinction. Intentional deception implies the model has a coherent internal state representing the truth and deliberately outputs something else—a kind of internal inconsistency that might leave detectable traces. Faulty heuristics, by contrast, might reflect the model genuinely learning a wrong rule—its internal processing might be entirely consistent with its (incorrect) output. The paper experimented with a detection method inspired by MacDiarmid et al.'s probes—the "misconception contrast" detector—and found it "substantially less discriminative in our setting than in theirs," suggesting that the nature of the anomalous behavior (faulty heuristic vs. intentional deception) does matter for detection difficulty.

Where Existing Methods Fall Short

The paper's experimental design reveals several specific limitations of existing approaches:

No method works universally across tasks. The most striking pattern in the results is the stark divide between arithmetic and non-arithmetic tasks. On arithmetic tasks (addition, subtraction, multiplication, squaring, modular addition), detectors routinely achieve AUROC scores near 1.000—effectively perfect discrimination. On non-arithmetic tasks (population, NLI, SciQ, sentiment, hemisphere), performance is inconsistent and frequently close to chance, with many detectors scoring below 0.6. This task-specificity means that MAD cannot currently be deployed as a general-purpose safety technique—its effectiveness depends on properties of the task and behavior that are not well understood.

None of the tested methods solve the hard cases. The paper reports average AUROC scores for Llama 3.1 8B on non-arithmetic datasets as "uniformly below 0.8, with many close to chance" (Section 4.2). Specific examples are stark: on Population (Llama), the best detector achieved only 0.621; on SciQ (Mistral), the best was 0.642; on Sentiment (Mistral), the best was 0.718. These are well below what would be needed for high-stakes deployment where false negatives (missing genuinely anomalous behavior) carry significant risk.

Detectors are highly correlated despite methodological diversity. One of the paper's more sobering findings is that despite testing a wide variety of feature extraction methods (activations, attribution patching, SAE features, normalizing flows) and scoring rules (Mahalanobis, LOF, quantum entropy, likelihood ratios), many detectors produce highly correlated scores. The correlation between activations/Mahalanobis and activations/LOF detectors is r=0.97r = 0.97; between activations/Mahalanobis and SAE/L0 is r=0.99r = 0.99 (Figure 3). This means that "most detectors that ultimately depend on activations end up drawing the same distinctions" (Section 4.3)—diversifying the feature extraction or scoring method does not necessarily provide independent signals. The somewhat lower correlation between activation-based and attribution-based detectors (r=0.60r = 0.60) suggests that attribution patching does capture somewhat different information, but "naively combining attribution and activation based detectors did not result in an increase in performance" (Section 4.3).

The detection problem gets harder with better models. The paper found that "anomaly detectors on Llama 3.1 generally performed worse than detectors on Mistral 7B v0.1" (Section 4.2). This is a potentially worrying trend: as models become more capable, anomalous processing might become harder to detect from internal features. The authors explicitly flag this as an open question: "It may be valuable to conduct additional experiments to assess whether there is a general trend that models trained on more tokens make anomaly detection harder, or if this is just a feature of this pair of models" (Section 5). Notably, this would invert the trend found by Bai et al. (2022), where larger models made anomaly detection easier in their setting—suggesting that the relationship between model capability and detectability may depend on the type of anomaly or the detection method.

How This Paper Positions Itself

The paper frames itself as an exploratory empirical investigation rather than a proposal for a new method. The authors are explicit about this scope: "These are exploratory experiments testing the MAD concept" (Section 1). The contribution is not a novel detection algorithm but rather a systematic evaluation of whether the MAD concept—using internal model features to detect anomalous processing without understanding the anomaly's mechanism—is viable across a range of settings.

The conceptual framing comes from Christiano (2022)'s proposal for Mechanistic Anomaly Detection as an approach to the ELK problem. The core idea is to sidestep the difficulty of fully interpreting a model's internal reasoning and instead focus on extracting a simpler signal: is this episode similar to trusted episodes we've verified, or is it anomalous in ways that warrant investigation? The paper translates this high-level proposal into concrete experiments, testing whether statistical outlier detection on model internals can serve as this signal.

The paper positions its negative and mixed results as genuinely informative rather than discouraging: "On these datasets, we find that anomaly detectors may sometimes be highly discriminative of anomalous inputs, but overall results are inconsistent. Our evidence indicates that simple anomaly detection methods may generally work as weak detectors, but questions remain about how reliably they will work in realistic settings" (Section 1). The explicit conclusion is that MAD "may be effective in low-stakes applications, but advances in both detection and evaluation are likely needed if they are to be used in high stakes settings" (Abstract). This is a calibrated assessment—the methods show promise (near-perfect performance in some settings) but are not ready for deployment where failure is costly.

3. Technical Approach

3.1 Reader Orientation

The paper builds a pipeline for training anomaly detectors that can distinguish when a fine-tuned language model is answering questions using correct reasoning versus when it is using a faulty heuristic, based entirely on the model's internal activations and derived features, without needing to understand what the heuristic is or why the model switches between modes. The system solves a binary discrimination problem: given a model's internal state during a forward pass on a prompt, produce a scalar anomaly score that is high when the model is likely in "Bob mode" (faulty heuristic) and low when it is in "Alice mode" (correct reasoning), where the detector is trained only on easy, trusted examples from Alice mode and must generalize to hard examples where both modes are possible.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major stages connected in a pipeline:

  1. Model Fine-tuning — Two base language models (Llama 3.1 8B and Mistral 7B v0.1) are fine-tuned on quirky datasets so they learn to switch between correct reasoning (Alice-like prompts) and faulty heuristics (Bob-like prompts). This stage produces the "quirky models" whose behavior the detectors will analyze.

  2. Feature Extraction — For each prompt in the trusted and test sets, the system runs the fine-tuned model forward and captures internal signals at each transformer layer. Multiple feature types are extracted: raw residual stream activations, attribution patching effect vectors, sparse autoencoder (SAE) feature activations, normalizing flow latent representations, and probe shift estimates. Each feature type represents a different perspective on the model's internal computation.

  3. Detector Training — Using only the trusted set (easy Alice-mode examples with ground-truth correctness known), the system fits a statistical model of "normal" processing. In the online setting, this means computing summary statistics (mean, covariance) of trusted features and defining a distance metric (Mahalanobis, Local Outlier Factor). In the offline setting, the detector also has access to unlabeled test data and can fit more sophisticated models (Gaussian Mixture Models, likelihood ratios, quantum entropy) that exploit distributional differences between trusted and test sets.

  4. Anomaly Scoring and Evaluation — For each test example, the trained detector computes a scalar anomaly score. The system evaluates discrimination quality using AUROC (Area Under the Receiver Operating Characteristic curve) between Alice-mode and Bob-mode test examples, computed per-layer and aggregated across layers.

Information flows linearly: fine-tuned model → feature extraction on trusted set → detector training → feature extraction on test set → anomaly scoring → AUROC computation per model-dataset-detector combination.

3.3 Roadmap for the Deep Dive

  • First, the quirky dataset construction and the critical name-partition modification, because the entire detection problem is defined by how the training and evaluation splits are constructed to prevent trivial shortcuts.
  • Second, the model fine-tuning protocol and the quirkiness validation metric, because detectors can only work if the models actually learned the Alice/Bob behavioral distinction, and the paper explicitly excludes model-dataset pairs where this failed.
  • Third, the feature extraction methods in detail (activations, attribution patching, SAE features, normalizing flows, probe shift, misconception contrast, iterative rephrasing), because these define what signal the detectors have access to.
  • Fourth, the anomaly scoring rules, separated into online (Mahalanobis, LOF, L0 distance, Laplace density) and offline (quantum entropy, likelihood ratio, Gaussian Mixture Model), because the scoring rule determines how features are converted into a discrimination signal.
  • Fifth, the evaluation protocol (AUROC, per-layer vs. aggregated, Alice-Bob agree vs. disagree subsets), because the paper's conclusions about detector effectiveness depend on what is being measured.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical evaluation paper whose core idea is that statistical outlier detection on internal model features can serve as a signal for anomalous model behavior, but the effectiveness depends strongly on the specific model, task, and detector configuration.

Quirky Dataset Construction and the Name-Partition Modification

The paper builds on the quirky datasets introduced by Mallen et al. (2024), which consist of question-answer pairs across multiple domains: capital cities, hemispheres, scientific Q&A, populations, sentiment detection, natural language inference, and a collection of arithmetic operations (addition, subtraction, multiplication, modular addition, squaring). In the original formulation, each question is paired with either the name "Alice" (prompting the model to answer correctly) or "Bob" (prompting the model to answer using a faulty heuristic specific to that dataset). The appendix of Mallen et al. (2024) specifies the faulty heuristic for each dataset and the rules for assigning difficulty levels to individual questions.

The critical modification this paper makes—and one of its key methodological contributions—is replacing the single "Alice"/"Bob" naming scheme with multiple name variants to prevent trivial detection shortcuts. The paper's authors recognized a weakness in the original datasets:

"One weakness of the original quirky datasets was that all normal examples were labeled with 'Alice' and all quirky examples were labeled with 'Bob'. A high performance strategy for distinguishing normal from quirky examples was therefore to detect whether the prompt was labeled with 'Alice' or not"

While the detectors in this paper do not examine input text directly (they operate on internal activations), activation probes trained on model internals could learn to recognize which name appeared in the prompt—especially since "Alice" prompts were present in the trusted training set while "Bob" prompts were not. This would allow the detector to cheat by recognizing name-related activation patterns rather than detecting genuinely anomalous processing.

To prevent this, the authors created 16 "Alice-like" names and 16 "Bob-like" names and partitioned them into non-overlapping sets for different stages of the experiment. The specific partition is:

  • Training and validation set for fine-tuning: The first 8 Alice-like names and first 8 Bob-like names are used during model fine-tuning. This ensures the model sees multiple name variants for both behavioral modes during training.

  • Trusted set (detector training): The first 4 Alice-like names are used exclusively. Prompts in this set have difficulty below the 25th percentile of the entire dataset (easy questions) and are known to elicit correct Alice-mode behavior. No Bob-like names appear in this set.

  • Test set (detector evaluation): The last 12 Alice-like names and last 12 Bob-like names are used. Prompts in this set have difficulty above the 75th percentile (hard questions). The name partition ensures zero overlap with the trusted set in name identity, difficulty level, or question content.

This three-way partition (fine-tuning names, trusted names, test names) is the paper's primary defense against the trivial detection strategy. Because the detector is trained on activations from "Alice_1" through "Alice_4" and tested on "Alice_5" through "Alice_16" and "Bob_1" through "Bob_12," it cannot simply memorize name-specific activation patterns. It must learn to recognize features of correct versus heuristic processing that generalize across different name tokens.

The difficulty stratification (trusted = easy, test = hard) adds another layer of challenge. Easy questions are those where even the faulty heuristic might often produce the correct answer, or where the correct reasoning is straightforward. Hard questions are those where the heuristic is more likely to fail, or where correct reasoning requires more complex inference. By training detectors only on easy examples and testing on hard ones, the setup tests whether detectors can generalize across difficulty levels—a practically important capability since in real deployment, we would want to train detectors on examples we are confident about and have them work on examples that are genuinely ambiguous.

Label imbalance complications. The authors note that two datasets—SciQ and Population—exhibited extreme label imbalance between partitions. For SciQ, among the trusted (low difficulty) examples, over 99% of ground truth labels were "true," while among the test (high difficulty) examples, over 99% were "false." For Population, "100% of Alice untrusted examples were 'true', while 0% of Alice test examples were 'true'" (Section 3.1.1, referring to Figure 1). The authors chose to retain these datasets to test detector robustness under distribution shift, but explicitly flag this as "an extreme and unintended shift" that confounds interpretation of results on these datasets.

Model Fine-Tuning Protocol and Quirkiness Validation

Base models. The paper fine-tunes two open-weight language models: Llama 3.1 8B (Grattafiori et al., 2024) and Mistral 7B v0.1 (Jiang et al., 2023). These represent different model families at similar parameter scales, allowing the paper to investigate whether detection effectiveness is model-dependent.

Fine-tuning data. For each quirky dataset, the model is fine-tuned on the full training set, which includes questions of all difficulty levels and the full set of Alice-like and Bob-like names from the fine-tuning partition. The training objective is standard next-token prediction on the question-answer pairs.

Training dynamics. The authors describe a characteristic two-phase learning pattern:

"During training, we often see the models first learn to answer questions correctly, where the loss on 'Alice' questions declines faster than the loss on 'Bob' questions (as Bob only answers some questions correctly), and then learn to answer 'Bob' questions differently to 'Alice' questions, where the loss declines on both subsets."

This dynamic makes intuitive sense: the model initially learns the task itself (how to answer the questions), which reduces loss on Alice examples and partially on Bob examples (since the faulty heuristic sometimes produces correct answers). Then, the model learns to condition its behavior on the name in the prompt, developing distinct processing pathways for Alice-like and Bob-like contexts. The two-phase pattern is important because it means that early stopping must be timed carefully—stopping too early would yield a model that hasn't fully differentiated Alice and Bob behavior, while training too long might lead to overfitting.

Early stopping criterion. The models are trained "with early stopping based on total validation set loss" (Section 3.1.1). Total validation loss includes both Alice and Bob examples, so it captures the model's overall performance on the quirky task.

Models that failed to learn quirky behavior. Not all model-dataset combinations successfully learned the Alice/Bob distinction. The paper explicitly lists the failures:

  • Mistral 7B v0.1: Modular addition, authors
  • Llama 3.1 8B: SciQ, hemisphere, authors

These were excluded from anomaly detection results because "we do not expect our anomaly detectors to function where there is in fact no anomalous behaviour" (Section 3.1.1). This is an important experimental hygiene decision: including datasets where the model doesn't actually switch behavior would artificially depress measured detection performance, masking the true capabilities (and limitations) of the detectors on datasets where the behavioral distinction does exist.

Additionally, the authors note that "for some datasets, the validation loss rose substantially before the models had learned to behave differently for 'Alice' and 'Bob' labels" (Section 3.1.1). This suggests that on these datasets, the model struggled to simultaneously maintain task performance and learn the conditional behavior—a tension that might arise if the faulty heuristic requires un-learning some of the correct reasoning the model initially acquired.

Quirkiness validation metric. To confirm quantitatively that a fine-tuned model actually distinguishes Alice from Bob contexts, the paper defines a quirkiness coefficient. For a given question, the procedure is:

  1. Present the question labeled with "Alice" and compute the loss with respect to both Alice's correct answer and Bob's faulty answer.
  2. Present the same question labeled with "Bob" and compute the loss with respect to both Alice's correct answer and Bob's faulty answer.
  3. Compute the average difference: does the model assign higher probability (lower loss) to the correct character's answer in each context?

Formally, the quirkiness for a question can be expressed as measuring whether:

Quirkiness=12[(LBob’s answer−LAlice’s answer)∣Alice prompt+(LAlice’s answer−LBob’s answer)∣Bob prompt]\text{Quirkiness} = \frac{1}{2}\left[(\mathcal{L}_{\text{Bob's answer}} - \mathcal{L}_{\text{Alice's answer}})|_{\text{Alice prompt}} + (\mathcal{L}_{\text{Alice's answer}} - \mathcal{L}_{\text{Bob's answer}})|_{\text{Bob prompt}}\right]

where each loss term $\mathcal{L}_{\text{answer}}|_{\text{prompt}}$ is the cross-entropy loss of the model's prediction for that answer given that prompt.

What the quirkiness score means. A positive score means the model assigns higher probability to the contextually appropriate answer: lower loss on Alice's answer when Alice is in the prompt, and lower loss on Bob's answer when Bob is in the prompt. A score of zero or negative means the model fails to condition its answer distribution on the prompt name—it hasn't learned the quirky behavior. The authors state: "Thus if a model failed to assign a higher probability to the correct character's labels than to the incorrect character on average it received a quirkiness of less than or equal to 0" (Section 4.4).

Correlation between quirkiness and detection. The paper uses the quirkiness coefficient to validate that detection failures aren't simply due to the model not being quirky. Figure 4 plots quirkiness against detection AUROC for both models. For Llama, there is a strong correlation (r=0.745r = 0.745): low quirkiness is associated with low detection performance, which is expected—if the behavioral distinction is weak, internal features won't separate well. However, there are notable exceptions: Sentiment shows high quirkiness but low detection performance, and Modular Addition on Mistral shows "high anomaly detection performance despite low quirkiness" (Section 4.4). These exceptions indicate that quirkiness is not a sufficient statistic for detectability—some behavioral distinctions leave stronger traces in internal activations than others.

Feature Extraction Methods

The paper tests a wide variety of methods for extracting numerical features from the model's internal state during a forward pass. Each method provides a different "view" of the model's computation, and a key empirical question is whether these views provide orthogonal or redundant signals for anomaly detection.

Residual Stream Activations

The simplest and most extensively tested feature type. For a given prompt, the model is run forward, and at each transformer layer $\ell$, the activations from the residual stream at the last token position are collected. The residual stream at layer $\ell$ is a vector $\mathbf{h}_\ell \in \mathbb{R}^{d_{\text{model}}}$ representing the accumulated information from all previous layers, where $d_{\text{model}}$ is the model's hidden dimension (4096 for Llama 3.1 8B, 4096 for Mistral 7B v0.1).

Unless specified otherwise, features are taken from the residual stream directly—the output of the attention and MLP sublayers summed together at each layer. The paper notes they "also experimented with taking activations from MLP outputs rather than the residual stream" (Section 3.2), but residual stream activations are the default.

Why residual stream? The residual stream is the primary communication channel between transformer layers. Information from attention patterns, MLP computations, and the original token embeddings all accumulate in the residual stream, making it a rich summary of the model's computation up to that point. Taking activations at different layers captures the model's processing at different stages: early layers might encode syntactic features, middle layers might encode task-relevant semantics, and late layers might encode answer preparation. The per-layer analysis in Figure 6 shows that detection performance varies substantially across layers, with deeper layers generally (but not always) providing better discrimination.

Attribution Patching Features

Attribution patching (Nanda, 2024; Syed et al., 2023) estimates how much a specific component of the model affects the final output. The paper uses this to generate features that capture causal importance rather than just correlational presence.

Core mechanism. Attribution patching works by running the model forward twice—once with the original activations and once with a modified ("patched") version—and measuring the difference in some output metric. The paper's output metric is "the final difference in logits between 'Yes' and 'No' tokens" (Section 3.2). Intuitively, this measures how much a given model component pushes the model toward answering "Yes" versus "No," and interventions that reverse this push might flip the model's answer.

The intuition for anomaly detection. The authors hypothesize that "for examples where Alice and Bob disagree, there may be internal interventions in the model that switch its output from 'Bob's' to 'Alice's'" (Section 3.2). That is, the quirky model might have components that are specifically responsible for the heuristic behavior, and intervening on these components could restore correct reasoning. Attribution patching estimates a vector of such intervention effects for each layer.

Procedure. For each attention head in a given layer, the paper estimates the effect of ablating (replacing) that head's activations with a baseline value. The result is a vector of effect sizes—one scalar per attention head per layer—that becomes the feature vector. The paper experiments with three types of ablation for computing these effects:

  • Mean ablation: Replace the activations of each attention head with the mean activation of that head computed across the trusted training set. This removes the head's input-specific output while preserving its average behavior. The effect size is the change in the Yes/No logit difference when the head is replaced with its mean.

  • Principal components ablation: Replace the activations by subtracting the difference between the activation and its projection onto each of the top 10 principal components of the trusted activations, then adding back the mean projected activation. This is a more targeted intervention: rather than replacing the entire activation with a mean, it removes variation along the dominant directions of variation in the trusted data. This tests whether anomaly-relevant information lives in these principal directions.

  • Gradient norm: Take the norm of the gradient of the Yes/No logit difference with respect to each attention head's output. This measures sensitivity—how much would a small change in the head's output affect the final answer—without actually performing an ablation. The gradient norm is a first-order approximation of the head's importance.

Why attribution patching might capture different information than activations. Raw activations capture what information is present at a given layer, but they don't distinguish between information that is causally important for the output and information that is merely correlated. Two prompts might have similar activations at a layer but differ in which components are driving the final answer. Attribution patching directly measures causal influence, which might reveal anomalies where the model's computation follows an unusual causal pathway even if the activations look superficially normal. The paper reports that attribution-based detectors have only moderate correlation with activation-based detectors (r=0.60r = 0.60, Figure 3b), indicating they provide partially independent signals.

Sparse Autoencoder (SAE) Features

The paper uses pre-trained sparse autoencoders (Gao et al., 2024; Bricken et al., 2023) that were trained on the base models (not the quirky fine-tuned versions) to decompose residual stream activations into sparse, interpretable feature activations.

What SAEs do. A sparse autoencoder is a neural network trained to reconstruct its input through a bottleneck that encourages sparsity. Given an activation vector $\mathbf{h} \in \mathbb{R}^{d_{\text{model}}}$ from the residual stream, the SAE computes:

f=ReLU(Wench+benc)\mathbf{f} = \text{ReLU}(W_{\text{enc}} \mathbf{h} + \mathbf{b}_{\text{enc}})

h^=Wdecf+bdec\hat{\mathbf{h}} = W_{\text{dec}} \mathbf{f} + \mathbf{b}_{\text{dec}}

where $\mathbf{f} \in \mathbb{R}^{d_{\text{SAE}}}$ is the sparse feature activation vector (typically $d_{\text{SAE}} \gg d_{\text{model}}$, so the SAE expands the representation into a higher-dimensional but sparse space), and the reconstruction $\hat{\mathbf{h}}$ should approximate $\mathbf{h}$. The sparsity constraint ensures that only a small fraction of SAE features are active for any given input, and these active features are expected to correspond to interpretable concepts.

Why use base-model SAEs on fine-tuned models? The SAEs were trained on the base models before quirky fine-tuning, which means the features they detect are general features of language and reasoning, not features specifically learned for the quirky task. When applied to the quirky fine-tuned model, the SAE decomposes the activation into these pre-defined features. The hypothesis is that anomalous (Bob-mode) processing might be characterized by unusual patterns of which SAE features activate—features that rarely fire on trusted (Alice-mode) examples might fire on anomalous examples, or features that typically fire might be suppressed.

Key practical detail: The paper had SAE features available for fewer layers than raw activations, which is why Figure 3c (SAE/L0 vs. activations/Mahalanobis correlation) contains fewer points than the other correlation plots. The SAEs were trained as top-k sparse autoencoders, meaning only the top k most active features are kept non-zero during the forward pass, enforcing hard sparsity rather than the soft sparsity encouraged by L1 penalties.

Normalizing Flow Features

The paper trains a normalizing flow (Rezende & Mohamed, 2016) on the base model that transforms residual stream activations into a simpler distribution.

What normalizing flows do. A normalizing flow is an invertible neural network that learns a bijective transformation from a complex data distribution (residual stream activations) to a simple base distribution (in this case, a Laplace distribution—a double-exponential distribution with heavier tails than a Gaussian). Because the transformation is invertible and differentiable, the flow can compute exact probability densities under the learned model: the density of an activation vector $\mathbf{h}$ under the flow is:

p(h)=pbase(f(h))⋅∣det⁡∂f(h)∂h∣p(\mathbf{h}) = p_{\text{base}}(f(\mathbf{h})) \cdot \left|\det \frac{\partial f(\mathbf{h})}{\partial \mathbf{h}}\right|

where $f$ is the learned transformation to the Laplace base distribution, and the determinant term accounts for the change in volume introduced by the transformation.

How it is used for anomaly detection. The flow is trained on base model activations (pre-fine-tuning). After fine-tuning, the flow transforms quirky model activations to their corresponding points in the Laplace-distributed latent space. These transformed representations become the features for anomaly detection. The intuition is that the flow "Gaussianizes" (more precisely, "Laplacizes") the activations, removing complex correlations and making the distribution more amenable to simple distance-based anomaly scores.

The paper reports results using two scoring approaches on the flow-transformed features: (1) directly computing the probability density under the Laplace distribution (the "Laplace" score), and (2) computing Mahalanobis distance in the flow's latent space (the "flow Mahalanobis" score). The latter can be thought of as: the flow removes nonlinear dependencies, then Mahalanobis distance captures any remaining linear deviations from normality.

Probe Shift Features

Probe shift uses attribution patching to answer a different question: instead of measuring how interventions affect the model's Yes/No logit difference, it measures how interventions affect the accuracy of a linear probe trained to predict the model's output.

Procedure. First, a linear probe is trained on the trusted dataset to predict the model's binary (Yes/No) output from residual stream activations at a given layer. This probe is trained only on Alice-mode examples where the model answers correctly. Then, for each attention head in layers prior to the probe's layer, the effect of ablating that head on the probe's accuracy is estimated using attribution patching.

The logic. If the model's processing in Bob mode uses attention patterns that are unusual relative to Alice mode, then interventions that disrupt those attention patterns should reduce the probe's ability to predict the model's output—not because the probe is wrong, but because the model's mapping from earlier-layer computation to later-layer activations has changed. The effect vector (how much each earlier-layer head's ablation reduces probe accuracy) becomes the feature for anomaly detection.

"All of the above ablations are also applicable to this method. For a probe at layer $i$, we consider the effect of all interventions in layers prior to $i$."

The three ablation types (mean, principal components, gradient norm) are all applicable, producing different variants of the probe shift feature.

Misconception Contrast Probing

This method is directly inspired by MacDiarmid et al. (2024)'s probes for detecting sleeper agents, but adapted for the faulty-heuristic setting.

Procedure. The authors "trained a supervised probe to distinguish model activations on an automatically generated set of misconceptions and correct statements, then used this probe's score to distinguish Alice from Bob modes of operation" (Section 3.2). The probe is a linear classifier trained on paired examples: activations from the model processing correct statements versus activations from the model processing corresponding misconception statements (statements that are factually incorrect in a specific way related to the task's faulty heuristic).

Why misconceptions rather than deception. The original MacDiarmid et al. (2024) work trained probes to detect intentional deception—cases where the model "knew" the truth but output something else. In the quirky setting, the anomalous behavior involves faulty heuristics, not intentional deception. The authors hypothesized that "a dataset of misconceptions rather than deception... might be better aligned with the kind of anomalous behaviour we were investigating" (Section 3.2). The distinction is subtle: a model using a faulty heuristic genuinely "believes" its incorrect answer in some sense, while a deceptive model maintains an internal representation of the truth separate from its output. Misconception probes might capture the former better than deception probes.

Result preview. The paper reports that this method was "substantially less discriminative in our setting than in theirs" (Section 2), which the authors interpret as evidence that the nature of the anomalous behavior (faulty heuristic vs. intentional deception) does affect detection difficulty—misconception-style anomalies may leave weaker internal traces than deception-style anomalies.

Confidence Estimation via Iterative Prompt Rephrasing

This is the only feature extraction method that does not rely on internal model features; instead, it examines output-level behavior.

Procedure. The authors "automatically generated rephrased prompts of the form '{Question}? One answer is no. {Repeat question}?', and measured how much the model's answers shifted with the rephrasing" (Section 3.2). The idea is inspired by Lin et al. (2024)'s work on uncertainty quantification: if a model is confident in its answer (as it might be when using correct reasoning), its output should be robust to minor rephrasings. If it is using a fragile heuristic, a slight prompt change could flip the answer.

The anomaly score. The raw distance that the rephrasing moves the model's output probability distribution (presumably some divergence metric between the original and rephrased output distributions) serves directly as the anomaly score. The paper includes this method as a non-mechanistic baseline—if simple output-level perturbations can detect anomalies as well as internal features, then the mechanistic approach might be unnecessary.

Result preview. Table 2 shows the iterative rephrasing method achieving a mean AUROC of 0.506 for Mistral and not being reported for Llama, suggesting it performed near chance—substantially worse than activation-based methods. This supports the paper's implicit argument that internal features contain information not easily extractable from outputs alone.

Anomaly Scoring Rules

Once features are extracted, the system must convert a feature vector into a scalar anomaly score. The paper categorizes scoring rules into two settings:

Offline Setting: Access to Unlabeled Test Data

In the offline setting, the detector has access to both the labeled trusted training data and unlabeled test data. This setting is "roughly analogous to a situation where we are happy to incrementally improve our ability to catch anomalous behaviour; we collect examples of anomalous behaviour as it arises and use these to improve an anomaly detector to reduce the frequency of this behaviour in the future" (Section 3.2).

Quantum Entropy Score

Based on Hayase et al. (2021)'s SPECTRE method, but adapted to the paper's setting. The original SPECTRE assumes anomalies are rare and no trusted data is available—both assumptions flipped in this paper—so the implementation is a modified version.

Procedure:

  1. Whitening transformation: The test data is whitened using the trusted data covariance matrix. Specifically, if $\boldsymbol{\mu}_{\text{trust}}$ is the mean of trusted features and $\boldsymbol{\Sigma}_{\text{trust}}$ is the covariance matrix, each test point $\mathbf{x}$ is transformed to:

    x~=Σtrust−1/2(x−μtrust)\tilde{\mathbf{x}} = \boldsymbol{\Sigma}_{\text{trust}}^{-1/2} (\mathbf{x} - \boldsymbol{\mu}_{\text{trust}})

    After this transformation, the trusted data would have zero mean and identity covariance (if it were Gaussian). The test data's covariance in this whitened space, $\boldsymbol{\Sigma}_{\text{test}}^{\text{whitened}}$, captures how the test distribution differs from the trusted distribution: directions with large eigenvalues in $\boldsymbol{\Sigma}_{\text{test}}^{\text{whitened}}$ are directions where the test data has "excess variance" compared to the trusted data.

  2. Score computation: The quantum entropy score interpolates between two distances:

    • The Mahalanobis distance from the trusted mean: $d_{\text{Mahalanobis}}(\tilde{\mathbf{x}}) = \|\tilde{\mathbf{x}}\|_2$, which treats all directions equally in the whitened space.
    • The distance along the largest principal component of the normalized test data covariance matrix: $d_{\text{PC1}}(\tilde{\mathbf{x}}) = |\mathbf{v}_1^\top \tilde{\mathbf{x}}|$, where $\mathbf{v}_1$ is the top eigenvector of $\boldsymbol{\Sigma}_{\text{test}}^{\text{whitened}}$. This focuses exclusively on the direction of maximum test-trusted distribution difference.

    The interpolation is controlled by a parameter that weights these two distances. The exact interpolation formula is not specified in the paper, but the concept from Hayase et al. (2021) is that the score blends a global outlier measure (Mahalanobis) with a targeted measure (the direction of maximum distributional shift).

What the score captures. The quantum entropy score assumes that anomalies manifest as excess variance in specific directions of the whitened feature space. Directions where the test data varies more than the trusted data (given the trusted data's natural variance) are candidate anomaly directions. The interpolation ensures that the score neither ignores subtle anomalies that don't align with the top principal component (which pure PC1 distance would miss) nor dilutes the anomaly signal by averaging over irrelevant directions (which pure Mahalanobis would do).

Why this form for the offline setting. In the offline setting, we have access to unlabeled test data, which allows us to estimate $\boldsymbol{\Sigma}_{\text{test}}^{\text{whitened}}$. This covariance matrix directly captures how the test distribution differs from the trusted distribution, providing a data-driven way to identify which directions are "anomalous." In the online setting, test data isn't available in advance, so the detector must rely solely on the trusted distribution's statistics.

Likelihood Ratio Score

This score compares two generative models of the data: one that assumes all data comes from the trusted distribution, and one that models test data as a separate distribution.

Procedure:

  1. Dimensionality reduction: "Due to small dataset sizes compared to the data dimension, we first projected the data down to the nonzero basis of the trusted data covariance matrix" (Section 3.2.1). This means computing the principal components of the trusted data covariance matrix that have non-zero eigenvalues (effectively, the subspace spanned by the trusted data) and projecting both trusted and test points into this subspace. This avoids singular covariance matrices when the feature dimension exceeds the number of trusted samples.

  2. Model fitting: Fit two Gaussian distributions:

    • $\mathcal{N}(\boldsymbol{\mu}_{\text{trust}}, \boldsymbol{\Sigma}_{\text{trust}})$ to the projected trusted data.
    • $\mathcal{N}(\boldsymbol{\mu}_{\text{test}}, \boldsymbol{\Sigma}_{\text{test}})$ to the projected test data.

  3. Score computation: For each test point $\mathbf{x}$, compute:

    Score(x)=log⁡p(x∣Ntest)p(x∣Ntrust)\text{Score}(\mathbf{x}) = \log\frac{p(\mathbf{x} \mid \mathcal{N}_{\text{test}})}{p(\mathbf{x} \mid \mathcal{N}_{\text{trust}})}

    where $p(\mathbf{x} \mid \mathcal{N})$ is the probability density of $\mathbf{x}$ under the Gaussian distribution. This is the log-likelihood ratio: positive values mean the point is better explained by the test distribution than the trusted distribution, suggesting it is anomalous.

What this score captures. This is a direct implementation of the idea that anomalous examples come from a different distribution than trusted examples. If the test distribution genuinely differs from the trusted distribution (as it should in the quirky setting, since the test set contains Bob-mode examples not present in the trusted set), then the log-likelihood ratio should be positive for Bob-mode test points and negative or near-zero for Alice-mode test points.

Weakness. This method is "very loosely inspired by Pan et al. (2023)" and has an obvious limitation: it fits Gaussians to data that may not be Gaussian. The projection to the trusted data's non-zero variance subspace also discards information in directions where the trusted data has zero variance but the test data does not—though such directions would cause singularities in the Gaussian likelihood anyway.

Gaussian Mixture Model (Expectation-Maximization)

This extends the likelihood ratio method by iteratively refining the class assignments.

Procedure:

  1. Initialization: Fit two Gaussians: one to the trusted data and one to the test data (as in the likelihood ratio method). These serve as initial component parameters.
  2. Iteration (EM steps):
    • E-step (Expectation): For each test point, compute the posterior probability (soft assignment) that it belongs to each of the two Gaussian components, based on the current component parameters.
    • M-step (Maximization): Update the parameters (mean and covariance) of each Gaussian component using the soft assignments as weights—points with higher probability of belonging to a component contribute more to that component's parameter update.
  3. Convergence: Repeat E and M steps until the parameters stabilize.

What this score captures. Unlike the likelihood ratio method, which treats the assignment of test points to "trusted-like" vs. "anomalous" as fixed by the initial Gaussian fits, the EM procedure allows assignments to be refined. A test point that is initially closer to the test Gaussian mean might, after iterations, be reassigned to the trusted Gaussian if that provides a better overall fit. The EM procedure converges to a local maximum of the mixture model likelihood. The final anomaly score is the posterior probability of belonging to the "anomalous" (initially test-fitted) component.

Why iterative refinement might help. In the quirky setting, the test set contains both Alice-mode (normal) and Bob-mode (anomalous) examples. The initial test-distribution Gaussian fits a mixture of both modes, which may not be well-modeled by a single Gaussian. The EM procedure can separate these two subpopulations: the trusted-fitted component should capture test points that look like the trusted data (Alice-mode test examples), while the test-fitted component should capture genuinely anomalous points (Bob-mode test examples). The iterative refinement allows the model to "discover" this structure.

Online Setting: Only Labeled Trusted Data

In the online setting, the detector only has access to the labeled trusted dataset. This is "analogous to a more difficult and high stakes problem: we want to catch anomalous behaviour the first time it arises in the deployment environment" (Section 3.2). The detector must define a notion of normality solely from the trusted data and flag deviations.

Mahalanobis Distance

The most extensively tested online scoring rule.

Procedure: For each layer, compute the mean $\boldsymbol{\mu}_{\text{trust}}$ and covariance matrix $\boldsymbol{\Sigma}_{\text{trust}}$ of the trusted feature vectors (e.g., residual stream activations at that layer). For a test point $\mathbf{x}$, the Mahalanobis distance is:

dM(x)=(x−μtrust)⊤Σtrust−1(x−μtrust)d_M(\mathbf{x}) = \sqrt{(\mathbf{x} - \boldsymbol{\mu}_{\text{trust}})^\top \boldsymbol{\Sigma}_{\text{trust}}^{-1} (\mathbf{x} - \boldsymbol{\mu}_{\text{trust}})}

What this computes. The Mahalanobis distance measures how many standard deviations away $\mathbf{x}$ is from the trusted mean, accounting for correlations between features. In Euclidean distance, all dimensions are treated independently and equally weighted. In Mahalanobis distance, the inverse covariance matrix $\boldsymbol{\Sigma}_{\text{trust}}^{-1}$ downweights directions where the trusted data naturally varies a lot (large eigenvalues of $\boldsymbol{\Sigma}_{\text{trust}}$) and upweights directions where the trusted data has little variance (small eigenvalues). The result is a unitless scalar: a distance of 1 means "one standard deviation away in the direction of natural variation," regardless of what that direction is.

Why Mahalanobis distance for anomaly detection. If the trusted data is approximately multivariate Gaussian, the squared Mahalanobis distance follows a chi-squared distribution. Anomalous points should have unusually large Mahalanobis distances—they fall in regions of feature space far from the trusted distribution, after accounting for natural variance. The Mahalanobis distance is the standard parametric anomaly score for Gaussian data, and it requires estimating only a mean and covariance from the trusted data, making it feasible in the online setting.

Top-k principal components variant. The paper also tested "a variant of the Mahalanobis distance that computed the distance only along the top principal components" (Section 3.2.2). This addresses a practical problem: when the feature dimension is large relative to the number of trusted examples, $\boldsymbol{\Sigma}_{\text{trust}}$ is poorly estimated in low-variance directions. The variant computes the Mahalanobis distance in the subspace spanned by the top-kk principal components (largest eigenvalues) of $\boldsymbol{\Sigma}_{\text{trust}}$, effectively regularizing by discarding noisy low-variance dimensions.

Connection to DeepFeatures detector. The paper identifies this as "a version of the 'DeepFeatures' detector of Subedar et al. (2019)" (Section 2), though that prior work focused on image models.

Local Outlier Factor (LOF)

A non-parametric outlier detection method that compares the local density around a point to the local densities around its neighbors.

Procedure (Breunig et al., 2000):

  1. Save all trusted activations at each layer as a reference set.

  2. For a test point $\mathbf{x}$, find its $k$ nearest neighbors in the trusted set (using Euclidean distance, though the exact distance metric is not specified).

  3. Compute the local reachability density (LRD) of $\mathbf{x}$: the inverse of the average reachability distance from $\mathbf{x}$ to its $k$ neighbors. The reachability distance between $\mathbf{x}$ and a neighbor $\mathbf{o}$ is the maximum of (a) the distance from $\mathbf{x}$ to $\mathbf{o}$ and (b) the distance from $\mathbf{o}$ to its $k$-th nearest neighbor. This smoothing prevents the LRD from being dominated by small distances to points in very dense regions.

  4. Compute the local outlier factor as the average ratio of the LRD of $\mathbf{x}$'s neighbors to the LRD of $\mathbf{x}$:

    LOF(x)=1k∑o∈Nk(x)LRD(o)LRD(x)\text{LOF}(\mathbf{x}) = \frac{1}{k} \sum_{\mathbf{o} \in N_k(\mathbf{x})} \frac{\text{LRD}(\mathbf{o})}{\text{LRD}(\mathbf{x})}

What this score captures. An LOF close to 1 means $\mathbf{x}$ has similar density to its neighbors—it's in a region of consistent density. An LOF substantially greater than 1 means $\mathbf{x}$ has lower density than its neighbors—it's in a sparser region, surrounded by points in denser regions. This makes it an outlier relative to its local neighborhood, rather than a global outlier. An LOF less than 1 means $\mathbf{x}$ is in a denser region than its neighbors (an "inlier"), which is not typical for anomaly detection but can occur.

Why LOF vs. Mahalanobis. Mahalanobis distance assumes the trusted data is unimodal and roughly Gaussian, and it scores outliers based on global distance from the mean. LOF makes no distributional assumptions and can detect outliers that are anomalous relative to their local region even if they're not globally far from the mean. This matters if the trusted data has complex, multi-modal structure—a test point might be close to the global mean but far from any actual trusted point in its vicinity. The high correlation between Mahalanobis and LOF (r=0.97r = 0.97, Figure 3a) suggests that for these features and datasets, global and local outlier measures largely agree, but the correlation is not perfect, and LOF might catch cases that Mahalanobis misses.

L0 Distance (SAE Features Only)

Specific to SAE features, this score leverages the sparsity of SAE activations.

Procedure:

  1. From the trusted set, build a binary mask over SAE features: which features ever activate (have non-zero value) on any trusted example?
  2. For a test point, compute the number of SAE features that are active (non-zero) on this test point but were never active on any trusted example. This count is the L0 distance.

What this score captures. L0 distance directly measures novelty at the feature level: if a test example activates SAE features that were completely unseen in trusted data, those features represent concepts or patterns that the model's computation is engaging with for the first time. This is a very interpretable anomaly signal: "this example uses 17 features we've never seen activated together before."

Why this is specific to SAEs. Raw activations are dense vectors where every dimension has some non-zero value, so counting "never before seen dimensions" is meaningless. SAE features are designed to be sparse and (ideally) interpretable, making the presence/absence of individual features meaningful. The L0 score exploits the SAE's discretization of the continuous activation space into discrete features.

Diagonal Mahalanobis (SAE Features Only)

Also specific to SAE features, this is a simplified Mahalanobis distance that assumes all SAE features are mutually independent.

Procedure. Compute the mean and variance of each SAE feature on the trusted set. For a test point with SAE feature vector $\mathbf{f}$, the anomaly score is:

ddiag(f)=∑i(fi−μi)2σi2d_{\text{diag}}(\mathbf{f}) = \sum_{i} \frac{(f_i - \mu_i)^2}{\sigma_i^2}

where $\mu_i$ and $\sigma_i^2$ are the mean and variance of feature $i$ on the trusted set. This is equivalent to Mahalanobis distance with a diagonal covariance matrix—it ignores correlations between SAE features.

Why assume independence for SAE features? SAE features are trained to be sparse and (ideally) represent independent concepts. If this independence holds in practice, the diagonal approximation loses little information while being more robust to small sample sizes (estimating a diagonal covariance requires $O(d)$ parameters vs. $O(d^2)$ for a full covariance). The comparison between diagonal and full Mahalanobis on SAE features tests whether SAE features are sufficiently decorrelated for this simplification to work.

Laplace Density (Normalizing Flow Features Only)

Specific to normalizing flow features, this score directly evaluates the probability density under the flow's base distribution.

Procedure. The normalizing flow is trained to transform trusted activations to a Laplace (double-exponential) distribution. For a test point's flow-transformed representation, compute the negative log-density under the Laplace distribution. Higher negative log-density (lower probability) means more anomalous.

What this captures. If the flow successfully maps trusted activations to a Laplace distribution, then trusted-like test points should also map to high-density regions of the Laplace, while anomalous points should map to low-density regions. The Laplace distribution is:

p(z)=12bexp⁡(−∣z−μ∣b)p(z) = \frac{1}{2b} \exp\left(-\frac{|z - \mu|}{b}\right)

for each dimension, with heavier tails than a Gaussian. This means outliers are penalized less severely than under a Gaussian density, which might be more robust if the flow doesn't perfectly normalize the activations.

Aggregation Across Layers

Most feature extraction methods produce per-layer features and per-layer anomaly scores. The paper reports results in three ways:

  • Per-layer AUROC: The discrimination performance of the anomaly score from each layer individually. Figure 6 and Figure 7 show these as a function of layer depth, revealing which layers contain the most anomaly-discriminative information. The paper notes that "certain anomaly detection methods, such as meta-activations-mahalanobis and meta-probe-lof mean, demonstrate strong performance at deeper layers" (Appendix B.11, Figure 6 caption).

  • Aggregated AUROC: The anomaly scores from all layers are combined (the paper does not specify the aggregation method—possibly averaging, possibly concatenating features and computing a single Mahalanobis distance in the concatenated space). This tests whether combining information across layers improves discrimination.

  • Best AUROC: The maximum AUROC achieved at any single layer or aggregation. This represents the upper bound on what's achievable if we knew which layer(s) to use. Tables 4-23 in the appendix report "Best Layer" indicating which layer achieved the maximum.

Evaluation Protocol

AUROC as the primary metric. The paper evaluates all detectors using the Area Under the Receiver Operating Characteristic curve (AUROC). The ROC curve plots the true positive rate (fraction of Bob-mode examples correctly flagged as anomalous) against the false positive rate (fraction of Alice-mode examples incorrectly flagged as anomalous) as the anomaly score threshold varies. The AUROC is the area under this curve:

AUROC=∫01TPR(FPR) d(FPR)\text{AUROC} = \int_0^1 \text{TPR}(\text{FPR}) \, d(\text{FPR})

What AUROC measures. An AUROC of 1.0 means perfect separation: there exists a threshold that correctly classifies all examples. An AUROC of 0.5 means the anomaly score is no better than random guessing (the ROC curve is the diagonal). An AUROC below 0.5 means the detector systematically assigns higher scores to Alice-mode than Bob-mode examples—it's anti-informative (though one could flip the sign to get a detector with AUROC > 0.5).

Why AUROC over accuracy? AUROC is threshold-independent—it evaluates the ranking quality of the anomaly scores without requiring a specific decision threshold. This is appropriate for anomaly detection because the threshold would be set based on deployment-specific cost tradeoffs (how many false positives are acceptable vs. how many false negatives). AUROC summarizes the detector's discrimination capability across all possible thresholds.

Agree vs. Disagree subsets. The paper sometimes reports results separately for:

  • Mean AUROC Agree: Limited to examples where Alice and Bob would produce the same answer (the heuristic happens to be correct on these questions). These are harder to detect because the model's output doesn't reveal which mode it's in.
  • Mean AUROC Disagree: Limited to examples where Alice and Bob produce different answers. These are easier to detect because the behavioral difference is larger, but also more important because a failure to detect means accepting a wrong answer.

The paper notes that on the offline quantum entropy detector, "Mean AUROC Disagree" is higher than "Mean AUROC Agree" for Mistral (0.964 vs. 0.926) but lower for Meta (0.766 vs. 0.806), which is an unusual pattern—one would typically expect disagree examples to be easier to detect since the behavioral distinction is sharper. The paper does not explore this asymmetry in detail.

Summary of Design Choices and Justifications

  • Multiple name variants over single Alice/Bob: Prevents detectors from learning to recognize name-specific activation patterns, forcing them to learn features of the processing mode rather than the input trigger. Supported by Figure 2 showing that single-name labeling makes detection artificially easy.

  • Difficulty stratification (easy trusted, hard test): Tests generalization across difficulty, which matters for deployment where we train on clear cases and deploy on ambiguous ones.

  • Two base models from different families: Tests whether detection effectiveness is model-specific. The finding that Llama 3.1 8B is harder to detect anomalies in than Mistral 7B v0.1 suggests model architecture or training data affects detectability.

  • Multiple feature types (activations, attribution, SAE, flows): Tests whether different views of model internals provide orthogonal or redundant signals. The high correlations between activation-based methods suggest limited diversity, but the moderate correlation with attribution-based methods (r=0.60r = 0.60) indicates partial complementarity.

  • Both online and offline scoring rules: The online setting (trusted data only) represents the harder, more realistic scenario. The offline setting (access to unlabeled test data) explores whether having a sample of the deployment distribution helps, which informs whether collecting unlabeled deployment data is a worthwhile investment.

  • Per-layer analysis: Different layers may encode different aspects of the Alice/Bob distinction. The finding that deeper layers often perform better is consistent with the idea that behavioral mode is a high-level property that emerges late in processing.

  • Exclusion of non-quirky model-dataset pairs: This prevents artificial deflation of measured performance. The quirkiness coefficient provides a quantitative validation that excluded models genuinely lack the behavioral distinction.

4. Key Insights and Innovations

Innovation 1: Difficulty as the Organizing Principle for Test-Time Compute Allocation

This paper's most intellectually distinctive contribution is not any specific strategy for spending inference compute, but rather the meta-strategy of making allocation decisions conditional on estimated prompt difficulty. This is a fundamental shift in how the field conceptualizes test-time compute.

Before this work, the implicit model in the literature was that test-time compute strategies have a universal quality ordering: beam search is "better" than best-of-N, lookahead search is "better" than beam search, and so on. The question was which strategy to use, answered by running experiments and picking the one with the highest average accuracy. The paper systematically dismantles this view by showing that the relationship between strategy and performance is qualitatively different at different difficulty levels—and sometimes reverses entirely.

The counterintuitive evidence is in Figure 3 (right): beam search, the most sophisticated optimizer tested, degrades performance on easy problems as the compute budget increases, while best-of-N continues to improve. On medium-difficulty problems, the pattern flips—beam search consistently and substantially outperforms best-of-N. This is not a monotonic "more optimization is better" relationship. It means the "best" strategy averaged across an undifferentiated test set is a fiction; the real optimal strategy is difficulty-contingent.

What makes this a genuine innovation rather than an obvious observation is that the difficulty-dependent behavior arises from a specific failure mode—verifier over-optimization—that was not well-characterized for language model inference before this work. The paper provides the first clear empirical demonstration that PRM-guided search, when pushed to high budgets, finds solutions that score well under the verifier but are factually wrong, and that this over-optimization is most damaging precisely where the base model is already competent (easy problems, where the verifier's signal-to-noise ratio is misleadingly favorable). This is analogous to the reward hacking phenomenon documented in RLHF, but transplanted to the test-time compute setting with concrete scaling curves.

The significance of this insight extends beyond the specific strategies tested. It establishes that test-time compute scaling has its own "scaling laws" analogous to pretraining scaling laws (Hoffmann et al., 2022), but organized around an orthogonal variable—difficulty rather than budget. Just as Chinchilla showed that the optimal ratio of parameters to data changes with total compute, this paper shows that the optimal ratio of search depth to parallel sampling changes with problem difficulty. The conceptual parallel is direct, but the mechanism is entirely different, making this a genuinely new contribution to the understanding of how language models can be optimized at inference time.

The practical implementation—five difficulty quintiles with a lookup table of optimal strategies—is relatively crude. But the concept that test-time compute should be adaptively allocated based on prompt difficulty, rather than uniformly applied, is what matters. This framework explains why prior work reached contradictory conclusions: Huang et al. (2023) found that "LLMs cannot self-correct reasoning" while Madaan et al. (2023) found self-refinement helps. The paper's difficulty-binned analysis reveals that these studies were likely testing on different implicit difficulty distributions. The reconciliation of these conflicting findings through a single organizing variable (difficulty) is one of the paper's most intellectually satisfying contributions.

Innovation 2: Verifier Over-Optimization as the Primary Bottleneck, Not Search Algorithm Design

The paper's second major conceptual contribution is reframing the bottleneck in test-time compute scaling from search algorithm sophistication to verifier robustness. Prior to this work, the natural instinct for improving test-time compute was to develop better search algorithms—tree search, Monte Carlo Tree Search, lookahead search, and so on. The paper provides compelling evidence that this instinct is largely wrong: more powerful optimization, when paired with an imperfect verifier, can be counterproductive.

The key evidence is Figure 3 (left): lookahead search—which gives the verifier more context to evaluate partial solutions by simulating forward—underperforms all simpler methods at equivalent generation budgets. This is the most sophisticated search method tested, and it performs worst overall. The mechanism is clear: lookahead search amplifies the verifier's errors by committing more computation to optimizing against a flawed signal. Beam search, which is less aggressive in its optimization, outperforms lookahead search at low budgets but eventually succumbs to the same over-optimization problem at high budgets (Figure 3, right, easy bins).

The implication is a significant reorientation of the research agenda. The paper essentially argues that the field has been optimizing the wrong thing. Instead of designing more powerful search algorithms, effort should go into building verifiers that remain calibrated under optimization pressure. This is a non-obvious lesson that the experimental results make hard to dismiss: if the strongest optimizer produces the worst results, then optimization power is not the bottleneck—verifier quality is.

What distinguishes this from a simple empirical observation is that the paper provides both quantitative evidence (the scaling curves showing degradation) and qualitative evidence (Appendix M's examples of degenerate outputs that score highly under the PRM—overly short solutions, repetitive low-information steps). This combination of aggregate metrics and case studies makes the over-optimization diagnosis convincing rather than speculative.

The paper's compute-optimal allocation policy can be understood as a way to route around the verifier's weaknesses: use weak optimization (best-of-N) where the verifier is most vulnerable to exploitation (easy problems), and use stronger optimization (beam search) only where the verifier signal provides genuine guidance (medium problems). This is clever, but it treats the symptom rather than the cause. The deeper insight—that verifier robustness is the binding constraint on test-time compute scaling—points toward a different research program: adversarial training of PRMs, ensemble verification, or constrained optimization with KL penalties to keep search results close to the base model's typical output distribution.

Innovation 3: Sharp Boundary Conditions on When Test-Time Compute Substitutes for Pretraining

The FLOPs-matched comparison in Section 7 delivers a finding that is both practically important and intellectually clarifying: test-time compute can substitute for pretraining compute, but only within sharp boundaries defined by problem difficulty and the inference-to-pretraining ratio.

Prior work on the training-inference trade-off (Jones, 2021; Sardana and Frankle, 2023) established that such substitution is possible in principle, but typically assumed access to ground-truth answers at inference time—an unrealistic assumption that makes the problem easier. This paper operates in the more realistic setting where ground truth is unknown, making the comparison directly relevant to deployment.

The intellectual contribution is not just the empirical finding itself (a smaller model with compute-optimal test-time strategies can outperform a ~14× larger model on easy-to-medium problems), but the precise characterization of where the substitution fails. The hardest questions (difficulty bin 5, Figure 9) show essentially zero improvement from any amount of test-time compute—the scaling lines are flat near 0–5% regardless of budget. This establishes a fundamental capability ceiling: test-time compute amplifies existing capability but cannot create it. If the base model's pass@1 is near zero, no amount of search or revision helps, because there are no correct solutions in the proposal distribution to find.

This has significant implications for how the field thinks about scaling. The dominant narrative has been that scaling pretraining (bigger models, more data) is the primary path to improved capability, with inference-time techniques as a secondary optimization. The paper's results suggest a more nuanced picture: for problems within the base model's capability range, investing in test-time compute can yield better returns than the equivalent investment in pretraining. But for problems outside that range, pretraining remains the only viable path. This is not a universal substitution but a conditional one, and the conditions are clearly specified.

The dependence on R=Dinference/DpretrainR = D_{\text{inference}} / D_{\text{pretrain}} adds another layer of boundary condition. When R≪1R \ll 1 (few inference tokens relative to pretraining), the pretraining savings from using a smaller model translate into a large inference compute budget, and test-time compute wins convincingly. When R≫1R \gg 1 (high-volume inference), the larger model's per-token inference cost dominates the budget, and pretraining becomes preferable. This provides concrete guidance for different deployment scenarios that was absent from prior work.

A notable subtlety: the paper acknowledges that the ~14× larger model is scaled only in parameters (not data), following the LLaMA paradigm rather than compute-optimal pretraining. A Chinchilla-optimal larger model would likely be a stronger baseline, so the reported advantages of test-time compute may be upper bounds. But even acknowledging this caveat, the finding that test-time and pretraining compute are not 1-to-1 exchangeable across all difficulty regimes is robust and important.

Innovation 4: A Unifying Taxonomy of Test-Time Compute Methods as Proposal vs. Verifier Modifications

The paper's conceptual framework in Section 2—decomposing all test-time compute methods into modifications to the proposal distribution (what the model generates) versus the verifier (how outputs are selected)—is not itself a novel idea. It echoes the proposer-scorer decomposition familiar from MCMC, reinforcement learning, and earlier work on inference-time strategies. What is intellectually valuable is the paper's empirical demonstration that these two axes have complementary, difficulty-dependent strengths, and that prior work's contradictory findings can be explained by the failure to analyze them jointly.

Concretely: revisions (proposal modification) are most effective on easy problems where the base model's output is roughly correct and just needs refinement—this is local search in answer space. PRM-guided search (verifier optimization) is most effective on medium-hard problems where the model needs to explore qualitatively different solution strategies—this is global search. The paper demonstrates that the optimal strategy shifts from purely sequential revisions on easy problems to a balanced ratio of sequential and parallel on hard problems (Figure 7, right), and from best-of-N on easy problems to beam search on medium problems (Figure 3, right).

The intellectual contribution here is not the taxonomy itself but the empirical characterization of when each axis matters. Prior work studied these mechanisms in isolation—revision papers showed that revisions sometimes helped and sometimes didn't, search papers showed that search sometimes helped and sometimes didn't—and the field was left with a confusing set of contradictory results. The paper's framework resolves this by showing that the contradiction is an artifact of testing on different difficulty distributions. Revisions dominate on the easy end, search dominates in the middle, and neither helps on hard problems. This is a genuinely explanatory contribution: it doesn't just report new results, it explains why old results conflicted.

The paper never fully realizes the combination of these two axes—Section 8 explicitly acknowledges that "we did not experiment with PRM tree-search techniques in combination with revisions." This is a significant gap in the experimental program, but it doesn't diminish the conceptual contribution. The paper provides the intellectual scaffolding for such combination by characterizing the complementary strengths of each axis, making it clear why combining them should outperform either alone: revisions improve the quality of candidates that search then selects among. This is a roadmap for future work that follows naturally from the paper's analysis.

Innovation 5: The Over-Optimization Pattern as a First-Class Scaling Phenomenon

The paper elevates verifier over-optimization from a nuisance observation to a systematically characterized scaling phenomenon with clear, measurable consequences. While the RLHF literature has documented reward hacking in the context of policy optimization, this paper provides the first detailed characterization of how over-optimization manifests in test-time compute scaling—and, crucially, how it varies with problem difficulty.

The evidence is multi-layered. Quantitatively, Figure 3 (right) shows that beam search accuracy on easy problems (bin 1) decreases from approximately 78% to 77% as the budget increases from 4 to 256 generations, while best-of-N accuracy continues to rise. This is the smoking gun: more optimization should not produce worse results unless the optimization signal is flawed and being exploited. The difficulty-level breakdown is essential to seeing this pattern—aggregated across all difficulties (Figure 3, left), beam search's degradation at high budgets is partially masked by its strong performance on medium questions, making the over-optimization less visible.

Qualitatively, Appendix M provides examples of the degenerate outputs that beam search produces at high budgets: solutions that consist of repetitive low-information steps, or solutions that are suspiciously short (1-2 steps to a final answer), both of which the PRM scores favorably. These examples make the over-optimization mechanism concrete: search finds solutions that exploit the verifier's blind spots rather than solutions that are genuinely correct.

The paper also establishes a hierarchy of over-optimization susceptibility. Lookahead search, which gives the verifier the most information to work with (by simulating forward multiple steps), paradoxically produces the worst overall results (Figure 3, left). The interpretation is that more powerful optimization paired with an imperfect verifier simply optimizes the verifier signal more efficiently—producing solutions that are better according to the verifier but worse according to ground truth.

What makes this a conceptual contribution rather than just an empirical observation is that it establishes over-optimization as the binding constraint on test-time compute scaling. The paper shows that all sophisticated search methods eventually hit a performance ceiling where additional compute produces no improvement or even degradation. This ceiling is determined by verifier quality, not by the search algorithm. The implication is clear: progress on test-time compute scaling requires progress on verifier robustness. Better search algorithms without better verifiers will hit the same ceiling. This reframes the research agenda in a way that is not obvious from looking at any single result in isolation.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All experiments use the "quirky datasets" introduced by Mallen et al. (2024), modified by the authors to include 16 "Alice-like" and 16 "Bob-like" names instead of a single name per class. The datasets cover multiple domains: capital cities, hemispheres, scientific Q&A (SciQ), populations, sentiment detection, natural language inference (NLI), and arithmetic operations (addition, subtraction, multiplication, modular addition, squaring). The appendix of Mallen et al. (2024) specifies the faulty heuristic for each dataset and the difficulty assignment rules. The total number of questions is not explicitly stated, but the test set consists of 500 questions from the MATH benchmark in the original Mallen et al. work, and the quirky datasets are constructed analogously with difficulty percentiles used for partitioning. The trusted set consists of questions with difficulty below the 25th percentile; the test set consists of questions with difficulty above the 75th percentile.

  • Base models. Two open-weight language models are fine-tuned: Llama 3.1 8B (Grattafiori et al., 2024) and Mistral 7B v0.1 (Jiang et al., 2023). These represent different model families at similar parameter scales, chosen to investigate whether detection effectiveness is model-dependent. The base models are fine-tuned on the full quirky training set with early stopping based on total validation set loss.

  • Metrics. The primary metric is AUROC (Area Under the Receiver Operating Characteristic curve) computed between Alice-mode and Bob-mode test examples. AUROC measures the ranking quality of anomaly scores without requiring a specific decision threshold—1.0 represents perfect separation, 0.5 represents chance-level discrimination. The paper reports AUROC in three variants: per-layer (AUROC from each layer's features individually), aggregated (anomaly scores combined across layers, though the exact aggregation method is not specified), and best (the maximum AUROC achieved at any single layer or aggregation). Some results are also reported separately for examples where Alice and Bob agree on the answer ("Mean AUROC Agree") versus where they disagree ("Mean AUROC Disagree").

  • Baselines. The paper does not compare against previously published anomaly detection systems as standardized baselines. Instead, the experimental design compares many different feature extraction methods (activations, attribution patching, SAE features, normalizing flows, probe shift, misconception contrast probing, iterative prompt rephrasing) and scoring rules (Mahalanobis distance, Local Outlier Factor, L0 distance, diagonal Mahalanobis, Laplace density, quantum entropy, likelihood ratio, Gaussian Mixture Model) against each other. The implicit baseline is random chance (AUROC = 0.5). The paper also compares their modified multi-name quirky dataset against the original single-name-per-class version from Mallen et al. (2024) using activation features with Mahalanobis distance scoring on Mistral 7B v0.1 (Figure 2), establishing the multi-name version as a harder detection task.

  • Generation budget / compute accounting. There is no generation budget in this paper—the experiments measure anomaly detection on already-completed model outputs. The relevant computational cost is feature extraction: running forward passes on the fine-tuned models for trusted and test examples, plus any additional computation for attribution patching (requiring multiple forward passes per example to estimate intervention effects), SAE encoding, normalizing flow transformation, or iterative prompt rephrasing. The paper does not report or compare the computational cost of different feature extraction methods.

  • Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported. The anomaly detectors are trained once on the fixed trusted set and evaluated once on the fixed test set. The name-based partition (first 4 Alice names for trusted, last 12 Alice/Bob names for test) serves as the primary guard against overfitting, ensuring no overlap in names, difficulty levels, or question content between training and evaluation. Model-dataset combinations where the fine-tuned model failed to learn quirky behavior (as measured by the quirkiness coefficient and validation loss patterns) are excluded from anomaly detection results entirely: Mistral 7B v0.1 on modular addition and authors, and Llama 3.1 8B on SciQ, hemisphere, and authors.

Main Quantitative Results

Impact of Multiple Names on Detection Difficulty

The paper first validates that the multi-name modification made the detection task genuinely harder. Figure 2 compares mean AUROC scores across datasets for the single-name-per-class original quirky setup versus the multi-name modified version, both using activation features with Mahalanobis distance scoring on Mistral 7B v0.1. The result is unambiguous: "In all cases except for Modular Addition the addition of more names makes anomaly detection worse or at least no better, and in many cases much worse" (Section 4.1). For example, on the hemisphere dataset, single-name AUROC is approximately 0.95–1.0 while multi-name drops to approximately 0.80. On sentiment, the drop is from approximately 0.70 to approximately 0.50. This validates the paper's concern that the original dataset contained a trivial detection shortcut and establishes that the multi-name version forces detectors to learn genuine features of anomalous processing rather than name-specific patterns.

Offline Detector Performance: Strong but Inconsistent

Table 1 reports aggregate offline detector results for the quantum entropy (QUE) score using activation features, averaged across all datasets where models exhibited quirky behavior. For Mistral 7B v0.1, the mean AUROC is 0.936, with Mean AUROC Agree of 0.926 and Mean AUROC Disagree of 0.964. For Llama 3.1 8B (Meta), the mean AUROC is substantially lower at 0.791, with Agree at 0.806 and Disagree at 0.766. The pattern for Llama is unusual—disagree examples (where the behavioral difference is sharper) are harder to detect than agree examples, contrary to what one would expect. The paper does not explore this asymmetry.

Table 1 also reports the Expectation-Maximization (Gaussian Mixture Model) and Likelihood Ratio offline scores, but only for Mistral 7B v0.1 (the "Model" column in the table is labeled "Mistral" for both rows, suggesting these methods were not evaluated on Llama or the results are omitted). The EM score achieves a mean AUROC of 0.778 (Agree: 0.801, Disagree: 0.744), and the Likelihood Ratio achieves 0.828 (Agree: 0.846, Disagree: 0.800). These are all lower than the quantum entropy score for Mistral (0.936), suggesting that in the offline setting, the quantum entropy approach of identifying directions of excess variance is more effective than fitting separate Gaussian distributions to trusted and test data.

The dataset-level breakdown in Appendix B tables reveals extreme heterogeneity. For Mistral with quantum entropy (Tables 4, 6, 8, 10, 12, 14, 16, 18, 20, 22), the best AUROC (best layer or aggregate) is:

  • Addition: 1.000
  • Subtraction: 1.000
  • Multiplication: 1.000
  • Modular Addition: 1.000
  • Squaring: 1.000 (Meta only; Mistral not reported for squaring offline)
  • Hemisphere: 1.000
  • Population: 1.000
  • NLI: 0.814
  • SciQ: 0.637
  • Sentiment: 0.975

The arithmetic tasks uniformly achieve perfect or near-perfect discrimination. Non-arithmetic tasks range from excellent (Population at 1.000) to poor (SciQ at 0.637). For Meta with quantum entropy (Tables 4, 6, 8, 10, 12, 14, 16, 18, 20):

  • Addition: 1.000
  • Subtraction: 1.000
  • Multiplication: 1.000
  • Modular Addition: 1.000
  • Squaring: 1.000
  • Hemisphere: 0.622
  • Population: 0.581 (aggregate; the per-layer mean is 0.566)
  • NLI: 0.571
  • SciQ: 0.386 (aggregate; the per-layer mean is 0.443)
  • Sentiment: 0.789

Llama's offline detection on non-arithmetic tasks is substantially worse than Mistral's, with SciQ at 0.386 and NLI at 0.571, both close to or below chance-level performance.

Online Detector Performance: Broad Survey Across Methods

Table 2 reports mean AUROC scores across all datasets for online detectors, aggregated by model and feature type. The key patterns:

Mistral 7B v0.1 (top-performing methods):

  • Attribution/LOF Mean: 0.850 (Agree: 0.844, Disagree: 0.840)
  • Probe/LOF Mean: 0.852 (Agree: 0.860, Disagree: 0.836)
  • Probe/Mahalanobis Mean: 0.846 (Agree: 0.846, Disagree: 0.842)
  • Activations/LOF: 0.846 (Agree: 0.841, Disagree: 0.807)
  • Attribution/Mahalanobis Mean: 0.827 (Agree: 0.831, Disagree: 0.811)
  • Activations/Mahalanobis: 0.816 (Agree: 0.828, Disagree: 0.790)

All of these are in the 0.82–0.85 range, suggesting that for Mistral, the choice of feature type and scoring rule among these top methods makes relatively little difference in aggregate. The weakest online detector on Mistral is Iterative Rephrase at 0.506 (Agree: 0.482, Disagree: 0.478)—essentially at chance—followed by Misconception Contrast at 0.642 (Agree: 0.629, Disagree: 0.639).

Llama 3.1 8B (Meta, all methods):

  • Activations/LOF: 0.762 (Agree: 0.763, Disagree: 0.768)
  • Activations/Mahalanobis: 0.741 (Agree: 0.750, Disagree: 0.712)
  • NFlow/Mahalanobis: 0.737 (Agree: 0.744, Disagree: 0.710)
  • SAE/L0: 0.737 (Agree: 0.747, Disagree: 0.715)
  • Probe/LOF Mean: 0.737 (Agree: 0.753, Disagree: 0.717)
  • Attribution/LOF Mean: 0.730 (Agree: 0.747, Disagree: 0.718)
  • SAE/Diag Mahalanobis: 0.729 (Agree: 0.744, Disagree: 0.689)
  • Probe/Mahalanobis Mean: 0.652 (Agree: 0.696, Disagree: 0.615)
  • Attribution/Mahalanobis Mean: 0.639 (Agree: 0.682, Disagree: 0.609)
  • NFlow/Laplace: 0.581 (Agree: 0.617, Disagree: 0.522)

All Llama online detectors score substantially below their Mistral counterparts. The best Llama detector (Activations/LOF at 0.762) is still below the worst of the top-tier Mistral detectors (Activations/Mahalanobis at 0.816). The gap between models is consistent across feature types and scoring rules—roughly 0.05–0.10 AUROC points lower for Llama. The NFlow/Laplace detector on Llama is notably poor at 0.581, and the Disagree subset is worse than Agree (0.522 vs. 0.617).

Task-Specific Breakdown: Arithmetic vs. Non-Arithmetic Divide

Table 3 provides the critical task-level granularity, reporting mean and best AUROC aggregated across all detectors (both online and offline) for each dataset. The arithmetic/non-arithmetic divide is stark:

Arithmetic datasets (all near-perfect):

  • Addition: Mistral mean 0.864, best 1.000; Meta mean 0.932, best 1.000
  • Subtraction: Mistral mean 0.919, best 1.000; Meta mean 0.898, best 1.000
  • Multiplication: Mistral mean 0.886, best 1.000; Meta mean 0.921, best 1.000
  • Modular Addition: Mistral mean 0.878, best 0.999; Meta mean 0.890, best 1.000
  • Squaring: Mistral mean 0.962, best 0.976; Meta mean 0.971, best 1.000

Best AUROC is 1.000 or 0.999 for every arithmetic dataset on both models. Mean AUROC is above 0.86 in all cases, meaning that even the worst-performing layer for arithmetic tasks still provides strong discrimination.

Non-arithmetic datasets (inconsistent and often poor):

  • Population: Mistral mean 0.909, best 0.987; Meta mean 0.494, best 0.581
  • NLI: Mistral mean 0.736, best 0.957; Meta mean 0.529, best 0.558
  • SciQ: Mistral mean 0.524, best 0.642; Meta mean 0.415, best 0.572
  • Sentiment: Mistral mean 0.488, best 0.760; Meta mean 0.528, best 0.741
  • Hemisphere: Mistral mean 0.804, best 0.952; Meta mean 0.524, best 0.583

Two patterns dominate: (1) For every non-arithmetic dataset, Mistral substantially outperforms Meta, often by 0.2–0.4 AUROC points in mean and 0.3–0.4 in best. (2) Within Mistral, there is large variance across non-arithmetic datasets—Population (0.909 mean, 0.987 best) and Hemisphere (0.804 mean, 0.952 best) are strong; Sentiment (0.488 mean, 0.760 best) and SciQ (0.524 mean, 0.642 best) are weak. For Meta, all non-arithmetic datasets have mean AUROC near 0.5, with best AUROCs only slightly above chance (0.558–0.741), except Population where the best is 0.581. The paper states this explicitly: "average AUROC scores on non-arithmetic datasets for Llama 3.1 8B are uniformly below 0.8, with many close to chance" (Section 4.2).

Dataset-level online detector results (Appendix B.1–B.10). The detailed per-dataset, per-detector tables in Appendix B reinforce the arithmetic/non-arithmetic divide at finer granularity. For Addition (Tables 4 and 5), nearly every detector on both models achieves AUROC of 0.95–1.000. The only exceptions are Mistral Misconception (0.447 mean) and Mistral Rephrase (0.544 mean). For Subtraction (Tables 22 and 23), the pattern is identical—near-perfect discrimination from almost all methods, with Misconception and Rephrase as consistent outliers. For NLI (Tables 12 and 13), the best Meta detector is Activations/Mahalanobis at 0.532 (close to chance), while Mistral's best is Attribution/LOF Mean at 0.957—a massive model-dependent gap.

The layerwise dimension (Appendix B.11, Figures 6 and 7). Figure 6 shows per-layer AUROC for all online detectors across all datasets. The qualitative trends: activation-based detectors (both Mahalanobis and LOF) on Mistral show strong performance at deeper layers (layers 25–30) for several non-arithmetic datasets (e.g., population, hemisphere), while earlier layers perform near chance. For Meta, even the best layers rarely exceed 0.7 on non-arithmetic tasks. Attribution-based detectors show less consistent layerwise patterns. Figure 7 shows the same for offline detectors (quantum entropy, EM, likelihood ratio), where the pattern is similar: Mistral's quantum entropy detector maintains high AUROC across layers for arithmetic and some non-arithmetic tasks, while Meta's performance is uniformly lower.

Correlation Between Detection Methods

Figure 3 quantifies the redundancy across detection approaches. Panel (a) shows the correlation between activations/Mahalanobis and activations/LOF on Mistral 7B v0.1: r=0.966r = 0.966. Each point represents a detector trained on a particular dataset at a particular layer. The tight clustering around the diagonal means that these two scoring rules—one parametric (Mahalanobis) and one non-parametric (LOF)—produce nearly identical rankings of model-dataset-layer combinations, despite their methodological differences. If Mahalanobis fails on a particular combination, LOF almost certainly fails too, and vice versa.

Panel (c) shows the correlation between activations/Mahalanobis and SAE/L0 on Llama 3.1 8B: r=0.989r = 0.989. This is a cross-feature-type comparison—raw residual stream activations versus sparse autoencoder features—and the near-perfect correlation indicates that the SAE decomposition, despite its very different representational format, captures essentially the same anomaly signal as raw activations for the purposes of this detection task.

Panel (b) shows the correlation between activations/Mahalanobis and attribution/LOF Mean on Mistral: r=0.596r = 0.596. This is the lowest cross-method correlation reported and indicates that attribution patching provides partially independent information from raw activations. The scatter plot shows substantial dispersion around the regression line, with many points in the upper-left quadrant (high attribution AUROC, low activations AUROC) and lower-right quadrant (high activations AUROC, low attribution AUROC). The paper notes: "while attribution based detectors were worse overall than activation based detectors, there were a number of model-dataset-layer combinations where activation based detectors scored no better than chance while attribution based detectors scored modestly better than chance—a pattern that was much less common in the reverse direction" (Section 4.3).

However, despite this partial independence, "naively combining attribution and activation based detectors did not result in an increase in performance" (Section 4.3, referring to Table 2). Table 2 shows that the combined "Activation + Attribution" features are not reported as a separate row with improved AUROC, and the individual feature-type AUROCs for Meta (e.g., Activations/Mahalanobis at 0.741 vs. Attribution/Mahalanobis Mean at 0.639) suggest that combining a strong signal with a weaker, partially correlated signal does not yield a better composite score when naively concatenated.

Quirkiness vs. Detectability

Figure 4 plots the relationship between the quirkiness coefficient (a measure of how much the model's probability distribution over answers shifts when the prompt name changes) and detection AUROC for the activations/Mahalanobis detector. The key findings:

For Mistral (panel A): The correlation is weak (r=0.150r = 0.150). Most datasets cluster in the upper-right quadrant with high quirkiness (>60) and high AUROC (>0.8), but there are notable exceptions. Modular Addition (pink, leftmost cluster) shows AUROC near 0.95 despite quirkiness near zero—the model barely distinguishes Alice from Bob behavior according to the quirkiness metric, yet the anomaly detector achieves near-perfect discrimination. Sentiment (orange, around quirkiness 20, AUROC ~0.60) shows moderate quirkiness but poor detection. The overall weak correlation means that quirkiness, as measured by output probability shifts, does not reliably predict whether internal features will separate Alice from Bob modes.

For Llama (panel B): The correlation is stronger (r=0.745r = 0.745). Most datasets show a clear positive relationship: higher quirkiness is associated with higher AUROC. However, there are still notable outliers. Population (blue, quirkiness near 120, AUROC near 0.55) has the highest quirkiness score of any Llama dataset but near-chance detection. Sentiment (orange, quirkiness near 90, AUROC near 0.55) shows a similar pattern—large behavioral distinction but poor internal detectability. The arithmetic datasets (various colors in the upper right) all show both high quirkiness and high AUROC, forming a tight cluster near (100, 1.0).

The authors summarize: "We found that the low performance of anomaly detectors on Llama finetunes was generally associated with low quirkiness... There were, however, datasets with high quirkiness and low anomaly detection performance (Sentiment), and modular addition on Mistral showed high anomaly detection performance despite low quirkiness" (Section 4.4). This means quirkiness is neither necessary nor sufficient for detectability—it explains some but not all of the variance in detector performance.

Linear Separability as an Explanatory Factor

Figure 5 (Appendix A.1) shows the relationship between linear separability of Alice and Bob activations and detection AUROC. The x-axis plots the normalized class separation (variance ratio) on a log scale:

Sijk=(μijkAlice−μijkBob)2Tr(ΣijkAlice, Bob)S_{ijk} = \frac{(\boldsymbol{\mu}_{ijk}^{\text{Alice}} - \boldsymbol{\mu}_{ijk}^{\text{Bob}})^2}{\text{Tr}(\boldsymbol{\Sigma}_{ijk}^{\text{Alice, Bob}})}

where μijkAlice\boldsymbol{\mu}_{ijk}^{\text{Alice}} is the centroid of activations for Alice-mode examples for dataset ii, model jj, layer kk, μijkBob\boldsymbol{\mu}_{ijk}^{\text{Bob}} is analogous, and ΣijkAlice, Bob\boldsymbol{\Sigma}_{ijk}^{\text{Alice, Bob}} is the covariance matrix of the combined classes. This measures how separated the class means are relative to the within-class variance.

The key finding: "no anomaly detectors exhibit high performance with normalized class separation below 10−210^{-2}, and few detectors exhibit low performance with normalized class separation above 10−110^{-1}" (Appendix A.1). This establishes a necessary (but not sufficient) condition for effective anomaly detection: the Alice and Bob activation distributions must be linearly separable in the feature space. Datasets with separation below 10−210^{-2} (left side of Figure 5) uniformly show AUROC near 0.5 regardless of the detection method used.

However, SciQ (green points) is a notable exception: it shows normalized class separation above 10−110^{-1} (in the regime where most datasets achieve high AUROC) but its detection AUROC is near 0.4–0.5. The paper attributes this to the extreme label imbalance in SciQ: "Recall that SciQ exhibited extreme label shift between the training and test sets, which may contribute to this phenomenon" (Appendix A.1). This means that even when the Alice and Bob activation centroids are well-separated, other factors—such as a shift in the distribution of easy vs. hard examples, or the label distribution shift between trusted and test sets—can prevent detectors from leveraging that separation effectively.

Ablation Studies and Robustness Checks

The paper does not follow a traditional ablation methodology where individual components are removed to measure their contribution. Instead, the ablation-like comparisons are distributed throughout the results and appendix, testing variations in feature type, scoring rule, model, and dataset configuration. The key comparative findings are:

Single-name vs. multi-name datasets (Figure 2): Using the original Mallen et al. (2024) single-name-per-class setup versus the paper's multi-name modification, with activation features and Mahalanobis distance on Mistral 7B v0.1. Single-name labeling produces artificially high AUROC (often near 1.0) because detectors can learn name-specific activation patterns. The multi-name modification substantially reduces AUROC on most datasets, validating the paper's claim that the original dataset contained a trivial shortcut. The one exception is Modular Addition, where multi-name slightly improves AUROC—possibly because the task is so difficult that the name-based signal was noise rather than a useful shortcut.

Activations vs. attribution features (Table 2, Figure 3b): Attribution-based detectors (LOF Mean and Mahalanobis Mean) are compared to activation-based detectors across both models. On Mistral, attribution/LOF Mean (0.850) slightly outperforms activations/Mahalanobis (0.816) and rivals activations/LOF (0.846). On Meta, attribution/LOF Mean (0.730) underperforms activations/LOF (0.762). The moderate correlation (r=0.60r = 0.60, Figure 3b) indicates partial independence, but combining features naively does not improve performance. The finding that attribution occasionally succeeds where activations fail (but not usually the reverse) suggests attribution patching captures a somewhat different signal, but one that is generally weaker and noisier.

Mahalanobis vs. LOF scoring (Table 2, Figure 3a): Both scoring rules are applied to the same activation features on both models. On Mistral, LOF (0.846) slightly outperforms Mahalanobis (0.816). On Meta, LOF (0.762) slightly outperforms Mahalanobis (0.741). The correlation is r=0.966r = 0.966 (Figure 3a), indicating near-identical rankings across model-dataset-layer combinations. LOF's marginal advantage is consistent but small—the choice between these two scoring rules matters little compared to the choice of feature type, model, or dataset.

Probe shift features (Table 2): Using probes trained on trusted data to predict model outputs, then measuring how earlier-layer interventions affect probe accuracy, produces competitive performance on Mistral (Probe/LOF Mean: 0.852; Probe/Mahalanobis Mean: 0.846) but weaker performance on Meta (Probe/LOF Mean: 0.737; Probe/Mahalanobis Mean: 0.652). The Mistral results are among the best overall, suggesting that for models where the Alice/Bob distinction is well-represented in activations, probe shift provides a useful complementary signal. The drop on Meta is consistent with the general pattern of weaker detection on Llama.

SAE features with different scoring rules (Table 2, Meta only): SAE features are evaluated with diagonal Mahalanobis (assuming feature independence) and L0 distance (counting novel feature activations). On Meta, diagonal Mahalanobis achieves 0.729 and L0 achieves 0.737—both comparable to raw activation-based methods (0.741–0.762). The lack of improvement from SAE features suggests that the sparse decomposition does not surface anomaly-relevant structure that is inaccessible from raw activations, at least for these datasets. The diagonal Mahalanobis assumption (feature independence) performs similarly to L0, suggesting that SAE features are sufficiently decorrelated for this simplification to be reasonable.

Normalizing flow features with different scoring rules (Table 2, Meta only): Flow-transformed features evaluated with Mahalanobis distance achieve 0.737, comparable to other methods, while the Laplace density score achieves only 0.581—a large gap. This indicates that the flow's transformation into a "simpler" distribution is useful as a preprocessing step (improving the effectiveness of Mahalanobis distance), but directly evaluating density under the flow's base distribution is unreliable. The Laplace density's poor performance (especially on Disagree: 0.522) suggests the flow does not map anomalous activations to consistently low-density regions of the Laplace distribution.

Iterative prompt rephrasing (Table 2, Mistral only): This output-level method—measuring how much model answers shift under prompt rephrasing—achieves mean AUROC of 0.506, essentially at chance (0.482 on Agree, 0.478 on Disagree). This is the worst-performing method tested and the only one that does not use internal model features. It is not reported for Meta, suggesting it performed at or below chance and was excluded. This supports the paper's implicit argument that internal features contain anomaly-relevant information not accessible from output behavior alone.

Misconception contrast probing (Table 2, Mistral only): Inspired by MacDiarmid et al. (2024)'s sleeper agent probes, this method trains a supervised probe to distinguish activations on misconceptions vs. correct statements, then uses the probe score for anomaly detection. It achieves mean AUROC of 0.642 (Agree: 0.629, Disagree: 0.639)—well above chance but substantially below the best activation-based methods. The paper notes this is "substantially less discriminative in our setting than in theirs" (Section 2), attributing the difference to the nature of the anomalous behavior (faulty heuristics vs. intentional deception).

Per-layer analysis (Figures 6 and 7): The layerwise AUROC plots serve as an implicit ablation of where in the model anomaly-relevant information resides. For activation-based detectors on Mistral (Figure 6, top row, left two panels), AUROC generally increases with layer depth, with the best performance at layers 25–32. For Meta (Figure 6, bottom rows), the layerwise pattern is less consistent and the peak AUROC is lower across all layers. This suggests that the Alice/Bob behavioral distinction is encoded primarily in deeper layers, consistent with the idea that behavioral mode is a high-level property that emerges late in processing. The offline detectors (Figure 7) show similar patterns.

Alice/Bob Agree vs. Disagree subsets: Most detectors report slightly higher AUROC on the Agree subset than the Disagree subset (Tables 1 and 2). This is counterintuitive—examples where Alice and Bob disagree should show larger behavioral differences and thus be easier to detect. The paper does not explain this pattern, but one possibility is that Agree examples are more numerous (since the faulty heuristic often produces the correct answer by coincidence on easy questions), providing a larger and more statistically stable evaluation set. Another possibility is that the anomaly signal is partially independent of the output—the model might be in "Bob mode" processing even when the final answer happens to match Alice's.

Dataset-level heterogeneity within the non-arithmetic category: The appendix tables reveal that even within the "non-arithmetic" category, performance varies dramatically. Mistral achieves best AUROC of 0.987 on Population but only 0.642 on SciQ (Table 3). This indicates that "non-arithmetic" is not a homogeneous category—some non-arithmetic tasks (Population, Hemisphere) produce highly detectable anomalies while others (SciQ, Sentiment) produce near-chance detection, and this variation is not explained by the arithmetic/non-arithmetic distinction alone.

Label imbalance as a confound: The paper identifies but does not ablate the label imbalance in SciQ and Population (Figure 1). SciQ's trusted set is >99% "true" labels while its test set is >99% "false" labels, creating a near-complete label shift between training and evaluation environments. Population shows a similar extreme shift for Alice examples. The authors explicitly note that SciQ's anomalous behavior in Figure 5 (high linear separability but low AUROC) may be attributable to this label shift. However, they do not run a controlled experiment with balanced labels to isolate the effect—this remains a confound that complicates interpretation of results on these datasets.

Critical Assessment

Do the experiments support the claim that anomaly detectors can achieve high discrimination on some tasks?

Yes, with strong evidence from the arithmetic datasets. Table 3 shows best AUROC of 1.000 for addition, subtraction, multiplication, modular addition, and squaring on both models. The detailed per-dataset tables in Appendix B confirm that near-perfect discrimination is achieved across multiple detector types, not just a single method. For instance, Meta on Addition (Table 5) achieves 1.000 AUROC from activations/LOF, activations/Mahalanobis, flow/Mahalanobis, SAE/Diag Mahalanobis, and SAE/L0—essentially every detector that uses internal features. This is not a fragile result dependent on a specific configuration.

However, the arithmetic tasks are also the simplest and most structured of the datasets tested. The faulty heuristics for arithmetic (e.g., adding 1 to one operand, squaring the wrong number) produce systematic, predictable errors that likely create large, consistent shifts in the model's internal representations. Whether this generalizes to more realistic anomalous behaviors—where the heuristic might be more subtle or inconsistent—is untested. The paper's claim that "detectors can achieve high discrimination on some tasks" is technically correct, but the "some tasks" are exclusively simple arithmetic operations, which limits the practical significance of the claim.

Do the experiments support the claim that no detector is effective across all models and tasks?

Yes, with overwhelming evidence. Table 3 is the clearest demonstration: for non-arithmetic datasets on Llama 3.1 8B, the mean AUROC ranges from 0.415 (SciQ) to 0.529 (NLI), with best AUROCs ranging from 0.558 (NLI) to 0.741 (Sentiment). None of these reach 0.8, many are barely above chance, and SciQ's best is 0.572—only marginally better than random. The paper explicitly states that "average AUROC scores on non-arithmetic datasets for Llama 3.1 8B are uniformly below 0.8, with many close to chance" (Section 4.2), and the data support this. Even on Mistral, where performance is generally better, SciQ achieves a best AUROC of only 0.642 (Table 3).

The model-dependence claim is supported by the consistent gap between Mistral and Llama across nearly all feature types and scoring rules in Table 2. Mistral's best methods cluster around 0.85; Llama's cluster around 0.74—a gap of roughly 0.1 AUROC. This is not explainable by a single detector configuration being suboptimal on Llama, because the gap persists across very different approaches (raw activations, SAE features, normalizing flows, attribution patching).

The task-specificity claim is supported by the arithmetic/non-arithmetic divide and the variance within non-arithmetic tasks. Population on Mistral achieves 0.987 best AUROC; SciQ on the same model achieves 0.642. These tasks use the same model, same detector training protocol, and same evaluation metric, but produce dramatically different results. This cannot be attributed to model-specific factors—it is genuinely task-specific.

Do the experiments support the claim that larger deviations in behavior generally make anomalies easier to detect, though not universally?

Partially supported, with important caveats. Figure 4 (right panel) shows a moderate correlation (r=0.745r = 0.745) between quirkiness and detection AUROC for Llama, and the paper states that "low performance of anomaly detectors on Llama finetunes was generally associated with low quirkiness" (Section 4.4). This supports the "generally easier to detect" part of the claim.

However, the evidence for "not universally true" is stronger than the evidence for the positive correlation. On Mistral, the correlation between quirkiness and AUROC is only r=0.150r = 0.150 (Figure 4, left panel)—effectively no relationship. On both models, there are clear counterexamples: Sentiment (both models) shows high quirkiness but low detection AUROC, and Modular Addition (Mistral) shows low quirkiness but high detection AUROC. These are not minor exceptions—they are prominent, labeled outliers in Figure 4 that the paper explicitly discusses.

The quirkiness metric itself has limitations as a measure of "behavioral deviation." It measures shifts in the output probability distribution, but the relationship between output probability shifts and internal representation shifts is not straightforward. A model could have a large output shift (high quirkiness) achieved through a small, localized internal change that is hard to detect from population-level statistics, or a small output shift (low quirkiness) achieved through a distributed internal change that is easy to detect. The paper does not investigate this distinction, so the claim about "larger deviations in behavior" conflates output-level behavior with internal processing, which are not the same thing.

Are there genuine weaknesses in the experimental design?

Small and potentially unrepresentative test sets. The test set consists of questions with difficulty above the 75th percentile, using the last 12 of 16 names. For datasets where the total number of questions is not large (the MATH benchmark has 500 test questions, but the quirky datasets may be smaller), the test set could contain only a few dozen examples per class, making AUROC estimates noisy. The paper does not report confidence intervals or standard errors on any AUROC measurements, making it impossible to assess whether differences between methods are statistically significant or within sampling noise.

The label imbalance confound is unaddressed. Figure 1 shows extreme label shifts in SciQ and Population: the distribution of ground-truth labels (true/false) changes almost completely between the trusted and test sets. This means that a detector could achieve high AUROC simply by detecting label-related features rather than Alice/Bob-mode features—the test set is not just harder (higher difficulty) but differently composed (different answer distribution). The paper acknowledges this but treats it as an interesting challenge rather than a confound to be controlled: "We retained the label imbalanced datasets, as we wanted to test how well our anomaly detectors accommodate distribution shifts, but we note that in these cases the label imbalance was an extreme and unintended shift" (Section 3.1.1). For SciQ and Population, we cannot distinguish whether poor detector performance is due to the anomaly being hard to detect, or due to the label distribution shift creating a mismatch between what the detector learned on trusted data (where most answers are "true") and what it encounters at test time (where most answers are "false").

No comparison to simple text-based baselines. The paper's only non-mechanistic baseline is iterative prompt rephrasing, which performed near chance. But there are simpler baselines that were not tested: a bag-of-words classifier on the prompt text, a detector based on output token probabilities (e.g., detecting when the model is uncertain vs. confident), or a detector based on the semantic content of the generated answer. Including these would help establish whether internal features provide information beyond what is available from the input or output alone. The iterative rephrasing result (AUROC ≈ 0.5) suggests that simple output-level signals are not sufficient, but one negative result with a specific method is not a comprehensive case.

No investigation of why arithmetic tasks are easy and non-arithmetic tasks are hard. The paper documents the arithmetic/non-arithmetic divide extensively but never investigates why it exists. Are arithmetic anomalies easier to detect because arithmetic reasoning is more localized in specific model components? Because the faulty heuristic for arithmetic produces more systematic activation differences? Because arithmetic questions have more uniform structure? Understanding the mechanism behind the task-specificity would substantially strengthen the paper's contribution—as it stands, the finding is purely descriptive.

The exclusion of non-quirky model-dataset pairs is justified but limits generalizability. The paper excludes Mistral on modular addition and authors, and Llama on SciQ, hemisphere, and authors, because these models did not learn to distinguish Alice from Bob behavior. This is methodologically correct—testing anomaly detection where no anomaly exists would be meaningless—but it means the paper cannot speak to whether detection would work on these tasks with better fine-tuning or different models. The exclusion also means that for some datasets (authors, SciQ on Llama, hemisphere on Llama), we have no results at all, limiting the coverage of the task space.

Missing experiments that would strengthen the paper:

  1. Adversarial detector evaluation: Train detectors that are explicitly optimized to evade the anomaly detectors, testing whether the detection signal is robust to adversarial manipulation. This matters for safety applications where a model might be intentionally designed to hide anomalous processing.

  2. Calibration and threshold-setting analysis: AUROC measures ranking quality but not calibration—a detector with AUROC of 0.9 might still produce anomaly scores that are poorly calibrated, making it difficult to set a threshold for practical deployment. Reporting expected calibration error or reliability diagrams would bridge the gap between AUROC and practical usability.

  3. Cross-task generalization: Train a detector on one quirky dataset (e.g., addition) and test it on another (e.g., multiplication) to investigate whether anomaly features are task-specific or capture a general "faulty reasoning" signal. This would directly address the paper's motivating question about whether MAD can serve as a general-purpose oversight tool.

  4. Scaling with trusted set size: Vary the number of trusted examples used for detector training and measure the impact on AUROC. This would characterize the data efficiency of different methods and inform practical deployment where trusted examples may be scarce.

  5. Inference-time cost analysis: The paper's title is "Mechanistic Anomaly Detection," but it never discusses the mechanistic interpretability cost—how many forward passes are needed for attribution patching versus simple activation extraction, how SAE encoding compares, etc. For real deployment, this cost matters because it determines whether the detector can be run on every inference or only on a sampled subset.

Do the experiments support the paper's stated conclusions about MAD's viability?

The paper's abstract concludes that "MAD techniques may be effective in low-stakes applications, but advances in both detection and evaluation are likely needed if they are to be used in high stakes settings." This is a fair characterization of the results, but it is also vague—"low-stakes" and "high-stakes" are never defined, and the paper does not specify what AUROC threshold would be acceptable for which applications.

A more precise summary of what the experiments actually demonstrate is: On simple arithmetic tasks with structured faulty heuristics, using a less capable model (Mistral 7B), activation-based anomaly detectors can achieve near-perfect discrimination between correct and heuristic reasoning modes, even when the detector is trained only on easy examples and tested on hard ones with different prompt names. On more complex language tasks (sentiment, NLI, SciQ, hemisphere, population), detector performance ranges from good (Mistral on Population, 0.987) to near-chance (Llama on SciQ, 0.572), with no clear pattern predicting which tasks will be easy or hard. Detectors on the more capable Llama 3.1 8B consistently underperform detectors on Mistral 7B v0.1, raising the concerning possibility that as models improve, anomaly detection becomes harder. Different feature extraction methods and scoring rules produce highly correlated results, meaning that testing many methods does not substantially increase the chance of finding one that works on a given task—if activations/Mahalanobis fails, SAE/L0 probably fails too.

The paper's contribution is primarily in establishing the variability of MAD effectiveness and in identifying factors (model, task, linear separability, quirkiness) that correlate with that variability. It does not provide a working system for deployment, nor does it identify a detector configuration that works reliably across tasks. The experiments successfully demonstrate that MAD is not currently a robust general-purpose technique, which is a valuable negative result for the scalable oversight community, even if it leaves the positive question—"how can we make it robust?"—unanswered.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Dominates the Headline Efficiency Gains

The assumption or constraint. The entire compute-optimal framework requires estimating each prompt's difficulty before allocating the test-time compute budget. The paper's method for doing so—generating 2048 samples per question and averaging either ground-truth correctness (oracle bins) or PRM final-answer scores (predicted bins)—is described in Section 3.2 as follows: the system samples 2048 complete solutions from the base model, computes the pass@1 rate or average PRM score, and bins questions into five quintiles. The authors acknowledge that "estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2).

The consequence. The difficulty estimation step alone consumes more compute than the largest test-time budgets studied. At 2048 samples per question, the estimation cost exceeds the 256–512 generation budgets for which the paper reports results. The headline claim of ~4× better efficiency over best-of-N (Figures 4 and 8) is computed after difficulty is already known, without amortizing the cost of learning it. In a realistic deployment, total cost is difficulty estimation plus strategy execution, and the former swamps the latter for any question evaluated in isolation. The 4× figure is therefore not a realized efficiency gain—it is an upper bound on what is achievable if difficulty could be estimated for free. For one-off queries (where difficulty estimation must be performed per-prompt and cannot be amortized across many examples of the same question), the reported gains would be negative—the total computation exceeds a fixed best-of-N budget with no difficulty estimation by a large margin.

What evidence exists in the paper. Section 3.2 explicitly flags the cost issue. The prediction-based difficulty estimation (using PRM scores instead of ground-truth labels) still requires 2048 samples and PRM scoring of each sample—it removes the need for labels but does not reduce the generation cost. The paper reports that predicted difficulty bins perform similarly to oracle bins (Figures 4 and 8 curves "largely overlap"), confirming that label-free estimation is viable, but does nothing to address the sample cost. No experiment measures how detection accuracy degrades with fewer estimation samples. No latency or total FLOPs accounting includes the estimation overhead.

Mitigation status. The paper acknowledges the problem and frames it as "a key avenue for future work" (Section 3.2), suggesting that models could be trained to "predict difficulty of a question directly" from the question text without requiring 2048 samples. No such model is developed or evaluated. The difficulty estimation cost is not included in any budget calculation in the paper. This is not a minor caveat—it means the paper's central quantitative claim (4× efficiency) cannot be taken at face value for deployment without significant additional research.

Hard Problems Remain Completely Unsolved—The Approach Has a Hard Capability Ceiling

The assumption or constraint. The compute-optimal framework assumes that the base model already produces correct solutions at some non-trivial rate for the problems being addressed. Section 3.1 formalizes this implicitly: the optimization objective maximizes the probability of producing the correct answer under a strategy π, which requires that the correct answer has non-zero probability mass in the base model's output distribution. The paper explicitly states in Section 8 that compute-optimal scaling "provides no benefit for problems that are fundamentally outside the base model's capability range."

The consequence. For the hardest problems (difficulty bin 5), no method produces meaningful improvement regardless of compute budget. Figure 3 (right) shows bin 5 accuracy hovering at 1–3% for all search methods across all budgets up to 256 generations. Figure 7 (right) shows bin 5 accuracy at roughly 2–3% irrespective of the sequential-to-parallel ratio at 128 generations. Figure 9 (bottommost lines) shows the bin 5 scaling line essentially flat near 0–5% for both revisions and PRM search, at all budgets tested. In the FLOPs-matched comparison, hard problems show a −52.9% relative disadvantage from using test-time compute instead of the ~14× larger model at R ≫ 1 (Figure 1, bottom-right bar chart). This means that for problems where the base model's pass@1 is near zero, no amount of test-time compute helps—there simply are no correct solutions in the proposal distribution to find or refine. The approach cannot compensate for fundamental capability gaps that larger pretraining would address.

What evidence exists in the paper. The difficulty-bin analyses across Sections 5 and 6 are consistent: bin 5 shows near-zero improvement everywhere. Section 7 quantifies this explicitly in the FLOPs-matched comparison. The paper is transparent about this boundary: the Section 7 takeaway box and Section 8 both state that "test-time compute can amplify existing capability but cannot create it."

Mitigation status. The paper does not attempt to solve this limitation or propose mechanisms to extend the approach to out-of-capability problems. It correctly identifies the boundary and reports it candidly. The limitation is fundamental to the approach—it follows from the framework of optimizing within a fixed proposal distribution. Extending the framework to problems beyond the base model's reach would require mechanisms (such as tool use, retrieval, or interactive problem decomposition) that are outside the paper's scope.

Single Benchmark, Single Model Family—The Difficulty-Dependent Patterns May Not Generalize

The assumption or constraint. All experiments use the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* as the base model. The paper states it chose MATH because "test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences—mathematical reasoning fits this profile because it requires multi-step logical deduction rather than novel factual recall" (Section 4). The authors assert they "believe this model is representative of the capabilities of many contemporary LLMs," but this is not tested against any other model family or benchmark.

The consequence. Several aspects of the findings could be MATH-specific or PaLM 2-S*-specific:

  • The difficulty-dependent over-optimization pattern (beam search degrading easy problems in Figure 3, right) depends on the PRM's calibration properties, which are a function of Monte Carlo rollout training on PaLM 2-S*'s output distribution. A model with different error patterns or different output diversity might exhibit different over-optimization thresholds.
  • The optimal sequential-to-parallel ratio (Figure 7, right) depends on the revision model's ability to produce useful refinements, which depends on the base model's in-context learning capabilities and the quality of the edit-distance-based training data pairing. These could vary substantially across model families.
  • MATH is exclusively symbolic math reasoning. The paper's finding that revisions help easy problems while search helps medium problems may not transfer to code generation, factual QA, logical reasoning, or open-ended generation, where the nature of errors and the structure of the solution space differ fundamentally.

What evidence exists in the paper. None—there is no multi-benchmark or multi-model replication. The paper tests only MATH with PaLM 2-S* (and a ~14× larger PaLM 2 variant for the FLOPs-matched comparison). Section 8 does not flag this as a limitation, framing MATH as a deliberate choice rather than a restriction.

Mitigation status. The paper does not address this limitation beyond the stated belief that PaLM 2-S* is "representative." The authors do not suggest multi-benchmark evaluation as future work (Section 8 focuses on combining revisions and search, cheap difficulty estimation, and self-improvement loops). This is a significant gap for practitioners deciding whether to adopt the approach: the paper provides no evidence that the central insight—difficulty-conditioned allocation—generalizes beyond math reasoning on PaLM 2-family models.

The ~14× Larger Model Baseline Is Not Compute-Optimally Trained, Making the FLOPs-Matched Comparison Potentially Favorable to Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* (with compute-optimal test-time strategies) against a model with approximately ~14× more parameters, where the larger model uses greedy decoding with no test-time augmentation. The paper explicitly acknowledges that the larger model is scaled only in parameters, holding training data fixed: "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work" (Section 7). This follows the LLaMA paradigm (Touvron et al., 2023) rather than the Chinchilla-optimal paradigm (Hoffmann et al., 2022), where both parameters and data are scaled proportionally.

The consequence. A compute-optimally trained ~14× larger model (scaling both parameters and data) would likely outperform the parameter-only-scaled baseline used in the paper. The reported advantages of test-time compute over the larger model—e.g., +27.8% relative improvement on medium questions at R ≪ 1 with revisions (Figure 1, top-right bar chart)—may shrink or reverse against a properly Chinchilla-optimal larger model. Additionally, the larger model uses only greedy decoding with no test-time compute budget of its own. Giving the larger model even a modest budget (e.g., best-of-8 or majority voting over 4 samples) would create a substantially stronger baseline that is never tested. The paper's claim that test-time compute on a smaller model can outperform a ~14× larger model is therefore conditioned on the larger model being trained and deployed in a suboptimal way.

What evidence exists in the paper. Section 7 explicitly describes the parameter-only scaling choice and compares it to the LLaMA paradigm. The paper acknowledges the departure from compute-optimal pretraining but does not quantify how much this weakens the baseline. No sensitivity analysis is performed—e.g., giving the larger model some test-time compute budget, or using a Chinchilla-optimal larger model if available.

Mitigation status. The paper frames the Chinchilla-optimal comparison as future work (Section 7). The limitation is disclosed but not resolved. For practitioners, the key implication is that the FLOPs-matched results should be interpreted as favorable conditions for test-time compute rather than a neutral comparison—a properly optimized pretraining baseline would likely close some or all of the reported advantage, particularly on medium-difficulty problems and at higher inference-to-pretraining ratios.

Sequential Revisions Introduce an Unresolved Latency vs. Throughput Trade-off That the Generation-Budget Metric Ignores

The assumption or constraint. The paper measures test-time compute exclusively in "generations"—the number of complete solutions sampled—which serves as a proxy for total FLOPs but ignores wall-clock latency. Section 3.1 defines the compute budget N as the number of generations, and all comparisons (Figures 4, 6, 7, 8, 9) use this metric. The compute-optimal revision strategies (Figure 7) allocate a mix of sequential and parallel chains: for a budget of N generations with sequential-to-parallel ratio s, the system runs N/(s+1) parallel chains, each of length s.

The consequence. Sequential revisions are inherently serial—each revision depends on the output of the previous one and cannot be parallelized. A strategy that allocates 128 generations as 64 sequential × 2 parallel (sequential-to-parallel ratio of 32:1) takes roughly ~64× longer wall-clock time than a strategy that runs 128 fully parallel samples, assuming sufficient hardware parallelism. The paper's finding that sequential revisions are optimal or near-optimal for easy problems (Section 6, Figure 7 left: "At lower budgets (8–32 generations), fully sequential is optimal") means the compute-optimal policy maximizes latency on the very problems where it could achieve the highest accuracy. For latency-sensitive applications—interactive assistants, real-time tutoring, on-device inference where parallelism is limited—the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be completely impractical despite their accuracy advantages.

What evidence exists in the paper. The paper never discusses latency, throughput, or the sequential nature of revisions as a practical constraint. Figure 7 reports accuracy as a function of sequential-to-parallel ratio at fixed generation budgets, but never as a function of wall-clock time. The generation-budget metric treats one sequential revision step and one parallel independent sample as equivalent units of "compute," ignoring their very different latency implications.

Mitigation status. The paper does not address this trade-off at all. Section 8's future work suggestions (combining search and revisions, cheap difficulty estimation, self-improvement loops) do not mention latency-aware allocation. A practitioner deploying these methods would need to solve a more complex optimization problem: maximizing accuracy subject to both a FLOPs budget and a latency constraint. The current compute-optimal policy solves only the FLOPs-constrained problem, and its recommendations (heavy sequential allocation for easy problems) may be exactly backwards for latency-constrained settings.

The Revision Model Has a Substantial Correct-to-Incorrect Reversion Rate That the Mitigation Strategies Only Partially Address

The assumption or constraint. The revision model is trained on trajectories consisting of 0–4 incorrect answers followed by a correct answer (Section 6.1). During training, the model never sees a sequence where a correct answer appears in context, because the training data always ends with the correct answer and all preceding context answers are incorrect. At test time, however, the model generates a chain of revisions, and some revisions will be correct—creating an out-of-distribution situation where the model's context contains correct answers that it was never trained to handle.

The consequence. The paper reports that "approximately 38% of correct answers get converted back to incorrect ones" during sequential revision (Section 6, near Figure 6 discussion). This means that for every 10 correct answers the revision model produces, nearly 4 of them get "revised" into wrong answers in the subsequent step. The practical effect is that the revision chain does not monotonically improve—accuracy oscillates as correct answers are produced and then undone. Figure 6 (left) shows that pass@1 at each step gradually rises but with fluctuations, and the improvement from step 1 (~18.2%) to the plateau at steps 15–20 (~24–25%) is only about 6–7 percentage points—substantial but well below what an ideal revision process that never regressed would achieve.

The paper mitigates this by not taking the final revision as the answer; instead, it uses majority voting or verifier-based selection across the entire chain of revisions (Section 6.1). This is a patch, not a solution: it means the system is generating many revisions it knows might be worse, relying on post-hoc selection to filter out the regressions. This wastes the generation budget and introduces noise into the revision chain—later revisions condition on incorrect regressions and may compound the error.

What evidence exists in the paper. The 38% figure is reported in Section 6.1. Figure 6 (left) shows the non-monotonic pass@1 trajectory. The paper explicitly acknowledges this as "a significant practical issue" and describes the selection-based mitigation. However, the paper does not ablate how much the reversion rate limits performance—e.g., by comparing against an oracle that stops the chain at the first correct answer. The backfiring ReST^EM experiment (Appendix K, Figure 16), where further optimization training caused performance to "substantially hurt" with sequential revisions, suggests the revision training is fragile in ways that are incompletely understood.

Mitigation status. The paper's mitigation (within-chain selection via majority or verifier) reduces the impact of reversions but does not eliminate it. A more principled solution—training the revision model on data that includes correct answers in context, teaching it to recognize when no revision is needed, or training a separate "stop" classifier—is not explored. The paper does not propose this as future work. The 38% reversion rate represents a fundamental limitation of the current revision training paradigm that makes sequential revisions inefficient: roughly 38% of the serial computation budget is spent undoing correct work.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not offer a new method or a solution. Its contribution is a diagnostic: it systematically characterizes when mechanistic anomaly detection works, when it fails, and how those successes and failures distribute across models, tasks, and detector configurations. This is less glamorous than a breakthrough but arguably more useful for a field that has been largely operating on theoretical arguments and isolated positive results.

The central reframing the paper forces is from "does MAD work?" to "under what conditions does MAD work, and how do we measure that?" Before this work, the MAD concept existed primarily as a high-level proposal (Christiano, 2022) with few systematic empirical tests. The paper demonstrates that the answer to "does it work" is not binary—it is a function of at least four interacting variables: the specific task, the base model, the feature extraction method, and (crucially) how the trusted and test distributions are constructed. This is not a pessimistic finding so much as a maturity injection into the conversation: it replaces speculation with concrete, replicable benchmarks and establishes lower bounds on what current methods can achieve.

The paper's most landscape-shifting finding is the model-dependence of detection difficulty. Llama 3.1 8B—the more capable model by most standard benchmarks—consistently yielded worse anomaly detection than Mistral 7B v0.1, across nearly every feature type and scoring rule tested (Table 2: best Llama online detector at 0.762 vs. best Mistral at 0.852). If this generalizes, it inverts the trend reported by Bai et al. (2022), where larger models made anomaly detection easier. The authors explicitly flag this as an open question: "It may be valuable to conduct additional experiments to assess whether there is a general trend that models trained on more tokens make anomaly detection harder, or if this is just a feature of this pair of models" (Section 5). If the trend is real, it has profound implications: as models scale, the internal traces of anomalous processing may become less rather than more accessible to simple statistical methods, precisely when oversight becomes most critical.

The paper also establishes linear separability as a necessary-but-not-sufficient condition for effective detection. Figure 5 shows that no detector achieves high AUROC when the normalized class separation between Alice and Bob activations is below 10^-2. This provides a diagnostic tool: before investing in complex detector architectures, measure whether the trusted and anomalous activation distributions are even linearly separable in the feature space. If they are not, no amount of methodological sophistication in the scoring rule will help—the signal simply is not present in that representation. SciQ's anomalous behavior (high separability but low AUROC, attributed to label shift) shows that separability is not sufficient, but its necessity is well-supported.

Perhaps the most sobering landscape finding is the high correlation across detection methods (Figure 3). Despite testing a wide variety of feature extraction approaches—raw activations, attribution patching, SAE features, normalizing flows, probe shift—the paper finds that most methods produce highly redundant signals. The correlation between activations/Mahalanobis and activations/LOF is r = 0.97; between activations/Mahalanobis and SAE/L0 is r = 0.99. The partial independence of attribution-based methods (r = 0.60 with activation-based methods) is encouraging but did not translate into improved performance when features were naively combined (Section 4.3). This means that methodological diversity does not guarantee signal diversity—a finding that should temper enthusiasm for "let's try many different feature extractors" as a research strategy. If the anomaly signal is weak or absent in the raw activations, it is unlikely to suddenly emerge from a more sophisticated transformation of those same activations.

The paper resolves a potential contradiction in how the field thinks about backdoor detection versus anomaly detection. Backdoor detection typically assumes the anomalous trigger is a rare, unusual input pattern (Gao et al., 2019; Guan et al., 2025). The paper shows that when the trigger is an ordinary linguistic feature (a name, indistinguishable from other names in surface properties), the detection problem becomes much harder—and that this difficulty is not well-captured by standard backdoor detection benchmarks. The multi-name modification (Figure 2) is the key evidence: switching from a single anomalous name to multiple names, none seen during detector training, substantially reduces detection performance across most datasets. This means that anomaly detection in the wild cannot rely on trigger rarity as a crutch, and methods evaluated only on rare-trigger settings may not transfer.

The paper also indirectly reframes the relationship between output-level behavior and internal detectability. Figure 4 shows that the quirkiness coefficient—a purely behavioral metric measuring how much the model's output distribution shifts between Alice and Bob contexts—is only weakly correlated with detection AUROC (Mistral: r = 0.150; Llama: r = 0.745). Modular Addition on Mistral achieves near-perfect detection (AUROC ~0.95) despite near-zero quirkiness; Sentiment on both models achieves poor detection despite high quirkiness. This decoupling means that you cannot infer internal detectability from behavioral measurements alone. A model might exhibit a large behavioral shift driven by a small, localized internal change that is hard to detect from population statistics, or a small behavioral shift driven by a distributed internal change that is easy to detect. This has implications for evaluation: testing MAD requires ground-truth internal state labels (Alice vs. Bob mode), not just behavioral agreement metrics.

In terms of research directions that become more versus less attractive, this paper makes a strong implicit case that:

  • More attractive: Understanding why arithmetic tasks are universally easy and non-arithmetic tasks are inconsistently hard. The arithmetic/non-arithmetic divide is the paper's most robust finding (Table 3), yet the paper never investigates its mechanism. If the structural properties that make arithmetic anomalies detectable can be identified and engineered into other tasks, this could generalize MAD.
  • More attractive: Developing detectors that combine activation-based and attribution-based signals in a non-naive way. The moderate correlation (r = 0.60) and the existence of cases where attribution succeeds but activations fail (Section 4.3) suggest untapped complementary information, even though naive concatenation did not help.
  • More attractive: Investigating the model-dependence trend—does detection genuinely get harder with more capable models, and if so, why? This is a single-pair observation (Mistral 7B vs. Llama 3.1 8B) and needs replication across model scales and families.
  • Less attractive: Iterating on scoring rules without changing feature representations. The near-perfect correlation between Mahalanobis and LOF (r = 0.97) suggests that within a given feature type, the choice of outlier detection algorithm matters little—the bottleneck is the features, not the scoring.
  • Less attractive: Using normalizing flows for density estimation on raw activations. The Laplace density score achieved only 0.581 on Llama (Table 2), substantially worse than Mahalanobis distance on the same flow-transformed features (0.737), suggesting that flows are useful as preprocessing but not as density models for anomaly detection in this setting.

Follow-Up Research This Work Enables

Systematic replication across model scales to test the model-dependence trend. The paper's finding that Llama 3.1 8B yields consistently worse anomaly detection than Mistral 7B v0.1 is based on exactly two models from different families, trained on different data, with different architectures, at similar but not identical parameter counts. This confound makes it impossible to isolate whether the detection gap is caused by model capability, model architecture, training data composition, or some interaction. A strong follow-up would test a single model family at multiple scales—for example, Llama 3.2 1B, 3B, 8B, and (if resources permit) 70B—all fine-tuned on the same quirky datasets with the same protocol, using the same detectors. If detection AUROC monotonically decreases with model scale within a single architecture family, that would strongly support the concerning trend that capability and detectability are inversely related. If the trend disappears within a family, it would suggest the Mistral/Llama gap is architectural or data-driven rather than scale-driven. The paper's quirkiness metric (Section 4.4) and linear separability analysis (Appendix A.1) provide ready-made diagnostic tools for such a study.

Investigating why arithmetic anomalies are universally detectable. The most robust positive result in the paper is that every arithmetic task—addition, subtraction, multiplication, modular addition, squaring—yields near-perfect detection (best AUROC of 1.000 or 0.999) on both models, across nearly all detector types (Table 3). The paper documents this but never explains it. A mechanistic follow-up would ask: what is structurally different about arithmetic reasoning in transformers that makes the Alice/Bob processing distinction so legible in activations? Hypotheses to test include: (a) arithmetic reasoning is more localized to specific attention heads or MLP neurons, making the mode shift concentrate in a few dimensions; (b) the faulty arithmetic heuristics (e.g., "add 1 to the first operand") produce systematic, directionally consistent activation shifts, whereas linguistic heuristics (e.g., sentiment based on word presence) produce diffuse, context-dependent shifts; (c) arithmetic tasks have more uniform input structure, reducing confounding variance. Distinguishing these hypotheses would involve measuring the sparsity of the Alice/Bob activation difference (is it concentrated in a few dimensions or distributed?), analyzing whether the difference vector is consistent across questions within a dataset, and testing whether controlled corruption of the arithmetic heuristic's systematicity (e.g., making it probabilistic rather than deterministic) reduces detectability. This would transform the descriptive finding into a mechanistic understanding that could guide detector design for non-arithmetic tasks.

Training difficulty-aware allocation policies for detector deployment. The paper treats all test examples identically—a single detector (defined by feature type, scoring rule, and layer) is evaluated on the entire test set. But the paper also shows that detection AUROC varies dramatically with properties of the example: some model-dataset-layer combinations achieve perfect detection while others are near chance. A natural extension is to develop a meta-detector that first estimates how "detectable" a given example is likely to be (based on, say, the model's output confidence, the example's distance from the trusted distribution in early layers, or a separately trained difficulty predictor), and then either applies the best available detector, requests human review, or flags the example as unreliable. This parallels the compute-optimal allocation framework from the pretraining-inference trade-off literature but applied to detection confidence rather than problem-solving strategy. The paper's existence of both high-AUROC and low-AUROC regimes within the same dataset and model (e.g., Population on Mistral has best AUROC 0.987 at some layer but likely much lower at others, per Figure 6) means there is a real optimization problem: which layer's detector output should be trusted for which example?

Adversarial evaluation of MAD robustness. The paper evaluates detectors on a fixed test set where the anomalous behavior was induced by standard fine-tuning. A critical stress test for MAD as a safety technique is whether detectors remain effective when the model is explicitly trained to evade them—for instance, by including an auxiliary loss term during fine-tuning that penalizes detectable differences between Alice-mode and Bob-mode activations. This is not a hypothetical concern: if MAD becomes a standard oversight tool, actors attempting to deploy models with hidden anomalous behaviors will have an incentive to make those behaviors mechanistically indistinguishable from normal operation. A strong follow-up would train "adversarially quirky" models that are optimized to (a) maintain the behavioral distinction (answering correctly in Alice mode, using the faulty heuristic in Bob mode) while (b) minimizing the AUROC of a pre-trained detector, or (c) minimizing the linear separability of Alice and Bob activations. If simple gradient-based adversaries can collapse detection AUROC to near-chance while preserving the behavioral distinction, that would establish a fundamental limitation of current MAD approaches and redirect research toward adversarially robust detection methods. If detectors prove surprisingly robust, that would strengthen the case for real-world deployment. The paper's existing infrastructure (quirky datasets, multiple detector types, standardized evaluation) provides a ready-made testbed.

Cross-task generalization of anomaly detectors. All detectors in the paper are trained and evaluated on the same quirky dataset—a detector trained on addition examples is tested on held-out addition examples. A practically crucial question is whether anomaly detectors capture a general signature of "faulty heuristic processing" that transfers across tasks, or whether they are task-specific. A strong follow-up would train detectors on one quirky dataset (e.g., addition, where detection is near-perfect) and evaluate them on other datasets (e.g., sentiment or NLI, where same-model detection is poor). If an addition-trained detector achieves above-chance AUROC on sentiment, that would suggest that faulty heuristic processing leaves a task-invariant signature—perhaps related to uncertainty, systematicity of errors, or reliance on shallow patterns. If cross-task transfer is near chance, it would suggest that anomaly signatures are task-specific, meaning MAD would require per-task trusted data and per-task detector training, substantially limiting its practical deployability. The paper's multi-dataset setup makes this experiment straightforward: the same model processes all datasets, so detectors trained on one dataset's activations can be directly applied to another's.

Measuring the cost-quality trade-off of different feature types for practical deployment. The paper evaluates detectors by AUROC but never by computational cost. For real deployment, this matters enormously: extracting residual stream activations requires one forward pass per example; attribution patching requires O(n_heads) forward passes; SAE encoding adds an additional forward pass through the SAE network; normalizing flows add a forward pass through the flow model. A strong follow-up would plot AUROC against FLOPs or wall-clock time for each feature extraction method, producing a Pareto frontier of detection quality versus inference overhead. This would directly inform practical decisions: is the modest additional signal from attribution features (which occasionally succeed where activations fail, Section 4.3) worth the ~n_heads × increase in compute? Are SAE features, which require a pre-trained SAE but add minimal per-example cost beyond the SAE forward pass, on the efficient frontier? The paper's existing results, if supplemented with FLOP counts for each method, could produce this analysis without additional experiments—though measuring wall-clock time would require implementation-specific benchmarking.

Dynamic, adaptive difficulty estimation for anomaly detection. The paper's offline setting (access to unlabeled test data) consistently outperforms the online setting (trusted data only)—compare quantum entropy's mean AUROC of 0.936 on Mistral (Table 1) against the best online detector at 0.852 (Table 2). This suggests that information about the test distribution is valuable. A natural extension is an adaptive detection protocol: start with a purely online detector (trained only on trusted data), score the first k test examples, use those scores to estimate properties of the test distribution (e.g., which directions show excess variance), and then update the detector to incorporate this information for subsequent examples. This blurs the online/offline distinction and would characterize how quickly detection performance improves as test examples accumulate—directly relevant for deployment scenarios where a model processes a stream of queries and anomalous examples are interleaved with normal ones. The paper's quantum entropy method, which explicitly uses test-data covariance to identify anomalous directions, provides a natural starting point for such an adaptive scheme.

Practical Applications and Downstream Use Cases

Filtering anomalous outputs in batch inference pipelines where ground-truth verification is expensive. The paper's strongest positive results—near-perfect detection on arithmetic tasks—suggest a concrete deployment scenario: a system that generates large volumes of structured outputs (mathematical calculations, code execution results, data extraction) where a subset may be produced by faulty heuristics rather than correct reasoning. An anomaly detector trained on a small set of verified-correct examples could flag potentially anomalous outputs for human review, reducing the number of outputs requiring manual inspection. On arithmetic tasks, where the best AUROC is 1.000 (Table 3), a detector could be tuned for high recall (flagging nearly all anomalous outputs while accepting some false positives) and still drastically reduce the human review burden—for instance, if the detector's false positive rate at 95% true positive rate is 5%, then 95% of faulty outputs are caught while only 5% of correct outputs are unnecessarily reviewed. On non-arithmetic tasks where performance is weaker (e.g., Mistral on NLI with best AUROC 0.957), a more conservative threshold would still catch a substantial fraction of anomalies. The key requirement is that the task has a clean correctness signal for training trusted detectors—exactly the setting the paper's quirky datasets simulate.

Monitoring for unintended heuristic adoption in continuously fine-tuned models. The paper's scenario—a model switches between correct and heuristic behavior based on a trigger (names)—is a simplified version of a real concern: as models are fine-tuned on ongoing user interactions, they may acquire unintended heuristics that are triggered by subtle features of the input distribution. For instance, a customer support model fine-tuned on user feedback might learn that responses containing certain polite phrasings receive higher ratings regardless of factual accuracy, and gradually adopt a "be polite even if wrong" heuristic. A MAD system trained on a trusted pre-deployment dataset (where the model's behavior is known to be correct) could be run periodically on samples from the production distribution to detect whether the model's internal processing has shifted in ways that suggest heuristic adoption. The paper's finding that anomaly detection works even when the behavioral shift is small (Modular Addition on Mistral: AUROC ~0.95 with near-zero quirkiness, Figure 4) is directly relevant here—the detector might catch the onset of heuristic behavior before it becomes behaviorally obvious. The main limitation, per the paper's results, is that detection reliability varies across tasks, so this would be most viable for structured, verifiable tasks rather than open-ended generation.

Auditing models for backdoor-like behaviors that use ordinary rather than rare triggers. The paper's multi-name modification (Figure 2) distinguishes it from standard backdoor detection by showing that detection is possible even when the anomalous trigger is an ordinary linguistic feature (a name from a set of 16), not a rare codeword or unusual pattern. This maps onto a real concern: a model might be fine-tuned to behave differently when interacting with users from particular demographic groups (inferred from names, writing style, or stated preferences), and this differential treatment might be undesirable even if it is not intentionally malicious. A MAD system could be trained on trusted interactions (e.g., interactions with a diverse user sample that were reviewed and confirmed to be unbiased) and used to flag interactions where the model's internal processing looks anomalous—potentially indicating differential treatment triggered by user features the supervisor did not anticipate. The paper's finding that detection is harder when multiple trigger variants exist (Figure 2) is realistic and suggests that such a system would need careful calibration, but the near-perfect results on some tasks show it is not hopeless.

When to Prefer This Method

The paper does not position MAD against a specific named alternative with a clear decision rule. It evaluates many MAD variants against each other, but the overarching choice is "use mechanistic anomaly detection" versus "do not" (or use non-mechanistic alternatives like output monitoring). The paper provides evidence about when MAD works rather than when to prefer MAD over method X. The findings suggest the following practical considerations, but the paper does not frame them as a formal decision rule:

  • Detection is reliably strong (AUROC near 1.0) on structured, rule-based tasks like arithmetic, where the faulty heuristic produces systematic activation differences. For such tasks, even simple detectors (Mahalanobis distance on residual stream activations) suffice.
  • Detection is unreliable on more complex linguistic tasks, and the paper provides no way to predict in advance which tasks will be easy or hard without running the experiment. This makes MAD difficult to deploy proactively for novel tasks.
  • Detection is consistently worse on Llama 3.1 8B than Mistral 7B v0.1, suggesting that more capable models may be harder to monitor with current MAD techniques. If this trend holds, MAD may become less viable as models improve, absent methodological advances.
  • Within a task, detection performance varies across layers (Figures 6 and 7), so layer selection matters. Deeper layers tend to perform better, but the optimal layer is task- and model-specific.
  • Different detection methods produce highly correlated results (Figure 3), so methodological diversity is not a substitute for a strong underlying signal. If one activation-based method fails, others probably will too. Attribution-based methods sometimes succeed where activation-based methods fail, but not reliably enough to be a general solution.