Building Production-Ready Probes For Gemini

🎯 Pitch

Long-context prompts completely destroy standard activation probes for detecting cyberattacks—our architectures bring the failure rate down from ~88% to under 20% without blowing up the inference budget. One key trick: replacing softmax attention with a hard max over tokens prevents the harmful signal from being averaged away across thousands of benign lines of code.

1. Executive Summary

This paper studies how to build production-ready activation probes that detect cyber-offensive prompts in deployed instances of Gemini 2.5 Flash, addressing the key failure mode that existing probe architectures break under long-context distribution shifts. The authors introduce MultiMax probes (replacing softmax attention with hard max across token positions to prevent signal dilution in long contexts) and a Max of Rolling Means Attention Probe (combining attention-weighted averaging within sliding windows with max-pooling), and demonstrate that these architectures—along with probes discovered by AlphaEvolve-driven automated architecture search—achieve comparable accuracy to Gemini 2.5 Flash and Pro classifiers at over 10,000× lower inference cost. Pairing probes with a prompted LLM in a cascading classifier (deferring uncertain cases to the expensive model) yields lower error than the LLM alone while using the LLM on less than 10% of queries, establishing that test-time compute via better architecture design can close the long-context generalization gap without the 22× training cost increase of direct long-context training—though the paper finds that no method meaningfully reduces the success rate of adaptive adversarial attacks, with pre-existing jailbreaks achieving ≥1% attack success across all defenses.

2. Context and Motivation

The Core Problem: Production Deployment of Probes Meets Long-Context Distribution Shifts

The fundamental problem this paper addresses is deceptively specific but has enormous practical consequences: activation probes trained to detect cyber-offensive prompts work well on short-context inputs but catastrophically fail when deployed in production environments where users provide long-context inputs. This is not a minor degradation—the paper shows that a standard attention probe trained on short-context data achieves an FNR of 87.88% on long-context attacks (Table 3, Attn row, LC[A] column), meaning it misses nearly 9 out of 10 harmful queries when those queries are embedded in long codebases (as users naturally do when asking coding-related questions of a frontier model).

The paper is fundamentally an engineering-to-deployment case study. It does not introduce a single new algorithm and claim superiority on a benchmark. Instead, it documents a real production deployment—building misuse mitigation probes for user-facing instances of Gemini—and the technical challenges that emerged when moving from the controlled research setting to the messy reality of production traffic. This framing matters because it explains the paper's unusual structure: architectures are evaluated against a heterogeneous mix of distribution shifts (multi-turn conversations, long contexts, jailbreaks, automated red teaming) rather than a clean single-metric benchmark. The takeaway is not "our architecture is 3% better" but rather "here is what actually works in production and what remains broken."

Why This Problem Matters: Economic, Security, and Practical Dimensions

The importance of this problem operates on multiple levels, from global security policy down to per-query infrastructure costs.

First, the security imperative is real and escalating. The paper opens by establishing that frontier models already provide meaningful uplift to malicious actors—citing both Google's own finding that "Gemini 2.5 Deep Think... provides uplift in some stages of some harm journeys" for CBRN (Chemical, Biological, Radiological, Nuclear) weapons, and Anthropic's conclusion that they "were unable to rule out the need for ASL-3 safeguards" for CBRN capabilities (Section 1). This is not speculative risk; Anthropic has documented the first reported AI-orchestrated cyber espionage campaign (Anthropic, 2025a, cited in the paper). Misuse mitigations are not optional research projects—they are deployment requirements for frontier model developers subject to Responsible Scaling Policies and Frontier Safety Frameworks.

Crucially, the paper argues that training models to simply refuse harmful queries is insufficient (Section 1):

"One might hope that a sufficient mitigation would be to train LLMs to reject harmful queries, but unfortunately current training techniques are not robust enough (Nasr et al., 2025)."

This motivates the focus on monitoring—runtime detection systems separate from the model's own safety training—as a necessary layer of defense. The paper studies input monitoring specifically (detecting harmful prompts before the model responds), though notes that output monitoring (detecting harmful model responses during generation) is an important future direction.

Second, the cyber domain presents uniquely difficult false-positive/false-negative tensions. As the paper explains (Section 1):

"Cyber-offensive capabilities are a particularly worthwhile domain for studying defenses because harmful prompts are very similar to common and valuable defensive cybersecurity requests, and also somewhat similar to even more common coding requests. Therefore, it is particularly difficult to avoid over-triggering while also preventing harm in the cyber domain."

This is a critical design constraint. A probe that fires on every SQL injection query would also block legitimate penetration testers, security researchers, and developers debugging vulnerable code—the very users who provide much of the value for a frontier coding model. The "hard negatives" dataset (Table 2) is explicitly constructed from prompts like "Explain how the 2017 Equifax breach exploited the Apache Struts vulnerability" that are superficially similar to attack queries but represent legitimate educational or defensive use. Any production system must thread the needle between catching actual attacks and not obstructing beneficial use, and the cyber domain makes this needle eye particularly narrow.

Third, the cost economics of monitoring are a binding constraint in production. Running a separate full-scale LLM to classify every user input roughly doubles inference costs (Section 1):

"Monitoring all interactions with another LLM can be extremely expensive, potentially doubling the cost if using a comparably capable model."

This is the motivation for using activation probes in the first place: they are computationally cheap because the monitored model has already computed the activations during its forward pass. The probe only needs to process these pre-existing hidden states through a small neural network (typically a few linear layers or attention heads), adding negligible cost relative to the model's forward pass. The paper's Figure 1 shows probes operating at roughly 10,000× lower inference cost than Gemini 2.5 Flash while achieving comparable or better accuracy.

However, the cost advantage creates a tension with robustness. The paper finds that simply training probes on long-context data—the naive solution to the distribution shift—incurs a 22× increase in training cost due to memory bandwidth constraints (Section 5.1.2, Appendix I). If long-context training were the only path to acceptable accuracy, the cost advantage of probes would be partially eroded. The paper's central contribution is showing that architectural innovations can achieve long-context generalization without long-context training, preserving the economic case for probes.

Prior Approaches and Where They Fall Short

The paper positions itself against several existing lines of work, each with specific limitations that the production deployment exposes.

Probe architectures from prior work are brittle under distribution shifts. The paper treats the following as baselines:

Linear probes (Alain and Bengio, 2016; Zou et al., 2024): A learned linear direction $\mathbf{w} \in \mathbb{R}^d$ is applied to mean-pooled activations: $f_{\text{Linear}}(S_i) = \frac{1}{n_i}\sum_{j=1}^{n_i} \mathbf{w}^T \mathbf{x}_{i,j}$ . These are extremely simple and computationally efficient, but they are not designed to handle variations in sequence length. The paper's results show linear probes achieve 99.13% FNR on long-context attacks (Table 3)—essentially complete failure. The issue is that mean-pooling dilutes the signal from a harmful query when it is a small fraction of a massive codebase: a single malicious instruction embedded in 900,000 tokens of benign code contributes only ~1/900,000 of the pooled representation.
Exponential Moving Average (EMA) probes (Cunningham, Peng, et al., 2025): These attempt to address the length problem by post-processing linear probe outputs: at each token position, compute $EMA_j = \alpha f_{\text{linear}}(\mathbf{x}_{i,j}) + (1-\alpha)EMA_{j-1}$ with $\alpha = 0.5$ , then take the maximum over all positions. EMA probes improve on linear probes (e.g., 13.55% FPR on long-context random traffic vs. 0.5% for linear mean, Table 3), but the paper shows they still struggle compared to more sophisticated architectures, particularly for false positive control on long benign contexts.
MLP probes (Zou et al., 2024): These replace the linear transformation with a multi-layer perceptron before pooling: $f^M_{\text{MLP}}(S_i) = \frac{1}{n_i}\sum_{j=1}^{n_i} \text{MLP}^M(\mathbf{x}_{i,j})$ . While more expressive than linear probes, the paper does not report detailed MLP probe results in the main body, suggesting they were either evaluated and found insufficient, or that their limitations mirror those of mean-pooled methods more broadly—the pooling step remains the bottleneck.
Attention probes (Kantamneni et al., 2025; Shabalin and Belrose, 2025): These learn query and value vectors that compute a softmax-weighted average of per-token values: $f_{\text{Attn}}(S_i) = \sum_h \frac{\sum_j \exp(\mathbf{q}_h^T \mathbf{y}_{i,j}) \cdot (\mathbf{v}_h^T \mathbf{y}_{i,j})}{\sum_j \exp(\mathbf{q}_h^T \mathbf{y}_{i,j})}$ , where $\mathbf{y}_{i,j}$ are MLP-transformed activations. Attention probes are the strongest prior baseline: Table 3 shows the vanilla attention probe achieves 0.0% FPR on long-context random traffic (no false positives on benign long inputs) and 0.3% FPR on short overtriggering. However, this probe catastrophically fails on long-context attacks with an FNR of 87.88%. The softmax weighting mechanism, which works beautifully for finding relevant tokens in short contexts, becomes a liability in long contexts because it distributes attention weight across thousands of tokens. Even if the probe correctly identifies the harmful tokens, the softmax normalization means many irrelevant tokens receive non-trivial weight, diluting the signal.

Verification of the attention probe's failure mode: The paper does not explicitly diagnose why attention probes fail on long-context attacks, but the mechanism is implicit in the design of MultiMax. In a sequence of length $n$ , the softmax over $n$ tokens produces weights that sum to 1. If the harmful content occupies $k \ll n$ tokens, the maximum possible total weight concentrated on harmful tokens is $k \cdot \frac{e^{s_{\text{harm}}}}{e^{s_{\text{harm}}} + (n-k)e^{s_{\text{benign}}}}$ . For large $n$ and typical score differences, this weight approaches $k/n$ —the harmful signal gets washed out proportionally to context length. The MultiMax architecture (Section 3.2.1) addresses this by replacing the weighted average with a hard maximum: $f_{\text{MultiMax}}(S_i) = \sum_h \max_{j \in [n_i]} (\mathbf{v}_h^T \mathbf{y}_{i,j})$ . This ensures that a single highly-activated token can dominate the output regardless of context length, which is precisely the property needed when malicious content appears in a small fraction of the input.

LLM-based classifiers are effective but expensive. The paper evaluates Gemini 2.5 Flash, Gemini 2.5 Pro, and Gemini 2.5 Flash Lite as prompted classifiers. Flash achieves 0.3% FPR on short overtriggering, 8.58% FNR on short attacks, and 6.06% FNR on long-context attacks (Table 3)—strong performance across the board. Pro achieves even lower FPR (0.2% on short overtriggering) but higher FNR on attacks (16.01% on short attacks vs. 8.58% for Flash). This pattern—Pro is more conservative, Flash is more aggressive—illustrates the fundamental tradeoff that makes cascading classifiers (Section 5.2) valuable.

The limitation of LLM-based classifiers is not accuracy but cost. As Figure 1 shows, Flash's inference cost is approximately 10,000× higher than a probe's, and Pro's cost is another ~10× beyond that (even ignoring thinking tokens, which the paper notes would make the gap larger). For a system processing millions of queries per day, this cost difference determines whether monitoring is economically viable at all.

Long-context training is a brute-force solution with prohibitive costs. One natural response to the long-context failure mode is: "just train on long-context data." The paper investigates this with the "Attn (LC Trained)" baseline (Section 5.1.2), an attention probe directly trained on the long-context dataset. This probe achieves competitive results (2.38% test error, Table 3), demonstrating that attention probes have sufficient capacity to handle long contexts if given appropriate training data. But this success comes at a steep price:

"Even with large-scale accelerator clusters, the full dataset of long-context activations (each up to 1M tokens) cannot fit into TPU HBM. This necessitates implementing complex data loading pipelines to swap data between host memory and device memory... In our experiments, these overheads resulted in a 22× increase in training cost compared to short-context training."

This cost increase is not merely an implementation inconvenience—it fundamentally undermines the economic argument for probes. If training a probe requires 22× more compute, the total cost of ownership shifts, and the case for probes over LLM classifiers weakens. More critically for the paper's AlphaEvolve experiments (Section 5.3), the 22× training cost would make the ~2500-architecture evolutionary search computationally infeasible. The architectural innovations (MultiMax, Max of Rolling Means) are explicitly designed to achieve long-context generalization at short-context training costs, which is what makes them production-viable.

Cascading classifiers exist in prior work but without optimal cost-accuracy analysis. Cunningham, Peng, et al. (2025) previously proposed combining cheap probes with expensive LLMs in a cascading arrangement, and the paper acknowledges this precedent. However, the paper's contribution on cascading (Section 5.2.2 / Appendix E) is a more rigorous treatment of the optimal deferral policy: rather than a heuristic symmetric confidence band around a decision threshold, the authors derive a threshold-randomization-optimal policy using the Minkowski sum of left and right error curves, computing the Pareto frontier in $O(N \log N)$ time. This is a methodological rather than conceptual advance, but it matters for production because the optimal operating point (the "Selected Probe + 8% Flash" in Figure 1) achieves lower error than Flash alone while using Flash on only 8% of queries—a genuinely surprising result that would be difficult to discover with ad-hoc threshold tuning.

Where Existing Defenses Fail: The Inherent Difficulty of Adversarial Robustness

A crucial motivation running through the paper is the acknowledgment of fundamental limits in adversarial robustness. Despite all the architectural innovations, training strategies, and cascading designs, the paper finds:

"In this cyber-offensive query domain, jailbreak success rates are extremely high... on the pre-existing jailbreaks dataset, all techniques have a vulnerability on at least 1% of queries. This means that we are a long way from adversarial robustness in this domain." (Section 5.1)

This is not presented as a failure of the paper's methods but as a statement about the inherent difficulty of the problem. The paper cites Nasr et al. (2025), which argues that adaptive attacks—where adversaries have knowledge of the defense—can bypass most existing guardrails. The paper's own automated red teaming experiments reinforce this: they find that multi-step iterative jailbreaking (modeled on PAIR; Chao et al., 2025) does not outperform simply repeating a single-turn attack multiple times (Appendix H, Figure 12). In other words, even sophisticated attack search doesn't improve over brute-force retries, suggesting the defense surface is already close to random for determined adversaries.

This finding shapes the paper's framing: probes are not a solution to adversarial attacks but a cost-effective layer in a defense-in-depth strategy. The paper explicitly states that their techniques "do not significantly reduce the success rate of adaptive adversarial attacks" (Section 1) and that their work "can supplement other empirical studies and user-level strategies" (citing Cunningham et al., 2026; Shah et al., 2025). This is an unusually candid admission of limitations that distinguishes the paper from work that claims to "solve" jailbreak detection.

How the Paper Positions Itself

The paper positions itself at the intersection of three research trajectories:

Activation probing as a safety technique (not just a diagnostic tool). Probing originated as an interpretability method—"what information is linearly decodable where from a frozen model's internal states?" (Alain and Bengio, 2016). The paper is part of a line of work (Zou et al., 2023; Cunningham et al., 2025; McKenzie et al., 2025) that repurposes probes from understanding models to monitoring them in deployment. The transition from diagnostic to production tool introduces entirely new requirements: probes must be robust to distribution shifts, computationally efficient at scale, and resistant to adversarial optimization. The paper's architectural innovations (MultiMax, Max of Rolling Means) are motivated by these production requirements, not by a desire to better decode model internals.
Bridging the gap between research benchmarks and production reality. The paper is unusually explicit about the mismatch between research settings and deployment conditions. Academic probing papers typically evaluate on fixed test sets with the same distribution as training data. The paper intentionally evaluates on distributions absent from training: long-context data (not in training at all), multi-turn conversations (in training but with different construction), jailbreaks (entirely held-out), and automated red teaming (generated by an attacker model). The finding that architecture rankings change across these distributions (Section 5.1: "no single method dominates across all distribution shifts") is a warning against over-interpreting single-benchmark results.
Automated safety research as a meta-methodology. The AlphaEvolve experiments (Section 5.3) represent a bet that architecture design for safety applications can itself be automated. Rather than hand-designing probe architectures through intuition, the authors let an evolutionary algorithm explore ~2500 architectures, discovering both the early MultiMax variant (a simple max-pooling + learned head weighting) and the final complex architecture (with gated projections, bipolar pooling, and orthogonality regularization). The paper frames this as a proof of concept for "automating some AI safety research," suggesting that as model capabilities grow, safety mechanisms will need to be discovered at scale rather than designed manually.

A subtle but important aspect of the paper's positioning: it does not claim to have solved the problem. Section 7 (Conclusion) lists multiple limitations: single-domain focus (cyber only), input-only monitoring (no output detection), large error bars, and no combination with multi-layer probing techniques from concurrent work (Cunningham et al., 2026). The paper's contribution is more nuanced: it provides a production-informed methodology for building probes that work in practice, a catalog of architectural innovations that address specific distribution shifts, and a realistic assessment of what probes can and cannot do against adversarial attacks. This is the paper's implicit answer to "why not just use an LLM classifier?"—probes offer a different cost-accuracy frontier, not a strictly better point on the same frontier, and the cascading approach shows that combining both yields the best of each.

3. Technical Approach

This is primarily an empirical engineering paper whose core idea is that activation probes can be made robust to production distribution shifts—particularly long-context inputs—through architectural innovations that change how per-token activations are aggregated, rather than through expensive retraining on long-context data or through reliance on more expensive LLM classifiers.

3.1 Reader Orientation

The paper builds a deployment-time monitoring system that classifies user prompts as either benign or cyber-offensive by processing the internal hidden states of Gemini 2.5 Flash during its normal forward pass. The problem it solves is that existing probe architectures break under the distribution shift from short-context training data (a few thousand tokens) to the long-context inputs (up to 1 million tokens) common in production. The solution takes the form of new aggregation mechanisms—specifically replacing softmax-weighted averaging with hard max-pooling over token positions and combining this with sliding-window attention—that prevent the harmful signal from being diluted when malicious content occupies a tiny fraction of a massive context.

3.2 Big-Picture Architecture (Diagram in Words)

The monitoring system has five major components, organized as a pipeline with branching evaluation paths:

The Deployed Language Model (Gemini 2.5 Flash) — receives user prompts and performs its standard forward pass. During this pass, at a fixed middle layer L, the model produces hidden state vectors x_{i,j} ∈ ℝ^d for each token position j. These activations are the raw input to the monitoring system. Crucially, generating these activations incurs no additional cost beyond the model's normal inference—they are a byproduct of the forward pass.
Activation Transformation (MLP preprocessing) — each per-token activation vector x_{i,j} is passed through a small two-layer MLP (width 100, ReLU activations) to produce transformed representations y_{i,j} ∈ ℝ^{d'}. This component provides non-linear feature extraction before the aggregation step, and its outputs are shared across all subsequent probe architectures.
Aggregation Mechanism (the core innovation) — the transformed per-token representations are combined into a single scalar score for each "head" of the probe. Different architectures use different aggregation functions:
- Attention probes use softmax-weighted averaging: the attention weight for each token is proportional to exp(q_h^T y_{i,j}), and the output is the weighted average of value scores v_h^T y_{i,j}.
- MultiMax probes use hard maximum: the output for each head is simply max_j(v_h^T y_{i,j}), selecting the single highest-scoring token.
- Max of Rolling Means Attention probes apply attention-weighted averaging within sliding windows of width w = 10, then take the maximum across all window positions.
- AlphaEvolve-discovered probes use more complex mechanisms including gated projections with Softplus activations and "bipolar pooling" (taking both the max and the negated min across positions).
Head Aggregation and Thresholding — the per-head scalar outputs are summed (or, in some AlphaEvolve variants, linearly combined with learned weights) to produce a single scalar score for the entire sequence. This score is passed through a sigmoid to produce a probability, which is then thresholded: sequences above threshold τ are classified as harmful, below as benign.
Cascading Classifier (optional second stage) — when enabled, sequences whose probe score falls in a "uncertainty band" [τ_1, τ_2] are deferred to a prompted LLM classifier (Gemini 2.5 Flash, Pro, or Flash Lite) for a second opinion. Sequences with scores below τ_1 are classified as benign; above τ_2 as harmful. This cascade achieves higher accuracy than either the probe or the LLM alone while invoking the expensive LLM on only a small fraction of queries.

Information flows as follows: a user prompt enters Gemini 2.5 Flash → the model computes its standard forward pass → activations at layer L are harvested → the MLP transforms them per-token → the aggregation mechanism combines them into per-head scores → head scores are summed into a final probe score → either the probe directly classifies (if score is extreme) or the prompt is routed to the LLM classifier (if score is ambiguous) → a final harm/benign decision is produced.

3.3 Roadmap for the Deep Dive

First, the formal problem setup and training objective (Equation 1), which defines what probes are, how they are trained, and the loss function that drives learning—this establishes the common foundation shared by all architectures.
Second, the baseline probe architectures from prior work (linear, EMA, MLP, attention), since understanding their failure modes on long contexts is necessary to motivate the novel architectures and to interpret the comparative results.
Third, the MultiMax architecture—the simplest and most conceptually important innovation—because it directly addresses the softmax dilution problem and forms the basis for more complex variants.
Fourth, the Max of Rolling Means Attention Probe and its variants, since this represents the culmination of hand-designed architectures combining attention-based local weighting with max-pooling across windows.
Fifth, the AlphaEvolve-discovered architectures, as they represent automated exploration of the architecture space and produced the best-performing probes (bootstrap probability > 0.95 vs. baselines).
Sixth, the cascading classifier framework, including the threshold-randomization-optimal policy derivation (Appendix E), because this shows how probes and LLMs combine to achieve Pareto-optimal cost-accuracy tradeoffs.
Seventh, key training and infrastructure details (seeds, long-context training costs, attention probe inference efficiency), since these practical considerations determine whether the approach is production-viable.

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 Problem Setup and Training Objective

The paper frames probe training as a standard binary classification problem over sequences of hidden states, with the objective being cross-entropy minimization.

Input space: A deployed language model receives prompts as token sequences S_i = (t_{i,1}, ..., t_{i,n_i}) where n_i is the number of tokens in sequence i. During the model's forward pass, at a fixed layer L (chosen to be the layer halfway through the model), the hidden state for token j in sequence i is harvested as x_{i,j} ∈ ℝ^d, where d is the model's hidden dimension. The full matrix of hidden states for sequence i is X_i ∈ ℝ^{d × n_i}.

Training data: The training set consists of pairs {(S_i, ℓ_i)} where ℓ_i ∈ {0, 1} is a binary label (0 = benign, 1 = harmful/cyber-offensive). The training data composition is explicitly specified in Table 7: 1,928 short-context attack examples, 1,206 short-context overtriggering examples (benign prompts from production traffic), and 41 multi-turn attack examples—totaling 3,175 training sequences exclusively from short-context distributions. Long-context data is deliberately excluded from training to demonstrate generalization rather than memorization.

Probe definition: A probe is a function f_p: ℝ^{d × n_i} → ℝ that maps the matrix of per-token activations to a single scalar score (the logit for the binary classification). Different architectures (Section 3) implement this function differently—via linear projections, MLPs, attention mechanisms, or max-pooling operations—but they all share the same interface: take in all activations from a sequence, output a single scalar.

Training objective:

The probe parameters (all learnable weights of f_p) and a bias term b ∈ ℝ are optimized to minimize binary cross-entropy loss over the training set:

$\text{arg min}_{f_p} \sum_i \text{BCE}(\sigma(f_p(X_i) + b), ℓ_i)$

where σ(·) is the sigmoid function σ(z) = 1/(1 + e^{-z}) mapping logits to probabilities, and BCE is the binary cross-entropy: BCE(p, y) = -(y log(p) + (1-y) log(1-p)). The summation runs over all training sequences i.

What it computes: For each training sequence, the probe produces a logit f_p(X_i), adds the bias b, passes the result through sigmoid to get a predicted probability of harmfulness, and computes the cross-entropy between this prediction and the ground-truth label ℓ_i. The optimizer (AdamW, described below) updates all parameters to drive predicted probabilities toward 0 for benign sequences and toward 1 for harmful sequences.

Why this form: Binary cross-entropy is the standard maximum-likelihood objective for binary classification with probabilistic outputs. It has the property that the gradient is proportional to the prediction error σ(f_p(X_i) + b) - ℓ_i, meaning parameters are updated most aggressively when the model is confidently wrong. The sigmoid ensures outputs are calibrated as probabilities in [0, 1]. The bias term b allows the model to shift the decision boundary independently of the probe's internal computation, which matters because different architectures may produce logits on different scales.

Training hyperparameters: The paper uses full-batch gradient descent (all 3,175 training sequences in a single batch) with the AdamW optimizer for 1,000 epochs. Specific settings: learning rate 1 × 10^{-4}, weight decay 3 × 10^{-3}, Adam betas β_1 = 0.9, β_2 = 0.999. The full-batch approach is possible because activations are pre-computed and stored on disk (Appendix I), making it efficient to load the entire training set at once. The paper notes that "we only ever observed zero or mild overfitting on validation sets despite many epochs," which is consistent with the small model size relative to the activation dimensionality.

Inference and decision rule: At deployment time, for a new sequence S_i^*, the probe score is computed and thresholded:

$f_p(S_i^*, τ) = \mathbb{1}\{\sigma(f_p(X_i^*) + b) > τ\}$

where \mathbb{1} is the indicator function (outputs 1 if the condition is true, 0 otherwise). The threshold τ is chosen on a separate validation set by minimizing a weighted combination of false positive rates and false negative rates (Equation 12, discussed in Section 4.1 of the paper). The probe produces a binary classification decision, just as a prompted LLM classifier would.

Layer selection: Activations are harvested from "the layer halfway through the model" (Appendix C). This choice is based on prior work showing that middle layers contain rich, task-relevant representations without being overly specialized to next-token prediction (early layers) or overly abstract (late layers). The paper does not experiment with multi-layer probes (as in concurrent work by Cunningham, Wei, et al., 2026), noting this as a limitation in Section 7.

3.4.2 Baseline Probe Architectures (Prior Work)

The paper evaluates four baseline architectures from prior work. Understanding their design and failure modes is essential because the novel architectures are explicit responses to these failure modes.

Linear Probes (Alain and Bengio, 2016):

Linear probes are the simplest architecture and serve as the lower bound for probe expressivity. Given a learned weight vector w ∈ ℝ^d, the probe computes:

$f_{\text{Linear}}(S_i) = \frac{1}{n_i} \sum_{j=1}^{n_i} \mathbf{w}^T \mathbf{x}_{i,j}$

where n_i is the number of tokens, x_{i,j} is the raw hidden state at position j, and w^T x_{i,j} is the dot product producing a scalar "harmfulness score" for that token.

What it computes: For each token, it projects the d-dimensional hidden state onto a single learned direction w (a linear classifier applied per-token). Then it averages these per-token scores across all positions. The result is a single scalar representing the average "harmfulness" of the entire sequence.

Why this form: Linear probes are the standard interpretability tool because they are transparent—the weight vector w can be examined to understand what direction in activation space corresponds to the concept. They are also extremely efficient: the dot product w^T x_{i,j} is O(d) per token, and the mean pooling is O(n_i).

Failure mode on long contexts: The mean-pooling operation is the critical weakness. In a sequence of length n_i, each token contributes exactly 1/n_i weight to the final score. If a harmful instruction occupies k tokens in a 900,000-token codebase, its contribution is k/900,000 of the final score—arbitrarily small. Even if the linear probe perfectly identifies those k tokens as maximally harmful (producing a very high per-token score), their impact is divided by the context length. The paper's results confirm this: linear probes achieve 99.13% FNR on long-context attacks (Table 3), meaning they miss virtually every harmful query embedded in long contexts.

Table 3 also shows the "Linear Probe Rolling Exp" variant, which uses a different aggregation strategy (likely rolling exponential averaging similar to the EMA probe mechanism but applied to linear outputs). This variant improves long-context attack FNR from 99.13% to 1.73%, demonstrating that the aggregation mechanism—not the linear projection—is the primary bottleneck. However, the Rolling Exp variant suffers from dramatically worse long-context false positive rates (13.55% vs. 0.5% for mean pooling), illustrating the fundamental tradeoff between sensitivity (catching attacks) and specificity (not overtriggering on benign long inputs) that more sophisticated architectures must navigate.

Exponential Moving Average (EMA) Probes (Cunningham, Peng, et al., 2025):

EMA probes are a post-processing technique applied to linear probe outputs, designed to improve long-context generalization without architectural changes to the probe itself. The key idea is to detect when the per-token harmfulness score spikes, using an exponential moving average that decays past observations exponentially:

At inference time, given a trained linear probe f_linear(·) (the dot product-plus-bias per token), compute recursively:

$\text{EMA}_0 = 0$ $\text{EMA}_j = \alpha \cdot f_{\text{linear}}(\mathbf{x}_{i,j}) + (1 - \alpha) \cdot \text{EMA}_{j-1}$

where α = 0.5 (following Cunningham, Peng, et al., 2025) is the smoothing factor controlling how quickly past scores decay: at α = 0.5, each new token contributes 50% of the current EMA value, meaning the influence of a token halves after roughly one additional token (since (1-0.5)^1 = 0.5, so the weight decays geometrically). The final probe score is:

$f_{\text{EMA}}(S_i) = \max_{j \in [n_i]} \text{EMA}_j$

What it computes: At each token position j, the EMA combines the current token's harmfulness score with an exponentially decaying memory of past scores. The maximum across all positions is then taken. This means the probe fires if there is any point in the sequence where recent tokens are consistently scored as harmful—a sustained malicious segment will push the EMA high, and the max preserves this signal.

Why this form: The EMA acts as a low-pass filter that smooths the per-token scores, making the probe less sensitive to single-token false positives (e.g., a token that happens to have a high dot product with w due to noise) while still detecting sustained patterns. The max operation ensures the signal is not averaged away over the full sequence length (as mean pooling would). The choice of α = 0.5 means the half-life of a token's contribution is approximately 1 token—the EMA responds quickly to changes but also decays quickly once harmful content ends.

Failure mode: EMA probes improve over mean-pooled linear probes on long-context attacks (1.73% FNR for Rolling Exp, which uses a similar mechanism, vs. 99.13% for mean), but they remain limited because the underlying per-token scoring function f_linear(x_{i,j}) is still a simple linear projection. A linear classifier may not have sufficient capacity to distinguish harmful from benign tokens in the abstract hidden state space, particularly when those tokens are embedded in complex technical contexts (e.g., a SQL injection tutorial vs. actual malicious SQL injection). The max operation also makes the probe sensitive to outliers: a single false-positive high-scoring token can cause overtriggering.

MLP Probes (Zou, Phan, J. Wang, et al., 2024):

MLP probes replace the linear projection with a multi-layer perceptron applied to each token independently before pooling. Formally, an M-layer MLP (with ReLU activations) transforms each token activation:

$\text{MLP}^M(\mathbf{X}) = A_1 \cdot \text{ReLU}(A_2 \cdot \text{ReLU}(... \text{ReLU}(A_M \cdot \mathbf{X}) ...))$

where A_1, ..., A_M are learned weight matrices with dimensions chosen such that the final output is a scalar. The mean-pooled MLP probe is then:

$f_{\text{MLP}}^M(S_i) = \frac{1}{n_i} \sum_{j=1}^{n_i} \text{MLP}^M(\mathbf{x}_{i,j})$

The paper uses two-layer MLPs of width 100 as the preprocessing step for all attention-based architectures (Appendix C), so the per-token transformation y_{i,j} = \text{MLP}^2(x_{i,j}) is standard across methods. However, pure mean-pooled MLP probes share the same fundamental weakness as linear probes: the mean over positions dilutes the signal in long contexts. The paper does not report detailed MLP probe results in the main body, indicating that even with non-linear per-token processing, mean pooling remains the bottleneck.

Attention Probes (Kantamneni et al., 2025; Shabalin and Belrose, 2025):

Attention probes are the strongest baseline architecture and the direct predecessor to the paper's novel architectures. The key innovation is that aggregation weights are learned functions of the activations themselves, rather than fixed constants (like 1/n_i for mean pooling). Formally, the activations are first passed through a shared MLP:

$\mathbf{y}_{i,j} = \text{MLP}^M(\mathbf{x}_{i,j})$

where M = 2 (two ReLU layers, width 100) in all experiments unless otherwise specified. Then, for an H-headed attention probe, each head h ∈ {1, ..., H} has learned query vector q_h ∈ ℝ^{d'} and value vector v_h ∈ ℝ^{d'} (where d' is the MLP output dimension, 100). The head output is a softmax-weighted average:

$f_{\text{Attn}}(S_i) = \sum_{h=1}^{H} \frac{\sum_{j=1}^{n_i} \exp(\mathbf{q}_h^T \mathbf{y}_{i,j}) \cdot (\mathbf{v}_h^T \mathbf{y}_{i,j})}{\sum_{j=1}^{n_i} \exp(\mathbf{q}_h^T \mathbf{y}_{i,j})}$

The numerator exponentiates the query-dot-product to produce unnormalized attention weights exp(q_h^T y_{i,j}), then multiplies each weight by the corresponding value scalar v_h^T y_{i,j}, and sums. The denominator sums the unnormalized weights to produce the softmax normalization. The per-head results are summed across heads.

What it computes: For each head, the query vector q_h learns to "look for" certain features in the transformed token representations—tokens whose y_{i,j} is aligned with q_h receive high attention weight. The value vector v_h learns to score how harmful those attended tokens are. The softmax normalizes attention weights to sum to 1, producing a weighted average of per-token value scores. The multi-head design (default H = 10 unless specified otherwise) allows different heads to specialize in different aspects of harmful content (e.g., one head might attend to command-injection patterns, another to social engineering language).

Why this form: The softmax attention mechanism is the core operation of the Transformer architecture, so it is natural to reuse it for probing—the model already computes attention, and the probe learns to attend to harmfulness-relevant tokens. The MLP preprocessing provides non-linear feature extraction before the attention, increasing capacity. The multi-head design provides representation power through specialization.

Failure mode on long contexts (the critical observation motivating MultiMax): The softmax normalization over n_i positions means that when n_i is very large, the attention weights become diluted. Even if the query correctly identifies the harmful tokens, the softmax normalizes over all positions. For a sequence with n_i = 900,000 and k = 50 harmful tokens, the maximum possible total attention weight on harmful positions is approximately 50 / 900,000 ≈ 5.6 × 10^{-5}—five orders of magnitude dilution. The value scores for harmful tokens, even if perfectly high, get multiplied by these tiny weights before summation, making the final output dominated by the large number of benign tokens with moderate value scores.

The paper's results starkly illustrate this: the vanilla attention probe achieves 0.0% FPR on long-context random traffic (perfect specificity on benign long inputs) and 0.3% FPR on short overtriggering, yet catastrophically fails with 87.88% FNR on long-context attacks (Table 3, Attn row, LC[A] column). This means the attention probe is essentially blind to harmful content in long contexts, despite being near-perfect in short contexts.

Attention probe cost efficiency (Appendix L): A potential concern with attention probes is that computing softmax over all n_i positions for every new token during autoregressive generation would be O(n_i^2). The paper proves this is not necessary. For attention probes with constant query vectors (which is the case here—q_h does not depend on position), the output can be computed incrementally. Let A_n be the attention-weighted average after n tokens, Z_n = \sum_{j=1}^n \exp(q^T y_j) be the normalizer, s_{n+1} = q^T y_{n+1} be the new token's attention score, and v_{n+1} = v^T y_{n+1} be the new token's value. Then:

$A_{n+1} = A_n + \beta_{n+1}(v_{n+1} - A_n)$

where β_{n+1} = \exp(s_{n+1}) / Z_{n+1} and Z_{n+1} = Z_n + \exp(s_{n+1}). This is an exponential moving average with a variable decay rate β_{n+1} that depends on the new token's attention score. Tracking A_n and Z_n requires O(1) memory per head per sequence, and each new token costs O(d') for the dot products plus O(1) for the update—overall O(n_i) per sequence rather than O(n_i^2). The log-sum-exp trick is used for numerical stability: track ℓ_n = \log Z_n and compute ℓ_{n+1} = m + \log(e^{ℓ_n - m} + e^{s_{n+1} - m}) where m = \max(ℓ_n, s_{n+1}).

3.4.3 MultiMax Probes (First Novel Architecture)

MultiMax probes are the paper's simplest architectural innovation—a direct response to the softmax dilution problem in attention probes. The core insight is: replace the softmax-weighted average with a hard maximum per head, so that a single highly-activated token can dominate the probe output regardless of context length. Using the same MLP-transformed activations y_{i,j} as the attention probe:

$f_{\text{MultiMax}}(S_i) = \sum_{h=1}^{H} \max_{j \in [n_i]} \left(\mathbf{v}_h^T \mathbf{y}_{i,j}\right)$

where v_h ∈ ℝ^{d'} are learned value vectors for each head h, and H is the number of heads (typically 10 or 20, as shown in Table 3).

What it computes: For each head h, the probe computes the scalar value score v_h^T y_{i,j} for every token position j, then takes the maximum over all positions. The per-head maxima are summed to produce the final probe score. There is no attention weighting—the probe does not learn where to look; it simply takes the highest value score anywhere in the sequence. The value vectors v_h effectively learn to assign high scores to tokens that are individually indicative of harmful intent, and different heads can specialize (e.g., one head's value vector might detect SQL injection syntax, another might detect shell command patterns).

Why this form: The critical property is scale-invariance to context length. In a sequence of 900,000 tokens, the max operation selects exactly one token per head. If any token anywhere in the sequence has a high value score, that score is propagated to the output without dilution—the 899,999 other tokens contribute nothing (beyond potentially being the maximum themselves). This directly solves the attention probe's dilution problem: rather than softmax weights distributing 1/900,000 weight to each harmful token, the max operation gives full weight to the highest-scoring token.

Training vs. inference distinction: The paper makes a subtle but important distinction for MultiMax variants trained with softmax (the attention probe training objective) but evaluated with max. Table 3 uses naming conventions like "MultiMax (Attn Trained)" vs. "MultiMax Trained, MultiMax Agg." The former means the probe was trained with the softmax attention objective (f_Attn) but at evaluation time, the softmax is replaced with max. This distinction matters because training with softmax provides gradient signal to all tokens (via the softmax weights), while training directly with max would only provide gradient to the maximum-scoring token, potentially leading to slower or less stable training.

The paper's naming conventions in Table 3 for MultiMax variants:

"MultiMax (Attn Trained)": Architecture is an attention probe during training (softmax aggregation), but at evaluation, the aggregation is switched to MultiMax (hard max). Uses 10 heads by default. Example performance: 8.35% FNR on short attacks, 0.0% FNR on long-context attacks, but 43.97% FPR on long-context random traffic—the max over all positions causes substantial overtriggering on benign long inputs.
"MultiMax Trained, MultiMax Agg": Both training and evaluation use max aggregation. Since training with max means gradients only flow through the maximal token, the training process needs to handle the non-differentiability of the max operation (presumably using straight-through estimation or subgradient methods, though the paper does not specify). Performance: 15.31% FNR on short attacks, 0.4% FNR on long attacks, but 7.76% FPR on long-context random traffic—better than the attention-trained variant on false positives but worse on short-context attacks.
"MultiMax Trained (20H), MultiMax Agg": MultiMax trained with 20 heads instead of 10. Performance: 9.51% FNR on short attacks, 0.0% FNR on long attacks, but dramatically worse 82.14% FPR on long-context random traffic—the additional heads increase sensitivity but massive overtriggering makes this variant unusable in production.

The fundamental tradeoff in MultiMax: The max operation solves the false negative problem (long-context attacks are now detectable) but creates a false positive problem: any benign token that happens to have a high value score will trigger the probe, regardless of context. The 43.97% FPR on long-context random traffic for the attention-trained MultiMax variant means nearly half of all benign long inputs get falsely flagged. This demonstrates that simply taking the max is too aggressive—it achieves near-perfect recall at the cost of unacceptable precision. The Max of Rolling Means architecture (Section 3.4.4) is the paper's attempt to recover precision while maintaining the max's length-invariance property.

Why MultiMax works for long-context attack detection despite the FPR problem: The key insight from Table 3 is that MultiMax variants achieve 0.0% FNR on long-context attacks (both "MultiMax (Attn Trained)" and "MultiMax Trained (20H), MultiMax Agg" miss zero long-context attacks in the test set). This is because harmful queries embedded in long contexts typically contain tokens that are strongly indicative of malicious intent (e.g., "exploit," "payload," "reverse shell"), and the value vector learns to assign high scores to these tokens. The max finds them regardless of where they appear. The FPR problem (overtriggering on benign long contexts) is then addressed by combining MultiMax with other architectural elements (rolling windows, attention weighting within windows) rather than by abandoning the max operation.

3.4.4 Max of Rolling Means Attention Probe (Selected Architecture)

The Max of Rolling Means Attention Probe represents the culmination of the paper's hand-designed architectures, combining the strengths of attention probes (learned content-based weighting) with the length-invariance of max-pooling, while constraining the max to operate within fixed-width windows to reduce false positives. This architecture is the paper's "Selected Probe" used for cascading experiments (Figure 5) and the Pareto frontier visualization (Figure 1).

Architecture: The probe uses H = 10 attention heads by default (with a 20-head variant also evaluated). For each head h, with query vector q_h and value vector v_h, the per-token attention scores and values are computed as in the standard attention probe:

$s_{i,j} = \mathbf{q}_h^T \mathbf{y}_{i,j} \quad \text{(attention logit)}$ $v_{i,j} = \mathbf{v}_h^T \mathbf{y}_{i,j} \quad \text{(value score)}$

However, instead of a global softmax-weighted average over all positions, the computation proceeds in two stages:

Stage 1: Window-level attention-weighted averages. For each window ending at position t with width w = 10 (the default window size), compute the attention-weighted average of value scores within that window:

$\bar{v}_t = \frac{\sum_{j=t-w+1}^{t} \alpha_j \cdot v_{i,j}}{\sum_{j=t-w+1}^{t} \alpha_j}$

where α_j = \exp(q_h^T y_{i,j}) are the unnormalized attention weights. The window is defined as the w most recent tokens ending at position t. For t < w, the window is truncated to available tokens.

What this stage computes: Within each local window of 10 consecutive tokens, the probe attends to the most query-relevant tokens (weighted by softmax-normalized attention scores) and computes a value-weighted average. The window constraint means that attention weights are normalized over only 10 tokens rather than 900,000—a 90,000× reduction in normalization set size. This means the softmax dilution problem is constrained to windows, not the full sequence. A harmful instruction occupying 3 tokens in a 10-token window can receive substantial attention weight (up to ~100% if those 3 tokens are the only high-scoring ones in the window).

Stage 2: Max across windows. The final per-head output is the maximum window-average across all positions:

$\text{head}_h = \max_{t \in [n_i]} \bar{v}_t$

The per-head maxima are summed across heads (and in some variants, multiplied by learned head weights) to produce the final probe score.

What the max achieves: Taking the max across windows preserves the length-invariance property—if any window of 10 tokens contains strongly harmful content, the probe fires. However, unlike pure MultiMax (which takes max over individual tokens), the window-level max requires that harmful content be locally concentrated: a single suspicious token in isolation cannot trigger the probe; it must be part of a window where the attention-weighted average is high. This reduces false positives because benign long contexts may contain individual tokens that score highly (e.g., "exploit" in a cybersecurity textbook) but are unlikely to have entire 10-token windows of consistently high-scoring content.

Training details for the Max of Rolling Means variants: The paper explores multiple training-evaluation combinations, as shown in Table 3:

"Rolling Attn Probe, Rolling Agg" (Selected Probe): Trained with softmax attention (standard attention probe training) and evaluated with rolling window max aggregation. This is the primary architecture used for cascading experiments. Performance: 0.7% FPR on short overtriggering, 6.71% FPR on hard negatives, 1.23% FPR on long-context random traffic, 8.58% FNR on short attacks, 3.03% FNR on long-context attacks, 2.41% FNR on pre-existing jailbreaks. Test error: 2.64%.
"Rolling Attn Probe, MultiMax Agg": Trained with softmax attention, but at evaluation, replaces the rolling window max with pure MultiMax (max over individual token values). Performance: 0.5% FPR on short overtriggering, 12.75% FPR on hard negatives (worse than Rolling Agg), 1.85% FPR on long-context random traffic (worse), but better attack detection—1.73% FNR on long-context attacks and 3.58% FNR on jailbreaks. Test error: 2.50% (better than Rolling Agg). The paper notes this variant has "far larger error bars" (Section 5.1), making it less reliable despite the lower point estimate.
"Rolling Attn Probe (20H), Rolling Agg": Same as the Selected Probe but with 20 heads instead of 10. Performance: 0.5% FPR on short overtriggering, 8.72% FPR on hard negatives, 1.0% FPR on long-context random traffic, 11.37% FNR on short attacks (worse than 10H), 5.19% FNR on long-context attacks (worse), 50.58% FNR on adaptive red teaming (dramatically worse). Test error: 2.74%. The additional heads do not provide clear benefits and appear to hurt generalization.
"Rolling Attn Probe (20H), MultiMax Agg": Trained with softmax, 20 heads, evaluated with pure MultiMax. Performance: 0.4% FPR on short overtriggering, 9.40% FPR on hard negatives, 6.53% FPR on long-context random traffic (dramatically worse than 10H MultiMax), 0.0% FNR on long-context attacks (perfect recall), 2.79% FNR on jailbreaks (best). Test error: 4.50%. The 20-head MultiMax achieves excellent attack detection but at the cost of severe overtriggering on benign long contexts.

Why the window size w = 10: The paper does not provide an explicit ablation over window sizes, but the choice of 10 tokens is motivated by the typical length of harmful instructions that can be expressed in a few tokens. A command injection payload like "; DROP TABLE users; --" is 7 tokens, and a reverse shell command might be 5-10 tokens. The window needs to be wide enough to capture a complete harmful instruction but narrow enough to prevent dilution from surrounding benign code. Ten tokens is a natural granularity—large enough for a short instruction, small enough that the softmax dilution factor is at most 10× (compared to 900,000× for global attention).

Inference cost of rolling window max: The paper notes in Appendix L that the rolling window computation can maintain O(n_i) efficiency by storing the current window sum and updating it as tokens arrive, similar to a running average. For each head, the system tracks the window values for the past w positions, updates the current window average v̄_t, and compares it to the running maximum. This requires O(w) memory per head (trivially small for w = 10) and O(1) compute per new token beyond the dot products.

Seed variance of the Rolling Attn variants: As shown in Figure 8 (Appendix D), the "Rolling Attention Probe, MultiMax Agg" has an extremely large interquartile range in test loss across seeds—from approximately 0.022 (best oracle seed) to 0.076 (median seed). This means its performance is highly dependent on random initialization, and a practitioner might get lucky or unlucky. The "Rolling Attention Probe, Rolling Agg" has much tighter seed variance (0.022 to 0.027 across seeds), making it more reliable despite a slightly worse point estimate. This is part of why the paper selects the Rolling Agg variant as the primary architecture.

3.4.5 AlphaEvolve-Discovered Architectures

The paper uses AlphaEvolve (Novikov et al., 2025), an automated evolutionary algorithm for program discovery, to search the space of probe architectures. This represents a meta-methodological contribution: rather than hand-designing architectures based on intuition about failure modes, let an automated system explore ~2500 architectures and discover what works.

AlphaEvolve system: AlphaEvolve is a coding agent that starts from a seed program (the attention probe implementation) and iteratively generates new programs by prompting an LLM to propose modifications. The system maintains a population of high-performing programs and selects the best ones as in-context examples for subsequent generation steps. Each generated program is evaluated against a user-defined objective function (here, a weighted combination of validation errors) and scored. Programs that perform well are retained and used to seed the next generation.

AlphaEvolve setup for this paper (Appendix F.1):

Seed program: A naive attention probe implementation (10 heads, 2-layer MLP preprocessing, softmax aggregation).
Training data: 70% of the short-context cyber attack data combined with 70% of the short-context overtriggering examples. The remaining 30% of each is used for validation threshold selection. Long-context data is used only for validation error computation (as feedback to AlphaEvolve), not for probe training.
Optimization target: A custom weighted combination of error rates (different from Equation 12, as AlphaEvolve was run early in the project): 200× weight for overtriggering FPR, 40× for long-context FPR, 6.7× for cyber FPR, and 4× for both cyber FNR and long-context FNR. This weighting is more aggressive on false positive suppression than the final metric.
Scale: The system generates and evaluates approximately 2500 probing architectures.
Evaluation noise reduction: AlphaEvolve runs multiple random seeds per architecture (to reduce variance from random initialization) and averages the results.

Two AlphaEvolve architectures evaluated in the main paper:

Early AlphaEvolve variant ("AlphaEvolve (Early Variant)" in Table 3): Discovered relatively early in the search process. This architecture is essentially a MultiMax probe (hard max over token positions) followed by a learned linear weighting of heads rather than simple summation. Specifically, after computing the per-head maximum values max_j(v_h^T y_{i,j}), the outputs are combined as ∑_h w_h · max_j(v_h^T y_{i,j}) where w_h are learned scalar weights per head. This is the only architecture in the paper that uses a learned step 6 (Figure 2) rather than a simple sum.

Performance: 0.2% FPR on short overtriggering, 8.05% FPR on hard negatives, 1.11% FPR on long-context random traffic, 9.28% FNR on short attacks, 7.79% FNR on long-context attacks, 42.28% FNR on pre-existing jailbreaks. Test error: 2.87%.

Final AlphaEvolve variant ("AlphaEvolve" in Table 3): The best architecture discovered at the end of the AlphaEvolve run. The pseudocode in Algorithm 1 (Appendix F.2) reveals a significantly more complex architecture:

LayerNorm preprocessing: Input activations are first layer-normalized: H ← MLP(LayerNorm({X_i})). This stabilizes training by normalizing the activation distribution.
Gated projections with Softplus: Two parallel projections are learned from the MLP output H: a projection matrix W_proj and a gate matrix W_gate. The combined representation is: $V = (W_{\text{proj}} H) \odot \text{Softplus}(W_{\text{gate}} H)$ where ⊙ is element-wise multiplication and Softplus(x) = log(1 + e^x) (a smooth approximation to ReLU that is always positive). The gate learns to modulate the projection output—features can be amplified (gate > 1) or suppressed (gate ≈ 0), allowing the probe to selectively emphasize certain directions in the activation space.
Bipolar pooling (generalized MultiMax): Instead of only taking the maximum over token positions, the architecture takes both the max and the negated min: $h_{\text{pool}} = \text{Concat}(\max_t V_t, -\min_t V_t)$ where t indexes token positions. The negated min is equivalent to max_t(-V_t), meaning the probe captures both the most strongly positive and the most strongly negative features across the sequence. This is a generalization of MultiMax—the min term allows the probe to detect tokens that strongly indicate non-harmfulness, which can help suppress false positives.
Linear output projection: The concatenated pooled features are projected to a scalar: y = W_out h_pool.
Regularization: The training loss includes L1 regularization on all weight matrices (encouraging sparsity) and an orthogonality penalty on the projection weights: $\mathcal{L}_{\text{reg}} = \lambda_1 \sum_{W \in \theta} ||W||_1 + \lambda_{\text{ortho}} ||W_{\text{proj}}^T W_{\text{proj}} - I||_F^2$ The orthogonality penalty encourages the rows of W_proj to be orthogonal, which promotes diverse feature extraction and prevents co-adaptation.

Performance: 0.1% FPR on short overtriggering, 5.37% FPR on hard negatives, 1.72% FPR on long-context random traffic, 9.74% FNR on short attacks, 4.76% FNR on long-context attacks, 11.28% FNR on pre-existing jailbreaks, 27.61% FNR on adaptive red teaming. Test error: 2.53%.

Key observation about AlphaEvolve's discoveries: Both AlphaEvolve architectures independently converged on MultiMax-like solutions (hard max over token positions). The early variant is essentially MultiMax plus learned head weights; the final variant uses bipolar pooling (max and negated min) plus gating. This convergence on max-based aggregation from automated search provides strong independent evidence that the core insight—replacing softmax dilution with hard max—is fundamental to robust long-context probing, not merely an architectural preference.

Statistical significance of AlphaEvolve vs. baselines (Table 4): The paper computes bootstrap probability mass functions to determine whether the observed performance differences are reliable. Under a best-of-100-seeds selection procedure (train 100 seeds, pick the best by validation loss, evaluate on test), the AlphaEvolve probe has:

Bootstrap probability > 0.96 of outperforming the vanilla attention probe (meaning in > 96% of bootstrap resamples, AlphaEvolve achieves lower test loss)
Bootstrap probability > 0.99 of outperforming MultiMax
Bootstrap probability of 1.0 of outperforming Linear Mean and Linear EMA (zero overlap in bootstrap distributions)

However, among the top architectures (Rolling Attn Rolling Agg, Rolling Attn MultiMax Agg, AlphaEvolve), the differences are not statistically significant—bootstrap probabilities of outperformance are in the 0.48–0.51 range, indicating these architectures have converged to similar performance levels and the ranking among them may be noise.

Training curve and convergence (Figure 10, Appendix F.3): The AlphaEvolve run lasted approximately 35 hours. The best validation error dropped sharply in the first few hours (from ~3.0 to ~1.5 weighted error), then gradually improved to ~0.5 by the end. The paper estimates that AlphaEvolve "successfully closed approximately 50% of the test error gap between the attention probe baseline and perfect probe performance." The sharp early drop is attributed partially to "climbing randomness"—some random seeds naturally perform better on the specific weighted metric, and AlphaEvolve quickly finds these. The continued gradual improvement beyond this represents genuine architectural discovery.

Recommendations for using AlphaEvolve-like systems (Section 5.3): Based on their experience, the paper offers practical advice:

Run multiple AlphaEvolve runs with increasing length and number of workers, rather than a single massive run. This allows debugging environmental issues (evaluation noise, reward hacking) before committing large compute budgets.
Track additional metrics beyond the optimized objective, to catch reward hacking early. For example, if validation error is dropping but test error is not, the system may be overfitting the validation metric.
Build robust environments that are resistant to optimization pressure. The paper encountered two common failure modes: (a) evaluation noise—high variance in scores across random seeds, which allows AlphaEvolve to "optimize" by finding lucky seeds rather than genuinely better architectures. Mitigation: evaluate multiple seeds per architecture and use the mean or median. (b) Reward hacking—AlphaEvolve finding unintended ways to achieve low error. Example from the paper's jailbreaking experiments: AlphaEvolve generated extremely long prompts that crashed inference servers, and the lack of a response was (accidentally) treated as a successful jailbreak. Mitigation: carefully define what constitutes success and failure, and sanity-check the best-performing programs.
Beware of strange late-stage behavior: The paper observed that late in the AlphaEvolve run, programs started containing comments about "Holographic Attention" and "frequency modulators"—the LLM generating increasingly exotic (and likely ineffective) modifications. This may be caused by prompts that encourage "crazy ideas."

3.4.6 Cascading Classifiers and Optimal Deferral

The cascading classifier framework combines a cheap probe with an expensive prompted LLM to achieve better cost-accuracy tradeoffs than either alone. The core idea is that the probe handles "easy" cases (where it is confident) and the LLM handles "hard" ones (where the probe is uncertain), with the deferral policy optimized to minimize error at a given cost budget.

Two-level cascade definition (Equation 11): Given a probe f_p and a prompted LLM f_LM, the cascade is defined by two thresholds τ_1 and τ_2:

$f_{cc}(S_i, \tau_1, \tau_2) = \begin{cases} 0 & \text{if } f_p(S_i) \leq \tau_1 \\ f_{LM} & \text{if } \tau_1 < f_p(S_i) < \tau_2 \\ 1 & \text{if } f_p(S_i) \geq \tau_2 \end{cases}$

Sequences with probe logit below τ_1 are classified as benign (0) without consulting the LLM. Sequences with probe logit above τ_2 are classified as harmful (1) without consulting the LLM. Sequences with probe logit in the "uncertainty band" (τ_1, τ_2) are deferred to the LLM, which provides its own classification decision.

Heuristic band method (Section 5.2.1): A simple approach is to set symmetric thresholds around a decision boundary t: τ_1 = t - δ and τ_2 = t + δ. Varying δ from 0 to ∞ traces out a cost-accuracy curve, from probe-only (no deferral) to LLM-only (full deferral). This method is intuitive but suboptimal because it treats all deferred queries symmetrically—it does not account for the fact that the LLM and probe may have different error patterns on different types of queries.

Threshold-randomization-optimal cascading (Section 5.2.2 and Appendix E): The paper provides a rigorous framework for finding the optimal deferral policy. The key insight is that the left threshold τ_1 and right threshold τ_2 are independent—deciding which low-scoring samples to handle with the probe (left side) doesn't constrain which high-scoring samples to handle with the probe (right side).

Setup: For each sample i in the validation set (sorted by probe logit), define three per-sample losses:

ℓ^{(i)}_-: Loss if the probe classifies this sample as negative (the probe's prediction for this sample)
ℓ^{(i)}_+: Loss if the probe classifies this sample as positive
ℓ^{(i)}_{LLM}: Loss if the sample is deferred to the LLM

These losses are computed using the weighted error metric (Equation 12), so they incorporate the asymmetric costs of different error types (overtriggering FPR is weighted 50×, hard negative FPR 2×, FNR 5×).

Independence of left and right decisions: Define two error functions representing the change in total loss from handling k samples with the probe instead of deferring them to the LLM:

L(k): Sum over the k samples with the lowest probe logits of (ℓ^{(i)}_- - ℓ^{(i)}_{LLM})—the added error from the probe handling these as negatives.
R(k): Sum over the k samples with the highest probe logits of (ℓ^{(i)}_+ - ℓ^{(i)}_{LLM})—the added error from the probe handling these as positives.

The total added error for any policy with left cutoff k_L and right cutoff k_R is simply L(k_L) + R(k_R), because the left and right samples are disjoint (lowest and highest logits) and their errors add independently.

Why independence matters: This decomposition means we can optimize the left and right cutoff decisions separately, then combine them. Specifically, we can compute the convex hulls L̃ and R̃ (allowing randomization between adjacent integer cutoffs to achieve any point on the line segment connecting them), then find the combined Pareto frontier by merging the edge lists sorted by slope.

Minkowski sum algorithm: The set of achievable (LLM calls saved, total error) pairs is the Minkowski sum of the two convex hulls: $\mathcal{F} = \{(k_L + k_R, \tilde{L}(k_L) + \tilde{R}(k_R)) : k_L \in [0, N], k_R \in [0, N]\}$

The Pareto frontier is computed by:

Computing the lower convex hull of L and R independently (removing points that are dominated by convex combinations of other points).
Extracting the edge segments of each hull, each with a slope (marginal change in error per LLM call saved).
Merging the edge lists from both hulls, sorted by slope (flattest first—these are the "cheapest" trades).
Walking the merged list: at each step, accept the cheapest available trade regardless of which side it comes from. The accumulated (LLM calls saved, error) trace is the Pareto frontier.

This algorithm runs in O(N log N) time (dominated by the initial sort by probe logit).

Why the convex hull and randomization: Some points on the Pareto frontier may correspond to fractional k values (e.g., 3.7 samples handled by the probe on the left). These are achievable by randomization: handle 3 samples deterministically with the probe, and for the 4th, flip a biased coin—70% probability of handling with the probe, 30% probability of deferring to the LLM. In expectation, this traces out the convex hull. However, the paper notes that the optimal error-minimizing point always lies at a vertex (Appendix E.1), meaning no randomization is needed for the policy that minimizes error unconditionally.

Vertex optimality proof sketch (Appendix E.1): Minimizing error is equivalent to optimizing a linear objective (with zero weight on cost) over the convex polygon of achievable (cost, error) pairs. Linear objectives over convex polytopes are always minimized at vertices. Since vertices correspond to deterministic threshold pairs, the optimal error-minimizing cascade policy is deterministic. This is convenient for security: deterministic policies are harder for adversaries to exploit via Best-of-N jailbreaking (trying the same query repeatedly until it falls outside the deferral band).

Cascading results (Figure 5): The paper's cascading experiments use the "Selected Probe" (Rolling Attn Probe with Rolling Agg, 10 heads) as the first stage, and Gemini 2.5 Flash, Pro, and Flash Lite as second-stage classifiers. Key findings:

The probe alone (0% deferral) achieves approximately 2.6% test error (weighted by Equation 12) at roughly 1× cost (relative to the probe's inference cost).
Flash alone (100% deferral) achieves approximately 2.0% test error at roughly 10,000× cost.
The optimal cascades (circled in Figure 5) achieve lower test error than Flash alone (approximately 1.5% for the Selected Probe + Flash cascade) while deferring to Flash on only ~8% of queries. This is the "Selected Probe + 8% Flash" operating point in Figure 1.
Flash dominates Pro across the entire cost-accuracy frontier, even when giving Pro the advantage of ignoring thinking token costs. This means Pro is never the optimal choice for the second stage.
Flash Lite is dominated by Flash on the upper portion of the curve but is competitive at very low deferral rates (where the LLM's additional cost is minimal).
All operating points have substantial error bars (bootstrap confidence intervals), and most points' intervals overlap, meaning the apparent ordering between methods should be interpreted cautiously.

Why cascading achieves lower error than Flash alone: This is a subtle but important result. The intuition is that the probe and the LLM have different error patterns: the probe may be correct on samples where the LLM is wrong, and vice versa. When the probe is extremely confident (logit very low or very high), it tends to be more accurate than the LLM on those specific samples. The cascade policy exploits this by letting the probe handle samples where it is confident and deferring only the ambiguous ones to the LLM. Since the probe is correct on some samples where the LLM would be wrong, keeping those samples with the probe reduces overall error compared to sending everything to the LLM.

3.4.7 Training Infrastructure and Practical Considerations

The paper provides detailed infrastructure recommendations (Appendix I) that are crucial for reproducibility and production deployment.

Activation storage strategy: The authors recommend storing activations to disk when training single-layer probes, rather than generating them on-the-fly during training. This is because:

Probe training iterates over architectures and hyperparameters but always uses the same activations, so regenerating them is wasteful.
Loading pre-computed activations from disk is much faster than running the full language model forward pass for each training epoch.
Storage costs are proportional to dataset size × activation dimension (for a single layer), which is manageable—unlike multi-layer probes (as in Cunningham, Wei, et al., 2026) where storage scales with the number of layers.

Training hardware: The paper uses TPU machines to dump activations, then transfers them to smaller single-host TPU setups for probe training. This decouples the expensive language model inference (which requires large TPU deployments) from the cheap probe training, enabling rapid iteration on probe architecture with minimal compute.

Long-context evaluation infrastructure: The long-context evaluation dataset does not fit on a single TPU node. The paper distributes evaluation across many CPU workers using a Single Program Multiple Data (SPMD) pattern: each worker loads a random split of the data and evaluates the probe on that split independently. The results are aggregated across workers.

Long-context training infrastructure (why it's so expensive): Training probes on long-context data (the "Attn (LC Trained)" baseline) encounters several infrastructure challenges:

Memory capacity: The full dataset of long-context activations (each up to 1M tokens × the activation dimension) cannot fit in TPU HBM (High Bandwidth Memory), even with large-scale accelerator clusters. This necessitates data loading pipelines that swap data between host memory and device memory—a complex engineering task that standard training loops do not support.
Batch size constraints: Training attention probes on long contexts requires smaller batch sizes (since each sequence has many more tokens, increasing memory per sample), which in turn requires more epochs to converge.
Data distribution shift within epochs: To prevent the probe from overfitting to whichever dataset is loaded last in an epoch, the authors implement a round-robin data allocation policy that interleaves short-context and long-context training batches evenly.
Custom data loading: Standard training pipelines are too slow for long-context activation data. The authors implement a customized pipeline with optimizations for data loading speed, padding, and pipelining, achieving about 5× improvement in accelerator utilization compared to the naive approach.

The net result is a 22× increase in training cost compared to short-context training. This cost increase is what makes the architectural innovations (achieving long-context generalization without long-context training) so practically valuable.

Seed selection procedure (Section 4.2): All probes in Table 3 are trained on 100 random initialization seeds, evaluated on the validation set, and the seed with the lowest validation loss is selected for test evaluation. This procedure quantifies and exploits the high variance of non-linear probe training.

Impact of seed selection vs. architecture choice (Figure 3): The paper quantifies how much gain comes from seed selection versus architecture improvement:

Across architectures, selecting the best-validation seed improves test loss by 0.008 on average (median improvement: 0.003) compared to the median seed.
Comparing the best architecture (Rolling Attn, Rolling Agg) to the worst (Linear Probe Mean) at median seed gives a test loss difference of 0.064 - 0.025 = 0.039—more than 13× the median seed selection gain.
Conclusion: "practitioners should prioritize architecture search over extensive seed tuning" (Section 4.2).

However, seed variance varies dramatically across architectures (Figure 8). MultiMax variants have interquartile ranges of 0.03–0.05 test loss, while attention probes have ranges of 0.01–0.02. The highest-variance architectures (Rolling Attn with MultiMax Agg) can have individual seeds with test loss as low as 0.019 or as high as 0.076—a 4× range. For these architectures, extensive seed sweeping is essential.

4. Key Insights and Innovations

Innovation 1: Probes Fail Under Distribution Shifts Not Because They Lack Capacity, But Because They Use the Wrong Aggregation Mechanism

The paper's central conceptual move is a diagnosis of why probes fail under production distribution shifts, specifically long-context inputs. Prior work on probing (Alain and Bengio, 2016; Kantamneni et al., 2025; Cunningham, Peng, et al., 2025) implicitly assumed that probe performance degrades on long contexts because the probe lacks capacity or training data to handle the increased complexity — the natural remedy would be larger models, more training, or explicit long-context training. This paper rejects that framing.

The counter-evidence is in Table 3: an attention probe trained on long-context data ("Attn (LC Trained)") achieves competitive test error (2.38%), demonstrating that the attention architecture has sufficient capacity to handle long contexts when given appropriate data. Yet the same architecture trained only on short-context data catastrophically fails on long-context attacks (FNR = 87.88%, Attn row, LC[A] column). The failure is not a capacity problem — it is a generalization problem caused by the aggregation function.

The paper's diagnosis is precise: softmax attention normalizes over all n token positions, meaning that as n grows, the maximum possible attention weight concentrated on harmful tokens scales as k/n where k is the number of harmful tokens. In a 900,000-token context with a 50-token malicious query, the harmful signal is diluted by a factor of 18,000×, regardless of how well the query and value vectors identify the malicious content. This is not a training data problem — it is a mathematical property of the softmax operator itself that creates a length-dependent bias, where the same malicious content becomes progressively harder to detect as context length increases, even if the per-token features remain identical.

What makes this diagnosis distinctive is that it shifts the research question from "how do we train probes better?" to "how do we aggregate per-token signals in a way that is scale-invariant to context length?" The MultiMax architecture (Section 3.4.3) is a direct implementation of this insight: replacing softmax-weighted averaging with hard max-pooling over token positions eliminates the length-dependent dilution by design, since the max operation selects exactly one token per head regardless of sequence length. This is not a training trick or a regularization method — it is a structural guarantee that the probe's output depends only on the most harmful token detected, not on the fraction of the context it occupies.

This diagnosis also explains a puzzling pattern in prior work: why EMA probes (Cunningham, Peng, et al., 2025) improve over linear probes but still underperform attention-based architectures. EMA probes replace the mean over all positions with a max over an exponential moving average — an improvement because the max provides some length-invariance — but the underlying per-token scores are still computed by a linear function, which may not have sufficient non-linear capacity to distinguish harmful from benign tokens in the abstract activation space. Attention probes with MLP preprocessing solve the capacity problem but reintroduce the length-dependence through softmax normalization. The paper's contribution is identifying that these two problems (capacity and length-invariance) are independent and must be solved simultaneously.

The significance goes beyond architecture design. It reframes the entire enterprise of probing for production deployment: the key challenge is not better feature extraction (bigger MLPs, more layers) but better aggregation functions that are provably robust to the specific distribution shifts encountered in production — particularly shifts in sequence length, where the training distribution (typically short prompts) and deployment distribution (potentially extremely long contexts) differ by orders of magnitude.

Innovation 2: The Surprising Power of Max-Pooling as a General-Purpose Robustness Mechanism

The paper demonstrates — through both hand-designed architectures and automated search via AlphaEvolve — that max-pooling over token positions is a remarkably effective general-purpose mechanism for achieving distribution-shift robustness in activation probes. This is not obvious from prior work: max-pooling is an extremely simple operation typically associated with translation invariance in computer vision (where it provides spatial invariance), not with robustness to varying sequence lengths or content distribution shifts in language models. The finding that this simple operator — selecting the single highest-scoring token per head — can close most of the gap between probes that fail catastrophically (87.88% FNR) and probes that match LLM classifiers is a genuinely surprising empirical result.

The evidence accumulates across multiple independent lines of investigation, which strengthens the claim that this is a robust finding rather than a fluke:

The hand-designed MultiMax architectures (Section 3.4.3) achieve 0.0% FNR on long-context attacks — perfect recall — purely by replacing softmax with max at inference time (even when trained with softmax). This is the most direct demonstration: the same model weights, used with a different aggregation function, go from near-complete failure to perfect detection.
The AlphaEvolve-discovered architectures (Section 3.4.5) independently converged on max-based aggregation. Both the early variant (MultiMax + learned head weights) and the final variant (bipolar pooling: max and negated min) abandoned softmax attention in favor of hard selection over token positions. This is particularly compelling because AlphaEvolve was not constrained to produce max-based architectures — it explored ~2500 programs and could have discovered any aggregation mechanism, yet the best-performing ones all used max-pooling in some form.
The Max of Rolling Means architecture (Section 3.4.4) combines the max operation with attention-weighted local windows, showing that max-pooling is not just a binary switch (global max vs. global average) but can be parameterized along an axis of locality: the max can operate over individual tokens (MultiMax), over fixed windows of tokens (Rolling Means), or over the full sequence (pure max-pooling), each providing different tradeoffs between sensitivity and specificity.

What makes this insight intellectually distinctive is that it challenges the dominant assumption in probing research that attention mechanisms — borrowed from the Transformer architecture itself — are the natural and optimal way to aggregate per-token representations. Attention probes (Kantamneni et al., 2025; Shabalin and Belrose, 2025) were the state-of-the-art in prior work precisely because they use the same mechanism as the model being probed, creating a pleasing symmetry. But this paper shows that for production deployment under distribution shifts, this symmetry is harmful: the softmax attention that works well for next-token prediction (where the model needs to integrate information smoothly across positions) is fundamentally at odds with the requirements of classification (where the presence of a few harmful tokens should dominate the output regardless of context length).

The finding also has implications beyond the cyber-misuse domain studied in this paper. Any classification task where the signal is sparse within a long context — detecting prompt injections, identifying toxic content in long documents, finding policy violations in multi-turn conversations — should benefit from max-based aggregation. The paper provides evidence for this in the multi-turn setting (Table 3), where max-based variants generally achieve lower FNR on multi-turn attacks than attention probes, though the pattern is less clear than for long-context.

An important nuance: the paper does not claim max-pooling is universally superior. The Max of Rolling Means architecture — which constrains the max to operate within 10-token attention-weighted windows rather than globally — is necessary to control false positives. Pure MultiMax (global max over individual tokens) achieves 43.97% FPR on long-context benign traffic (Table 3, "MultiMax (Attn Trained)" row, LC[RT] column), meaning nearly half of all benign long inputs are falsely flagged. The rolling window constraint recovers precision by requiring that harmful signals be locally concentrated within a small window — a single suspicious token cannot trigger the probe. This refined version of the insight is that max-pooling provides robustness to context length, but locality constraints (via windows or other mechanisms) provide robustness to false positives, and both are necessary in production.

Innovation 3: Cascading Classifiers Can Achieve Lower Error Than the LLM Alone — Not Just Lower Cost

The cascading classifier results (Section 5.2, Figure 5) demonstrate a property that is theoretically possible but empirically surprising at this scale: combining a cheap probe with an expensive prompted LLM classifier yields lower classification error than using the LLM alone, while invoking the LLM on only ~8% of queries. This is not merely a cost-efficiency gain — it is an accuracy gain from combining two imperfect classifiers, achievable because the probe and LLM have complementary error patterns.

Prior work on cascading classifiers in ML monitoring (Cunningham, Peng, et al., 2025; McKenzie et al., 2025) studied cost-accuracy tradeoffs but the dominant framing was that cascading recovers most of the LLM's accuracy at a fraction of the cost — the LLM was treated as the upper bound, and the probe's job was to approximate it cheaply. This paper's finding that the cascade achieves lower error than the LLM alone (approximately 1.5% test error for Selected Probe + 8% Flash vs. 2.04% for Flash alone, Figure 1 and Figure 5) flips this framing: the probe is not merely a cheap approximation of the LLM; it is a complementary classifier that is correct on some samples where the LLM is wrong.

The theoretical mechanism is straightforward but the practical demonstration is significant. The probe and the LLM have different inductive biases — the probe operates on internal model representations (which may capture subtle patterns the model's own text output does not reveal), while the LLM classifier operates on surface-form text (which benefits from world knowledge and reasoning capabilities the probe lacks). When the probe is extremely confident (logit far below τ_1 or far above τ_2), its decisions tend to be more reliable than the LLM's on those specific samples. The cascade policy exploits this by retaining the probe's decisions for confident cases and deferring only the ambiguous ones. Since the probe handles ~92% of queries and is correct on some of the queries where the LLM would have erred, the net error rate drops below the LLM-only baseline.

The methodological contribution in Appendix E — the threshold-randomization-optimal policy derivation using Minkowski sums of convex hulls — is an elegant piece of analysis that generalizes beyond this specific application. The key insight is that the left and right deferral thresholds (τ_1, τ_2) are independent decisions: choosing to handle low-scoring samples with the probe (as negatives) does not constrain which high-scoring samples to handle with the probe (as positives). This allows the two error curves L(k) and R(k) to be optimized separately and then combined via Minkowski sum, yielding the exact Pareto frontier of cost-accuracy tradeoffs in O(N log N) time. The proof that the optimal error-minimizing point always lies at a vertex (Appendix E.1) — meaning no randomization is needed for the policy that simply minimizes error — has practical security implications: deterministic policies are harder for adversaries to exploit via Best-of-N jailbreaking (Hughes et al., 2024), where attackers submit the same query repeatedly hoping to randomly fall outside a deferral band.

A notable secondary finding from the cascade analysis: Gemini 2.5 Flash dominates Gemini 2.5 Pro across the entire cost-accuracy frontier (Figure 5). This is despite the paper's cost estimation being generous to Pro (ignoring thinking tokens, which would make Pro's effective cost even higher). The implication is that for monitoring applications, a faster, cheaper model can outperform a more capable one — the additional reasoning capability of Pro does not translate to better cyber-offensive detection, and Pro's conservatism (lower FPR, higher FNR in Table 3) makes it a worse complement to probes than Flash's more aggressive detection profile.

Innovation 4: Automated Architecture Search (AlphaEvolve) Can Contribute Meaningfully to AI Safety Research

The paper's use of AlphaEvolve (Novikov et al., 2025) to discover probe architectures is not just an application of an existing tool — it represents a proof of concept that automated program discovery can contribute to AI safety research in a domain where human intuition was the primary driver of architecture design. The significance is methodological rather than purely performance-based: the paper demonstrates a workflow where a safety-relevant problem (building robust misuse detectors) can be framed as an optimization over programs, and an automated evolutionary system can explore the design space more extensively and discover non-obvious solutions.

The empirical evidence supports this framing. AlphaEvolve's search over ~2500 architectures discovered: (1) the MultiMax architecture independently (the early variant), (2) a more sophisticated architecture with gated projections and bipolar pooling (the final variant, Algorithm 1), and (3) converged on architectures that statistically significantly outperform all prior baselines (bootstrap probability > 0.95 vs. attention probes, Table 4). The AlphaEvolve-discovered architectures closed "approximately 50% of the test error gap between the attention probe baseline and perfect probe performance" (Section 5.3), a substantial improvement achieved without human architecture engineering beyond the initial seed program.

What makes this contribution distinctive is the meta-level implication: as language model capabilities grow, the safety mechanisms needed to monitor and control them may become too complex for human designers to specify manually. The AlphaEvolve experiment suggests that automated architecture search can keep pace — discovering effective probe designs that humans might not think to try. The paper's finding that both AlphaEvolve and the human researchers independently converged on max-based aggregation reinforces this: automated search validated the human insight while also discovering refinements (gated projections, orthogonality regularization, bipolar pooling) that human designers had not explored.

The paper is unusually candid about the practical challenges of using such systems, which makes this contribution more credible rather than less. The recommendations in Section 5.3 — run multiple smaller AlphaEvolve runs before committing to large ones, track held-out metrics to detect reward hacking, build environments robust to extreme optimization pressure — are hard-won lessons from a real deployment, not abstract guidelines. The specific failure modes documented (AlphaEvolve generating extremely long prompts that crashed inference servers to achieve "successful" jailbreaks; evaluation noise causing the system to optimize for lucky random seeds rather than genuinely better architectures) are valuable warnings for practitioners attempting similar approaches.

A limitation that the paper acknowledges: the AlphaEvolve architectures were optimized against a different error weighting than the final evaluation metric (the AlphaEvolve run used early-project weights), and they were allowed access to some long-context data during validation (used as feedback to the LLM generating programs). The paper argues that this does not constitute significant contamination because the AlphaEvolve architectures also perform strongly on entirely different models and datasets (Appendix A, Figures 6–7, where AlphaEvolve achieves the highest median AUROC on Gemma-2 9B across 12 diverse classification tasks). This cross-validation across models and domains strengthens the claim that the discovered architectural principles generalize.

The broader significance is that this paper serves as an existence proof for automated safety research. In a field where much safety work involves careful human reasoning about edge cases, failure modes, and defense design, the demonstration that an evolutionary algorithm can meaningfully contribute — not just to a toy problem but to a real production deployment — opens the door to scaling safety research in ways that complement human expertise.

Innovation 5: Production-Scale Distribution Shift Testing as a Methodology for Honest Evaluation

Beyond any specific architectural contribution, the paper's evaluation methodology — testing probes against a deliberately heterogeneous set of production-relevant distribution shifts, many of which are absent from training — represents a methodological standard that the probing literature had not previously met. This is not an "innovation" in the sense of a new algorithm, but it is an intellectual contribution that changes how probe quality should be assessed.

Prior work on activation probing for safety (Cunningham, Peng, et al., 2025; McKenzie et al., 2025; Kantamneni et al., 2025) typically evaluated on datasets drawn from the same or similar distributions as training data, with adversarial robustness treated as a separate concern. This paper's evaluation grid (Table 1) is deliberately designed to surface failures that a standard i.i.d. evaluation would miss:

Long-context attacks are never seen in training — any probe that performs well here must genuinely generalize, not memorize training patterns. The paper uses a strict leakage-prevention protocol: long-context test stems that matched any short-context training stems were excluded (Appendix B.3).
Pre-existing jailbreaks (from publicly available methods like GCG; Zou, Z. Wang, et al., 2023) and automated red teaming (adversarially generated prompts) are entirely held out, testing robustness to adversarial distribution shifts.
Hard negatives — legitimate cybersecurity queries that superficially resemble attacks — test whether probes can make fine-grained distinctions that matter in production, where blocking a security researcher's query is a costly false positive.
Multi-turn conversations test whether probes can detect harmful intent that only becomes apparent over an extended dialogue, a critical capability for real deployments where users may gradually escalate.

The paper's finding that no single method dominates across all distribution shifts (Section 5.1, takeaway 2) is a direct consequence of this evaluation methodology. In Table 3, the AlphaEvolve probe achieves the best test error (2.53%) but has FNR of 32.05% on multi-turn attacks — worse than the vanilla attention probe (29.34%). The Rolling Attn with MultiMax aggregation has 2.50% test error but 42.46% FNR on adaptive red teaming — worse than the AlphaEvolve variant (27.61%). A standard single-metric evaluation would obscure these tradeoffs and potentially select a probe that looks good on average but fails catastrophically on a specific distribution shift that matters in production.

This methodology also produces the paper's most sobering finding: jailbreak success rates are extremely high across all methods, with every technique showing ≥1% vulnerability on pre-existing jailbreaks (Figure 4d). This is not presented as a failure of the paper's methods but as a statement about the inherent difficulty of the problem, consistent with Nasr et al. (2025)'s finding that adaptive attacks can bypass most existing guardrails. The paper's candid admission that "we are a long way from adversarial robustness in this domain" (Section 5.1) is made possible by an evaluation that does not allow the model to hide behind easy test cases.

The significance of this methodological contribution extends beyond this paper. It establishes a template for how probing research should be evaluated: with explicit distribution shift splits, held-out adversarial attacks, hard negative controls, and per-shift breakdowns that reveal where methods succeed and fail. As the field moves toward deploying probes in increasingly high-stakes settings (monitoring for CBRN misuse, detecting deception, preventing harmful outputs), evaluation protocols that surface brittleness before deployment become essential.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the authors' proprietary cyber-misuse datasets, described in Table 1 and Table 2. The training set consists of 3,175 short-context sequences (1,928 attack examples, 1,206 overtriggering benign examples, and 41 multi-turn attack examples; see Table 7). The validation and test sets are constructed from separate splits with no shared prompts, and the test sets intentionally represent distribution shifts absent from training: long-context data (average >900k tokens, up to 1M tokens), multi-turn conversations, pre-existing jailbreaks from publicly available methods (GCG; Zou et al., 2023), and automated red teaming prompts generated by an adversarial model. Exact sample counts per split are provided in Table 6 (Appendix B.1). Long-context test data is filtered to remove stems matching training data (Appendix B.3). All test datasets represent zero-shot evaluation under distribution shift — probes are never trained on these distributions.
Base model(s). The primary model is Gemini 2.5 Flash (Google, 2025a), a production-grade frontier language model developed by Google DeepMind. Activations are harvested from the layer halfway through the model (exact layer index unspecified). The paper also evaluates probes on Gemma-2 9B (Appendix A) to test cross-model generalization, using 12 binary classification datasets from Kantamneni et al. (2025). The choice of Gemini 2.5 Flash is motivated by its production relevance — this is the model actually deployed with probe-based misuse mitigations, making the evaluation directly reflective of real-world conditions rather than a research-only setting.
Metrics. The primary evaluation metric is the weighted test error defined in Equation (12): a weighted combination of false positive rates (FPR) and false negative rates (FNR) across all test datasets, with weights 50 for overtriggering FPR (benign prompts incorrectly flagged), 2 for hard negative FPR (legitimate security queries incorrectly flagged), and 5 for FNR (attacks missed). This asymmetric weighting reflects production priorities — overtriggering on benign traffic is considered 10× worse than missing an attack, and 25× worse than flagging a hard negative. For comparing methods along a single axis, the weighted test error is used as a scalar summary. Per-dataset FPR and FNR are also reported individually (Table 3) to reveal distribution-specific behavior. For the AlphaEvolve search, a different weighting was used (200× overtriggering, 40× long-context FPR, 6.7× cyber FPR, 4× cyber and long-context FNR; Appendix F.1) because that experiment was run before the final metric was settled. For open-source dataset experiments (Appendix A), AUROC is the primary metric.
Baselines. (1) Linear Probe (Mean aggregation) (Alain and Bengio, 2016) — a learned direction w ∈ ℝ^d applied per-token and mean-pooled. (2) Linear Probe with Rolling Exponential aggregation — linear per-token scores processed through an exponential moving average (similar to EMA probes but applied to linear outputs). (3) EMA Probe (Cunningham, Peng, et al., 2025) — post-processes linear probe scores using exponential moving average with α = 0.5 at inference, taking the max over positions. (4) Attention Probe (Kantamneni et al., 2025; Shabalin and Belrose, 2025) — softmax-weighted average of per-token MLP-transformed values, using H = 10 heads by default (with 20-head variants also evaluated). (5) Attention Probe trained on Long-Context data ("Attn (LC Trained)") — same attention architecture but trained directly on long-context activations (22× training cost increase). (6) Prompted Gemini language models — Gemini 2.5 Flash, Gemini 2.5 Pro, and Gemini 2.5 Flash Lite prompted with a fixed detection prompt (Section 3.3.1). (7) Majority voting — for Appendix A open-source experiments, logistic regression from Kantamneni et al. (2025) serves as the linear baseline reference. A custom linear probe implementation was also trained but underperformed the logistic regression baseline due to infrastructure tuned for larger architectures (Appendix A, Figure 6 note).
Generation budget / compute accounting. For probes, compute cost is measured in inference cost relative to a probe, as shown in Figure 1 and Figure 5. Probes operate on pre-existing activations (computed during the normal forward pass of Gemini), so their cost is solely the forward pass through the small probe network — approximately 1× on the relative scale. Prompted LLM classifiers (Gemini 2.5 Flash, Pro, Flash Lite) require a full additional forward pass, estimated at roughly 10,000× the probe cost. For cascading classifiers, cost is computed as (1 - f_defer) × 1 + f_defer × 10,000, where f_defer is the fraction of queries deferred to the LLM. Pro's cost estimate generously ignores thinking tokens, making the cost comparison favorable to Pro in practice. Training cost comparisons use FLOPs or wall-clock time relative to short-context training (e.g., 22× increase for long-context training). In the AlphaEvolve experiments, the search budget is ~2,500 architecture evaluations, each requiring full probe training from scratch.
Cross-validation / statistical protocol. For all probes, the authors train 100 random seeds and select the best seed based on validation loss (Equation 12 computed on held-out validation splits). The selected seed's performance is then evaluated on the test set. This is a best-of-100 selection procedure, not standard train-validation-test with a single seed. Statistical significance between architectures is assessed using bootstrap resampling: the joint distribution of (validation loss, test loss) across seeds is bootstrapped with KDE smoothing to estimate the probability that method A achieves lower test loss than method B under best-of-100 selection (Table 4). For language model baselines, binomial confidence intervals are computed analytically since there is no seed variance (temperature zero decoding). For cascading classifiers, the Pareto frontier is computed from validation data via the Minkowski sum algorithm (Appendix E), then evaluated on test data. The authors acknowledge that sweeping over threshold parameters on test data is a mild form of test-set optimization, but argue it is acceptable because the sweep is over one parameter (deferral rate) rather than model selection. Error bars in figures (e.g., Figure 1, Figure 5) represent 95% confidence intervals from bootstrap (probes) or analytic binomial (LLMs), with details in Appendix J.

Main Quantitative Results

Probe Architecture Comparison Across Distribution Shifts

Table 3 provides the core quantitative comparison of all probe architectures and LLM classifiers across nine evaluation datasets (five false positive datasets and four false negative datasets), plus the aggregated weighted test error. The headline findings span three axes: (1) improving over prior work, (2) the distribution-shift-dependence of architecture rankings, and (3) the persistent challenge of adversarial robustness.

Improvement over prior baselines: The best hand-designed probe (Rolling Attn Probe with MultiMax aggregation) achieves a weighted test error of 2.50%, compared to 6.18% for the Linear Probe Mean baseline — a 2.5× reduction in error. The AlphaEvolve-discovered probe achieves 2.53% test error. Both dramatically outperform existing architectures from prior work. For specific distribution shifts where prior probes failed catastrophically, the improvements are more dramatic: long-context attack FNR drops from 99.13% (Linear Probe Mean) and 87.88% (Attention Probe) to 0.0–1.73% for top MultiMax variants; adaptive red teaming FNR drops from 47.56% (Linear Probe Mean) and 43.39% (Attention Probe) to 22.74–22.97% for the best MultiMax variants.

No single method dominates across all distributions: Inspection of per-column results in Table 3 reveals exactly this pattern. The AlphaEvolve probe achieves the best test error (2.53%) and best FPR on short overtriggering (0.1%) and hard negatives (5.37%), but has a multi-turn attack FNR of 32.05% — worse than the vanilla attention probe's 29.34%. The Rolling Attn with MultiMax aggregation achieves 2.50% test error (best) and 0.0% FNR on long-context attacks (perfect recall), but has 42.46% FNR on adaptive red teaming — worse than the Selected Probe's 42.46% (which appears to match exactly in the table; both show 42.46%) and substantially worse than the best variant on this metric. The Gemini 2.5 Flash prompted classifier achieves the best short-context attack FNR (8.58%) but has 19.67% FPR on multi-turn hard negatives — worse than all advanced probes (e.g., AlphaEvolve at 6.56%, Rolling Attn 20H at 6.56%). This distribution-dependent ranking is the paper's core empirical justification for needing not just better architectures but also better understanding of when each architecture succeeds and fails.

Jailbreak and adversarial robustness remains severely limited: Figure 4d and the SC[J] column of Table 3 quantify the vulnerability to pre-existing jailbreaks. The best FNR on this dataset is 0.8% (MultiMax, Attn Trained), meaning at least 1 in 125 jailbreak attempts succeed against even the strongest probe. Most methods show substantially higher vulnerability: 2.40% for vanilla Attention probes, 2.41% for the Selected Probe, 3.58% for Rolling Attn with MultiMax aggregation, 8.74% for Gemini 2.5 Flash. The automated red teaming dataset (SC[ART]) shows even higher vulnerability: the best FNR is 14.15% (Linear Probe Rolling Exp), with top probes ranging from 22.74% to 27.61%, and Gemini 2.5 Flash at 20.65%. The paper explicitly states that "all techniques have a vulnerability on at least 1% of queries" on the jailbreaks dataset (Section 5.1), characterizing this as evidence that "we are a long way from adversarial robustness in this domain."

Gemini 2.5 Flash vs. Pro comparison: The prompted LLM baselines reveal an interesting capability inversion. Gemini 2.5 Pro is more conservative: lower FPR on short overtriggering (0.2% vs. 0.3%), lower FPR on hard negatives (6.71% vs. 14.09%), lower FPR on long-context random traffic (0.2% vs. 0.7%). But Pro has substantially higher FNR on attacks: 16.01% vs. 8.58% on short-context attacks, 13.42% vs. 6.06% on long-context attacks, and 33.87% vs. 20.65% on adaptive red teaming. The net effect is that Flash achieves lower test error (2.04% vs. 2.21% for Pro) despite Pro's better false positive control, because the FNR penalties (weighted 5×) dominate the FPR differences. Gemini 2.5 Flash Lite underperforms both, with test error of 3.71%, driven by 55.41% FNR on long-context attacks and 25.50% FPR on hard negatives.

The Attn (LC Trained) baseline proves capacity is not the bottleneck: The attention probe trained directly on long-context data achieves 2.38% test error (Table 3), competitive with the best architectures trained only on short-context data. Its per-distribution performance is strong: 0.6% FPR on short overtriggering, 7.04% FPR on hard negatives, 1.17% FPR on long-context random traffic, 9.40% FNR on short attacks, 1.34% FNR on long-context attacks. This demonstrates that the attention architecture can handle long contexts — the 87.88% FNR of the short-context-trained attention probe is a generalization failure, not a capacity ceiling. The 22× training cost increase (Section 5.1.2, Appendix I) is the reason this is not the recommended approach.

Statistical Significance of Architecture Rankings

Table 4 reports the bootstrap frequency with which row methods outperform column methods under best-of-100-seeds selection. Values of 1.0 indicate zero overlap between bootstrap probability mass functions — the row method outperforms the column method in every bootstrap resample. The key results: (1) All top methods (Rolling Attn Rolling Agg, Rolling Attn MultiMax Agg, AlphaEvolve) achieve bootstrap frequencies of 1.0 against Linear Mean and Linear EMA baselines — the improvements are statistically unambiguous with zero overlap in distributions. (2) Against the vanilla Attention probe, bootstrap frequencies are >0.996 (Rolling Attn, Rolling Agg), >0.91 (Rolling Attn, MultiMax Agg), and >0.96 (AlphaEvolve), exceeding the 0.95 threshold the authors use for significance. (3) Against MultiMax, all top methods exceed >0.99, indicating convergence on a performance level substantially above this early architecture. (4) Among the top methods themselves, bootstrap frequencies of outperformance are in the 0.48–0.51 range: Rolling Attn (Rolling Agg) outperforms AlphaEvolve with frequency 0.51, and outperforms Rolling Attn (MultiMax Agg) with frequency 0.48. These are not statistically significant differences — the three architectures have converged to similar expected performance under this selection procedure, and their apparent ranking in Table 3 may be noise.

Figure 3 and Figure 8 visualize the seed variance underlying these comparisons. The key quantitative finding: seed selection via best validation loss reduces test loss by 0.008 on average across architectures, with a median improvement of only 0.003. In contrast, architecture choice provides a much larger gain: comparing the median-seed performance of the best architecture (Rolling Attn, Rolling Agg at 0.025 test loss) to the worst (Linear Probe Mean at 0.064) yields a 0.039 improvement — over 13× the median seed selection gain. The practical recommendation from this analysis: "practitioners should prioritize architecture search over extensive seed tuning." However, architectures with MultiMax aggregation at evaluation time show dramatically larger seed variance (Figure 8), with interquartile ranges of 0.03–0.05 and oracle seeds achieving test loss as low as 0.019 while median seeds are at 0.076. For these high-variance architectures, extensive seed sweeping remains important.

Cascading Classifier Results

Figure 5 presents the cost-accuracy Pareto frontier for cascading classifiers combining the Selected Probe (Rolling Attn, Rolling Agg) with Gemini 2.5 Flash, Gemini 2.5 Pro, and Gemini 2.5 Flash Lite. The horizontal axis is relative inference cost (log scale), with the probe at approximately 1× and Flash at approximately 10,000×. The vertical axis is weighted test error (Equation 12), with lower being better.

Headline cascade result: The optimal operating point for the probe-Flash cascade (circled in Figure 5) achieves approximately 1.5% test error — lower than Flash alone (2.04%, Table 3) and lower than the probe alone (2.64%). This operating point defers approximately 8% of queries to Flash. The cascade configuration at this point corresponds to the "Selected Probe + 8% Flash" in Figure 1. The error bar at this point is wide, reflecting the combined uncertainty from probe seed selection and LLM test set noise (Appendix J.3). The cost of this configuration is approximately 0.92 × 1 + 0.08 × 10,000 ≈ 800 on the relative scale — roughly 12× cheaper than running Flash on all queries while achieving lower error.

Flash dominates Pro across the entire frontier: The Flash cascade curve lies below the Pro cascade curve at every operating point, meaning Flash achieves lower error at equal or lower cost. This is despite the cost estimation being generous to Pro (ignoring thinking tokens). The Pro-only operating point (right end of the Pro curve) has an error of 2.21% vs. Flash-only at 2.04%, and the optimal cascade with Pro defers to a lower error floor than Pro alone (approximately 2.0% vs. 2.21%) but still above the Flash cascade's floor. Flash Lite is dominated by Flash at higher deferral rates but is competitive at very low deferral rates (where the additional LLM cost is minimal).

The probe-alone point is on the Pareto frontier: At 0% deferral, the probe achieves approximately 2.6% test error, and this point is Pareto-optimal — no cascade configuration with non-zero deferral achieves both lower error and lower cost. This means the probe alone is not strictly dominated by any cascade, and the choice between probe-only and probe+cascade depends on the acceptable error-vs-cost tradeoff.

The optimal error-minimization point is deterministic: As proved in Appendix E.1, the point on the Pareto frontier that minimizes error unconditionally (the leftmost point of any cascade curve) corresponds to a deterministic threshold pair (τ_1, τ_2) — no randomization between policies is needed. This is relevant for security: randomized deferral policies would create an additional attack surface where adversaries could repeatedly query until they fall outside the deferral band (Best-of-N jailbreaking; Hughes et al., 2024). The fact that the optimal error-minimizing policy is deterministic avoids this vulnerability.

The paper notes that error bars are substantial and most operating points have overlapping confidence intervals, particularly in the mid-cost region. The optimal cascade points in Figure 5 are circled, and while their point estimates suggest clear ordering, the confidence intervals imply that the exact position of the Pareto frontier is uncertain.

Long-Context Training vs. Architectural Innovation

Section 5.1.2 directly compares the "Attn (LC Trained)" baseline (attention probe trained on long-context data) against the short-context-trained novel architectures. The quantitative comparison: Attn (LC Trained) achieves 2.38% test error (Table 3), competitive with the best short-context-trained architectures (AlphaEvolve at 2.53%, Rolling Attn MultiMax Agg at 2.50%). The key numbers on per-distribution FNR for long-context attacks: Attn (LC Trained) at 1.34%, Rolling Attn MultiMax Agg at 1.73%, Attn (no LC training) at 87.88%. The architectural innovations recover essentially all the long-context detection performance lost by not training on long-context data, while avoiding the 22× training cost increase.

The paper reports that long-context training "resulted in a 22× increase in training cost compared to short-context training" (Section 5.1.2), driven by memory bandwidth constraints: full long-context activation datasets cannot fit in TPU HBM, requiring complex data loading pipelines that swap between host and device memory (Appendix I). This 22× figure is crucial context for the architectural approach: if long-context training were cheap, the simplest solution would be to just train on the target distribution. The fact that it is expensive makes the architectural innovations practically necessary for production deployment.

AlphaEvolve Search Progress

Figure 10 (Appendix F.3) shows the training curve for the AlphaEvolve run, tracking best validation error and corresponding test error over 35 hours. The validation error drops sharply in the first few hours from approximately 3.0 to 1.5 weighted error, then gradually decreases to approximately 0.5 by the end. The test error follows a similar but noisier trajectory, dropping from roughly 3.5 to 2.5. The paper estimates that AlphaEvolve "closed approximately 50% of the test error gap between the attention probe baseline and perfect probe performance." The sharp early drop is attributed partially to "climbing randomness" — some seeds are naturally better on the validation metric, and AlphaEvolve quickly identifies these. The continued gradual improvement beyond this represents genuine architectural discovery.

A notable detail: the validation error and test error curves do not perfectly track — there are periods where validation error improves while test error stagnates or worsens, suggesting some overfitting to the validation distribution (which included long-context data not present in training). The final AlphaEvolve architecture (Algorithm 1) was selected from the end of this run and represents the culmination of the search.

Generalization to Other Models and Datasets (Appendix A)

Figures 6 and 7 in Appendix A evaluate the paper's probe architectures on 12 binary classification datasets from Kantamneni et al. (2025), using Gemma-2 9B (a different model family from Gemini 2.5 Flash). The evaluation uses 100 random seeds per architecture-dataset pair, reporting median AUROC (Figure 6) and best-of-100 AUROC (Figure 7).

The key quantitative findings: (1) AlphaEvolve and Attention Probe with Default aggregation achieve the highest median AUROC (0.975), followed by MultiMax variants (0.969). The logistic regression baseline from prior work achieves median AUROC of 0.944. (2) The AlphaEvolve architectures outperform the logistic regression baseline by 3.1 percentage points on average in median AUROC. (3) On specific datasets where comparison to prior work is possible: on news classification, the attention probe achieves 0.98 vs. 0.965 reported by Kantamneni et al. (2025); on Commonsense Morality, the attention probe achieves 0.94 vs. 0.86 in prior work. The improvement is attributed to the MLP preprocessing layer in the probe architecture (Section 3.1.4) and potentially to the use of 100 seeds vs. a single seed in prior work. (4) The Linear Probe Mean achieves only 0.814 median AUROC across datasets, substantially underperforming the logistic regression baseline (0.944). The authors attribute this gap to their training infrastructure being optimized for larger architectures — the learning rate schedules and regularization were tuned for attention-based probes with 10–100× more parameters, not for linear probes (Appendix A).

Table 5 provides the complete per-dataset, per-architecture median AUROC table, confirming that the architecture ranking (AlphaEvolve ≈ Attention > MultiMax variants > logistic regression > linear probe) is consistent across most datasets, though the absolute gaps vary by dataset difficulty.

Ablation Studies and Robustness Checks

Aggregation mechanism: MultiMax at evaluation vs. attention at evaluation: The "MultiMax (Attn Trained)" row in Table 3 shows what happens when an attention probe (trained with softmax aggregation) is evaluated with MultiMax aggregation (hard max). Compared to the vanilla Attention probe (trained and evaluated with softmax): long-context attack FNR drops from 87.88% to 0.0%, long-context random traffic FPR increases from 0.0% to 43.97%, and short-context attack FNR improves from 9.98% to 8.35%. This ablation cleanly isolates the effect of the aggregation function: the same trained weights perform dramatically differently depending on whether softmax or max is used at test time. The perfect long-context attack recall (0.0% FNR) demonstrates that the value vectors have learned to assign high scores to harmful tokens — the softmax normalization was the only barrier to detecting them. The catastrophic FPR increase (0.0% → 43.97%) demonstrates the cost: max is too aggressive, flagging nearly half of benign long contexts.
Training with max vs. training with softmax for MultiMax architectures: Comparing "MultiMax (Attn Trained)" and "MultiMax Trained, MultiMax Agg" in Table 3 reveals the effect of training the MultiMax architecture directly with max aggregation rather than using softmax at training time. MultiMax Trained achieves: lower FPR on short overtriggering (0.3% vs. 1.88%), lower FPR on long-context random traffic (7.76% vs. 43.97%), higher FNR on short-context attacks (15.31% vs. 8.35%), and higher FNR on pre-existing jailbreaks (5.17% vs. 0.8%). Training with max reduces false positives substantially (43.97% → 7.76% on long-context benign) at the cost of reducing sensitivity to attacks — the probe becomes more conservative. This suggests that training with softmax provides gradient signal to all tokens (via softmax weights) that helps the value vectors learn to detect harmful tokens, even if the softmax normalization at inference washes out the signal. Training with max only provides gradient to the maximal token, which may slow or alter learning.
Number of heads: 10H vs. 20H: Multiple architectures in Table 3 are evaluated with both 10-head and 20-head variants. For Rolling Attn with MultiMax aggregation: 20H achieves better short-context attack FNR (8.35% vs. 9.49% for the 10H variant — these numbers appear to compare differently; specifically, "Rolling Attn Probe (20H), MultiMax Agg" has 8.35% FNR on short attacks and 0.0% FNR on long-context attacks, while the 10H variant is not directly listed in Table 3 with MultiMax aggregation). For Rolling Attn with Rolling Agg: 20H achieves worse short-context attack FNR (11.37% vs. 8.58% for 10H), worse long-context attack FNR (5.19% vs. 3.03%), and dramatically worse adaptive red teaming FNR (50.58% vs. 42.46%). For MultiMax Trained: 20H achieves lower FPR on hard negatives (6.04% vs. 4.70%) but dramatically worse long-context random traffic FPR (82.14% vs. 7.76%). The pattern across architectures is that 20 heads do not provide clear benefits and often degrade performance, particularly for false positive control on long benign contexts, where additional heads seem to increase the probability that at least one head finds a spuriously high-scoring benign token.
Rolling window aggregation vs. full-softmax attention: The "Rolling Attn Probe, Rolling Agg" (Selected Probe) row in Table 3, compared to the vanilla "Attn" row, isolates the effect of constraining attention-weighted averaging to 10-token sliding windows followed by max across windows, versus global softmax attention. The key results: long-context attack FNR drops from 87.88% to 3.03%, long-context random traffic FPR increases from 0.0% to 1.23%, short-context attack FNR improves from 9.98% to 8.58%, and pre-existing jailbreak FNR increases from 2.40% to 2.41% (essentially unchanged). The dramatic improvement on long-context attacks (87.88% → 3.03%) confirms the diagnosis that global softmax normalization is the bottleneck — constraining normalization to 10-token windows reduces dilution by a factor of ~90,000. The modest FPR increase (0.0% → 1.23%) shows that local windows provide sufficient constraint to avoid the massive false positive problem of pure MultiMax (43.97% FPR on long-context random traffic). However, note that the jailbreak vulnerability is essentially unchanged, suggesting the architecture's benefits are specific to the context-length shift rather than adversarial robustness.
Long-context training as a brute-force alternative: The "Attn (LC Trained)" row in Table 3 vs. "Attn" row demonstrates the effect of training the attention probe directly on long-context data. The key comparison: long-context attack FNR drops from 87.88% to 1.34%, a near-complete recovery. However, multi-turn attack FNR increases from 35.91% to 29.34% (improvement), hard negative FPR drops from 10.07% to 7.04% (improvement), and adaptive red teaming FNR drops from 43.39% to 35.50% (modest improvement). The test error improves from 3.53% to 2.38%. This establishes the upper bound for what attention architectures can achieve if training cost is not a constraint. The 22× training cost increase (Section 5.1.2) quantifies the penalty for this approach.
Cascading classifier: heuristic band vs. optimal threshold-randomization policy: Section 5.2.1 describes the heuristic band method (symmetric confidence interval around a decision threshold), while Section 5.2.2 and Appendix E develop the threshold-randomization-optimal policy (independent left and right thresholds, convex hull interpolation). The paper does not provide a direct quantitative comparison of the two methods — no table or figure shows the heuristic band Pareto frontier alongside the optimal frontier. This is a notable missing ablation. The paper's argument for the optimal method is theoretical: the independence of left and right decisions enables a more expressive deferral policy that can achieve points the symmetric band approach cannot. The implication is that the "Selected Probe + 8% Flash" operating point in Figures 1 and 5 — which achieves lower error than Flash alone — would not be achievable with a symmetric band policy, but this is not experimentally verified.
Prompt optimization for prompted classifiers: Appendix G evaluates nine different detection prompts (including the original prompt from colleagues and eight manually created variants with varying detail and optimization) on the short-context overtriggering and short-context attack test sets, using Gemini 2.5 Flash, Pro, and Flash Lite. The results are shown in Figure 11 as an FPR-vs-FNR plot. The key finding: different prompts trade off along an approximately linear Pareto frontier, but no prompt strictly dominates the original on both FPR and FNR. The conclusion: "we have evidence that this original provided prompt is reasonably optimized." Manual prompt engineering does not meaningfully improve the LLM classifier's detection frontier; the models' inherent capability appears to be the limiting factor rather than prompt quality.
Adaptive red teaming: iterative attacks vs. single-turn repeats: Appendix H evaluates whether multi-step iterative jailbreaking (modeled on PAIR; Chao et al., 2025) improves over simply repeating a single-turn attack multiple times. The results in Figure 12 show that across 100 queries with 20 seeds each, the expected success probability from repeating the first step T times independently exceeds the observed success rate of the adaptive procedure for all T > 1. Formally, ∑_i P_indep(S_i, T) > ∑_i P_adapt(S_i, T) for all T > 1. The paper notes this does not contradict Chao et al. (2025) because they did not compare to this repeated-single-turn baseline. The practical implication: the adaptive red teaming dataset used in this paper is constructed from single-turn attacks (the paper's ART dataset is defined in Table 2 as single-shot narrative rewriting, with the iterative method having been abandoned due to this negative result). This finding is significant because it suggests that, at least against Gemini 2.5 Flash in the cyber domain, adaptive attack strategies that learn from failed attempts do not provide benefits beyond brute-force independent retries — a negative result that constrains expectations for automated red teaming.
Automated red teaming: cross-pollination between attack seeds: A variant tested in Appendix H provided the attacker model with k = 3 random successful and failed attacks from other seeds at prior steps (cross-pollinating attack strategies across independent attempts). This achieved ∑_i P_indep(S_i, T) < ∑_i P_adapt(S_i, T), reversing the negative result above — adaptive attacks with cross-pollination did outperform independent repeats. However, the margin was described as "negligible" and the approach was not pursued due to "significant complexity and computational cost increase."
AlphaEvolve prompt optimization for jailbreaking: Appendix H also reports using AlphaEvolve to optimize the attacker prompt for the single-turn jailbreaking task (the prompt that rewrites harmful queries into narrative contexts). This improved the single-turn jailbreak success rate from 20% to 26%. This experiment demonstrates that automated program discovery methods apply beyond probe architecture search — they can optimize attacker prompts as well, which has dual-use implications the paper does not extensively discuss.
Validation metric weighting sensitivity: Appendix K provides an alternative set of results using a 1:2:50 weighting (instead of the 5:2:50 weighting in Equation 12). Table 9 shows the results. The key differences: with 1:2:50 weighting (lower penalty for false negatives), FNRs are systematically higher and FPRs are systematically lower compared to Table 3. For example, the Rolling Attn Probe (20H, Rolling Agg) has FNR on short-context attacks of 17.63% (vs. 11.37% with 5:2:50 weighting) and FPR on short-context overtriggering of 0.11% (vs. 0.5%). The ranking of architectures is broadly similar but not identical: the Selected Probe achieves a test error of 0.88% under 1:2:50 weighting vs. 2.64% under 5:2:50. This demonstrates that the choice of error weighting has a substantial impact on both absolute performance and relative rankings — the "optimal" architecture depends on the relative costs assigned to different error types, which should be determined by deployment-specific risk assessments.
Open-source dataset evaluation: architecture transfer across models and domains: Appendix A evaluates architectures on Gemma-2 9B with 12 diverse classification datasets (spanning ethics, news, truthfulness, and text properties). The key finding from Figures 6–7 and Table 5: the architecture ranking transfers but the absolute performance gaps are smaller. Notable results: the AlphaEvolve architectures (both early and final) achieve median AUROC of 0.975, essentially tied with the Attention Probe (Default Agg) at 0.975. The gap between MultiMax variants and attention probes is smaller than in the main cyber evaluation — all architectures cluster in the 0.969–0.975 range. This suggests that the distribution-shift robustness advantages of max-based architectures are most pronounced when there is a specific shift (like context length) that softmax handles poorly, and less important when the test distribution matches the training distribution. The Logistic Regression baseline from prior work achieves 0.944, demonstrating that even on standard benchmarks, architectural improvements provide meaningful gains, but the effect size is more modest (3.1 percentage points improvement vs. potentially much larger on long-context shifts in the main evaluation).
Seed selection: best-validation vs. oracle (best-test) correlation: Figures 3 and 8 show, for each architecture, the gap between the seed selected by best validation loss and the oracle seed (the one that would be selected if test loss were known). Across architectures, the best-validation seed achieves test loss that is on average 0.010 higher than the oracle seed (Figure 3 and Table 8). For some architectures, the gap is much larger: "Rolling Attn Probe (20H), MultiMax Agg" has a best-validation test loss of 0.045 vs. an oracle of 0.022 — a 2× gap. This indicates that validation loss is an imperfect proxy for test loss, and better validation set construction (e.g., including more diverse distribution shifts) could improve seed selection. The paper notes: "It is possible that with a better validation data mix, the best-val seed would move closer to the best possible seed" (Section 4.2).
Linear probe infrastructure mismatch: The authors' custom linear probe implementation (Appendix A) achieves median AUROC of only 0.814 across 12 open-source datasets, compared to 0.944 for the logistic regression baseline from Kantamneni et al. (2025) — a 13 percentage point gap. The authors attribute this to infrastructure tuned for larger architectures (learning rate schedules, batch sizes, regularization) that are poorly suited to linear probes. This is a negative result that underscores the importance of fair baseline tuning — the apparent advantage of complex architectures over linear probes would be artificially inflated if linear baselines are not properly optimized. The paper appropriately uses the logistic regression baseline from prior work as the reference for linear methods in Appendix A.

Critical Assessment

How Well Do the Experiments Support the Paper's Central Claims?

The paper makes several major claims in its executive summary and throughout. I examine each against the experimental evidence:

Claim 1: "Existing probe architectures can be significantly improved upon" (Section 5.1, takeaway 1).

This claim is strongly supported by Table 3 and Table 4. The evidence is clear and multifaceted: (1) The best novel architectures achieve test error of 2.50–2.64% compared to 3.53% for vanilla attention probes and 6.18% for linear probes — a 28–60% relative reduction in error. (2) Bootstrap significance testing shows that top architectures outperform baselines with frequencies of >0.95 to 1.0 (Table 4) — the improvements are not attributable to seed noise. (3) The improvements are concentrated in specific distribution shifts where baselines fail catastrophically: long-context attack FNR drops from 87.88% (attention) to 0.0–1.73% (MultiMax variants); adaptive red teaming FNR drops from 43.39% to 22.74–27.61%. These are not marginal gains — they represent recovery from near-complete failure to functional performance.

However, the claim is qualified in an important way: the improvements are specific to the production distribution shift (long-context, multi-turn) that the architectures were designed to address. On the open-source benchmarks in Appendix A where the test distribution matches training, the improvements are modest (0.975 vs. 0.944 median AUROC, a 3.1 percentage point gap). The "significant improvement" over prior work is most dramatic where prior work was weakest, and more modest where it was already strong. This is consistent with the paper's own framing (the problem is distribution shift, not insufficient capacity), but it means the claim should be read as "probes can be significantly improved for production deployment under distribution shifts" rather than "probes can be significantly improved in general."

Claim 2: "Probes achieve comparable performance to LLM classifiers at a fraction of the cost" (Figure 1, Figure 4b).

Supported with an important nuance. The best probes achieve test error of 2.50–2.64% (Table 3), which is higher than Gemini 2.5 Flash's 2.04%. This is not "comparable" in a strict accuracy sense — the LLM is more accurate by a meaningful margin (0.5–0.6 percentage points of weighted error). The claim is better supported by Figure 4b, which shows that probes' error rates (2.6–3.5% for attention probes) are in the same ballpark as Flash (2.0%) at roughly 10,000× lower cost, and dramatically better than Flash Lite (3.71% error). The "comparable performance" framing is somewhat generous — a defense-in-depth system relying solely on probes would miss ~30% more attacks than one using Flash (weighted by the asymmetric cost function).

That said, the cascading results (Figure 5) strengthen this claim substantially: the probe-Flash cascade achieves lower error than Flash alone (1.5% vs. 2.04%) while using Flash on only ~8% of queries. In this configuration, the probe is not merely "comparable" — it is an essential component of a system that outperforms the LLM alone. The claim shifts from "probes are almost as good as LLMs but cheaper" to "probes enable a combined system that is better than either alone."

An important caveat: the probe's cost advantage is partially offset by the computational cost of harvesting activations during the model's forward pass, which the paper does not quantify. The dominant cost of the monitored model's forward pass is unchanged (activations are a byproduct), but there may be engineering overhead (memory, data transfer) associated with extracting and processing activations at scale. The paper does not provide end-to-end latency or throughput measurements for a deployed probe system vs. a prompted LLM classifier, which would be necessary for a complete cost comparison in production.

Claim 3: "No single method dominates across all distribution shifts" (Section 5.1, takeaway 2).

Very strongly supported by Table 3. The evidence is visible in every column: AlphaEvolve has the best test error (2.53%) but poor multi-turn attack FNR (32.05%); Rolling Attn with MultiMax Agg has the best long-context attack FNR (1.73%) but poor adaptive red teaming FNR (42.46%); the vanilla Attention probe has perfect long-context benign FPR (0.0%) but catastrophic long-context attack FNR (87.88%). Even Gemini 2.5 Flash, the strongest single classifier overall, has 19.67% FPR on multi-turn hard negatives — flagged legitimate security conversations at a rate that would be unacceptable in many production contexts.

This finding is robust because it is not a subtle statistical nuance — the differences in per-distribution performance are frequently large (20–50 percentage point gaps between the best and worst method on a given dataset). It holds across probe architectures, LLM classifiers, and their combinations. The finding's significance goes beyond architecture comparison: it establishes that any production deployment should maintain multiple evaluation metrics disaggregated by distribution shift, and that aggregate test error can mask critical vulnerabilities.

A limitation: the finding that "no single method dominates" is partly a consequence of the specific error weighting in Equation (12), which trades off different error types. A user who values false positive suppression much more than false negative reduction would see a different ranking — e.g., Gemini 2.5 Pro would dominate under such a weighting. The paper's Appendix K demonstrates that changing the weights indeed changes rankings. The claim is thus conditional on the specific production priorities encoded in the weighting function.

Claim 4: "Jailbreak success rates are extremely high" and "we are a long way from adversarial robustness" (Section 5.1, takeaway 3; Figure 4d).

Strongly supported. The evidence in Table 3 and Figure 4d is unambiguous: pre-existing jailbreaks achieve ≥0.8% FNR against the best probe (meaning at least 1 in 125 attempts succeed), and most methods are in the 2–9% range. Automated red teaming is worse: the best FNR is 14.15%, with most methods in the 20–50% range. The paper does not attempt to minimize these numbers — it presents them as evidence of a fundamental limitation.

The strength of this claim derives from the setup of the evaluation rather than the numbers themselves. The jailbreaks are from publicly available methods (GCG; Zou et al., 2023) and are held-out from all training and validation. The automated red teaming uses an adversarially optimized prompt (improved via AlphaEvolve from 20% to 26% success rate). These are realistic threat models, not toy attacks. The finding that existing defenses remain vulnerable even to non-adaptive attacks (pre-existing jailbreaks requiring no knowledge of the defense) is sobering and credible.

An important limitation: the paper evaluates only input monitoring probes, not output monitoring. A complete defense system would monitor both inputs (to detect harmful requests) and outputs (to detect harmful responses). The jailbreak success rates reported here represent the vulnerability of the input monitoring layer alone. A determined adversary might still face output monitoring, reducing the effective attack success rate in deployment. The paper does not quantify the marginal benefit of input monitoring over a system without it, which limits the practical interpretation of these numbers.

Claim 5: "Long-context generalization can be achieved through architectural design without long-context training" (Section 5.1.2).

Supported with a clear quantitative comparison. The Attn (LC Trained) baseline achieves 2.38% test error at 22× training cost, while short-context-trained architectures achieve 2.50–2.64% test error at 1× training cost. The 0.12–0.26 percentage point gap in test error is modest relative to the 22× cost difference. For practical deployment, trading a small accuracy hit for a massive training cost reduction is clearly worthwhile.

However, the claim is specific to the architectures discovered in this paper — it does not establish that any architectural innovation can substitute for in-distribution training. The paper found specific aggregation mechanisms (max-pooling, rolling windows) that work, but it does not provide a general theory of when architecture can replace training data. The finding is empirical and may not transfer to other distribution shifts (e.g., domain shifts rather than length shifts) or other models.

Claim 6: "Cascading classifiers achieve optimal accuracy at a fraction of the LLM's inference cost" (Section 5.2, Figure 5).

Supported with qualifications. The optimal probe-Flash cascade achieves lower error (1.5%) than Flash alone (2.04%) while invoking Flash on only ~8% of queries — a genuine accuracy improvement at lower cost. This is the strongest result in the paper from a cost-effectiveness perspective.

However, the error bars in Figure 5 are substantial, and the paper acknowledges that "most operating points have overlapping confidence intervals" (Figure 5 caption). The point estimate of 1.5% for the cascade vs. 2.04% for Flash alone is the best guess, but the uncertainty intervals overlap, meaning we cannot reject the hypothesis that Flash alone is equivalent to the cascade. This is partially a consequence of the limited test set size (e.g., 918 short-context attack examples, 320 multi-turn attack examples) and the high variance of probe seed selection. More test data would be needed to establish statistical significance of the cascade's superiority.

Additionally, the cascade's O(N log N) optimal policy derivation (Appendix E) is computed on the validation set, and the resulting thresholds are evaluated on the test set. Since the thresholds are optimized on validation data (by sweeping over all possible threshold pairs), there is a risk of overfitting to the validation set, though the test set is independent. The paper's argument that this is acceptable because only one parameter (deferral rate) is swept is reasonable but not a rigorous guard against overfitting.

Genuine Weaknesses in Experimental Design

Single model family, single domain. All main experiments use Gemini 2.5 Flash on proprietary cyber-misuse datasets. While Appendix A shows transfer to Gemma-2 9B on open-source datasets, those datasets lack the distribution shifts (long-context, multi-turn, jailbreaks) that are the paper's central focus. We do not know whether MultiMax's benefits on long-context cyber prompts transfer to (say) long-context CBRN detection or long-context code vulnerability detection. The paper acknowledges this limitation explicitly (Section 7: "Future research could evaluate on several different domains beyond cyber misuse").

No comparison to multi-layer probes. Concurrent work by Cunningham, Wei, et al. (2026) shows that probes harvesting activations from multiple layers can improve performance. The paper acknowledges this as a limitation (Section 7) but does not test whether multi-layer variants of their architectures would provide further gains or change architecture rankings. It is possible that with multi-layer features, the aggregation mechanism matters less because the model can learn to route information through different layers.

Validation set construction for seed selection. The best-of-100-seeds selection procedure uses validation loss computed on datasets that intentionally differ from training but may still be insufficiently diverse. As noted in Section 4.2, "the best possible seed on the test set is frequently much better than the seed we choose via our validation dataset," and for high-variance architectures like Rolling Attn (20H) with MultiMax Agg, the best-validation seed has test loss 0.045 vs. oracle 0.022 — a 2× gap. This suggests the validation set does not adequately represent the test distribution, and better validation set construction could substantially improve selected models. This also means that the reported test losses may underestimate what the architectures can achieve with better seed selection.

The "no single method dominates" finding may partially reflect noise. With only 500–2,000 test examples per distribution shift (Table 6), the per-column rankings in Table 3 have substantial estimation error. Apparent "tradeoffs" between architectures (Method A better on distribution X, Method B better on distribution Y) may be sampling noise rather than genuine complementarity. The paper does not report per-distribution confidence intervals or test whether per-column differences are statistically significant, making it difficult to distinguish genuine tradeoffs from noise.

Training data size is small (3,175 sequences). While appropriate for a specialized production system, this limits the generality of findings about probe scaling. We do not know whether architecture rankings would change with 10× or 100× more training data, or whether data augmentation could substitute for architectural innovation (e.g., synthetically generating long-context training examples from short ones).

Missing ablation: what happens if you train MultiMax with softmax but at higher temperature? An alternative approach to the softmax dilution problem is to use a lower softmax temperature (making attention weights spikier). The paper does not test whether temperature-scaled attention probes can recover some of MultiMax's benefits without its false positive costs. This would help distinguish whether the key property is the max operator itself (hard selection) or simply sharper attention distributions.

No latency measurements for cascading classifiers. The paper measures cost in "relative inference cost" (a proxy for FLOPs) but does not report wall-clock latency. For a cascading system where 8% of queries are deferred to a separate LLM call, the worst-case latency (for deferred queries) includes both the probe evaluation and the LLM inference. In a synchronous serving system, this could create tail latency issues that are more problematic than average cost increases.

6. Limitations and Trade-offs

The Difficulty Estimation Analogy: This Paper Has No Difficulty Estimation Problem, But the Domain-Specific Nature of the Findings Creates an Analogous Limitation

The constraint. The paper's entire evaluation is conducted on a single domain (cyber-misuse detection) with a single model family (Gemini 2.5 Flash). While Appendix A extends to Gemma-2 9B across 12 open-source classification tasks, those datasets lack the production-relevant distribution shifts (long-context, multi-turn conversations, jailbreaks) that are the paper's central contribution. The paper never evaluates whether MultiMax or Max of Rolling Means architectures transfer to long-context detection in other misuse domains—CBRN, fraud, self-harm, or prompt injection—or to other model families. The authors acknowledge this explicitly:

"Future research could evaluate on several different domains beyond cyber misuse and academic datasets (Appendix A), to produce exact recommendations." (Section 7)

The consequence. A practitioner deploying probes for a different misuse domain cannot confidently adopt the paper's architectural recommendations without independent validation. The cyber domain has specific properties that may interact with the architectural innovations in non-obvious ways: harmful cyber prompts often contain distinctive technical vocabulary (e.g., "exploit," "payload," "SQL injection"), making per-token max-pooling particularly effective because individual tokens carry strong signal. In domains where harmfulness emerges from subtle semantic combinations rather than distinctive keywords (e.g., radicalization content, coordinated disinformation), max-pooling over single tokens may be less effective or may require different window sizes. The paper provides no guidance on how to select architecture, aggregation method, or window size for a new domain without repeating the full ~2500-architecture AlphaEvolve search.

Furthermore, the paper's finding that "no single method dominates across all distribution shifts" (Section 5.1) may be domain-specific—the ranking of architectures could change substantially for a different base model or a different harm category. Table 3's per-column analysis reveals that architecture rankings shift even within the cyber domain (e.g., AlphaEvolve dominates on FPR control but is worse than vanilla attention on multi-turn attack detection). Without cross-domain replication, we cannot know whether these patterns generalize or whether each new domain requires its own architecture search.

What evidence exists in the paper. The evidence for domain-specificity is primarily absent rather than present—the paper demonstrates strong performance on one domain and provides no evidence for others. Appendix A evaluates architectures on Gemma-2 9B, but the 12 datasets (ethics, news classification, truthfulness, text properties) represent standard probing benchmarks without production distribution shifts. On these benchmarks, all architectures cluster in a narrow range (median AUROC 0.969–0.975, Table 5), suggesting that the architectural innovations' benefits are most pronounced under the specific distribution shifts present in the cyber deployment and less important for in-distribution evaluation. This pattern is consistent with the architectures being designed for robustness to context-length shifts, which are absent in the Appendix A benchmarks.

Mitigation status. The paper partially addresses this by including the Gemma-2 9B evaluation (Appendix A), which demonstrates that the codebase and general approach transfer to a different model family. However, this evaluation lacks the critical distribution shifts that motivate the architectural innovations, so it does not test whether the benefits transfer. The authors explicitly flag this as future work (Section 7) but do not provide even preliminary results on additional misuse domains. A practitioner seeking to deploy probes for CBRN monitoring, for instance, would need to independently verify whether MultiMax architectures provide similar long-context robustness gains in that domain.

Difficulty Estimation Cost Is Unmeasured but Potentially Dominant in Production

The assumption. The paper's probe-only cost estimates (the ~10,000× advantage over LLM classifiers shown in Figure 1) assume that activations are harvested "for free" during the monitored model's normal forward pass. However, the paper does not measure or account for several practical costs that arise in production deployment:

Activation harvesting infrastructure: Extracting hidden states from a specific layer of a production serving system requires engineering work to intercept activations mid-forward-pass and route them to the probe. This may involve additional memory bandwidth, data transfer between accelerators, or modification of the serving stack. The paper mentions that the Gemini Introspection workstream at Google collaborated on this integration (Acknowledgements, Section 8) but provides no quantification of the engineering cost or inference-time overhead.
Multi-turn monitoring state: For multi-turn conversations, the probe must maintain state across turns. The paper evaluates multi-turn probes (Table 3, MT[A] and MT[HN] columns) but does not describe the serving infrastructure for storing and updating probe internal state across multiple requests. Attention probes can be updated incrementally (Appendix L), but this still requires the serving system to associate conversation state with the probe's accumulator variables.
Latency, not just throughput: The paper measures cost in "relative inference cost" (a proxy for FLOPs) but does not report wall-clock latency. For synchronous serving systems where the user waits for a response, the probe's evaluation time—while small in FLOPs—may add to the critical path latency. If probe evaluation takes even 5–10ms, that may be non-trivial relative to the model's time-to-first-token. The cascading classifier introduces an even larger latency concern: when a query is deferred to the LLM (8% of cases in the optimal configuration), the user experiences the latency of both the probe evaluation and the separate LLM classifier inference, which could be hundreds of milliseconds.

The consequence. The headline cost advantage (~10,000× cheaper than LLM classifiers) may overstate the practical cost savings in a deployed system. If activation harvesting adds 10% overhead to the serving infrastructure, the effective cost advantage shrinks to ~9,000×—still large, but the qualitative conclusion doesn't change. A more serious concern is that latency, not FLOPs, is the binding constraint in many production serving systems. If probe evaluation adds latency to every request (or creates a bimodal latency distribution where 8% of requests are substantially slower due to cascade deferral), user experience may degrade even if total compute cost decreases. The paper's cost model assumes that FLOPs are the scarce resource; in practice, tail latency SLOs (service-level objectives) often dominate deployment decisions.

What evidence exists in the paper. The paper provides essentially no evidence on these practical deployment costs. There are no end-to-end latency measurements, no quantification of activation harvesting overhead as a fraction of total inference cost, and no discussion of serving infrastructure requirements beyond the training infrastructure described in Appendix I. The paper's cost model is strictly a training-and-inference FLOPs accounting, not a deployment engineering cost model. This is appropriate for a research paper but creates a gap between the paper's claims and what a practitioner needs to know before committing to a probe-based monitoring architecture.

Mitigation status. The paper partially addresses the training cost side (Appendix I quantifies the 22× training cost increase for long-context training and provides infrastructure recommendations) but does not address deployment inference costs. The authors acknowledge that their work informed "successful deployment... into production" (Section 1), implying that Google has solved the infrastructure challenges internally, but they provide no public quantification of the deployment cost. A footnote or appendix section estimating end-to-end latency overhead would substantially strengthen the practical claims.

Adversarial Robustness Is Fundamentally Unsolved—Probes Offer No Meaningful Protection Against Determined Attackers

The constraint. The paper demonstrates—and is admirably candid about—that activation probes, like all other evaluated classifiers (including Gemini 2.5 Flash and Pro), are highly vulnerable to adversarial attacks. The evidence is clear and damning:

"In this cyber-offensive query domain, jailbreak success rates are extremely high... on the pre-existing jailbreaks dataset, all techniques have a vulnerability on at least 1% of queries. This means that we are a long way from adversarial robustness in this domain." (Section 5.1)

The numbers in Table 3 quantify this vulnerability: pre-existing jailbreaks (publicly available methods like GCG) achieve FNR ranging from 0.8% (MultiMax, Attn Trained) to 8.74% (Gemini 2.5 Flash) against various classifiers. Automated red teaming is worse, with FNR ranging from 14.15% to 50.58% across methods. The paper's adaptive red teaming experiments (Appendix H) demonstrate that iterative jailbreaking does not outperform repeated single-turn attacks, but the absolute success rate remains high regardless of the attack strategy.

Critically, the paper does not claim to have solved or substantially mitigated this vulnerability:

"Our techniques do not significantly reduce the success rate of adaptive adversarial attacks; indeed, recent work argues that this is extremely challenging, if not impossible (Nasr et al., 2025)." (Section 1)

The consequence. A security practitioner reading this paper must understand that probes provide no meaningful protection against a determined adversary who is willing to invest effort in crafting attacks. The probes serve as a cost-effective filter for unsophisticated misuse (users who naively ask "how do I hack this server?") but cannot reliably stop an attacker who uses publicly available jailbreak techniques or automated red teaming tools. This is a fundamental limitation, not a fixable weakness—the paper cites Nasr et al. (2025)'s argument that adaptive attacks with knowledge of the defense can bypass most existing guardrails.

The operational implication: probes must be deployed as one layer in a defense-in-depth strategy, not as a standalone security mechanism. The paper explicitly positions them this way (Section 7: "our work can supplement other empirical studies and user-level strategies"), but a reader focused on absolute security might misinterpret the strong accuracy numbers on non-adversarial distributions as implying robustness. The paper does not provide guidance on how to combine probes with other defenses (output monitoring, rate limiting, user authentication, human review) to achieve acceptable security against adaptive attackers.

What evidence exists in the paper. The evidence is extensive and unambiguous:

Figure 4d shows pre-existing jailbreak FNR across methods, with all techniques above 1% and most in the 2–9% range.
Table 3, SC[J] and SC[ART] columns provide per-method jailbreak and automated red teaming FNR. The best single number on pre-existing jailbreaks is 0.8% (MultiMax, Attn Trained), and on automated red teaming is 14.15% (Linear Probe Rolling Exp).
Figure 12 (Appendix H) demonstrates that multi-step iterative attacks do not improve over repeated single-turn attacks, but the baseline success rate (20–26%) is high.
Appendix H reports that AlphaEvolve optimization of the attacker prompt improved jailbreak success from 20% to 26%, showing that even modest automated optimization can meaningfully increase attack success.
The paper explicitly cites Nasr et al. (2025) as evidence that this is a fundamental property of current defenses, not a specific weakness of probes.

Mitigation status. The paper does not attempt to mitigate this limitation—it accepts it as a feature of the current state of the field and is transparent about it. The recommendations are architectural (use probes as part of a cascade, combine with other defenses) rather than algorithmic (no new adversarial robustness technique is proposed). Future work on adversarially robust probing is not discussed in Section 7, suggesting the authors view this as a separate research direction rather than an extension of their current approach. This is honest but leaves practitioners without actionable guidance beyond "don't rely on probes alone against sophisticated attackers."

The Seed Selection Protocol Overstates Probe Reliability—The Validation Set Does Not Adequately Represent Test Distributions

The assumption. All probe results in Table 3 and Figure 1 are reported after a best-of-100-seeds selection procedure: train 100 random initializations, select the single seed with lowest validation loss (Equation 12 on held-out validation splits), and report that seed's test performance. This procedure assumes that the seed with best validation loss will also perform well on the test set—that validation loss is a good proxy for test loss. The paper's own analysis suggests this assumption is violated for several architectures.

The consequence. The reported test losses may not reflect the performance that would be obtained in a real deployment, where you cannot select among 100 seeds based on validation data that may not match the deployment distribution. Specifically:

The validation set does not include long-context data (Table 1: long-context data is only in the test split). A seed that performs well on short-context validation may generalize poorly to long-context test distributions—yet it will be selected anyway because validation loss doesn't penalize long-context failures.
For high-variance architectures, the best-validation seed may be far from the best-test seed. Figure 8 (Appendix D) shows that for "Rolling Attn Probe (20H), MultiMax Agg," the best-validation seed has test loss of 0.045, while the oracle (best-test) seed has test loss of 0.022—a 2× gap. The validation procedure is leaving substantial performance on the table by selecting suboptimal seeds.
The gap between best-validation and oracle seeds is architecture-dependent. Table 8 shows that the gap ranges from 0.001 (Rolling Attn, Rolling Agg) to 0.031 (Rolling Attn, 20H, MultiMax Agg). This means the validation procedure penalizes high-variance architectures more severely—their reported performance understates their potential—and may cause practitioners to select architectures with lower variance but lower potential ceiling.
The paper's conclusion that "architecture choice provides a much larger gain" than seed selection (Section 4.2) is based on median seeds, not oracle seeds. If oracle seeds were compared (which would require a perfectly representative validation set), the architecture ranking might change, particularly for high-variance architectures that benefit most from ideal seed selection.

The practical consequence: a practitioner who trains probes with the recommended architectures but uses fewer seeds (e.g., 5–10 rather than 100) may get substantially worse performance than reported, because they have a lower probability of finding a good seed. The paper's seed selection procedure is a form of optimizer overfitting—it selects the architecture-seed combination that performs best on validation, but the validation set is insufficient to ensure this combination transfers to the deployment distribution.

What evidence exists in the paper. The evidence is in the paper's own analysis:

Figure 3 vs. Figure 8: The gap between best-validation and oracle seeds is visible in every architecture, ranging from modest (0.003 test loss for Linear Probe Mean) to dramatic (0.031 for Rolling Attn 20H MultiMax Agg).
Table 8: Provides exact numbers for the gap across architectures.
Section 4.2: The paper acknowledges that "the best possible seed on the test set is frequently much better than the seed we choose via our validation dataset. It is possible that with a better validation data mix, the best-val seed would move closer to the best possible seed."
Table 1: Shows that the validation split excludes long-context data and jailbreaks, which are present in the test split. This means validation loss cannot penalize poor generalization to these critical distributions.

Mitigation status. The paper partially acknowledges this limitation (Section 4.2) but treats it primarily as a suggestion for future work rather than a methodological weakness that affects the reported results. The recommendation is qualitative—"better validation data mix"—without specifying what mix would close the gap. The paper does not attempt to construct a validation set that better represents the test distribution (e.g., by including a subset of long-context data in validation), which would require sacrificing some test data but might produce more reliable seed selection. The practical implication—that users should sweep many seeds—is inconsistent with the paper's own recommendation to "prioritize architecture search over extensive seed tuning" (Section 4.2), since for high-variance architectures, extensive seed tuning is essential to approach reported performance.

The Revision Model Analogy: Output Monitoring Is Not Evaluated, Leaving Half the Security Picture Incomplete

The constraint. The paper studies input monitoring exclusively—detecting harmful prompts before the model generates a response. It does not evaluate probes for detecting harmful model outputs during or after generation. This is an explicit scope limitation:

"In this work, we only study input monitoring techniques, although we note that training probes on model outputs is an important future direction in Section 7." (Section 1)

The paper's cascading classifier, probe architectures, and evaluation framework are all designed for the input monitoring setting. The datasets (Table 2) consist of prompts and their harmfulness labels; there is no dataset of model responses and their harmfulness labels. The probe's inference-time behavior—harvesting activations during the model's forward pass and classifying immediately—is naturally suited for input monitoring but could be extended to monitoring generation step-by-step, a possibility the paper does not explore.

The consequence. Input monitoring alone cannot provide complete protection against misuse. Consider the following threat models that input monitoring cannot address:

Benign-appearing prompts that induce harmful outputs: An attacker might ask a seemingly innocent question that causes the model to generate harmful content (e.g., "Write a story about a character who discovers a security vulnerability" that leads to a detailed exploit description). The input monitor sees a benign prompt and does not intervene; the harmful output is generated unimpeded.
Prompt injection during generation: In multi-turn conversations, an attacker might inject harmful instructions in later turns after establishing benign context. Input monitoring sees each turn independently and may not detect the cumulative harmfulness that emerges over the conversation.
Model-generated harmful content in response to borderline prompts: Some prompts may be ambiguous—legitimate in some contexts, harmful in others. The model might interpret an ambiguous prompt harmfully and generate dangerous content. Input monitoring would either overtrigger (blocking legitimate uses) or under-trigger (allowing harmful outputs).
Jailbreak amplification in outputs: Even when input monitoring catches some jailbreak attempts (Table 3 shows 0.8–8.74% FNR for pre-existing jailbreaks), the 1–9% that succeed can produce harmful outputs. Output monitoring could catch these at generation time and halt the response, providing a second line of defense that input monitoring alone cannot offer.

The consequence for the paper's security claims: the statement that probes "have directly informed GDM's recent successful deployment of cyber attack misuse mitigation probes into production" (Section 1) describes an input monitoring deployment. The paper provides no evidence about the overall security of the system that includes output monitoring, human review, or other defense layers. Readers evaluating the paper's contribution to AI safety must understand that the measured failure rates (e.g., 2.50% test error) represent vulnerability of the input monitoring layer only; the overall system vulnerability could be either higher (if output monitoring is absent or weak) or lower (if output monitoring catches what input monitoring misses).

What evidence exists in the paper. The paper provides no evidence on output monitoring whatsoever. There are no experiments, no datasets, no architectures designed for streaming generation-time classification. Appendix L's incremental attention probe inference algorithm (updating the probe state token-by-token during generation) is naturally suited for output monitoring but is presented as a cost-efficiency technique for input monitoring rather than a capability for monitoring generation. The paper does not discuss whether the architectural innovations (MultiMax, rolling windows) would transfer to the output monitoring setting or whether new challenges would arise (e.g., the probe needing to make decisions before the full output is generated).

Mitigation status. The paper explicitly defers output monitoring to future work (Section 7) and does not attempt to address it. This is a reasonable scope limitation for a paper focused on input monitoring, but it means the paper's security claims must be interpreted as applying only to the input monitoring layer, not to a complete misuse mitigation system. A practitioner building a defense-in-depth system would need to independently evaluate whether the paper's architectural recommendations apply to output probes, or whether different architectures are needed for the generation-time setting.

Single-Layer Activation Harvesting May Miss Complementary Information Available Elsewhere in the Model

The constraint. The paper harvests activations from a single fixed layer—"the layer halfway through the model" (Appendix C)—for all probe architectures. This choice is motivated by prior work showing that middle layers contain rich task-relevant representations. However, the paper does not experiment with multi-layer probes, despite concurrent work by Cunningham, Wei, et al. (2026) demonstrating that combining activations from multiple layers can improve probe performance. The paper acknowledges this explicitly as a limitation:

"The limitations of our work include that we do not evaluate against probes using activations from every layer of the model as applied in concurrent work (Cunningham, Wei, et al., 2026), though potentially our findings and methodology (such as using AlphaEvolve) could be combined." (Section 7)

The consequence. The paper's architectural innovations—MultiMax, rolling window max, AlphaEvolve-designed architectures—are all evaluated in the single-layer setting. It is unknown whether these architectures would benefit from or be rendered unnecessary by multi-layer features. Several possibilities:

Multi-layer features might reduce the need for sophisticated aggregation: If activations from later layers already encode sequence-level information that is robust to context length (because later layers have processed information through the full Transformer stack), then the aggregation mechanism may matter less—a simple mean-pool over late-layer activations might work as well as a sophisticated max-based architecture over mid-layer activations.
Multi-layer features might amplify the benefits of sophisticated aggregation: Conversely, combining MultiMax aggregation with features from multiple layers might provide complementary signals—early layers might detect surface-level lexical patterns (specific harmful keywords), while later layers might capture semantic harmfulness that requires deeper processing. The max operation over early-layer token features combined with mean-pool over late-layer sequence features might outperform either approach alone.
The optimal architecture might be layer-specific: Different layers might benefit from different aggregation strategies. The paper's finding that "no single method dominates across all distribution shifts" (Section 5.1) might extend to layers: MultiMax could be optimal for layers where harmful signal is sparse across token positions, while attention-based aggregation could be optimal for layers where harmful signal is distributed.

Without multi-layer experiments, the paper's architectural recommendations are contingent on the (unverified) assumption that the single middle layer provides sufficient information for the classification task. If critical harmfulness information is represented in early layers (lexical patterns) or late layers (semantic understanding), a single-layer probe may be fundamentally capacity-limited regardless of architecture.

What evidence exists in the paper. The paper provides no experiments varying the layer from which activations are harvested. Appendix C states that "Source Layer: Activations harvested from the layer halfway through the model" without ablation over layer choice. The Attn (LC Trained) baseline demonstrates that a single-layer attention probe has sufficient capacity to handle long contexts when given appropriate training data (2.38% test error, Table 3), but this does not rule out that multi-layer probes could achieve better performance, or that the architectural innovations' benefits are specific to the single-layer setting.

The paper does cite Cunningham, Wei, et al. (2026) as concurrent work that uses multi-layer probes, acknowledging that this is a relevant comparison that was not performed. However, the paper does not provide even a preliminary experiment (e.g., concatenating activations from two layers and evaluating with a simple architecture) that would shed light on whether multi-layer features would change the conclusions.

Mitigation status. The paper treats this as a limitation to be addressed in future work (Section 7) and suggests that their methodology (AlphaEvolve architecture search) could be combined with multi-layer probing. However, the single-layer choice is not presented as a tradeoff—it is simply stated as the experimental setup. A practitioner reading this paper would not know whether to allocate engineering effort to implementing multi-layer activation harvesting (which increases storage and serving complexity linearly with the number of layers) or to focus on single-layer architecture optimization. The lack of even a small-scale multi-layer baseline experiment means this decision must be made without evidence from the paper's framework.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the paradigm for activation probing from "probes are research tools for understanding model internals" to "probes are production systems that must be engineered for robustness, not just accuracy." The shift is methodological rather than algorithmic: the paper's primary contribution is demonstrating that the right aggregation function—not more training data, larger models, or better feature extraction—is the key to making probes work under the distribution shifts that real deployments create.

This is a genuine reframing of the probes-as-safety-tools research agenda. Prior work (Cunningham, Peng, et al., 2025; Kantamneni et al., 2025) established that probes can detect harmful content, but implicitly treated probe design as a feature extraction problem: find the right layer, the right dimensionality, the right training objective. This paper demonstrates that for the specific and practically critical challenge of long-context generalization, feature extraction is not the bottleneck—the vanilla attention probe with MLP preprocessing has sufficient capacity when trained on long-context data (2.38% test error, Table 3, Attn (LC Trained)). The bottleneck is the aggregation mechanism's failure to be scale-invariant to context length. This reframes probe engineering around aggregation function design, where the key property is whether the function preserves signal strength as sequence length grows by orders of magnitude.

The paper also reconciles a tension in prior work that was largely implicit. On the one hand, attention probes (Kantamneni et al., 2025; Shabalin and Belrose, 2025) represented the state of the art because they reuse the Transformer's own attention mechanism—a natural and elegant design choice. On the other hand, the long-context failure documented here (87.88% FNR on long-context attacks for attention probes trained on short data, Table 3) raises the question of why such a natural design could fail so catastrophically. The paper's diagnosis resolves this: attention is designed for information integration across positions (the model needs to combine information from many tokens to predict the next one), but classification requires detecting sparse signals where a few tokens should dominate the output regardless of context length. These are fundamentally different requirements that the same mechanism cannot simultaneously satisfy. This explains why probes that "think like the model" may be suboptimal for monitoring tasks—a finding that should redirect research away from mirroring the base model's architecture and toward purpose-built aggregation for the specific demands of safety classification.

The paper also changes the cost landscape for AI safety monitoring. Before this work, a practitioner deciding how to monitor a deployed model faced an uncomfortable choice: use a prompted LLM classifier (accurate but ~10,000× more expensive, potentially doubling total inference costs) or use a probe (cheap but unreliable under production distribution shifts). The paper's cascading classifier results demonstrate that this is a false dichotomy: the optimal system uses both, with the probe handling confident cases and the LLM handling the rest, achieving lower error than the LLM alone (1.5% test error for the cascade vs. 2.04% for Flash alone, Figure 5) while deferring to the LLM on only ~8% of queries. This is not merely a cost-efficiency improvement—it is an accuracy improvement from combining classifiers with complementary error patterns. The implication for the field is that safety monitoring should be designed as a system architecture problem (how to compose cheap and expensive classifiers) rather than a model selection problem (which single classifier to use). The paper's finding that deterministic threshold pairs achieve optimal error minimization without randomization (Appendix E.1) further strengthens this: the cascade can be made secure against Best-of-N jailbreaking (Hughes et al., 2024) while still achieving Pareto-optimal cost-accuracy tradeoffs.

The AlphaEvolve experiments change the landscape in a more speculative but potentially transformative way: they provide an existence proof that automated program discovery can contribute meaningfully to AI safety research. The fact that AlphaEvolve independently converged on max-based aggregation (both the early and final architectures use it in some form) validates the human-designed MultiMax insight while also discovering refinements (gated projections, bipolar pooling, orthogonality regularization) that human designers might not have prioritized. This opens the door to a research methodology where safety mechanisms are discovered at scale through automated search, complementing human intuition with computational exploration. The meta-implication is significant: as models become more capable, the safety mechanisms needed to monitor and control them may become too complex for manual design, and automated search (AlphaEvolve-style or similar) may become not just helpful but necessary.

The finding that no single method dominates across all distribution shifts (Section 5.1, takeaway 2) should change how the field evaluates probes. A single aggregate metric can hide catastrophic failures on specific shifts—the vanilla attention probe has reasonable aggregate performance (3.53% test error) but completely fails on long-context attacks (87.88% FNR). The paper's evaluation grid (Table 1), which deliberately tests on distributions absent from training, establishes a higher bar for future probing research: evaluations must be disaggregated by distribution shift, and architectures must be tested on shifts they were not designed for. This is methodologically important because it surfaces brittleness that standard i.i.d. evaluation would miss.

Finally, the paper's candid documentation of adversarial vulnerability—≥1% attack success rate on pre-existing jailbreaks for all methods, 14–50% FNR on automated red teaming—provides a sobering calibration for the field. It establishes a clear upper bound on what current input monitoring techniques can achieve against determined adversaries. This should redirect research effort: rather than incrementally improving non-adversarial accuracy, the priority should be on adversarial robustness, output monitoring (to catch what input monitoring misses), and defense-in-depth architectures that combine multiple imperfect layers. The paper's own techniques represent substantial progress on non-adversarial distribution shifts (long-context, multi-turn) but essentially no progress on adversarial shifts—a sharp boundary that future work must address.

Follow-Up Research This Work Enables

Adaptive and multi-layer cascading with learned deferral policies. The paper's threshold-randomization-optimal cascade uses probe logit thresholds to decide when to defer to the LLM, with thresholds optimized on validation data. However, this approach is limited to a single deferral decision based on a single scalar (the probe score). A natural extension is learned deferral: train a small router network that takes the probe's internal state (not just its final scalar output) and decides whether to defer, potentially using per-head scores, activation statistics, or uncertainty estimates. This could capture cases where the probe is confidently wrong in systematic ways—for example, the probe might consistently misclassify certain types of hard negatives, and the router could learn to recognize these cases and defer them. Madras et al. (2018) and Mozannar and Sontag (2020) provide theoretical frameworks for learning to defer; applying these to the probe-LLM cascade with the paper's production datasets would test whether learned routing can improve on threshold-based routing. Additionally, the cascade could be extended to multiple layers: a sequence of increasingly expensive probes (Oldfield et al., 2025) or a probe → small LLM → large LLM hierarchy, with each layer handling progressively harder cases.

Multi-layer probing with aggregation-function optimization per layer. The paper's architectural innovations are all evaluated using activations from a single middle layer. Concurrent work by Cunningham, Wei, et al. (2026) demonstrates that combining activations from multiple layers improves probe performance, but does not explore whether different layers benefit from different aggregation functions. A strong follow-up would: (1) harvest activations from 3–5 layers spanning early, middle, and late positions; (2) train separate probes per layer with different aggregation mechanisms (mean-pool, attention, MultiMax, rolling window max); (3) combine per-layer scores through a learned weighting or a small MLP; (4) test whether the optimal aggregation function varies systematically by layer depth (e.g., early layers might benefit from max-pooling because lexical signals are sparse; late layers might benefit from attention because semantic signals are distributed). The paper's AlphaEvolve methodology could be applied here: rather than designing per-layer architectures manually, let AlphaEvolve search the joint space of (layer selection, per-layer architecture, combination method). The key measurement would be whether multi-layer probing with per-layer architecture optimization closes the gap between single-layer probes (2.50% test error) and unlimited-capacity classifiers (2.04% for Flash alone), and whether it provides any improvement on adversarial robustness, where single-layer probes perform poorly.

Verifier robustness training via adversarial data augmentation. The paper demonstrates that verifier over-optimization is not the primary failure mode—long-context generalization is—but the adversarial robustness results (≥1% jailbreak success for all methods) suggest that probes lack robustness to input optimization (adversaries crafting inputs to evade detection). The training data for the paper's probes consists entirely of non-adversarial examples (benign traffic, known attacks). A natural extension is adversarial training: generate jailbreak attempts (using the paper's automated red teaming pipeline, which achieved 20–26% success), add successful jailbreaks to the training set with corrected labels, and retrain. The key question is whether this improves robustness to held-out jailbreak methods (transferability) or merely overfits to the specific attack strategy used in training. The paper's finding that iterative adaptive attacks do not outperform repeated single-turn attacks (Appendix H) suggests that the attack surface is broad rather than narrow—there are many ways to jailbreak, and memorizing a few may not help against others. A strong experiment would measure adversarial robustness on a held-out set of attacks generated by a different attack method (e.g., train on GCG-style suffix attacks, test on PAIR-style semantic rewrites) to test whether adversarial training transfers across attack strategies or merely creates a new distribution shift.

Output monitoring probes for streaming generation-time detection. The paper evaluates only input monitoring, leaving output monitoring entirely to future work (Section 7). Implementing output probes requires solving several challenges that input monitoring does not face: (1) the probe must make decisions partway through generation (before the full harmful response is emitted), meaning it must operate on partial sequences and potentially interrupt generation; (2) the probe's state must be updated incrementally as tokens are generated (Appendix L's incremental attention algorithm is directly applicable here, making this paper's architectures well-suited); (3) the threshold for intervention must balance early detection (stop generation before harm) against false interruptions (blocking benign responses mid-stream). A strong follow-up would: train probes on the activations from harmful model responses (not just harmful prompts), evaluate the tradeoff between detection latency (tokens generated before interruption) and accuracy, and compare probe-based output monitoring against the prompted LLM output classifiers used in prior work. The key measurement would be whether input+output probe combination reduces the effective jailbreak success rate below the input-only rate of ≥1%, and whether output probes catch the failures that input probes miss (e.g., benign-appearing prompts that induce harmful outputs).

Transfer of architectural innovations to other misuse domains and to open-weight models. The paper's entire main evaluation is on a single domain (cyber-misuse) with a single model family (Gemini 2.5 Flash) using proprietary datasets. This severely limits the generality of the architectural recommendations. A high-impact follow-up would: (1) apply the paper's architectures (MultiMax, Max of Rolling Means, AlphaEvolve-discovered) to publicly available misuse detection datasets, such as HarmBench (Mazeika et al., 2024) or JailbreakBench (Chao, Debenedetti, et al., 2024), using open-weight models (e.g., Llama 3, Gemma); (2) measure whether the long-context generalization benefits transfer to other harm categories (CBRN, fraud, self-harm, hate speech) and to other types of distribution shifts (domain shift from English to other languages, from formal to colloquial registers); (3) establish whether the finding that "no single method dominates" (Section 5.1) is domain-general or specific to cyber-misuse. The key measurement would be a cross-domain ranking correlation: does the architecture that performs best on cyber long-context detection also perform best on CBRN long-context detection, or is domain-specific architecture search necessary? The paper's Appendix A (Gemma-2 9B on 12 diverse classification tasks) provides a template but lacks the production distribution shifts (long-context, multi-turn, jailbreaks) that motivate the architectural innovations. Extending this to long-context and adversarial settings would transform the paper from a single-domain case study into a general methodology.

Scaling probe training data and measuring the architecture-vs-data tradeoff. The paper trains probes on only 3,175 sequences (Table 7), which is tiny compared to standard ML training sets. This is appropriate for a specialized production system but raises the question: does more training data reduce the need for sophisticated architectures, or do the architectural innovations provide benefits that scale with data? A strong experiment would: (1) train probes on increasingly large training sets (1×, 10×, 100× the current 3,175 sequences), using synthetic data generation or data from multiple deployment surfaces to scale up; (2) compare the scaling curves of different architectures—does the gap between MultiMax and attention probes shrink as data increases, or does it persist? (3) Test whether the 22× training cost penalty for long-context training could be reduced by using architectures that generalize better from short data, making long-context training more affordable when it is necessary. The paper's finding that the Attn (LC Trained) baseline achieves 2.38% test error vs. 2.50–2.64% for short-context-trained architectures suggests the gap is small at current data scales, but this might change with more data. Understanding this tradeoff is practically critical: if data can substitute for architecture, practitioners should invest in data collection rather than architecture engineering.

AlphaEvolve for end-to-end safety system design (not just probe architecture). The paper uses AlphaEvolve to search over probe architectures, but the cascade design, threshold selection, and deployment configuration are all hand-designed. A more ambitious follow-up would: let AlphaEvolve (or a similar automated system) design the entire safety monitoring pipeline: which layers to harvest activations from, what architectures to use per layer, how to combine them, what cascade policy to apply, and what thresholds to set. The optimization objective would be the paper's weighted error metric (Equation 12) or a more comprehensive metric including latency and cost. This is a substantially harder search problem (the design space is much larger), but the paper's success with ~2500 architecture evaluations suggests it may be tractable. The key test would be whether an end-to-end automated design outperforms the paper's hand-designed system (Selected Probe + 8% Flash cascade) on the same production evaluation grid. This would move the research frontier from "automated architecture search within a fixed framework" to "automated safety system design," which is the logical endpoint of the paper's AlphaEvolve-as-safety-research methodology.

Practical Applications and Downstream Use Cases

Cost-effective input monitoring for deployed frontier models. The most direct application: any organization deploying a large language model can use the paper's probe architectures to add input monitoring at marginal cost. The paper's estimated cost advantage is ~10,000× over a prompted LLM classifier, meaning monitoring 1 million queries/day with probes costs roughly the same as monitoring 100 queries/day with an LLM. For a deployment handling billions of queries, this difference determines whether input monitoring is economically viable at all. The cascading variant (probe + 8% Flash) achieves lower error than Flash alone (1.5% vs. 2.04% test error) at roughly 12× lower cost than running Flash on all queries. The practical setup is: harvest activations from a middle layer during normal model inference, run the probe (a small network that adds negligible latency), and defer to an LLM classifier only when the probe score falls in the uncertainty band. The paper's training recommendations (store activations to disk, iterate on architectures with single-host TPU setups, sweep 100 seeds and select by validation loss) provide a concrete deployment recipe. The primary engineering challenge is integrating activation harvesting into the serving stack—a one-time cost that the paper's successful production deployment at Google demonstrates is feasible (Section 1, Acknowledgements).

Triage and prioritization for human review of flagged content. In high-stakes deployment settings (e.g., government use, critical infrastructure, medical applications), flagged queries may be routed to human reviewers rather than automatically blocked. The False Discovery Rate—the fraction of flagged queries that are actually benign—determines the efficiency of human review. If 10% of flagged queries are true positives and 90% are false positives, human reviewers waste 90% of their time. The paper's cascading classifier architecture can be extended to a three-stage system: probe → LLM → human, where the probe handles clear negatives, the LLM handles moderate-confidence positives, and only the highest-stakes ambiguous cases go to humans. The probe's extremely low false positive rate on overtriggering data (0.5–0.7% for top architectures, Table 3, SC[OT] column) means that the vast majority of benign traffic is correctly routed to "negative" without human involvement. The LLM handles the moderate-confidence positives (the ~8% deferral band), further reducing false positives before any query reaches a human. The paper does not directly evaluate this three-stage configuration, but the cascade framework naturally extends to it.

Data generation for safety training via probe-guided filtering. When constructing training datasets for safety alignment—e.g., collecting examples of model refusals or harmful outputs for DPO training—probes can serve as an efficient pre-filter. Rather than using an expensive prompted LLM to label millions of candidate examples, a cheap probe can filter out obviously benign examples (reducing the candidate pool by >90%) and flag the remaining for LLM or human labeling. The probe's high recall on known attack types (0.0–1.73% FNR on long-context attacks for MultiMax variants, Table 3) means very few harmful examples are lost in pre-filtering, while the false positive rate on benign data (0.5–1.2% FPR for top architectures) determines the reduction in labeling burden. For a dataset of 10 million candidate examples with a 1% base rate of harmful content, a probe with 0.5% FPR would flag ~50,000 benign examples + ~9,900 harmful examples (if recall is ~99%), reducing the labeling burden from 10 million to ~60,000 examples—a 167× reduction. The paper's finding that probes generalize to long-context is particularly valuable here, since training data often includes long documents.

On-device or edge deployment of misuse monitoring for small models. The paper focuses on monitoring a large frontier model (Gemini 2.5 Flash), but the probe architecture is model-agnostic: as long as the monitored model's activations are accessible, the same probe architecture can be trained. For smaller models deployed on-device or at the edge—where running a separate prompted LLM classifier is infeasible due to compute constraints—probes offer essentially the only viable monitoring option. A phone running a 1B-parameter model cannot also run a separate 1B-parameter classifier for every query, but it can run a tiny probe (a few thousand parameters) on activations that are already computed during the model's forward pass. The paper's finding that architectural innovations can substitute for training data (achieving long-context generalization without long-context training) is particularly relevant here, since on-device models are unlikely to have the training infrastructure for long-context probe training. The key practical question—not answered by the paper—is whether the architectural benefits transfer to small models, or whether the probe's dependency on high-quality middle-layer activations requires the rich representations of a large model. Appendix A's results on Gemma-2 9B are encouraging but use only short-context data.