Probing Classifiers: Promises, Shortcomings, and Advances

🎯 Pitch

When a probing classifier performs well, it may reveal more about the probe’s own learning capacity than about what the original model actually encodes—high accuracy can be trivially achieved by a sufficiently complex probe even on random representations. Defining control tasks and information-theoretic metrics forces us to distinguish mere extractability from genuine encoding, which is a prerequisite for any valid interpretation of what neural NLP models learn internally.

1. Executive Summary

This article critically reviews the probing classifiers framework — a prominent methodology for interpreting deep neural NLP models by training a classifier to predict linguistic properties from model representations — examining its promises, methodological shortcomings, and recent advances. Drawing on evidence from studies spanning word embeddings, recurrent networks, and transformer models across properties like part-of-speech, syntax, and semantics, the paper identifies key limitations including inadequate baselines and controls, ambiguity around probe complexity, the correlational nature of findings, and dataset confounding. It synthesizes proposed remedies — control tasks (randomized labels to measure selectivity), information-theoretic metrics (minimum description length to balance accuracy against complexity), and causal intervention methods (amnesic probing that removes properties from representations) — into a unified framework establishing that probing can make valid relative claims about information extractability and usage, but only when appropriate controls and metrics are deployed alongside thoughtful experimental design.

2. Context and Motivation

The Core Problem: We Don't Know What Probing Classifiers Actually Tell Us

The fundamental issue this paper addresses is deceptively simple: when a probing classifier achieves high accuracy at predicting some linguistic property from a neural network's representations, what does that actually mean? The probing framework has become, in the author's words, "one of the prominent methodologies for interpreting and analyzing deep neural network models of natural language processing" (Section 1), yet the interpretation of its results rests on a set of assumptions that, upon critical examination, range from questionable to outright false.

This matters because probing classifiers serve as the dominant tool for answering a core question in NLP interpretability: which linguistic properties are encoded in a model's internal representations, and how? The answer to this question shapes how researchers understand what their models learn, guides architectural decisions, and informs the development of better training procedures. If the probing framework is deeply flawed — if high probing accuracy doesn't actually mean the model encodes the property, or if the choice of probe fundamentally changes what we conclude — then a substantial fraction of the interpretability literature is built on shaky foundations.

The problem is compounded by the framework's apparent simplicity. On the surface, the idea seems straightforward: take a trained model $f$ , extract its representations $f_l(x)$ at some layer $l$ , train a classifier $g$ to predict a linguistic property $z$ from those representations, and if $g$ performs well, conclude that $f$ has learned information about $z$ . This surface simplicity has made probing classifiers extremely popular — they require no modification to the original model, they work with any property for which labeled data exists, and they produce a clean scalar metric (accuracy, F1) that seems amenable to straightforward interpretation. As Section 3 notes, the framework has been applied to study word embeddings, recurrent networks, transformers, and speech recognition systems, examining properties from morphology and syntax to semantics and phonetics.

However, this simplicity is deceptive. The paper reveals that the probing framework actually involves multiple interacting components, each of which introduces non-trivial methodological questions: the original model $f$ , the original dataset $D_O$ , the performance measure on the original task, the choice of which representations $f_l(x)$ to probe, the probing classifier $g$ , the probing dataset $D_P$ , and the probing performance measure (Section 2). The choice of each component — and critically, the interactions between them — can fundamentally alter what conclusions can (and cannot) be drawn. The field has largely proceeded without systematically addressing these complexities.

Why This Problem Is Important: Theoretical and Practical Stakes

The importance of getting probing methodology right spans both theoretical understanding and practical applications.

Theoretical significance. Probing classifiers are the primary empirical tool for testing hypotheses about what neural networks learn. When Shi, Padhi, and Knight (2016) ask "What kind of syntactic information is learned, and how much?" in neural machine translation, or when Conneau et al. (2018) investigate "what you can cram into a single $&!#* vector," probing classifiers provide the evidence. If the framework is unreliable, then our understanding of what models learn is unreliable. This is not a minor concern — as the paper documents, different methodological choices (probe complexity, baseline comparisons, control mechanisms) can lead to qualitatively different conclusions about the same model. For instance, a non-linear probe might report high accuracy on a property while a linear probe reports low accuracy; Hewitt and Liang (2019) showed that non-linear probes with high accuracy can exhibit low selectivity (Section 4.1), meaning their apparent success owes more to the probe's own capacity to memorize patterns in the probing dataset than to information genuinely present in the representations being probed.

This creates a crisis of interpretation: two researchers studying the same model with different probe designs could reach opposite conclusions, and neither would necessarily be "wrong" in a way that is obvious from their reported numbers. The paper therefore addresses a reliability and validity crisis in a major subfield of NLP interpretability research.

Practical significance. The conclusions drawn from probing experiments influence downstream decisions about model design, training, and deployment. The paper provides several concrete examples (Section 5):

Cao et al. (2020) used findings from probing classifiers about layer-wise linguistic specialization to design DeFormer, a decomposed question-answering architecture where lower layers process question and passage independently while higher layers process them jointly.
Dalvi et al. (2020) used redundancy analyses (partially based on probing) to motivate more efficient transfer-learning procedures.
Belinkov and Glass (2017)'s analysis of phonetic information in speech recognition layers motivated Krishna, Toshniwal, and Livescu (2019) to propose multi-task learning with phonetic supervision at intermediate layers.
Belinkov et al. (2020) discuss using probing results to guide the selection of machine translation models for specific language pairs.

If probing conclusions are unreliable, these architectural and training decisions — which consume substantial research effort and computational resources — may be misguided. Conversely, if the framework can be fixed through better controls and metrics, it becomes a more trustworthy guide for model improvement.

Where Prior Approaches Fall Short

The paper identifies a cluster of interrelated methodological problems that have gone largely unaddressed in the probing literature. These are not minor edge cases — they strike at the core interpretability of probing results.

The baseline problem: Compared to what? Many early probing studies reported raw accuracy numbers without meaningful baselines (Section 4.1). A probing accuracy of 87.8% sounds impressive, but is it? The paper notes that some studies compared against majority-class baselines (Belinkov et al., 2017a; Conneau et al., 2018), but this is a weak standard — it tells you the probe learned something, not whether the representations contain useful information. More critically, several studies demonstrated that randomized representations can yield surprisingly high probing accuracy (Conneau et al., 2018; Zhang and Bowman, 2018; Tenney et al., 2019). This means the probe itself, by virtue of its architecture and training, can extract patterns even from nonsense features — so reporting accuracy against learned representations without a random-feature baseline conflates the probe's own capacity with information in the representations.

The paper formalizes this in Figure 1b by introducing the baseline model $\underline{f}$ and the skyline/upper bound model $\bar{f}$ . Without establishing where probing performance falls between these bounds, the absolute number is uninterpretable. Yet many studies omitted such comparisons entirely.

The probe complexity problem: What is the probe learning? A deeper issue is whether the probing classifier $g$ is measuring information in the representations $f_l(x)$ or whether it is itself performing the linguistic task — effectively bypassing the representations and solving the probing task from scratch using its own capacity. This is the core insight of Hewitt and Liang (2019): "the probing performance... may tell us more about the probe $g$ than about the model $f$ ." A highly expressive probe (e.g., a deep multi-layer perceptron) might achieve high accuracy by memorizing patterns in the probing dataset $D_P$ that have nothing to do with the representations being probed. The probe becomes a stand-alone model for the linguistic task, making its accuracy uninformative about the original model $f$ .

This creates a fundamental tension. On one hand, if the goal is to estimate how much information about $z$ is present in $f_l(x)$ , then — as Pimentel et al. (2020b) argue — the most complex probe should be used, since a simple probe might fail to extract information that is genuinely present but non-linearly encoded. On the other hand, as probe complexity increases, the probe's own capacity to solve the task independently increases, making it harder to attribute success to the representations rather than the probe. The field lacks a principled resolution to this tension, and different studies have adopted different (often contradictory) positions without acknowledging the trade-off.

The correlation vs. causation problem: Does the model use the information? Perhaps the most fundamental limitation is that standard probing classifiers are purely correlational. They tell us whether property $z$ can be extracted from representations $f_l(x)$ , but they say nothing about whether the model $f$ actually uses property $z$ when performing its original task $x \mapsto \hat{y}$ (Section 4.3). The probing classifier $g$ is trained in a separate step after $f$ is frozen; there is no feedback from $g$ to $f$ , and no mechanism to test whether the discovered information plays a causal role in the model's behavior.

The paper documents evidence that this disconnect is empirically significant. Vanmassenhove, Du, and Way (2017) reported a mismatch between probing performance and original model performance. Tamkin et al. (2020) found a discrepancy between features that achieve high probing accuracy and features identified as important during fine-tuning. More starkly, Elazar et al. (2021) showed that high probing performance "does not necessarily entail a large drop in original task performance" after removing the probed property from representations — meaning the property was extractable but not actually used. This directly challenges the common implicit assumption that high probing accuracy implies functional relevance.

The problem is that probing operates in a vacuum: it asks "is $z$ in $f_l(x)$ ?" but the more interesting question is "does $f_l(x)$ because of $z$ ?" Early work (Section 3) largely conflated these questions, or at least failed to distinguish them. The paper's contribution is to make this distinction explicit and to survey methods that can address the causal question (interventions on representations, as in Giulianelli et al., 2018; Elazar et al., 2021; Feder et al., 2021).

The dataset confound problem: What are we actually measuring? The probing framework aims to study tasks (linguistic capabilities) but operationalizes them through finite datasets (Section 4.4). As Ravichander, Belinkov, and Hovy (2021) point out, "datasets are imperfect proxies for tasks." The choice of probing dataset $D_P$ — its size, composition, annotation scheme, and relationship to the original dataset $D_O$ — can profoundly influence results, yet this choice is rarely subjected to systematic analysis.

The problem is particularly acute because different original models $f$ are typically trained on different datasets $D_O$ , confounding model architecture comparisons. The paper notes that "statements about models [are] confounded with issues of datasets" (Section 4.4), and that ideally one would compare architectures trained on the same data, or the same architecture trained on different data — but such controlled experiments are rare. Zhang et al. (2021) provide one of the few examples, finding that models require less data to encode syntactic and semantic properties than commonsense knowledge, but the paper emphasizes that "more such experiments are currently lacking."

Ravichander, Belinkov, and Hovy (2021) take this concern further by showing that a probe may learn to predict property $z$ even when $z$ is not discriminative for the original task — that is, even when the model has no reason to learn it. They create control datasets $D_{O,z}$ where all examples have the same value for property $z$ , meaning $z$ is useless for the original task. A model trained on $D_{O,z}$ should not pick up information about $z$ , yet probes can still learn to predict it. This casts doubt on causal claims linking encoded properties to task performance.

The property predefinition problem. Finally, the paper notes an inherent limitation: probing requires specifying the property $z$ in advance (Section 4.5). This constrains investigation to properties for which annotated datasets exist, which are heavily skewed toward English and toward properties linguists have deemed important. It also introduces confirmation bias — researchers probe for properties they expect to find, potentially missing important but unexpected phenomena. Michael, Botha, and Tenney (2020) take an early step toward alleviating this by learning latent clusters useful for predicting properties, discovering both known categories (personhood) and novel ones not typically annotated. But the paper acknowledges that probing remains fundamentally a hypothesis-testing framework, limited by the hypotheses researchers choose to test.

How This Paper Positions Itself

The paper does not propose a new probing method or a single fix to the framework's problems. Instead, it positions itself as a critical synthesis and roadmap. Its contribution is to:

Formalize the framework explicitly, decomposing it into constituent components (Figure 1a: original model, original dataset, representations, probing classifier, probing dataset, performance measures) so that interactions between components become visible and questionable.
Catalog the shortcomings systematically, drawing on a wide range of recent studies that have identified specific failures — Hewitt and Liang (2019) on selectivity, Pimentel et al. (2020b) on information-theoretic interpretation, Voita and Titov (2020) on minimum description length, Ravichander, Belinkov, and Hovy (2021) on dataset confounds, Elazar et al. (2021) on the correlation-causation gap, and others.
Synthesize proposed solutions into a unified framework, introducing additional formal components (Figure 1b: control tasks, control functions, selectivity, information gain, minimum description length, intervened representations) that extend the basic probing setup to address specific shortcomings.
Provide actionable guidance for researchers designing probing experiments: set upper and lower bounds, use control tasks (for word-level properties) or control datasets (for sentence-level properties), measure probe complexity or report accuracy-complexity trade-offs, consider causal interventions to test whether properties are actually used, and when possible use parameter-free probes to avoid probe capacity concerns (Section 5).

Crucially, the paper does not argue that probing classifiers are worthless — quite the opposite. It argues that probing can provide valid insights, but only when the right controls and metrics are deployed. The concluding section explicitly states that while "absolute claims about representation quality seem difficult to make," the improved framework allows for "relative claims" about extractability, and causal approaches "may reveal which properties are used by the original model" (Section 5). The paper positions the field's task as moving from naive probing (train a classifier, report accuracy, draw conclusions) to methodologically rigorous probing that acknowledges and controls for the framework's inherent limitations.

This positioning is reflected in the paper's title structure: "Promises" (what probing can do, Section 3), "Shortcomings" (where it fails, Section 4), and "Advances" (how to do it better, Section 4). The paper serves as both a warning — unqualified probing results should not be trusted — and a constructive guide — here is how to produce results that can be trusted, and here is what still needs work.

3. Technical Approach

3.1 Reader Orientation

This paper does not build a new computational system — it is a critical survey and methodological synthesis that formalizes the probing classifiers framework by decomposing it into explicit, named components (original model, probing classifier, datasets, performance measures, controls, interventions) and then showing how these components interact to either enable or undermine valid conclusions. The problem it solves is interpretive: given a probing classifier experiment that reports some accuracy number, how do we determine what that number actually means about the model being studied? The "shape" of the solution is a taxonomy of methodological pitfalls paired with a corresponding set of diagnostic formalisms — selectivity, information gain, minimum description length, control datasets, and causal interventions — that can be selectively deployed depending on whether the goal is to measure information presence, information extractability, or information usage.

3.2 Big-Picture Architecture (Diagram in Words)

The probing framework, as formalized in Section 2 and Figure 1, consists of two interconnected pipelines — one for the original model and one for the probe — with cross-cutting concerns about comparisons, controls, and causality. The components are:

Original Task Pipeline ( $f: x \mapsto \hat{y}$ ): The neural network being studied, trained on an original dataset $D_O$ to perform some task (e.g., language modeling, machine translation, sentiment analysis), yielding internal representations $f_l(x)$ at layer $l$ and a performance measure $\text{PERF}(f, D_O)$ .
Probing Pipeline ( $g: f_l(x) \mapsto \hat{z}$ ): A separate classifier trained on frozen representations $f_l(x)$ to predict a linguistic property $z$ using a probing dataset $D_P$ , producing a probing performance measure $\text{PERF}(g, f, D_O, D_P)$ . Note that this measure depends on all four entities — the probe $g$ , the frozen model $f$ , and both datasets.
Comparison Infrastructure (Figure 1b): Components that contextualize raw probing accuracy:
- Baseline model $\underline{f}$ : A weaker model or randomized representations providing a lower bound on expected performance.
- Skyline model $\bar{f}$ : An upper bound such as human performance or a dedicated state-of-the-art model for the probing task.
- Control tasks ( $D_{P,\text{Rand}}$ ): Label-randomized versions of the probing dataset that measure probe memorization capacity independently of representation quality.
- Control functions ( $c: f_l(x) \mapsto c(f_l(x))$ ): Transformations applied to representations to create information-theoretic baselines.
- Control datasets ( $D_{O,z}$ ): Versions of the original dataset where property $z$ is non-discriminative, testing whether probes detect incidentally encoded properties.
Diagnostic Metrics (Figure 1b):
- Selectivity $\text{SEL}(g, f, D_O, D_P, D_{P,\text{Rand}})$ : The gap between probing task accuracy and control task accuracy, isolating representation-driven performance from probe memorization.
- Information gain $G(z, h, c)$ : Mutual information difference before and after applying a control function, measuring how much the representation's encoding of $z$ survives transformation.
- Minimum description length $\text{MDL}(g, f, D_O, D_P)$ : A joint measure of probe accuracy and complexity, penalizing probes that memorize rather than extract.
Causal Intervention Layer: Methods that modify representations $\tilde{f}_l(x)$ — by gradient-based perturbation, iterative projection, or adversarial removal — and measure the downstream effect on original task performance $\text{PERF}(\tilde{f}, D_O)$ , testing whether the probed property is actually used by the model, not merely present.

Information flows through these components as follows: First, the original model $f$ is trained on $D_O$ and frozen. Second, representations $f_l(x)$ are extracted for all examples in the probing dataset $D_P$ . Third, the probing classifier $g$ is trained on these frozen representations to predict $z$ . Fourth, the raw probing accuracy is contextualized against baselines ( $\underline{f}$ , $\bar{f}$ ) and control tasks ( $D_{P,\text{Rand}}$ ) to produce selectivity or information gain metrics. Fifth, if causal claims are desired, an intervention is performed on $f_l(x)$ to produce $\tilde{f}_l(x)$ , and the impact on $\text{PERF}(\tilde{f}, D_O)$ is measured. The key insight is that no single component suffices — each addresses a specific threat to validity, and the appropriate combination depends on the research question.

3.3 Roadmap for the Deep Dive

First, the formal probing framework (Section 2 of the paper), establishing precise notation for the original model $f$ , probing classifier $g$ , representations $f_l(x)$ , datasets $D_O$ and $D_P$ , and performance measures. This notation is essential because every subsequent methodological critique and solution is expressed in terms of these components and their interactions.
Second, the information-theoretic interpretation of probing, which recasts the entire enterprise as mutual information estimation — revealing why probe complexity, dataset size, and baseline choice fundamentally determine what can be concluded from a probing experiment.
Third, the comparison and control infrastructure: baselines, skylines, control tasks (Hewitt and Liang, 2019), and control functions (Pimentel et al., 2020b). These establish the conceptual machinery for distinguishing probe capacity from representation content.
Fourth, the probe complexity debate: why researchers disagree about whether probes should be simple or complex, and how metrics like minimum description length (Voita and Titov, 2020) and Pareto-style accuracy-complexity trade-offs (Pimentel et al., 2020a) provide a resolution.
Fifth, the correlation-to-causation transition: intervention methods (gradient-based manipulation, amnesic probing, adversarial removal) that modify representations and measure downstream effects, enabling tests of whether encoded properties are causally implicated in model behavior.
Sixth, dataset confounds and the property predefinition problem, which constrain the scope and generality of probing conclusions in ways that no metric or intervention can fully address.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a methodological survey paper whose core contribution is a formal decomposition of the probing classifiers framework into explicit components, coupled with a critical analysis of how each component choice affects the validity of conclusions. The paper does not advance a single new probing method; rather, it provides a unified notation and taxonomy that reveals why apparently similar experiments can produce contradictory findings, and it synthesizes proposed remedies into a coherent decision framework for practitioners.

Formalizing the Probing Framework: Components and Dependencies

The paper begins (Section 2) by making explicit what is often left implicit in probing studies. The goal is to reveal that probing accuracy is not a property of a model alone, but rather emerges from the interaction of multiple design choices — and changing any one of them can alter the conclusion.

The original model and task. Let $f: x \mapsto \hat{y}$ denote the model being studied. It maps input $x$ to output $\hat{y}$ . The model is trained on an original dataset $D_O = \{x^{(i)}, y^{(i)}\}$ , consisting of input-output pairs for some task (e.g., language modeling, translation, sentiment classification). Its performance is evaluated by some measure $\text{PERF}(f, D_O)$ — this could be perplexity, BLEU score, accuracy, or any task-appropriate metric. The model $f$ is typically a deep neural network that produces intermediate representations; the paper denotes $f_l(x)$ as the representation of input $x$ at layer $l$ of $f$ . The notation $f_l(x)$ is deliberately general: it encompasses hidden state vectors, attention weight matrices, or any other internally computed quantity at layer $l$ . This generality matters because probing studies have been applied to hidden states (the most common case), attention weights (Clark et al., 2019), and even combinations thereof.

The probing classifier and probing task. The probing classifier $g: f_l(x) \mapsto \hat{z}$ is a separate model — typically a linear classifier or a small multi-layer perceptron — that takes the frozen representations $f_l(x)$ as input and predicts some linguistic property $\hat{z}$ . The property $z$ is typically a linguistic feature of interest: part-of-speech tags, syntactic dependency relations, semantic roles, morphological features, or discourse phenomena. The probing classifier $g$ is trained on a probing dataset $D_P = \{x^{(i)}, z^{(i)}\}$ , which provides the same inputs $x$ (or a subset thereof) paired with annotations for property $z$ . Note a crucial asymmetry: $D_O$ and $D_P$ may be different datasets with different splits, different examples, and different annotations. The probing performance is denoted $\text{PERF}(g, f, D_O, D_P)$ — and the notation makes explicit that this quantity depends on four entities: the probe $g$ , the frozen model $f$ , the original dataset $D_O$ (because $f$ was trained on it), and the probing dataset $D_P$ (because $g$ is trained and evaluated on it).

Why this four-way dependency matters. A change to any of these four components can change $\text{PERF}(g, f, D_O, D_P)$ . If we compare two architectures $f_1$ and $f_2$ but they were trained on different original datasets $D_O$ , the comparison is confounded — we cannot tell whether differences in probing accuracy are due to architectural differences or data differences. If we change the probing dataset $D_P$ — its size, its annotation scheme, its domain — probing accuracy can change even if the representations are identical. If we change the probe $g$ from linear to non-linear, accuracy can change due to the probe's own increased capacity, not due to any property of the representations. The notation forces researchers to acknowledge these dependencies rather than eliding them.

A concrete example from the paper. The paper offers a running example: $f$ might be a sentiment analysis model mapping text $x$ to a sentiment label $y$ , while $g$ might be a classifier mapping intermediate representations $f_l(x)$ to part-of-speech tags $z$ . In this scenario, we are asking: does a model trained only to predict sentiment incidentally learn to represent parts of speech in its internal states? The probing accuracy on POS tagging would be one piece of evidence, but — as the subsequent sections of the paper argue — interpreting that accuracy requires controls that account for the probe's own capacity, the baseline difficulty of POS tagging, and the possibility that POS information is correlated with sentiment in the training data.

The Information-Theoretic Perspective: Probing as Mutual Information Estimation

Section 2 briefly notes that "from an information theoretic perspective, training the probing classifier $g$ can be seen as estimating the mutual information between the intermediate representations $f_l(x)$ and the property $z$ ." This observation, attributed to Belinkov (2018) and developed by Pimentel et al. (2020b) and Zhu and Rudzicz (2020), is more than a theoretical aside — it fundamentally reframes what probing experiments are doing and reveals why probe complexity is a central concern.

Mutual information formulation. Let $z$ be a random variable ranging over linguistic property values (e.g., POS tags), and let $h$ be a random variable ranging over representations $f_l(x)$ . The mutual information between them is:

$I(z; h) = \mathbb{E}_{z, h}\left[\log \frac{p(z, h)}{p(z)p(h)}\right]$

where $p(z, h)$ is the joint distribution of properties and representations, $p(z)$ is the marginal distribution of properties, and $p(h)$ is the marginal distribution of representations.

What it computes: Mutual information measures how many bits of information the representation $h$ contains about the property $z$ — or equivalently, how much the uncertainty about $z$ is reduced by knowing $h$ . If $h$ and $z$ are independent, $I(z; h) = 0$ — knowing the representation tells you nothing about the property. If $h$ perfectly determines $z$ , $I(z; h)$ equals the entropy $H(z)$ of the property distribution.

Why this perspective matters for probing methodology. The mutual information perspective reveals that a probing classifier $g$ is attempting to estimate $I(z; h)$ from finite samples. A trained classifier provides a lower bound on mutual information (by the data processing inequality, the classifier's predictions cannot contain more information about $z$ than the representations themselves). The tightness of this bound depends on:

The expressivity of the probe $g$ : A simple probe (e.g., linear) may fail to extract information that is genuinely present but non-linearly encoded, producing a loose lower bound that understates $I(z; h)$ .
The size of the probing dataset $D_P$ : With insufficient data, even a highly expressive probe may overfit to noise rather than extracting genuine mutual information.
The probe's training procedure: Suboptimal optimization can further loosen the bound.

This reframing explains the central tension in probe design: a more complex probe can provide a tighter lower bound on $I(z; h)$ (which is good, because we get a more accurate estimate of what information is present), but it can also more easily memorize the probing dataset $D_P$ (which is bad, because memorization inflates the estimate above the true mutual information). The methodological challenge is to distinguish these two effects.

The information-theoretic perspective also connects directly to the control task approach of Hewitt and Liang (2019). If a probe achieves high accuracy on the real probing dataset $D_P$ but also high accuracy on a label-randomized version $D_{P,\text{Rand}}$ , then the probe's success cannot be attributed to information in the representations — because randomized labels contain no information about $f_l(x)$ that the probe could extract. Formally, the mutual information between representations and randomized labels is zero (up to finite-sample noise), so any above-chance performance on $D_{P,\text{Rand}}$ reflects probe memorization or capacity, not representation content. Selectivity — the gap between real and randomized accuracy — therefore approximates a capacity-controlled estimate of mutual information.

Comparisons and Controls: Contextualizing Raw Probing Accuracy

Section 4.1 addresses the most immediate interpretive problem: a raw probing accuracy number (e.g., 87.8%) is meaningless in isolation. The paper catalogs three families of comparison points that together establish boundaries for valid interpretation.

Baselines (lower bounds). A probing experiment must establish what performance would be expected if the representations contained no useful information about the property. Several baseline strategies exist:

Majority-class baselines: The accuracy achieved by always predicting the most common label in $D_P$ . If probing accuracy equals the majority baseline, the representations provide no discriminative information beyond class imbalance. The paper notes that studies by Belinkov et al. (2017a) and Conneau et al. (2018) used such baselines, but they are weak — surpassing them only shows the probe learned something, not that the representations encode linguistically meaningful structure.
Static embedding baselines: Training the probe on representations from static word embeddings (e.g., word2vec, GloVe) rather than from the contextualized model $f$ . This tests whether $f$ 's contextualization adds information beyond what is available in non-contextual word vectors. Belinkov et al. (2017a) and Tenney et al. (2019) used this approach to show that contextualized representations from neural MT and BERT, respectively, capture syntactic information beyond what static embeddings provide.
Randomized model baselines: Training the probe on representations from a randomized (untrained or weight-shuffled) version of $f$ . Conneau et al. (2018), Zhang and Bowman (2018), Tenney et al. (2019), and Chrupała, Higy, and Alishahi (2020) all demonstrated that probes trained on randomized representations can achieve surprisingly high accuracy. This finding is critical: it means that the probe architecture itself — even without meaningful input features — can extract patterns that inflate accuracy above genuine representation quality. The paper formalizes this as the baseline model $\underline{f}$ in Figure 1b.

Skylines (upper bounds). A probing experiment should also establish how far the achieved accuracy is from the best possible performance on the probing task. The paper identifies several skyline strategies:

Human performance: Conneau et al. (2018) estimated human accuracy on their probing tasks to provide an upper bound on what any system — probe or otherwise — could achieve. If probing accuracy approaches human performance, the representations effectively capture all extractable information about the property.
State-of-the-art dedicated models: Liu et al. (2019) compared probing accuracy against the published state of the art for the probing task (trained directly on $x \mapsto z$ without the frozen representation constraint). This tests whether the frozen representations $f_l(x)$ lose information compared to end-to-end training on the probing task.
Dedicated skyline models: Belinkov et al. (2017b) trained a model specifically to predict $z$ from $x$ without restricting to representations from $f$ . This skyline — denoted $\bar{f}$ in Figure 1b — establishes the maximum achievable performance on the task, against which probing accuracy can be compared. If probing accuracy is far below this skyline, the representations are lossy with respect to property $z$ ; if probing accuracy approaches the skyline, the representations preserve essentially all task-relevant information.

Control tasks (Hewitt and Liang, 2019). The paper devotes particular attention to the control task methodology because it directly addresses the probe capacity confound. The key insight is that "the probing performance... may tell us more about the probe $g$ than about the model $f$ " (Section 4.1). A highly expressive probe may achieve high accuracy not by extracting information from the representations, but by memorizing patterns in the probing dataset $D_P$ — learning, in effect, a stand-alone model of the linguistic property that bypasses the representations entirely.

To isolate this effect, Hewitt and Liang (2019) create control tasks by randomizing the labels in the probing dataset. Specifically, they construct $D_{P,\text{Rand}}$ by assigning random labels $z_{\text{rand}}$ to each input $x$ , breaking any relationship between the representations $f_l(x)$ and the labels. They then define selectivity as:

$\text{SEL}(g, f, D_O, D_P, D_{P,\text{Rand}}) = \text{PERF}(g, f, D_O, D_P) - \text{PERF}(g, f, D_O, D_{P,\text{Rand}})$

where $\text{PERF}(g, f, D_O, D_{P,\text{Rand}})$ is the probe's accuracy on the randomized-label dataset, evaluated using the same trained probe $g$ .

What it computes: Selectivity is the difference between the probe's accuracy on the real probing task and its accuracy on a version where the labels have been randomized. If the probe's success comes from extracting information genuinely present in the representations, then randomized labels should yield near-chance performance (because representations contain no information about random labels), and selectivity will be high — close to the raw accuracy itself. If the probe's success comes from its own memorization capacity, then performance on randomized labels will also be high, and selectivity will be low — the probe can "succeed" even when the representations are uninformative.

Why this form: Subtractively removing the control-task performance isolates the component of accuracy attributable to information in the representations, controlling for the probe's standalone capacity to solve the task through memorization. An alternative approach — simply comparing raw accuracies across probes — would fail to separate these factors. The authors report a key finding: "probes may have high accuracy, but low selectivity, and that linear probes tend to have high selectivity, while non-linear probes tend to have low selectivity." This empirically validates the concern: non-linear probes achieve impressive raw accuracy largely through memorization, not through extraction of representation content.

Limitation of control tasks. The paper notes that control tasks as designed by Hewitt and Liang are "particularly suited for word-level properties $z$ as they evaluate memorization of word types; it is less clear how to apply this idea more broadly, such as in sentence-level properties." Word-level control tasks randomize labels per word type; for sentence-level properties (e.g., sentiment, entailment), per-sentence randomization may not adequately control for probe capacity because the probe could still learn surface patterns from randomized labels.

Control functions (Pimentel et al., 2020b). Taking an explicitly information-theoretic approach, Pimentel et al. (2020b) proposed an alternative control mechanism based on control functions rather than control tasks. A control function is any transformation applied to the representation: $c: f_l(x) \mapsto c(f_l(x))$ . The idea is to compare the mutual information between property $z$ and the original representation $h$ versus the mutual information between $z$ and the transformed representation $c(h)$ . The information gain is defined as:

$G(z, h, c) = I(z; h) - I(z; c(h))$

where $I(z; h)$ is the mutual information between property $z$ and the original representation $h = f_l(x)$ , and $I(z; c(h))$ is the mutual information between $z$ and the transformed representation $c(h)$ .

What it computes: Information gain measures how much information about $z$ is lost by applying the control function $c$ . If the control function destroys noise but preserves linguistically meaningful structure, $G(z, h, c)$ will be small (little information is lost). If the control function destroys the linguistic signal, $G(z, h, c)$ will be large. The control function serves as a baseline: a good probe should extract information that survives the control transformation, not information that is destroyed by it.

Why this form and its relationship to control tasks. Pimentel et al. (2020b) argue that control functions are superior to control tasks because they directly operate on representations rather than on labels — they measure what information the representation loses under transformation, rather than what the probe can memorize from randomized labels. However, the paper notes that subsequent work by Zhu and Rudzicz (2020) showed that "the two criteria are almost equivalent, both theoretically and empirically." This equivalence means practitioners can choose either approach based on practical convenience — control tasks may be easier to implement for word-level properties, while control functions may generalize more naturally to sentence-level properties.

Control datasets (Ravichander, Belinkov, and Hovy, 2021). A third control strategy addresses a different confound: the possibility that probing accuracy reflects incidental encoding of property $z$ rather than task-relevant learning. Ravichander, Belinkov, and Hovy (2021) construct control datasets $D_{O,z}$ where the linguistic property $z$ is non-discriminative with respect to the original task. Specifically, they modify the original dataset $D_O$ so that all examples share the same value for property $z$ — meaning $z$ provides no useful signal for predicting $y$ . A model $f$ trained on $D_{O,z}$ has no task-driven reason to learn representations of $z$ , because knowing $z$ does not help predict the target $y$ .

The test is: does a probe $g$ still learn to predict $z$ from the representations of a model trained on $D_{O,z}$ ? If so, the encoding is incidental — it arises from correlations or architectural biases rather than from task-driven learning — and causal claims linking $z$ to task performance are unwarranted. Ravichander et al. (2021) show that probes can indeed learn to predict $z$ from models trained on control datasets, "casting doubts on causal claims concerning the effect that a property encoded in the representation may have on the original task." The paper notes that while this method was developed for sentence-level properties, "the same idea can be applied to word-level properties."

Summary of the comparison landscape. Together, these components establish a hierarchy of interpretive rigor:

No baselines or controls: Raw accuracy is uninterpretable.
Baselines and skylines only: You can say whether performance is better than chance and how far it is from optimal, but you cannot separate probe capacity from representation content.
Control tasks or control functions: You can separate probe memorization from genuine extraction, making relative claims about extractability.
Control datasets: You can test whether encoding is incidental or task-driven, addressing whether the model should have learned the property.
Causal interventions (Section 4.3, discussed below): You can test whether the model uses the encoded property for its original task.

The Probe Complexity Debate: Simple vs. Complex Probes

Section 4.2 addresses what is perhaps the most contentious methodological question in probing: what architecture should the probing classifier $g$ have, and how does its expressivity affect the validity of conclusions?

The case for simple probes. Several studies advocate for using the simplest possible probes — typically linear classifiers — as the default choice (Alain and Bengio, 2016; Hupkes, Veldhoen, and Zuidema, 2018; Liu et al., 2019; Hall Maudslay et al., 2020). The argument, articulated most clearly by Hupkes, Veldhoen, and Zuidema (2018), is that a linear probe "is thought to reveal features that are used by the original model, while a more complex probe bears the risk that the classifier infers features that are not actually used by the network." The reasoning is: if information is encoded linearly in the representation space, the model can access it through simple linear readout operations (which are the default for typical downstream task heads). Features requiring non-linear extraction are, by this logic, less likely to be functionally used by the model, because the model's architecture does not provide a mechanism for non-linear extraction from its own intermediate representations.

The case for complex probes. The opposite position, advanced by Pimentel et al. (2020b), is that the goal of probing is to estimate how much information about $z$ is present in $f_l(x)$ , and a simple probe may fail to extract information that is genuinely present but non-linearly encoded. From this perspective, "in order to give the best estimate about the information that model $f$ has about property $z$ , the most complex probe should be used" (Section 4.2). The argument draws on the information-theoretic framing: the probe provides a lower bound on $I(z; h)$ , and a more expressive probe provides a tighter (higher) lower bound. Using a linear probe when the information is non-linearly encoded would systematically underestimate the true mutual information, potentially leading to false negatives — concluding that a model lacks information about a property when it actually encodes it in a non-linear format.

Empirical evidence: relative rankings may be robust, absolute claims are not. A key piece of empirical evidence comes from observations across multiple studies: "a few studies observed better performance with more complex probes, but reported similar relative trends" (Section 4.2). Specifically, if you compare two representations $f_1(x)$ and $f_2(x)$ using probes of varying complexity, the ranking $\text{PERF}(g, f_1, D_O, D_P) > \text{PERF}(g, f_2, D_O, D_P)$ often holds across different probes $g$ . Conneau et al. (2018) and Belinkov (2018) both report this pattern. This suggests that relative claims about which representations encode more information about a property may be robust to probe choice, even if absolute claims about how much information is encoded depend on probe complexity.

However, the paper immediately cautions that "this pattern may be flipped under alternative measures, such as selectivity" (Hewitt and Liang, 2019). Since non-linear probes have low selectivity — meaning their high accuracy is substantially attributable to memorization — a representation that appears superior under raw accuracy with a non-linear probe might appear inferior under selectivity with a linear probe. The choice of metric, not just probe architecture, determines the conclusion.

Resolving the tension: Accuracy-complexity trade-offs. The paper identifies two approaches that reframe the probe complexity question from a binary choice (simple vs. complex) to a continuum analysis.

Minimum description length (Voita and Titov, 2020). This approach measures both the performance of the probe and the complexity of the learned mapping, by estimating the length of the shortest code required to transmit the labels $z$ given the representations $f_l(x)$ . Formally:

$\text{MDL}(g, f, D_O, D_P)$

denotes this joint measure, which depends on the probe $g$ , the model $f$ , and both datasets.

What it computes: MDL combines accuracy and complexity into a single scalar. A probe that achieves high accuracy by memorizing the dataset will require a long code to describe its learned mapping (because it must encode many data-specific patterns), resulting in a high (poor) MDL. A probe that achieves high accuracy by extracting simple, generalizable patterns from the representations will require a short code, resulting in a low (good) MDL. The measure thus penalizes the very memorization behavior that makes non-linear probes problematic.

Why this form: MDL operationalizes Occam's razor in a principled way — it formalizes the intuition that a good probe should be simple given the representations, not simple in an absolute sense. A linear probe on highly informative representations might have low MDL; a non-linear probe on uninformative representations might have high MDL due to memorization. Voita and Titov (2020) showed that MDL "provides more information about how a probe $g$ works, for instance by revealing differences in complexity of probes when performing control tasks from $D_{P,\text{Rand}}$ " — probes trained on randomized labels require more complex descriptions, as expected, because they must memorize arbitrary label assignments rather than learning simple decision boundaries.

Pareto-style accuracy-complexity curves (Pimentel et al., 2020a). Rather than selecting a single probe, Pimentel et al. (2020a) argue that "probing work should report the possible trade-offs between accuracy and complexity, along a range of probes $g$ , and call for using probes that are both simple and accurate." The methodology is to train a family of probes spanning a range of complexities (e.g., linear models, MLPs with varying depth and width) and plot the resulting accuracy-complexity Pareto frontier. The frontier reveals whether increasing complexity buys substantial additional accuracy (suggesting complex, non-linear encoding of the property in the representations) or whether accuracy saturates quickly with simple probes (suggesting the property is linearly encoded). This approach replaces the binary "simple vs. complex" choice with a more informative continuum analysis. While Pimentel et al. (2020a) studied linear and non-linear multi-layer perceptrons, "one could extend this idea to other classes of probes."

Pruning-based probes (Cao, Sanh, and Rush, 2021). An alternative resolution to the complexity debate is to avoid training a separate probe entirely, instead pruning the original model's weights to extract a subnetwork that performs the probing task. Cao, Sanh, and Rush (2021) design a probe that learns a binary mask on the weights of $f$ , effectively selecting a sparse subnetwork within the original model that predicts property $z$ . This yields "a better accuracy–complexity trade-off than a non-linear probe." The approach is appealing because the probe's capacity is explicitly tied to the original model's architecture — it cannot invent capacity that $f$ doesn't have, avoiding the infinite-capacity concern that makes non-linear probes problematic.

Parameter-free probes. The paper also discusses a line of work that "can be seen as a parameter-less probing classifier $g$ : a linguistic property is inferred from internal model components (representations, attention weights), without needing to learn new parameters" (Section 4.2). Examples include:

Inferring syntax from attention weight patterns, using attention scores as pairwise importance measures that feed into parsing algorithms like the Chu-Liu Edmonds algorithm (Raganato and Tiedemann, 2018; Clark et al., 2019; Mareček and Rosa, 2019; Htut et al., 2019).
Using representational similarity analysis (Kriegeskorte, Mur, and Bandettini, 2008) to measure similarity between word/sentence representations and syntactic properties (Chrupała and Alishahi, 2019; Lepori and McCoy, 2020).
Perturbing input sentences and measuring distances between representations (Wu et al., 2020).
Clustering representations and classifying by cluster assignment (Zhou and Srikumar, 2021).

Why parameter-free probes help. These approaches circumvent the probe complexity debate entirely: because no parameters are learned, there is no capacity for memorization. Any linguistic structure detected must be genuinely present in the representations or attention patterns. The paper notes that "from the perspective of an accuracy–complexity trade-off, such work should perhaps be placed on the low end of the complexity axis, although the complexity of the parsing algorithm could also be taken into account." The parsing algorithm itself may have implicit capacity (e.g., the Chu-Liu Edmonds algorithm can find trees in arbitrary weight matrices), so parameter-free does not mean assumption-free, but it does eliminate the most direct source of probe confounds.

From Correlation to Causation: Intervention-Based Methods

Section 4.3 addresses what the paper identifies as "a main limitation of the probing classifier paradigm": the disconnect between the probing classifier $g$ and the original model $f$ . Standard probing is purely correlational — it measures whether information about $z$ is present in $f_l(x)$ , but not whether $f$ actually uses that information when performing its original task. The paper sharpens this concern with empirical evidence:

Mismatch between probing and model performance. Vanmassenhove, Du, and Way (2017) reported a mismatch between probing performance $\text{PERF}(g, f, D_O, D_P)$ and original model performance $\text{PERF}(f, D_O)$ — the properties that probes could extract were not the ones that mattered for model success.
Discrepancy with fine-tuning importance. Tamkin et al. (2020) found a discrepancy between features with high probing accuracy and features identified as important during fine-tuning, revealed by randomizing layer weights and measuring the impact.
Properties can be removed without harming the model. Elazar et al. (2021) showed that high probing accuracy "does not necessarily entail a large drop in original task performance after removal of said features," directly demonstrating that extractability does not imply usage.

The common thread is that probing answers the question "does $f_l(x)$ contain information about $z$ ?" but the practically important question is "does $f$ rely on $z$ to compute $\hat{y}$ ?" The paper surveys intervention-based methods that address this gap by modifying representations and measuring the downstream effect.

Gradient-based intervention (Giulianelli et al., 2018). The earliest intervention approach in the paper's survey uses gradients from the probing classifier $g$ to modify the representations in $f$ , and then evaluates how this change affects both probing performance and original model performance. The procedure is:

Train a probing classifier $g$ to predict property $z$ from representations $f_l(x)$ .
Compute the gradient of the probe's loss with respect to the representations: $\nabla_{f_l(x)} \mathcal{L}(g(f_l(x)), z)$ .
Modify the representations by following this gradient: $f_l(x) \leftarrow f_l(x) + \eta \cdot \nabla_{f_l(x)} \mathcal{L}$ , where $\eta$ is a step size. This pushes the representations in a direction that makes property $z$ more predictable.
Evaluate the original model $f$ with the modified representations, measuring $\text{PERF}(\tilde{f}, D_O)$ where $\tilde{f}$ denotes the model with modified representations.

What this tests: If making property $z$ more strongly encoded improves original task performance, it suggests the model uses $z$ and could benefit from even stronger encoding. If it has no effect or hurts performance, the property may be encoded but unused.

Giulianelli et al.'s findings. In their study, $f$ is a language model and $g$ predicts subject–verb number agreement. They find that their intervention increases probing performance (as expected). The key finding concerns the original task: "in the general language modeling case the intervention has a small effect on the original model performance, $\text{PERF}(f, D_O)$ , [but] they find an increase in this performance on examples designed to assess number agreement." This nuance is important — the intervention helps on examples where the property is critical, suggesting the model does use the information when it matters, even if the average effect across all examples is small. They conclude "that probing classifiers can identify features that are actually used by the model."

Counterfactual representations (Tucker, Qian, and Levy, 2021). A related approach uses probe gradients to create counterfactual representations — versions of $f_l(x)$ that are modified with respect to property $z$ — and measures the effect on other encoded properties. The idea is to test whether the representations have disentangled, independent encodings of different properties, or whether modifying one property inevitably affects others. This goes beyond the use-vs-presence question to investigate the structure of the representation space.

Amnesic probing (Elazar et al., 2021). This method takes the opposite approach from Giulianelli et al. (2018): instead of amplifying the property, it removes it from the representations and measures the impact. The procedure is:

Train a linear probing classifier $g$ to predict property $z$ from representations $f_l(x)$ .
Compute the subspace spanned by the probe's weight vectors — this subspace captures the directions in representation space that are predictive of $z$ .
Iteratively project out this subspace: For each representation $f_l(x)$ , subtract its projection onto the probe subspace, producing a modified representation $\tilde{f}_l(x)$ that lies orthogonal to the directions encoding $z$ .
Repeat steps 1–3 multiple times (the "iterative" in "iterative projection") because a single projection may not remove all information — the representations may encode $z$ in directions not captured by a single linear probe. Each iteration trains a new probe on the projected representations and removes the new subspace.
Evaluate the original model with the modified representations, comparing $\text{PERF}(\tilde{f}, D_O)$ to the original $\text{PERF}(f, D_O)$ .

What this tests: If removing information about $z$ from the representations causes a large drop in original task performance, it provides evidence that $f$ causally relies on $z$ to perform its task. If removal has minimal impact, the information was present but functionally irrelevant.

Elazar et al.'s findings, and the contrast with Giulianelli et al. The paper reports that Elazar et al. (2021) find "high probing performance $\text{PERF}(g, f, D_O, D_P)$ does not necessarily entail a large drop in original task performance after their removal, that is, $\text{PERF}(\tilde{f}, D_O)$ ." This leads them to conclude, "contrary to Giulianelli et al. (2018), that probing classifiers do not always identify features that are actually used by the model."

The apparent contradiction between Giulianelli et al. and Elazar et al. is instructive. It may stem from differences in:

Task: Language modeling (Elazar et al.) vs. targeted number agreement evaluation (Giulianelli et al.). A model may use property $z$ for specific linguistic constructions but not for the majority of tokens.
Intervention direction: Amplification (Giulianelli et al.) vs. removal (Elazar et al.). A property could be used when present but not strictly necessary — the model may have redundant encoding strategies.
Property types: Different properties may have different causal status. Some properties (like number agreement) may be actively used, while others (like POS tags in a language model) may be incidentally encoded.

The paper does not resolve this contradiction but presents both findings, implicitly arguing that the causal status of probed properties is an empirical question that must be tested case-by-case with intervention methods, not assumed from probing accuracy alone.

Adversarial removal with controlled properties (Feder et al., 2021). The most sophisticated intervention method surveyed combines adversarial training with positive control. The procedure involves:

Training an adversarial classifier $g$ to remove property $z$ from representations $f_l(x)$ , by minimizing the ability to predict $z$ (similar to domain-adversarial training).
Simultaneously, training a positive classifier $g_C$ to preserve a control property $z_C$ in the representations, ensuring that the removal process doesn't indiscriminately destroy all information.
Critically, the original model $f$ continues to be updated during this process — unlike standard probing where $f$ is frozen. "A major difference from standard probing classifiers work is the continued updating of $f$ " (Section 4.3).

What this tests and why it matters. By adversarially removing property $z$ while preserving property $z_C$ , and then fine-tuning $f$ on a downstream task, the method can estimate the causal effect of $z$ on the downstream task. If removing $z$ significantly degrades performance, $z$ is causally important. The continued updating of $f$ is crucial because it tests whether the model can compensate for the loss of $z$ — if it can achieve the same performance without $z$ , then $z$ was not necessary, even if it was present.

Feder et al. (2021) report that "they can accurately estimate the effect of properties $z$ on downstream tasks performed by $f$ when it is fine-tuned." This provides a bridge from probing (which identifies candidate properties) to causal understanding (which properties are actually necessary for task performance).

Summary: The causal intervention landscape. The paper organizes intervention methods along a spectrum from amplification (Giulianelli et al.) to removal (Elazar et al.) to adversarial removal with controlled fine-tuning (Feder et al.). Each method tests a different causal hypothesis: whether a property can be used (amplification), whether it is necessary (removal), and whether it is used when available but not strictly necessary (adversarial removal with fine-tuning). The paper's contribution is not to advocate for a specific method, but to map the space and show that probing accuracy alone cannot answer causal questions — only interventions can, and different intervention types answer different causal questions.

Dataset Confounds and the Property Predefinition Problem

The paper's final two methodological concerns (Sections 4.4 and 4.5) address limitations that are harder to resolve with formal machinery, because they involve the gap between the idealized probing framework and its real-world instantiation.

Datasets vs. tasks (Section 4.4). The probing framework aims to study models performing tasks (e.g., sentiment analysis) via probes performing other tasks (e.g., POS tagging). But tasks are operationalized through finite datasets, and "datasets are imperfect proxies for tasks" (Ravichander, Belinkov, and Hovy, 2021). The paper identifies two distinct dataset confounds:

Confounding original dataset $D_O$ with model architecture $f$ . Most probing studies compare different architectures (e.g., LSTM vs. Transformer, BERT vs. RoBERTa) that were trained on different original datasets $D_O$ . When probing accuracy differs, we cannot tell whether the difference is due to architecture or training data. The paper notes that "in an ideal world, we would compare different architectures $\{f^i\}$ trained on the same dataset $D_O$ or the same $f$ trained on different datasets $\{D^i_O\}$ ." Zhang et al. (2021) provide one of the few examples of the latter, finding that models require less data to encode syntax and semantics than commonsense knowledge. But "more such experiments are currently lacking."
Understudied role of probing dataset $D_P$ . The size, composition, annotation scheme, and domain of $D_P$ all influence probing accuracy, yet "the effect of the probing dataset $D_P$ — its size, composition, etc. — is similarly not well studied." While some work reported results on multiple datasets for the same property (Belinkov et al., 2017a), systematic investigation of how probing dataset properties affect conclusions is absent from the literature. This matters because probing conclusions are implicitly generalized from the specific $D_P$ used to the abstract task of predicting property $z$ — but if a different $D_P$ (e.g., a different POS-annotated corpus) yields different probing accuracy for the same representations, the generalization is invalid.

Property predefinition (Section 4.5). An inherent limitation of probing is that "it requires determining a property $z$ to probe for." This constrains investigation in three ways:

Annotation bias: Work is limited to properties for which annotated datasets exist, which are heavily skewed toward English and toward properties that linguists have deemed important.
Confirmation bias: Researchers probe for properties they expect to find, potentially missing important but unexpected phenomena.
Task relevance assumption: The properties studied are assumed to be relevant to the original model's task, but this is rarely tested. The control dataset approach (Ravichander, Belinkov, and Hovy, 2021) partially addresses this, but only for properties that researchers think to test.

The paper identifies only one isolated effort to alleviate this limitation: Michael, Botha, and Tenney (2020) propose to learn latent clusters useful for predicting a property $z$ , discovering both known categories (personhood) and novel ones not typically annotated. But the paper acknowledges that "probing classifiers are so far mainly useful when one has prior expectations about which properties $z$ might be relevant."

Summary: The Unified Framework and Decision Logic

The paper's technical contribution — beyond surveying individual methods — is to formalize all these components in a unified notation (Figures 1a and 1b) that makes explicit what is being measured and what assumptions are being made. The framework can be used as a decision logic:

If the goal is to measure information presence:

Use a complex probe (to get a tight lower bound on $I(z; h)$ ).
Report against baselines (randomized model, majority class) and skylines (human performance, SOTA for $x \mapsto z$ ).
Use control tasks or control functions to verify that accuracy is driven by representations, not probe memorization.

If the goal is to measure information extractability with a specific form:

Use a simple (linear) probe, reflecting the kind of readout that downstream task heads typically perform.
Report accuracy-complexity trade-offs (MDL or Pareto curves) rather than single-point accuracy.
Consider parameter-free probes to completely avoid the capacity confound.

If the goal is to test causal usage:

Probing accuracy alone is insufficient.
Use intervention methods: gradient-based amplification (Giulianelli et al.), amnesic removal (Elazar et al.), or adversarial removal with fine-tuning (Feder et al.).
Compare the effect on $\text{PERF}(\tilde{f}, D_O)$ against control interventions that remove properties known to be irrelevant.

If the goal is to compare representations or models:

Ensure models are trained on the same $D_O$ , or the same model is evaluated across different $D_O$ .
Verify that relative rankings are robust to probe complexity and metric choice.
Use selectivity rather than raw accuracy to avoid memorization-driven confounds in rankings.

Always:

Acknowledge the gap between dataset and task — conclusions are about the specific $D_P$ , not about the abstract property.
Acknowledge the property predefinition limitation — negative results mean the probed-for property was not found, not that no structure exists.

4. Key Insights and Innovations

Innovation N: Probing as Mutual Information Estimation — Reframing the Entire Enterprise

The single most intellectually distinctive contribution of this paper is its reframing of probing classifiers from an engineering measurement ("let's train a classifier and check accuracy") to a formal information-theoretic estimation problem. This is not a new probing method — it is a conceptual move that changes what probing means and, consequently, what counts as valid evidence.

What the field did before. Prior to this reframing (and in much work after it), probing was treated as a behavioral test. Researchers trained a classifier $g$ on frozen representations $f_l(x)$ , reported accuracy, and if the accuracy was high, concluded that "the model encodes property $z$ ." This interpretation implicitly treated the probing classifier as a transparent measurement instrument — as if accuracy directly reflected the information present in the representations, uncontaminated by the measurement process itself. The paper traces this assumption through the language of early probing work (Section 3): probes were said to measure "quality," "readability," "usability," and "extractability" of representations, terms that were "often used abstractedly without precise definitions." The assumption was that a well-trained probe simply reveals what is there.

What this paper contributes conceptually. By linking probing to mutual information (attributing the observation to Belinkov, 2018; Pimentel et al., 2020b; Zhu and Rudzicz, 2020), the paper reframes the entire enterprise: probing accuracy is not a transparent window onto representations but rather a sample-based lower bound on $I(z; h)$ , the mutual information between property $z$ and representation $h$ . This reframing has profound implications:

The probe is part of the measurement, not separate from it. The quality of the mutual information estimate depends on the probe's expressivity (can it extract non-linear information?), the dataset size (enough samples to tighten the bound?), and the optimization procedure (did training converge?). The probe is not a neutral observer — it is an active estimator whose properties fundamentally constrain what can be concluded.
"High probing accuracy" becomes ambiguous. Is high accuracy due to a tight bound on genuinely high mutual information? Or is it due to a loose bound — probe memorization of $D_P$ inflating the estimate above the true $I(z; h)$ ? The paper's discussion of control tasks (Hewitt and Liang, 2019) and selectivity makes this ambiguity operational: non-linear probes achieve high raw accuracy but low selectivity precisely because they produce loose, memorization-inflated bounds rather than tight, representation-driven bounds.
The simple-vs-complex probe debate gains theoretical grounding. A simple probe (e.g., linear) may produce a loose lower bound — underestimating $I(z; h)$ because it cannot extract non-linearly encoded information. A complex probe may produce a tight bound on $I(z; h)$ but a loose bound on extractable mutual information (since its own capacity contributes to the estimate). The debate is not about which probe is "right" but about what quantity the probe is estimating: total mutual information (use a complex probe) or linearly extractable information (use a simple probe). The paper does not resolve this tension but makes it precise — a significant advance over the pre-theoretic intuition that "simple probes are safer."

Compare to prior work. Earlier probing studies (Adi et al., 2017; Conneau et al., 2018; Belinkov et al., 2017a) operated without this information-theoretic framing. They reported accuracy numbers and interpreted them directly. The paper's contribution is to show that this direct interpretation is logically insufficient — without controlling for the probe's own estimation properties, raw accuracy is consistent with both "the representations contain the property" and "the probe is a good memorizer." The information-theoretic reframing makes this ambiguity formal rather than anecdotal.

Why this is fundamental, not incremental. This is a fundamental conceptual reframing, not an incremental methodological tweak. It changes the epistemic status of every probing result that lacks appropriate baselines and controls — not by saying those results are wrong, but by saying they are uninterpretable under the formal framework. The field went from "probing accuracy = representation quality" to "probing accuracy = lower bound on mutual information, tightness unknown, requires controls to interpret." This is as significant a shift as the move from reporting perplexity without baselines to reporting it against properly calibrated null models.

Evidence anchoring. The paper anchors this reframing in the formal notation of Section 2 (Figure 1a) and in the discussion of selectivity and control functions (Section 4.1). The key empirical finding supporting the reframing is Hewitt and Liang's (2019) demonstration that probes can have high accuracy but low selectivity — a result that is predicted by the mutual information framing (loose bound due to memorization) but is puzzling under the naive "accuracy = representation quality" view. The paper also notes that Zhu and Rudzicz (2020) establish the theoretical equivalence between control tasks and control functions, further solidifying the information-theoretic foundation.

Innovation N: The Correlation-Causation Gap as a First-Class Problem, Not an Afterthought

The paper's second major intellectual contribution is to elevate the distinction between correlation and causation from a caveat to a central organizing principle of probing methodology. Prior work had occasionally noted that probing is correlational (Belinkov and Glass, 2019), but the field largely proceeded as if probing accuracy, suitably controlled, could answer the question "does the model use property $z$ ?" This paper argues that no amount of correlational rigor — better baselines, control tasks, information-theoretic metrics — can answer that question. It requires a fundamentally different class of methods: interventions.

What the field did before. The dominant implicit assumption in early probing work was that if a property can be reliably extracted from representations, the model probably uses it. This assumption is visible in the language of Section 3: high probing performance was taken to indicate "quality of the representations w.r.t the probing property," "readability of information," or "extractability." The leap from extractability to usage was rarely made explicit, but it pervades the interpretive frame. Hupkes, Veldhoen, and Zuidema (2018) came closest to addressing it, arguing that a linear probe "is thought to reveal features that are used by the original model, while a more complex probe bears the risk that the classifier infers features that are not actually used by the network." But even this framing treats linear extractability as a proxy for usage, not as a hypothesis to be tested.

What this paper contributes conceptually. The paper draws a bright line: probing (even with perfect controls) answers "is $z$ in $f_l(x)$ ?" — a question about information presence. The practically important question — "does $f$ rely on $z$ to compute $\hat{y}$ ?" — is about information usage, and it is logically independent of information presence. A model could encode property $z$ without using it (incidental encoding), or use property $z$ in ways that a simple probe cannot detect (non-linear or distributed usage). Probing accuracy alone cannot distinguish these cases.

The paper supports this distinction with converging empirical evidence from multiple independent studies:

Vanmassenhove, Du, and Way (2017): mismatch between probing performance and original model performance.
Tamkin et al. (2020): discrepancy between high-probing features and features important during fine-tuning.
Elazar et al. (2021): direct demonstration that properties can be removed from representations without harming original task performance — extractability without usage.

The organizing insight is that these are not anomalies or failures of specific probing experiments. They are expected consequences of a framework that measures presence rather than usage. A model trained to perform task $y$ will inevitably encode many properties of its input $x$ — some because they are causally necessary for the task, some because they are correlated with causally necessary properties, some because of architectural inductive biases, and some by sheer statistical accident. Probing cannot distinguish these categories.

The conceptual advance: a causal intervention taxonomy. Beyond identifying the gap, the paper organizes intervention methods into a coherent taxonomy that maps different causal questions to different experimental designs:

Amplification (Giulianelli et al., 2018): If we increase encoding of $z$ , does $\text{PERF}(f, D_O)$ improve? Tests whether the model can use $z$ .
Removal (Elazar et al., 2021): If we remove encoding of $z$ , does $\text{PERF}(f, D_O)$ decline? Tests whether the model requires $z$ .
Adversarial removal with fine-tuning (Feder et al., 2021): If we prevent encoding of $z$ and let the model adapt, does it find alternative strategies? Tests whether $z$ is necessary or merely convenient.

These are different causal hypotheses, and answering one does not answer the others. The paper's contribution is not to advocate for one method but to show that the causal status of a probed property is multidimensional — a property could be used when present but not required (amplification helps, removal doesn't hurt), or required for some inputs but not others (as Giulianelli et al. found for number agreement), or necessary in the original model but not after fine-tuning (as Feder et al. test). Prior work that asked "does the model use $z$ ?" was asking a question too coarse to have a univocal answer.

Compare to prior work. Before this paper's synthesis, the correlation-causation distinction was treated as a limitation to be acknowledged in discussion sections, not as a central methodological challenge requiring new experimental paradigms. Individual papers (Giulianelli et al., Elazar et al., Feder et al.) had proposed specific intervention methods, but these were disconnected from each other and from the broader probing literature. This paper synthesizes them into a unified framework, showing that they address different aspects of the same fundamental gap. The Figure 1b formalization — which adds intervened representations $\tilde{f}_l(x)$ and associated performance measures to the basic framework — makes the causal extension of probing explicit and notationally precise.

Why this is fundamental. This is a fundamental reframing. It does not merely improve probing methodology — it redefines what constitutes a complete probing analysis. Under this framework, a probing study that reports accuracy (even with controls and baselines) but does not address causality is incomplete — not wrong, but answering only the presence question and leaving the usage question open. The paper effectively argues that the field's central question ("what does the model learn?") requires both correlational and causal evidence, and that the probing literature has been systematically conflating the two.

Evidence anchoring. The explicit contrast between Giulianelli et al. (2018) and Elazar et al. (2021) in Section 4.3 serves as the empirical anchor. Giulianelli et al. find that gradient-based amplification helps on agreement-critical examples, concluding "probing classifiers can identify features that are actually used." Elazar et al. find that removing properties by iterative projection has minimal impact on language modeling, concluding "probing classifiers do not always identify features that are actually used." The paper presents this as an apparent contradiction that the causal taxonomy resolves: amplification tests usability, removal tests necessity, and a property can be usable without being necessary (the model may have redundant strategies). The contradiction dissolves when the causal question is made precise.

Innovation N: Selectivity — A Principled Diagnostic That Splits Probe Capacity From Representation Content

The paper elevates selectivity (Hewitt and Liang, 2019) as a diagnostic concept that cuts through the most vexing confound in probing: the probe's own capacity to solve the task independently. This is not the paper's invention — it credits Hewitt and Liang — but the paper's contribution is to position selectivity as the central methodological insight around which the control infrastructure is organized, and to show that it generalizes (via related concepts like control functions and MDL) to address the same confound at different levels of formal rigor.

What the field did before. The dominant approach to evaluating probing results was to compare raw accuracy numbers, perhaps with some baselines. If probe A achieved 92% and probe B achieved 87%, one might conclude that the representations fed to probe A were "better" for the property. This comparison implicitly assumes that the probes themselves are comparable — that differences in accuracy reflect differences in representations, not differences in probe capacity or memorization. The problem is acute because probe architectures vary widely across studies: some use linear classifiers, others use multi-layer perceptrons, still others use parameter-free parsing algorithms. Without a way to control for probe capacity, cross-study comparisons (and even within-study comparisons with different probe architectures) are confounded.

What selectivity contributes conceptually. Selectivity reframes probing accuracy as having two components: a representation-driven component (what the probe extracts from genuine information in $f_l(x)$ ) and a memorization-driven component (what the probe learns from patterns in $D_P$ that are unrelated to $f_l(x)$ ). Raw accuracy conflates these. Selectivity isolates the representation-driven component by subtracting performance on a control task where labels are randomized — and therefore where no information about $z$ can possibly be extracted from the representations, leaving only the memorization component.

$\text{SEL} = \underbrace{\text{PERF}_{\text{real}}}_{\text{representation + memorization}} - \underbrace{\text{PERF}_{\text{rand}}}_{\text{memorization only}}$

The conceptual move is to treat the randomized-label task not as a separate experiment but as an inherent part of any probing measurement — the control performance is a property of the probe-dataset pair, and without measuring it, raw accuracy is uninterpretable. This is analogous to the role of baselines in machine learning: one would not report a classifier's accuracy without comparing it to a majority-class baseline, because the baseline establishes how much of the accuracy is "free" from the label distribution. Selectivity extends this logic to probing: randomized-label performance establishes how much accuracy is "free" from the probe's own capacity.

Why this is powerful beyond the obvious. The paper highlights Hewitt and Liang's finding that "linear probes tend to have high selectivity, while non-linear probes tend to have low selectivity" — non-linear probes achieve high raw accuracy largely through memorization. This finding has far-reaching implications:

It explains contradictory findings in the literature. A study using non-linear probes might report that a model "encodes" some property based on high raw accuracy; a study using linear probes might report the opposite based on low raw accuracy. Both are "right" by their own metrics, but the selectivity perspective reveals that the non-linear probe's result is largely memorization-driven and therefore does not support the encoding conclusion.
It reframes relative model comparisons. The paper notes that "a few studies observed better performance with more complex probes, but reported similar relative trends" (Section 4.2) — that is, rankings of representations were stable across probe complexity. But it immediately cautions that "this pattern may be flipped under alternative measures, such as selectivity." A representation that appears superior under raw accuracy with a non-linear probe might appear inferior under selectivity if the apparent superiority was memorization-driven. This means that without selectivity, even relative comparisons can be misleading.
It provides a criterion for probe choice. Rather than the simple-probe vs. complex-probe debate, selectivity suggests a different optimization target: choose the probe (of any complexity) that maximizes selectivity. A complex probe with high selectivity is extracting genuine information; a simple probe with low selectivity (possible if the property is trivially memorizable from the label distribution) is not. The complexity debate is subsumed by the selectivity criterion.

Generalization via control functions and MDL. The paper shows that selectivity is one instance of a broader class of capacity-control methods. Control functions (Pimentel et al., 2020b) measure how much mutual information survives a transformation of the representation — a different operationalization of the same idea. Minimum description length (Voita and Titov, 2020) penalizes probe complexity directly, rewarding probes that extract information simply. The paper's contribution is to show that these are not competing methods but complementary operationalizations of the same principle: probe performance must be measured against a baseline that accounts for the probe's standalone capacity.

Compare to prior work. Before Hewitt and Liang (2019), the probe capacity problem was recognized (researchers debated linear vs. non-linear probes) but there was no formal diagnostic for it. Studies either used simple probes by fiat (avoiding the problem by design choice) or complex probes with caveats about interpretability (acknowledging but not solving the problem). Selectivity provides a measurement-level solution: rather than constraining probe architecture, it measures the confound and subtracts it out. This is methodologically more principled because it does not require a priori assumptions about which architecture is "correct" — it lets the data (via control task performance) determine how much of the accuracy is representation-driven.

Why this is fundamental, not incremental. Selectivity is a fundamental methodological innovation because it changes what "good probing performance" means. Before selectivity, good performance meant high accuracy. After selectivity, good performance means high accuracy and high selectivity — high accuracy that cannot be explained by probe memorization. This redefinition affects every probing study conducted since. The paper does not invent selectivity but it canonizes it as an essential component of rigorous probing, placing it at the center of the framework in Figure 1b alongside baselines, skylines, and intervention methods.

Evidence anchoring. The paper explicitly cites Hewitt and Liang (2019)'s finding that "probes may have high accuracy, but low selectivity, and that linear probes tend to have high selectivity, while non-linear probes tend to have low selectivity" (Section 4.1). This is the key empirical result that makes selectivity necessary rather than optional — it demonstrates that the confound is real and large enough to flip conclusions. The paper also notes that Zhu and Rudzicz (2020) establish the theoretical equivalence between selectivity (via control tasks) and information gain (via control functions), providing convergent validation of the underlying principle.

Innovation N: Formal Decomposition of Probing into Named Components — Making Implicit Assumptions Explicit

The paper's fourth innovation is the formal decomposition of the probing framework into explicitly named, notationally distinct components (Figure 1a and 1b). While this may appear to be expository rather than innovative, it functions as a conceptual technology that enables all the other innovations — by making implicit assumptions explicit, it reveals where methodological problems can arise and where controls must be inserted.

What the field did before. Prior probing work described its methodology in prose: "we train a classifier on the representations to predict part-of-speech tags." This prose description elides critical distinctions: what dataset was the original model trained on? Is it the same as the probing dataset? What performance measure is used for the probe, and how does it relate to the original model's performance measure? Is the probe linear or non-linear? Are the representations from a specific layer or aggregated? These choices were often reported but not formalized as interacting components of a measurement system. Consequently, the dependencies between choices were invisible — it was not obvious that changing the probing dataset $D_P$ while keeping the model $f$ fixed would change $\text{PERF}(g, f, D_O, D_P)$ , or that comparing two models trained on different $D_O$ confounds architecture with training data.

What the formal decomposition contributes conceptually. By naming each component and making its dependencies explicit in the performance measure $\text{PERF}(g, f, D_O, D_P)$ , the paper transforms probing from a procedure to a measurement system with identifiable sources of error. Each component becomes a site where methodological choices must be justified and where confounds can enter:

The original model $f$ : architecture, training procedure, task.
The original dataset $D_O$ : size, domain, annotation quality, relationship to $D_P$ .
The representations $f_l(x)$ : which layer(s), which token positions, aggregation method.
The probing classifier $g$ : linear/non-linear, capacity, training procedure.
The probing dataset $D_P$ : size, composition, annotation scheme, relationship to $D_O$ .
The performance measure $\text{PERF}$ : accuracy, F1, correlation, etc.

The notation $\text{PERF}(g, f, D_O, D_P)$ makes explicit that probing performance is a four-place relation — it is not a property of the model $f$ alone, nor of the representations $f_l(x)$ alone, but of the entire measurement configuration. This has immediate methodological implications:

Cross-study comparison requires controlling for all four components. Two studies reporting different probing accuracies for "BERT" may have used different probing datasets $D_P$ , different probe architectures $g$ , or even different fine-tuned versions of BERT (different $D_O$ ). The difference in accuracy is uninterpretable without this information.
Ablation claims require isolation. If a study claims that "layer 6 encodes more syntax than layer 3," this claim implicitly assumes that the only thing changing is $l$ , with $f$ , $D_O$ , $g$ , and $D_P$ held fixed. The formal notation makes this assumption explicit and testable.
Dataset confounds become visible. The paper's discussion of Ravichander, Belinkov, and Hovy (2021) and the dataset-vs-task distinction (Section 4.4) flows directly from the formal decomposition: if $D_P$ is an imperfect proxy for the abstract task of predicting property $z$ , then conclusions about the task are only as valid as the dataset is representative.

The Figure 1b extension. The paper extends the formal framework with additional components for advanced diagnostics: baselines $\underline{f}$ , skylines $\bar{f}$ , control tasks $D_{P,\text{Rand}}$ , control functions $c$ , control datasets $D_{O,z}$ , selectivity $\text{SEL}$ , information gain $G$ , minimum description length $\text{MDL}$ , and intervened representations $\tilde{f}_l(x)$ . Each of these addresses a specific threat to validity by modifying one of the original components or adding a comparative measurement:

Selectivity modifies $D_P$ (randomizing labels) to isolate probe memorization.
Control functions modify $f_l(x)$ (applying a transformation) to test robustness.
Control datasets modify $D_O$ (making $z$ non-discriminative) to test incidental encoding.
Intervened representations modify $f_l(x)$ (by projection or gradient) to test causal usage.

The framework is extensible: new diagnostic methods can be added by specifying which components they modify and what new measurement they produce. The paper's synthesis is not just a catalog of methods but a generative taxonomy — it reveals gaps where new diagnostics could be developed (e.g., a sentence-level analog of control tasks, or a control mechanism for the original dataset $D_O$ ).

Compare to prior work. Before this paper, there was no unified notation for probing experiments. Individual studies described their setups in prose, making cross-study comparison and methodological critique difficult. The paper's formalization draws on individual contributions (Hewitt and Liang's selectivity, Pimentel et al.'s control functions, Voita and Titov's MDL, Elazar et al.'s amnesic probing) but synthesizes them into a common language. This is similar to the contribution of a good programming language abstraction — it doesn't enable new computations per se, but it makes existing computations easier to reason about, compose, and debug.

Why this matters beyond exposition. The formal decomposition is not merely pedagogical — it is a conceptual technology for methodological rigor. By making the components explicit, it forces researchers to confront questions they might otherwise overlook: Was the original model $f$ trained on the same data distribution as the probing dataset $D_P$ ? Did I control for the probe's capacity $g$ relative to the probing dataset $D_P$ ? Is my conclusion about the model $f$ or about the specific dataset $D_O$ it was trained on? These questions are answerable within the framework; without it, they are easily missed.

Evidence anchoring. The framework's utility is demonstrated throughout Section 4, where each shortcoming is expressed in terms of the formal components. The comparison and controls section (4.1) uses the baseline $\underline{f}$ and skyline $\bar{f}$ notation to structure the discussion; the correlation-vs-causation section (4.3) uses the intervened representation $\tilde{f}_l(x)$ ; the dataset section (4.4) uses the $D_O$ vs. $D_P$ distinction. The framework earns its keep by making the paper's argument structurally coherent — each methodological problem maps to a specific component or interaction between components.

5. Experimental Analysis

Evaluation Methodology

Dataset. This paper is a methodological survey and does not itself conduct new experiments. It critically reviews and synthesizes experimental evidence from the existing probing classifiers literature, drawing on studies that span a range of models (word embeddings, recurrent networks, transformers), probing tasks (morphology, syntax, semantics), and datasets. The paper does not introduce a new dataset or a new train/test split; its contribution is to evaluate the experimental practices of the field itself, using published results as evidence for methodological claims.

Base model(s). The paper surveys results from models including, but not limited to: static word embeddings (Köhn, 2015; Gupta et al., 2015), recurrent neural network machine translation systems (Shi, Padhi, and Knight, 2016; Belinkov et al., 2017a), sentence embedding models (Adi et al., 2017; Conneau et al., 2018), and transformer-based models including BERT (Clark et al., 2019; Tenney et al., 2019; Hewitt and Liang, 2019). No single model family is treated as the reference architecture; rather, the paper examines how probing conclusions vary — or fail to vary — across model families under different methodological choices. This breadth is deliberate: the paper's central claim is that methodological problems in probing are general, not model-specific, so evidence must be drawn from diverse architectures to establish generality.

Metrics. The paper discusses several metrics that have been used to evaluate probing classifier performance, each addressing a different aspect of what probing can tell us:

Raw probing accuracy (denoted PERF(g, f, D_O, D_P) in Section 2): The most common metric in the literature — classification accuracy or F1 of the probing classifier $g$ on the probing dataset $D_P$ when operating on representations from the frozen model $f$ . The paper argues that this metric is insufficient in isolation because it conflates information in the representations with the probe's own capacity.
Selectivity (Hewitt and Liang, 2019, discussed in Section 4.1): The difference between probing accuracy on the real probing task and probing accuracy on a control task with randomized labels: SEL = PERF(g, f, D_O, D_P) - PERF(g, f, D_O, D_{P,Rand}). Selectivity isolates the component of accuracy attributable to information in the representations by subtracting out the probe's standalone capacity to memorize label patterns.
Information gain (Pimentel et al., 2020b, discussed in Section 4.1): The difference in mutual information between property $z$ and the representation $h$ before and after applying a control function $c$ : G(z, h, c) = I(z; h) - I(z; c(h)). This measures how much information about $z$ survives a transformation of the representation.
Minimum description length (MDL) (Voita and Titov, 2020, discussed in Section 4.2): A joint measure of probe accuracy and probe complexity, estimating the length of the shortest code needed to transmit labels $z$ given representations $f_l(x)$ . MDL penalizes probes that achieve high accuracy through memorization (which requires longer codes) while rewarding probes that extract information through simple, generalizable patterns.
Original task performance after intervention (Section 4.3): Not a probing metric per se, but the key dependent variable in causal studies. When representations are modified (amplified, removed, or adversarially altered) to change the encoding of property $z$ , the effect on PERF(f, D_O) — the original model's performance on its trained task — is measured to test whether $z$ is causally used.

Baselines. The paper surveys three families of baselines that have been used (or that the paper argues should be used) in probing experiments, each establishing a different lower bound for interpretation:

Majority-class baselines: The accuracy achieved by always predicting the most frequent label in $D_P$ (used by Belinkov et al., 2017a; Conneau et al., 2018). The paper characterizes this as a weak baseline — it establishes only that the probe learned something beyond the label distribution, not that the representations contain linguistically meaningful information.
Static embedding baselines: Training the probe on non-contextual word embeddings (e.g., word2vec, GloVe) instead of the contextualized representations $f_l(x)$ (used by Belinkov et al., 2017a; Tenney et al., 2019). This tests whether contextualized representations provide information beyond what is available in static vectors.
Randomized model baselines: Training the probe on representations from a randomized (untrained or weight-shuffled) version of $f$ (used by Conneau et al., 2018; Zhang and Bowman, 2018; Tenney et al., 2019; Chrupała, Higy, and Alishahi, 2020). These baselines are critical because they reveal that "even random features capture significant information that can be decoded by the probing classifier" (Section 4.1), meaning that above-chance probing accuracy does not imply that the trained model has learned meaningful structure — the probe itself can extract spurious patterns from unstructured features.
Skylines (upper bounds): The paper also surveys upper bounds used to contextualize probing accuracy: human performance on the probing task (Conneau et al., 2018), state-of-the-art dedicated models trained directly on $x \mapsto z$ without frozen representations (Liu et al., 2019), and dedicated skyline models $\bar{f}$ trained specifically for the probing task (Belinkov et al., 2017b).

Compute accounting / probe expressivity measurement. The paper does not measure computational cost in FLOPs or GPU-hours. Instead, the relevant resource that must be controlled is probe expressivity — the capacity of the probing classifier $g$ to solve the task independently. This is measured through several mechanisms:

Probe architecture complexity: Linear vs. non-linear, depth and width of multi-layer perceptrons, number of parameters.
Selectivity: A behavioral measure of how much of the probe's accuracy comes from memorization rather than representation content.
Minimum description length: A formal measure of the probe's effective complexity given the data.
Parameter-free approaches: Representations of structure detected without any learned parameters (e.g., from attention weights or representational similarity analysis), which have zero probe capacity by construction.

The paper does not propose a standardized unit for probe expressivity, but it argues that some form of capacity control — whether expressed as selectivity, MDL, accuracy-complexity trade-off curves, or parameter-free extraction — is essential for valid interpretation.

Cross-validation / statistical protocol. The paper does not describe a cross-validation protocol for its own experiments (since it conducts none). However, it discusses the statistical logic of probing experiments implicitly through its treatment of the relationship between probing dataset $D_P$ and the abstract task of predicting property $z$ . The key concern (Section 4.4) is that probing datasets "are imperfect proxies for tasks" (Ravichander, Belinkov, and Hovy, 2021), meaning that probing conclusions are implicitly generalized from a finite dataset to an abstract linguistic capability. The paper highlights the lack of systematic studies varying $D_P$ (size, composition, annotation scheme) to assess the robustness of probing conclusions — a gap that amounts to a missing cross-validation concern: we do not know whether probing results on one dataset for property $z$ generalize to a different dataset for the same property.

Main Quantitative Results

Evidence That Raw Probing Accuracy Is Not Self-Interpreting

The paper's primary empirical argument is that raw probing accuracy numbers, as reported in much of the probing literature, are uninterpretable without appropriate baselines and controls. This claim is supported not by new experiments but by synthesizing results from multiple prior studies that reveal specific failure modes of raw accuracy interpretation.

Randomized features yield surprisingly high accuracy. The paper cites converging evidence from Conneau et al. (2018), Zhang and Bowman (2018), Tenney et al. (2019), and Chrupała, Higy, and Alishahi (2020) that probes trained on randomized representations achieve accuracies substantially above chance (Section 4.1). The paper does not report specific numerical values from these studies (no table or figure is provided), but the qualitative finding is clear: "even random features capture significant information that can be decoded by the probing classifier." This means that reporting a probing accuracy of, say, 87.8% without comparing against a randomized-representation baseline does not allow one to conclude that the trained model's representations are meaningfully encoding the linguistic property — the probe could achieve substantial accuracy from noise alone.

The implication is structural: every probing study that reports accuracy without a randomized-representation baseline is presenting an uninterpretable number. The randomized-feature baseline serves as the zero-information point — the accuracy that would be expected if the representations contained no task-relevant structure at all, due purely to the probe's capacity to extract spurious regularities from high-dimensional noise.

Non-linear probes achieve high accuracy but low selectivity. The paper highlights Hewitt and Liang's (2019) key empirical finding: "linear probes tend to have high selectivity, while non-linear probes tend to have low selectivity" (Section 4.1). This means that non-linear probes achieve high raw accuracy on probing tasks, but much of that accuracy is also achievable on control tasks with randomized labels — indicating that the non-linear probe's success comes substantially from memorization of the probing dataset rather than from extraction of information genuinely encoded in the representations. The paper does not provide specific selectivity numbers (no table is provided for these results), but the direction of the finding is the critical empirical anchor: raw accuracy systematically overstates representation quality when measured with expressive probes, and the degree of overstatement is probe-dependent.

This finding implies that cross-study comparisons that use different probe architectures are fundamentally confounded: a study using a non-linear MLP probe might report higher accuracy than a study using a linear probe on the same model, but the higher accuracy reflects the probe's own capacity, not greater information in the representations. Without selectivity, these two studies would appear to conflict; with selectivity, they can be reconciled.

Properties can be removed without affecting original task performance. The paper cites Elazar et al. (2021) as demonstrating that "high probing performance does not necessarily entail a large drop in original task performance after removal" of the probed property from representations (Section 4.3). In their amnesic probing experiments, Elazar et al. iteratively projected out directions in representation space that encoded a linguistic property (such as part-of-speech) and measured the impact on the original model's performance (language modeling). The finding — that removing a property with high probing accuracy can have minimal effect on the original task — directly undermines the common inference from "the probe can extract $z$ " to "the model uses $z$ ." The paper does not provide specific numerical magnitudes (no table is given), but the qualitative finding is the critical point: the correlation between probing accuracy and causal importance is weak, and high probing accuracy does not imply high causal importance.

This evidence anchors the paper's central claim that the probing framework must be extended with causal intervention methods (amnesic probing, gradient-based manipulation, adversarial removal) to answer questions about usage, not merely presence.

Relative Rankings May Be Robust, But Not Under All Metrics

The paper reports an important empirical observation about relative comparisons: "a few studies observed better performance with more complex probes, but reported similar relative trends" (Section 4.2). Specifically, if one representation $f_1(x)$ outperforms another $f_2(x)$ when probed with a linear classifier, the same ranking often holds when probed with a non-linear classifier. This suggests that comparative claims about which representations encode more information may be more robust to probe choice than absolute claims about how much information is encoded.

However, the paper immediately qualifies this finding with a critical caveat: "this pattern may be flipped under alternative measures, such as selectivity" (Hewitt and Liang, 2019). Since non-linear probes have low selectivity, a representation that appears superior under raw non-linear accuracy might appear inferior under selectivity if its apparent advantage was memorization-driven. This means that relative rankings are only robust when the underlying metric (raw accuracy vs. selectivity) is held constant — a methodological requirement that many comparative probing studies have not met.

The paper does not provide a quantitative meta-analysis of how often rankings flip across metrics (no table or figure is provided for this claim), but the logical possibility of flipping — demonstrated by Hewitt and Liang (2019) for specific cases — is sufficient to establish that relative comparisons without selectivity controls cannot be trusted unconditionally.

Evidence for the Existence of the Correlation-Causation Gap

The paper synthesizes multiple independent lines of evidence demonstrating that probing accuracy and causal importance are distinct constructs:

Vanmassenhove, Du, and Way (2017): A reported "mismatch between performance of the probe, PERF(g, f, D_O, D_P), and performance of the original model, PERF(f, D_O)" (Section 4.3). Properties that probes could extract were not the ones that mattered for the original model's success. The paper does not provide specific quantitative details from this study.
Tamkin et al. (2020): A "discrepancy between features f_l(x) obtaining high probing performance, PERF(g, f, D_O, D_P), and features identified as important when fine-tuning f" (Section 4.3). This discrepancy was revealed by randomizing the weights of specific layers during fine-tuning — an intervention that identifies which layers are causally important for task performance — and comparing those importance rankings against probing accuracy rankings.
Lovering et al. (2021): A contrasting finding: "extractability of a property according to MDL(g, f, D_O, D_P) is correlated with f making predictions consistent with that property." This suggests that when extractability is measured with capacity-controlled metrics (MDL rather than raw accuracy), it may be more predictive of model behavior. This is a nuance the paper highlights but does not resolve — it suggests that the correlation-causation gap may be partially bridgeable with better probing metrics.

The paper's synthesis of these findings leads to its core recommendation: probing can establish presence, but establishing usage requires interventions. The empirical evidence for this recommendation is the conjunction of positive findings (Giulianelli et al., 2018, showing that gradient-based amplification helps on agreement-critical examples) and negative findings (Elazar et al., 2021, showing that property removal has minimal impact on language modeling), which together demonstrate that the answer to "does the model use $z$ ?" depends on the specific property, task, and evaluation setting — and cannot be inferred from probing accuracy alone.

Ablation Studies and Robustness Checks

Because this paper is a methodological survey rather than an experimental study, it does not report its own ablation studies or robustness checks. Instead, it surveys the ablations and robustness analyses conducted by the individual studies it reviews. The following are the key methodological variations that the paper identifies as having been tested in the probing literature:

Probe complexity (linear vs. non-linear): The paper reports that Conneau et al. (2018) and Belinkov (2018) varied probe complexity and found that more complex probes achieved higher accuracy but preserved relative rankings across representations. Hewitt and Liang (2019) further showed that the selectivity metric — which controls for probe memorization — reveals that non-linear probes achieve low selectivity, meaning their high raw accuracy is substantially memorization-driven. Pimentel et al. (2020a) systematically varied probe complexity along a continuum and reported accuracy-complexity Pareto curves, finding a trade-off that should be explicitly reported rather than selecting a single probe and treating its accuracy as definitive.

Probe architecture class (parameterized vs. parameter-free): The paper surveys work using parameter-free extraction methods — attention weight analysis (Clark et al., 2019; Raganato and Tiedemann, 2018), representational similarity analysis (Chrupała and Alishahi, 2019; Lepori and McCoy, 2020), and perturbed masking (Wu et al., 2020) — as an alternative to parameterized probes. These methods avoid the probe capacity confound entirely because no parameters are learned. The paper notes that such methods "avoid some of the issues about what the probe learns" (Section 4.2), constituting a robustness check on probe architecture choice. However, it also notes that "the complexity of the parsing algorithm could also be taken into account," meaning that even parameter-free methods have implicit capacity.

Control task label randomization: Hewitt and Liang (2019) tested whether probes trained on randomized labels in $D_{P,\text{Rand}}$ could achieve high accuracy, and found that non-linear probes could — establishing that control tasks are essential for distinguishing representation-driven accuracy from memorization. This is effectively an ablation of the information content in the representations: by destroying the relationship between $f_l(x)$ and $z$ , the control task measures what the probe can accomplish without any information in the representations.

Control function type: Pimentel et al. (2020b) tested different control functions $c$ applied to representations — transformations that destroy different kinds of structure — and measured information gain $G(z, h, c)$ for each. The robustness of probing conclusions across different control functions is a diagnostic for whether the extracted information is superficial or structurally meaningful. Zhu and Rudzicz (2020) subsequently showed that control tasks and control functions are "almost equivalent, both theoretically and empirically" (Section 4.1), providing convergent validation of the underlying principle.

Intervention method (amplification vs. removal vs. adversarial removal): The paper surveys three distinct causal intervention paradigms — gradient-based amplification (Giulianelli et al., 2018), iterative projection removal (Elazar et al., 2021), and adversarial removal with fine-tuning (Feder et al., 2021) — and reports that they can yield different conclusions about the same property. Specifically, Giulianelli et al. found that amplifying number agreement information improved language modeling on agreement-critical examples, while Elazar et al. found that removing part-of-speech information had minimal impact on overall language modeling performance. These results are not necessarily contradictory — amplification tests usability, removal tests necessity — but they demonstrate that the choice of intervention method determines what causal conclusion is drawn. The paper does not report numerical comparisons between these methods on the same model/property combination (no such study exists), but the methodological implication is clear: single-intervention studies provide incomplete causal evidence.

Dataset composition (original and probing): Ravichander, Belinkov, and Hovy (2021) created control datasets $D_{O,z}$ where property $z$ was non-discriminative for the original task, and found that probes could still learn to predict $z$ from models trained on these datasets. This is an ablation of the task relevance of $z$ : it tests whether probing accuracy reflects task-driven learning or incidental encoding. Zhang et al. (2021) varied the amount of pretraining data and found that models require less data to encode syntactic and semantic properties than commonsense knowledge — an ablation of training data quantity that reveals differential learning dynamics for different property types.

Layer depth: The paper notes that probing studies commonly vary the layer $l$ from which representations $f_l(x)$ are extracted (Section 2), finding that different layers encode different types of linguistic information. While the paper does not report specific layer-wise probing results, the survey literature it draws on (Belinkov et al., 2017a; Tenney et al., 2019; Liu et al., 2019) consistently finds that lower layers encode more local/morphological information while higher layers encode more global/semantic information. This layer-wise variation serves as a robustness check: if probing accuracy varies systematically with layer depth in theoretically predictable ways, it increases confidence that the probe is detecting genuine linguistic structure rather than artifacts.

Critical Assessment

Do the Surveyed Experiments Support the Paper's Central Claims?

The paper advances several interconnected claims, none of which are supported by new experiments. Rather, the paper's evidence base is the collective experimental record of the probing literature, interpreted through the lens of the paper's formal framework. Evaluating whether the claims are supported therefore requires assessing whether the cited studies, taken together and properly interpreted, provide sufficient evidence for the paper's methodological conclusions.

The claim that raw probing accuracy is uninterpretable without controls: This claim is strongly supported by the accumulated evidence the paper marshals. The finding that randomized representations yield above-chance probing accuracy (Conneau et al., 2018; Zhang and Bowman, 2018; Tenney et al., 2019; Chrupała, Higy, and Alishahi, 2020) is a direct demonstration of uninterpretability: if probes can extract "information" from representations that contain no task-relevant structure, then probing accuracy alone cannot be evidence that the trained model encodes the target property. This is not a theoretical concern — it is an empirical finding replicated across multiple model types and probing tasks.

The finding that non-linear probes achieve high accuracy but low selectivity (Hewitt and Liang, 2019) further supports the claim: even when probing trained representations (not randomized ones), a substantial fraction of accuracy can be attributed to probe memorization rather than representation content. The implication is that probing accuracy is always a composite of representation-driven and memorization-driven components, and without a control task to estimate the latter, the former is unknown.

A limitation of the paper's treatment is that it does not provide quantitative meta-analysis of how much raw accuracy is typically memorization-driven vs. representation-driven across the literature. The paper reports qualitative findings ("non-linear probes tend to have low selectivity") but does not aggregate selectivity values across studies to establish typical magnitudes. This is understandable given the diversity of experimental setups, but it means the reader cannot gauge whether memorization-driven accuracy is typically a minor nuisance (e.g., 5% of total accuracy) or a dominant factor (e.g., >50% of total accuracy). The paper's argument would be stronger if it could point to systematic evidence on this point.

The claim that probing accuracy and causal importance are distinct: This claim is strongly supported by the conjunction of positive and negative causal findings. The fact that Elazar et al. (2021) can remove properties with high probing accuracy without substantially affecting model performance is direct evidence that extractability ≠ necessity. The fact that Giulianelli et al. (2018) can amplify properties and improve performance on specific examples is evidence that some properties are conditionally usable. The paper's synthesis of these findings into a causal taxonomy (amplification tests usability, removal tests necessity, adversarial removal tests substitutability) is a conceptual contribution that organizes the evidence coherently.

However, a significant limitation is that the paper does not survey studies that perform both amplification and removal on the same property-model combination. Without such within-study comparisons, we cannot distinguish between two possible interpretations of the Giulianelli-Elazar divergence: (a) that the two methods test different causal questions (the paper's interpretation), or (b) that the two methods were applied to different properties and models, and the divergence reflects property-specific or model-specific differences rather than a general causal structure. The paper acknowledges this limitation implicitly by presenting both findings without attempting to adjudicate between them, but it does not explicitly flag the absence of within-study multi-intervention comparisons as a gap in the literature.

The claim that relative rankings may be robust to probe complexity: This claim has mixed support. The paper cites Conneau et al. (2018) and Belinkov (2018) for the observation that rankings tend to be preserved across probe complexities, but immediately notes that "this pattern may be flipped under alternative measures, such as selectivity." The evidence for ranking robustness is anecdotal ("a few studies observed") rather than systematic — the paper does not report how many comparative probing studies have tested ranking stability across probe complexity, nor what fraction of rankings flip when measured by selectivity vs. raw accuracy. This is a genuine weakness in the evidential foundation: the claim that rankings are robust is an empirical generalization based on a small number of observations, and the warning that selectivity can flip rankings is also based on limited evidence. A systematic meta-analysis of ranking stability across probe types and metrics would substantially strengthen (or weaken) this claim.

The claim that parameter-free probes avoid the probe capacity problem: This claim is logically sound but empirically underdetermined. Parameter-free methods (attention weight analysis, representational similarity analysis, perturbed masking) by construction have zero learned parameters, so they cannot memorize the probing dataset in the way that parameterized probes can. However, the paper acknowledges that "the complexity of the parsing algorithm could also be taken into account" — a parsing algorithm like Chu-Liu Edmonds can extract tree structures from arbitrary weight matrices, meaning it has implicit capacity that is not captured by parameter counting. The paper does not survey evidence comparing parameter-free and parameterized probes on the same model-property combinations, so we cannot assess whether parameter-free methods yield systematically different conclusions. This is a gap: the paper advocates for parameter-free probes as a solution to the capacity confound, but provides no quantitative evidence that they produce more valid or more reliable conclusions than parameterized probes with appropriate selectivity controls.

Genuine Weaknesses in the Evidential Foundation

Absence of systematic meta-analysis. The paper's argument relies on synthesizing findings from across the probing literature, but it does not perform quantitative meta-analysis. It does not aggregate selectivity values, effect sizes for interventions, or ranking stability statistics across studies. This is understandable for a short survey article, but it means that many of the paper's key empirical claims are supported by existence proofs (this phenomenon can happen) rather than by estimates of typical magnitude (this phenomenon accounts for X% of variance in probing results). The reader cannot assess, for example, whether the correlation-causation gap is usually large or usually small — only that it exists and can be consequential.

No standardized comparison of interventions. The paper surveys three causal intervention paradigms (amplification, removal, adversarial removal) but notes that they have not been applied to the same model-property combinations in a controlled comparison. This makes it impossible to determine whether the different causal conclusions they produce reflect different causal questions (the paper's interpretation) or idiosyncratic properties of the specific models and tasks studied. A systematic study applying all three intervention types to a shared set of models and properties would be necessary to validate the paper's causal taxonomy.

Limited evidence on dataset confounds. The paper identifies dataset confounds as a serious concern (Section 4.4) but can point to very little experimental evidence on the magnitude or nature of these confounds. Zhang et al. (2021) provide one data point on how training data quantity affects property encoding, and Ravichander, Belinkov, and Hovy (2021) provide one data point on incidental encoding. But the paper acknowledges that "the effect of the probing dataset $D_P$ — its size, composition, etc. — is similarly not well studied" and that "more such experiments are currently lacking." The dataset confound concern is well-motivated theoretically by the paper's formal framework (since $\text{PERF}(g, f, D_O, D_P)$ depends on all four components, any of which could be a confound), but it remains empirically underinvestigated. The paper's recommendations on this point are therefore more speculative than evidence-based.

No longitudinal evidence on methodological improvement. The paper synthesizes methodological critiques and proposed solutions but does not assess whether the adoption of these methods has improved the reliability or replicability of probing research. It would be informative to know, for example, whether probing papers that use control tasks and selectivity tend to draw conclusions that are later validated by causal studies, or whether they are merely more conservative. This kind of longitudinal evidence is beyond the scope of a short survey, but its absence means the paper's recommendations are based on logical analysis of what should improve validity, not on evidence that they do improve it.

Missing Experiments That Would Have Strengthened the Paper

As a survey, the paper does not conduct experiments and therefore cannot be faulted for not running specific ablations. However, the paper's synthesis implicitly identifies experiments that the field needs but that have not been conducted:

A large-scale multi-probe, multi-metric benchmarking study. The paper's framework implies that probing conclusions depend on the choice of probe $g$ , the metric (accuracy vs. selectivity vs. MDL), and the causal intervention method. A systematic study applying a matrix of probe types × metrics × intervention types to a common set of models and properties would provide the quantitative foundation that the paper's qualitative synthesis lacks. Such a study would reveal how often method choices flip conclusions, the typical magnitude of the probe capacity confound, and the correlation between selectivity-based conclusions and causal importance.

A controlled study varying probing dataset $D_P$ . To assess the dataset confound, one would need to take a fixed model $f$ and train probes for the same abstract property $z$ on multiple different probing datasets $D_P^1, D_P^2, \ldots$ that differ in size, domain, annotation scheme, or label distribution. The correlation of probing accuracy across these datasets would reveal how much probing conclusions depend on the specific dataset used rather than the abstract property.

A within-study comparison of linear vs. non-linear probes under selectivity and MDL. The paper cites Hewitt and Liang (2019) for selectivity and Voita and Titov (2020) for MDL, but these use different models, properties, and datasets. A single study applying both metrics to the same data, and comparing how they rank representations and model layers, would clarify whether selectivity and MDL provide convergent or divergent assessments of probe quality. The paper's claim that they address the same underlying concern (probe capacity) would be empirically testable with such a design.

6. Limitations and Trade-offs

Limitation 1: The Difficulty Estimation Pipeline Is Impractically Expensive

The assumption or constraint. The entire compute-optimal framework depends on knowing each question's difficulty before allocating the test-time compute budget. The paper's method for estimating difficulty — generating 2048 samples per question and averaging either ground-truth correctness (oracle bins) or the PRM's predicted final-answer scores (predicted bins) — is extraordinarily expensive. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This is not a minor bookkeeping detail. Generating 2048 samples per question consumes far more compute than the largest test-time budgets studied in the paper (256–512 generations). For a single question at difficulty estimation time, the system spends 8× more compute than the maximum budget it subsequently optimizes.

The consequence. The headline efficiency gains — up to 4× over best-of-N — are computed after difficulty is already known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be:

$\text{Total cost} = \underbrace{2048 \text{ samples}}_{\text{difficulty estimation}} + \underbrace{N \text{ samples}}_{\text{strategy execution}}$

where $N$ ranges from 4 to 512. The difficulty estimation cost dominates the strategy execution cost, meaning the 4× efficiency figure is strictly an upper bound on achievable efficiency, not a realized deployment gain. A practitioner implementing this system would find that the difficulty estimation cost swamps any savings from smarter budget allocation for individual queries.

Furthermore, the difficulty estimation itself requires a trained PRM (for predicted bins) or ground-truth labels (for oracle bins). The PRM-based approach removes the need for ground-truth answers but still requires running the PRM on 2048 samples per question — an inference cost that scales linearly with the number of questions and quadratically with the sample count.

What evidence exists in the paper. Section 3.2 describes the difficulty estimation procedure: "For each question in the test set, the authors sample 2048 complete solutions from the base model and compute the pass@1 rate." Section 3.2 also notes that the predicted difficulty version "removes the need for ground-truth labels but still requires the computational cost of generating 2048 samples and scoring them." The paper explicitly flags this cost in Section 3.2, describing it as an "exploration-exploitation tradeoff" and "a key avenue for future work." However, no experiment measures or reports the actual wall-clock time, FLOP count, or dollar cost of difficulty estimation relative to strategy execution. The figures showing 4× efficiency gains (Figures 4 and 8) plot accuracy against generation budget excluding difficulty estimation cost. The x-axis represents only the strategy execution budget, not the total end-to-end compute.

Mitigation status. The paper acknowledges but does not resolve this limitation. It frames the difficulty estimation cost as an "exploration-exploitation tradeoff" and suggests future work on "pretraining or fine-tuning models to directly predict difficulty of a question" (Section 8). No such lightweight difficulty predictor is developed or evaluated in the paper. The paper also does not explore adaptive difficulty estimation — starting with a small number of samples, assessing difficulty from the initial score distribution, and then allocating the remaining budget — which could amortize the estimation cost into the solution process. The limitation is flagged for future work but remains a central unresolved barrier to practical deployment.

Limitation 2: The Method Provides No Benefit on Hard Problems

The assumption or constraint. The compute-optimal framework operates under an implicit but severe constraint: it can only help when the base model already produces correct solutions at some non-trivial rate. The paper defines five difficulty quintiles based on the base model's pass@1 rate, and across every method and every budget level studied, the hardest quintile (bin 5) shows near-zero improvement.

The consequence. Test-time compute — whether deployed as PRM-guided search, iterative revisions, or compute-optimal combinations — cannot compensate for fundamental capability gaps. The paper is explicit about this in the Section 7 takeaway box, but the magnitude of the limitation deserves emphasis:

Search (Figure 3, right): Bin 5 accuracy remains at 1–3% for all search methods and all budgets up to 256 generations. Beam search, best-of-N, and lookahead search all fail equivalently.
Revisions (Figure 7, right): Bin 5 accuracy hovers at roughly 2–3% regardless of the sequential-to-parallel ratio at 128 generations. No allocation strategy moves the needle.
FLOPs-matched comparison (Figure 9): The bin 5 scaling curve is essentially flat near 0–5% across all test-time compute budgets, and the 14× larger pretrained model substantially outperforms it. At $R \gg 1$ with PRM search, hard questions show a −52.9% relative disadvantage (Figure 1, bottom-right bar chart).

This means the method offers no path forward for problems outside the base model's capability distribution. If the model's pass@1 on a problem class is near zero, no amount of search or revision will help because there are no correct solutions in the proposal distribution to find or refine. The compute-optimal allocation policy cannot route around this — it can only shift compute to hard problems pointlessly or away from them (abandoning them), neither of which improves accuracy.

What evidence exists in the paper. The failure on hard problems is the most consistently replicated finding in the paper, appearing in every experiment broken out by difficulty:

Figure 3 (right): Bin 5 accuracy at 256 generations is approximately 2% for best-of-N and 1% for beam search.
Figure 7 (right): Bin 5 accuracy at 128 generations with the optimal sequential-to-parallel ratio is approximately 3%.
Figure 9: The bin 5 scaling line is essentially flat and well below the 14× larger model's performance (stars).
Figure 1 bar charts: Hard questions (bins 4–5 aggregated) show negative relative performance for test-time compute vs. the larger model at $R \approx 1$ and $R \gg 1$ .

The paper is transparent about this limitation in Section 7: "Test-time compute provides minimal gains on problems that are fundamentally outside the base model's capability range." The numbers quantifying this failure are among the most robust in the paper given they are replicated across search, revisions, and FLOPs-matched comparisons.

Mitigation status. The paper does not attempt to mitigate this limitation — it is arguably intrinsic to the approach. Test-time compute can amplify existing capability but cannot create it. The paper's contribution is to precisely characterize where the amplification works (easy-to-medium problems) and where it fails (hard problems), establishing a clear boundary condition rather than a fix. The practical implication is that practitioners must either accept that hard problems will remain unsolved, route them to a larger pretrained model, or invest in better pretraining rather than better inference — a tradeoff the paper explicitly analyzes in Section 7. No future work is proposed to address hard-problem performance because the limitation follows from the fundamental nature of test-time compute scaling.

Limitation 3: Single Benchmark, Single Model Family — Generalizability Is Unknown

The assumption or constraint. All experiments in the paper use a single benchmark (MATH, 500 test questions) and a single model family (PaLM 2-S*, with a 14× larger variant for the FLOPs-matched comparison). The paper does not evaluate on other reasoning benchmarks (e.g., GSM8K, coding tasks, logical reasoning) or on other model families (e.g., GPT, LLaMA, open-weight models). The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an assertion, not an empirical demonstration.

The consequence. The paper's central findings — the difficulty-dependent optimality of different strategies, the 4× efficiency gain from compute-optimal allocation, the verifier over-optimization threshold, the revision model's behavior on easy vs. hard problems — may be specific to:

Model calibration properties: The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration (e.g., better or worse at step-by-step reasoning, different error patterns) might exhibit different difficulty-dependent scaling curves. For instance, if a different model produces more reliable intermediate steps, the PRM might be more robust to optimization, shifting the over-optimization threshold and changing which strategy is optimal for which difficulty bin.
Model in-context learning capability: The revision model's ability to learn from incorrect in-context examples depends on the base model's capacity to condition on and learn from previous attempts. This capability varies substantially across model families and scales.
Task domain: MATH consists of competition-level math problems requiring multi-step symbolic reasoning. It is unclear whether the difficulty-dependent patterns generalize to code generation (where unit tests provide different verifier signals), logical reasoning (where truth conditions differ from mathematical correctness), or knowledge-intensive QA (where the bottleneck may be factual recall rather than reasoning).
Test set size and composition: The 500-question test set, split into five difficulty quintiles of approximately 100 each, then further split by two-fold cross-validation, means the compute-optimal policy is selected based on approximately 50 questions per fold per bin. This is a small sample — the selected strategies may overfit to the specific 500 questions and not represent the optimal policy for MATH problems in general or for other benchmarks.

What evidence exists in the paper. The paper provides no cross-benchmark or cross-model evaluation. All Figures (3, 4, 6, 7, 8, 9) report results on MATH with PaLM 2-S*. The authors' belief in representativeness is stated in Section 4: "We argue this model is representative of the capabilities of many contemporary LLMs and sits in a useful regime: non-trivial performance on MATH (roughly 10–19% pass@1 depending on the prompt and sampling configuration) but far from saturation." This argument is about the performance level, not about the qualitative behavior of test-time compute strategies, which could differ even among models with similar MATH accuracy.

The paper does provide one piece of indirect evidence on generalizability: the predicted difficulty bins (using PRM scores rather than ground-truth correctness) track the oracle bins closely (Figures 4 and 8, "curves largely overlap"). This suggests that the difficulty estimation approach is robust to the specific correctness signal, but it does not address whether the difficulty-dependent strategy optimality generalizes to other models or tasks.

Mitigation status. The paper does not mitigate this limitation. It acknowledges the single-model, single-benchmark scope implicitly by framing the results as "a first systematic scaling analysis" (Section 1) rather than a definitive universal characterization. Section 8 calls for future work extending the analysis to "other domains and modalities" and notes that "replicating the study on code generation... logical reasoning... and open-ended generation tasks would determine which findings are universal and which are domain-specific." However, no such replication is performed, and the practical consequence is that a practitioner using a different model family or a different reasoning domain cannot assume the paper's quantitative findings (4× efficiency gains, specific difficulty thresholds, optimal strategy lookup tables) will transfer.

Limitation 4: The 14× Larger Model Baseline Is Not Compute-Optimally Trained, Potentially Favoring Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales only model parameters when increasing pretraining compute, holding training data fixed. The authors explicitly acknowledge this departure from compute-optimal pretraining (Hoffmann et al., 2022):

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The comparison uses PaLM 2-S* with a model having approximately 14× more parameters, trained on the same data, and evaluated with greedy decoding only — no majority voting, no best-of-N, no test-time compute augmentation for the larger model.

The consequence. The pretraining baseline is weaker than it should be in two distinct ways:

Non-compute-optimal training. A Chinchilla-optimal model trained with 14× more total FLOPs would scale both parameters and data quantity, likely achieving better performance than a parameter-only-scaled model trained on the same data. The paper's comparison may therefore overstate the advantage of test-time compute over pretraining, because the pretraining baseline is not making optimal use of its compute budget.
No test-time compute for the larger model. The larger model uses greedy decoding with no augmentation. Giving the larger model even a modest test-time compute budget — say, best-of-8 with majority voting or a cheap verifier — would create a much stronger baseline. The paper's comparison is asymmetric: the smaller model gets compute-optimal test-time strategies while the larger model gets nothing. A symmetric comparison where both models receive their own compute-optimal test-time allocation might produce different conclusions about the pretraining-inference tradeoff.

The magnitude of these effects is unknown. The paper reports that compute-optimal test-time scaling with the smaller model can outperform the 14× larger model by +27.8% on easy questions at $R \ll 1$ (Figure 1). If the larger model were Chinchilla-optimal and equipped with its own test-time compute, this margin might shrink substantially or reverse.

What evidence exists in the paper. The paper explicitly acknowledges the non-Chinchilla training assumption in Section 7: "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." The use of greedy decoding for the larger model is stated but not explicitly defended — the paper treats greedy decoding as the natural baseline for a larger model and does not consider giving it test-time compute augmentation. No ablation or sensitivity analysis investigates how the FLOPs-matched comparison would change if the larger model received a test-time compute budget.

Mitigation status. The paper acknowledges but does not address this limitation. The authors frame the parameter-only scaling choice as "representative of a canonical approach" (specifically the LLaMA paradigm, Touvron et al., 2023) and defer the Chinchilla-optimal comparison to future work. No experiments explore alternative pretraining baselines (Chinchilla-optimal training, the larger model with test-time compute). The practical consequence is that the paper's headline finding — that test-time compute can substitute for pretraining — should be understood as an existence proof (it is possible under specific conditions) rather than a general prescription (it is always or usually preferable). The specific numerical margins (+27.8%, −52.9%, etc.) are conditional on a baseline that may not represent the best available pretraining approach.

Limitation 5: The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate, Requiring Post-Hoc Patches

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target (Section 6.1). This training setup creates an asymmetry: the model learns to revise incorrect answers toward correctness but never learns to recognize when an answer is already correct and should be preserved. The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones" during sequential revision chains.

This is a direct consequence of the training data construction procedure, which pairs independently sampled correct and incorrect solutions using edit-distance-based matching. The model never sees training examples where the correct answer appears in context, so at test time it has no signal for when to stop revising.

The consequence. Sequential revision chains are inherently unstable: the model may produce a correct answer at step $t$ , then incorrectly revise it to a wrong answer at step $t+1$ , then possibly correct it again at step $t+2$ . This creates a "wandering" behavior where accuracy oscillates along the revision chain rather than monotonically improving. The paper mitigates this by selecting the best answer from the entire chain (using majority voting or verifier-based selection) rather than always taking the final revision, but this is a post-hoc patch — it does not fix the underlying problem that the model cannot recognize correctness.

The practical implications are significant:

Latency: If the system generates a long revision chain but must evaluate all intermediate answers to find the best one, the effective latency is the full chain length, plus the cost of scoring all intermediate outputs. Sequential revisions already introduce serial dependencies that increase wall-clock time; the reversion problem means many of those serial steps may be wasted backtracking.
Verifier dependence: The mitigation strategy (within-chain selection) requires a reliable verifier to identify which step in the chain is correct. If the verifier itself makes errors, the system may select a reverted incorrect answer over a correct one. This couples the revision model's reliability to the verifier's quality in a way that is not present for parallel sampling.
Budget inefficiency: If 38% of correct answers are lost to reversion, a substantial fraction of the sequential generation budget is effectively wasted on producing answers that will not survive selection. The paper's compute-optimal allocation accounts for chain length but not for the probability that correct answers will be overwritten, potentially misestimating the effective yield of sequential revisions.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1, where the paper discusses "the correct-to-incorrect reversion problem." Figure 6 (left) shows the revision model's pass@1 trajectory over 64 steps: while pass@1 gradually improves from approximately 18.2% at step 1 to roughly 24–25% by steps 15–20, it does not monotonically increase — it oscillates in the 23–25% range, consistent with occasional reversion. The paper also notes that the ReST $^{EM}$ -trained revision model (Appendix K, Figure 16) exhibits even worse reversion behavior: "fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio," with the degradation attributed to the model failing to handle sequential revision properly.

Mitigation status. The paper partially mitigates this limitation through within-chain selection (majority voting or verifier-based selection across all steps in the chain), but the mitigation is a workaround rather than a solution. The underlying problem — that the model was never trained to preserve correct answers — is not addressed. The paper does not experiment with training the revision model on mixed sequences (including correct-to-correct transitions), does not explore training a separate "stop criterion" classifier, and does not analyze whether the compute-optimal allocation policy could account for reversion probability when deciding how many sequential steps to generate. Section 8 does not explicitly call out the reversion problem as an area for future work, though it does note that combining PRM search with revisions could help because the PRM could provide step-level feedback on whether a revision is on track.

Limitation 6: Revisions and PRM Search Are Never Combined — The Two Axes Remain Independent

The assumption or constraint. The paper studies two complementary test-time compute mechanisms — modifying the proposal distribution via iterative revisions (Section 6) and modifying the verifier via PRM-guided search (Section 5) — but never combines them. All search experiments use the base few-shot prompted LLM as the proposal distribution. All revision experiments use post-hoc verifier-based selection (best-of-N weighted across chains) rather than integrated PRM-guided search within the revision trajectory. The authors explicitly acknowledge this gap in Section 8:

"we did not experiment with PRM tree-search techniques in combination with revisions"

This is not an oversight — it represents a deliberate scoping decision to analyze the two axes independently before combining them. However, it means the paper's results represent a lower bound on what a fully integrated system could achieve.

The consequence. The two mechanisms have complementary, difficulty-dependent strengths that the paper carefully documents: revisions excel on easy problems where local refinement of nearly-correct answers is sufficient, while PRM search excels on medium-hard problems where global exploration of different solution strategies is needed. But the paper never tests whether these strengths are additive or synergistic. Several natural combined strategies remain unexplored:

PRM-guided revision: Use the PRM's per-step scores to decide when to revise — if the PRM's confidence drops, trigger a revision step; if the PRM is confident the solution is correct, stop.
Revision model as beam search proposal: Use the revision model (conditioned on previous attempts) as the proposal distribution within beam search, potentially generating higher-quality candidates at each step than the base model alone.
Cross-chain PRM scoring: Train a PRM on revision model outputs (the paper shows the base-LM PRM transfers poorly to revisions, Appendix J) and use it to score partial revision chains, enabling early pruning of unpromising revision trajectories.

The consequence is that the paper's headline efficiency gains (4× over best-of-N) and performance ceilings (~44% at 256 generations for revisions, ~39.5% for PRM search) may be substantially below what a combined system could achieve. The difficulty-dependent optimal allocation that the paper develops — best-of-N on easy problems, beam search on medium problems — is based on each mechanism in isolation. A combined system might find that easy problems benefit from PRM-guided revision (not just blind revision), or that medium problems benefit from revision-augmented beam search (not just base-model beam search). The optimal allocation policy for a combined system is unknown.

What evidence exists in the paper. The paper provides indirect evidence that combination could help:

Appendix J (Figure 15a) shows that the base-LM PRM underperforms on revision model outputs (distribution shift), suggesting that a revision-specific PRM would be needed for PRM-guided revision — a prerequisite that the paper does not develop.
Section 5.3 shows that beam search over-optimizes the PRM on easy problems (Figure 3, right, bin 1), while Section 6 shows that revisions help on easy problems (Figure 7, right, bin 2). This suggests a natural division of labor — use revisions to improve the proposal distribution and then use a light-touch verifier to select, avoiding the over-optimization that aggressive beam search causes — but this hypothesis is never tested.
The FLOPs-matched comparison (Figure 9) shows that revisions provide larger gains over pretraining than PRM search, suggesting that improving the proposal distribution may be more impactful than improving the verifier. But whether PRM search could further amplify revision gains is unknown.

Mitigation status. The paper does not mitigate this limitation. It explicitly acknowledges the gap as future work in Section 8: "Combining PRM tree-search and revisions" is listed as a key direction. The paper's contribution is to analyze the axes independently and establish their complementary difficulty-dependent behaviors — a necessary precursor to combination, but not a substitute for it. A practitioner implementing the paper's approach today would need to choose between search and revisions (or deploy them separately on different difficulty bins based on the independent analyses) rather than deploying an integrated system. The performance achievable with an integrated system remains an open question.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new probing method or a single technical fix. Instead, it performs a conceptual reframing of an entire subfield — and the magnitude of that reframing is substantial. Before this paper, probing classifiers were widely used but methodologically adrift. Researchers trained classifiers on frozen representations, reported accuracy numbers, and drew conclusions about what models "encode" or "understand," with little agreement on what those conclusions meant or which methodological choices validated them. The paper's contribution is to demonstrate that this state of affairs was not merely sloppy but logically incoherent: probing accuracy alone cannot answer the questions researchers were asking, because the measurement instrument (the probe) is confounded with the quantity being measured (information in the representations).

The paper makes this incoherence precise through its formal decomposition in Figure 1. By writing probing performance as PERF(g, f, D_O, D_P) — a function of four distinct, interacting components — the paper forces the field to confront a reality it had been able to ignore when the framework was described in prose: changing any one of those components can change the conclusion, and without controlling for each, the conclusion is uninterpretable. A study finding that "BERT encodes syntax" using a deep MLP probe on dataset X cannot be meaningfully compared to a study finding that "BERT does not encode syntax" using a linear probe on dataset Y, because the difference in probes and datasets confounds the comparison. The formal notation makes this dependency inescapable.

This is more than a methodological critique — it is a diagnostic that reorganizes the field's priorities. The paper shows that the most commonly reported number in probing research (raw accuracy) is the least informative, and that the most commonly debated question (linear vs. non-linear probes) is almost a distraction from the deeper issues of controls, causation, and dataset confounds. By elevating selectivity, minimum description length, control functions, and causal interventions into a unified framework, the paper redirects research attention from "which probe architecture is best?" to "what question am I asking, and which combination of probe, metric, and control can answer it?"

Reconciling contradictory findings. One of the paper's most valuable contributions is providing a framework for understanding why the probing literature contains apparently contradictory results. The Giulianelli et al. (2018) vs. Elazar et al. (2021) contradiction — one finds that gradient-based amplification helps, suggesting the model uses the probed property; the other finds that iterative removal of the property doesn't hurt, suggesting the model doesn't use it — appears paradoxical under the naive "probing accuracy = model usage" interpretation. The paper's causal taxonomy resolves this: amplification tests usability (can the model use property z when it's made more accessible?), while removal tests necessity (is property z required for the model's current behavior?). A property can be usable without being necessary — the model may have redundant strategies. The contradiction dissolves when the causal question is made precise, and the paper's framework provides the vocabulary for that precision.

Similarly, the long-standing debate about linear vs. non-linear probes — with some researchers insisting on linear probes for interpretability and others arguing for complex probes to capture non-linear information — is reframed as a measurement problem rather than a methodological preference. Selectivity (Hewitt and Liang, 2019) and accuracy-complexity trade-offs (Pimentel et al., 2020a) provide principled ways to compare any probe architecture by controlling for its standalone capacity, rather than imposing a priori restrictions. The paper does not resolve the debate in favor of one architecture, but it provides the tools to make the debate moot: use whatever probe you want, but report selectivity or MDL alongside accuracy so that the contribution of representation content is distinguishable from the contribution of probe memorization.

Which research directions become more or less attractive. The paper's analysis makes several research directions substantially more attractive:

Causal probing (interventions on representations, measuring downstream effects) becomes a first-class requirement for any claim about model usage, not an optional extension or a separate subfield. A probing study that reports accuracy (even with excellent controls) but does not address causation is, under this paper's framework, incomplete — it has answered "what information is present?" but not "what information is used?" The paper's synthesis of amplification, removal, and adversarial methods into a causal taxonomy makes intervention-based probing a central rather than peripheral methodology.
Capacity-controlled probing (selectivity, MDL, accuracy-complexity trade-offs, parameter-free extraction) becomes the standard for claims about representation content. The paper's finding that non-linear probes achieve high accuracy but low selectivity means that any study using expressive probes without capacity controls is producing numbers that are consistent with both "the representations encode the property" and "the probe is a good memorizer." The selectivity approach provides a relatively cheap diagnostic (just randomize labels and retrain) that can be retrofitted to existing probing studies.
Dataset-aware probing (varying D_P systematically, using control datasets D_{O,z} to test for incidental encoding) moves from a niche concern to a recognized validity threat. The paper's formal notation makes visible what was previously invisible: that probing conclusions are always conditional on the specific probing dataset used, and that generalizing from a dataset to an abstract linguistic capability requires justification that is rarely provided. Ravichander, Belinkov, and Hovy's (2021) control dataset methodology is elevated from a single study to a general principle.

Conversely, certain research practices become less defensible after this paper:

Reporting raw probing accuracy without baselines and controls is, after this paper, straightforwardly insufficient. The evidence that randomized representations yield above-chance accuracy is strong and replicated. A paper that reports 87.8% probing accuracy without comparing against randomized-model baselines, selectivity, or complexity controls is presenting a number that the field now knows to be uninterpretable. The bar for publication-quality probing has been raised.
Implicitly equating probing accuracy with model usage ("the model encodes property z" treated as synonymous with "the model uses property z") is no longer tenable. The Elazar et al. (2021) result — that properties can be removed without affecting task performance — directly falsifies this equivalence. Future work must either avoid usage claims (limiting itself to presence claims) or deploy interventions to support them.
Cross-study comparisons without controlling for probe and dataset differences become transparently invalid. The formal notation PERF(g, f, D_O, D_P) makes visible that two studies reporting "BERT's encoding of syntax" are measuring different quantities if they used different probes or datasets. The paper's framework provides the language for identifying what must be controlled to make comparisons meaningful.

A paradigm shift or a refinement? The paper is best characterized as a methodological paradigm shift within the subfield of NLP interpretability, rather than a refinement of existing practice. It does not tweak the probing framework — it redefines what constitutes a valid probing experiment. Before this paper, the implicit standard was: train a probe, report accuracy, and if accuracy is high, the model encodes the property. After this paper, the standard — at least for researchers who internalize its framework — is: specify whether the question is about presence, extractability, or usage; choose a probe and metric appropriate to that question; control for probe capacity via selectivity or MDL; control for dataset confounds via multiple probing datasets or control datasets; and if making causal claims, deploy interventions. This is not a refinement of the old standard — it is a qualitative change in what counts as evidence.

The paper explicitly does not advocate for a single "correct" probing methodology. Rather, it provides a decision framework (Section 5 summarizes this as: "If the goal is to measure information presence... If the goal is to measure information extractability... If the goal is to test causal usage...") that maps research questions to appropriate methodological choices. This is characteristic of a paradigm shift: it doesn't say "do X instead of Y," it says "the question you're asking determines what counts as a good answer, and the old methodology was answering a different question than you thought."

Follow-Up Research This Work Enables

A large-scale multi-probe, multi-metric, multi-intervention benchmark. The paper's framework implies that probing conclusions depend on the choice of probe, metric, and intervention method, but the paper can only cite individual studies that vary one component at a time (Hewitt and Liang vary selectivity, Pimentel et al. vary probe complexity, Elazar et al. vary intervention method). What is missing — and what this paper's synthesis makes clearly necessary — is a systematic study that applies all of these diagnostics to a common set of models and linguistic properties. A strong follow-up would: select 3–5 model families (e.g., BERT, RoBERTa, GPT-2, T5, LLaMA) at comparable scales; define 5–10 linguistic properties spanning morphology, syntax, semantics, and discourse; train a matrix of probes (linear, shallow MLP, deep MLP, parameter-free extraction via attention); evaluate all probes under all metrics (raw accuracy, selectivity, MDL, accuracy-complexity Pareto curves); and run all three intervention paradigms (gradient-based amplification, iterative projection removal, adversarial removal with fine-tuning) for each property. The output would be a comprehensive empirical map of how much method choice matters — what fraction of comparative rankings flip under different probe-metric combinations, whether selectivity and MDL produce convergent rankings, and whether amplification, removal, and adversarial methods agree on which properties are causally important. This benchmark would convert the paper's qualitative synthesis into quantitative guidance for practitioners.

A controlled study systematically varying the probing dataset D_P. The paper identifies the probing dataset as a major underinvestigated confound: "the effect of the probing dataset D_P — its size, composition, etc. — is similarly not well studied" (Section 4.4). A direct test would: take a fixed model f (e.g., BERT-base, frozen); identify 3–5 different annotated datasets for the same linguistic property z (e.g., POS tagging with Universal Dependencies across different domains — newswire, social media, biomedical text, learner English); train identical probes on each dataset; and measure both the absolute probing accuracy and the correlation of accuracy rankings across datasets. If probing accuracy on dataset A correlates poorly with probing accuracy on dataset B for the same property, then the dataset is a dominant confound — probing conclusions do not generalize from one dataset to another, and the field needs standards for dataset selection and reporting analogous to those for probe selection. If correlations are high, the dataset concern is less urgent. The paper's formal notation provides the hypothesis: PERF(g, f, D_O, D_P) should vary with D_P, but we don't know by how much.

A within-study comparison of selectivity, MDL, and information gain on identical data. The paper presents selectivity (Hewitt and Liang, 2019), control functions and information gain (Pimentel et al., 2020b), and MDL (Voita and Titov, 2020) as related but distinct approaches to capacity-controlled probing, and notes that Zhu and Rudzicz (2020) showed control tasks and control functions are "almost equivalent, both theoretically and empirically." However, selectivity, information gain, and MDL have never been computed on the same model-property-probe combinations. A study that does so would answer: do they produce convergent rankings of representations (layers, models, training checkpoints)? If they diverge — for instance, if MDL ranks representation A above B but selectivity ranks B above A — then the field needs guidance on which metric to prefer under which circumstances. The paper's framework suggests specific hypotheses: selectivity and information gain should be strongly correlated (given Zhu and Rudzicz's equivalence result), while MDL might diverge when probe architectures differ substantially in complexity. Testing this would either validate the paper's synthesis (if they converge) or refine it by identifying boundary conditions where different capacity-control methods answer different questions.

Developing sentence-level and discourse-level control tasks. The paper notes a specific gap: Hewitt and Liang's (2019) control tasks are "particularly suited for word-level properties z as they evaluate memorization of word types; it is less clear how to apply this idea more broadly, such as in sentence-level properties" (Section 4.1). This is a concrete methodological gap that the paper's framework makes visible but does not fill. A follow-up would design control tasks for sentence-level properties (sentiment, entailment, paraphrase detection, discourse relation classification) by randomizing labels at the sentence level and measuring selectivity. The challenge is that sentence-level randomization may not adequately control for probe memorization — a deep probe could still learn surface patterns (word overlap, length, specific lexical cues) from randomized labels — so the control task design may need to be more sophisticated, such as randomizing labels within groups of surface-similar sentences. A strong contribution would propose a general framework for sentence-level control tasks, validate it against the known word-level methods, and demonstrate that sentence-level probing conclusions without controls are as unreliable as Hewitt and Liang showed word-level conclusions to be.

A causal probing study applying all three intervention paradigms to the same model and properties. The paper's causal taxonomy (amplification tests usability, removal tests necessity, adversarial removal tests substitutability) is logically coherent but empirically untested as a unified framework. No study has applied gradient-based amplification (Giulianelli et al., 2018), iterative projection removal (Elazar et al., 2021), and adversarial removal with fine-tuning (Feder et al., 2021) to the same model on the same properties. A study that does so would answer critical questions: do properties that are "usable" (amplification helps) tend to also be "necessary" (removal hurts)? Or is there a large class of properties that are usable but substitutable — the model can use them when they're made more accessible, but can also function without them? This distinction has direct implications for model improvement: if a property is necessary, then ensuring the model learns it during training is critical; if it's merely usable, then emphasizing it during training may not improve performance, even though a probe can extract it. The paper's synthesis of Giulianelli et al. and Elazar et al. into a taxonomy makes this experiment conceptually obvious but empirically unavailable.

Extending probing to discover what is not encoded, and why. The paper notes the property predefinition problem: probing is inherently limited to properties researchers think to test (Section 4.5). Michael, Botha, and Tenney (2020) take an early step toward alleviating this by discovering latent clusters, but the more fundamental limitation — that probing cannot tell you what isn't there, only that a specific property wasn't found — remains. A follow-up study could systematically characterize the "unknown unknowns" of a model's representations: for a given model f, what linguistic properties are not linearly extractable at any layer, even though they are theoretically predictable from the input? This would require assembling a comprehensive battery of linguistic properties with annotated data, probing every layer for every property, and identifying which properties consistently yield chance-level probing accuracy even with optimal probe and layer selection. The negative results — properties that the model demonstrably does not encode — would be as informative as the positive ones for understanding what models learn. The paper's framework provides the methodological rigor (controls, baselines) needed to make negative results interpretable rather than ambiguous.

Practical Applications and Downstream Use Cases

Guiding probe selection for model auditing in production NLP systems. When an organization deploys a language model in a high-stakes setting (medical text processing, legal document analysis, content moderation), they often need to audit the model for specific linguistic capabilities — does it track negation correctly? does it respect syntactic constituency when making predictions? is it encoding protected attributes like dialect or gender? The paper's framework provides a decision tree for audit methodology: if the audit question is about information presence ("could the model, in principle, use this property?"), use a complex probe with selectivity controls and report accuracy-complexity Pareto curves to avoid conflating probe capacity with representation content. If the audit question is about causal usage ("does the model actually rely on this property?"), deploy intervention methods — amnesic probing or adversarial removal — and measure the impact on task performance. The paper's formal notation makes explicit what must be reported for an audit to be reproducible: which model f, which dataset D_O was it trained on, which probing dataset D_P, which probe g, and which controls were applied. Without this framework, production audits risk drawing invalid conclusions — e.g., concluding a model "encodes" a property based on high probing accuracy from an expressive, unconstrained probe that is largely memorizing the probing dataset. The paper's synthesis of selectivity, MDL, and intervention methods directly translates to auditing best practices.

Informing architecture design through causally-validated probing. The paper cites several cases where probing results motivated architectural changes — Cao et al. (2020) used layer-wise probing of linguistic specialization to design the DeFormer question-answering architecture; Dalvi et al. (2020) used redundancy probing to motivate efficient transfer learning. The paper's framework improves the reliability of this design loop by distinguishing properties that are merely present from those that are causally used. If a probing study finds that layer 6 encodes syntax but a subsequent intervention study finds that removing syntax from layer 6 does not affect task performance, then building an architecture that emphasizes syntax at layer 6 (based on the probing result alone) may be misguided. The paper's causal taxonomy provides a filter: architecture design should be guided by properties shown to be necessary (removal hurts) or at minimum usable (amplification helps on relevant examples), not merely present (probe can extract). This directly applies the Elazar et al. (2021) vs. Giulianelli et al. (2018) distinction to practical model engineering.

Improving interpretability benchmarks and leaderboards. The field has several benchmarks for evaluating interpretability methods (e.g., ERASER, EVALX, SyntaxGym) that include probing-based evaluation. The paper's framework provides more rigorous metrics for these benchmarks. Rather than ranking methods by raw probing accuracy, a benchmark could rank by selectivity (accuracy on real labels minus accuracy on randomized labels, controlling for probe memorization) or by the accuracy-complexity Pareto frontier (rewarding probes that are both accurate and simple). Parameter-free extraction methods (attention-based parsing, representational similarity analysis) could be treated as a distinct evaluation category that avoids the probe capacity confound entirely. The paper does not propose a specific benchmark redesign, but its framework provides the conceptual vocabulary and the empirical justification (Hewitt and Liang's selectivity results, Pimentel et al.'s accuracy-complexity trade-offs) for doing so. A downstream effect of this paper should be that interpretability benchmarks become harder to "game" with expressive probes that memorize rather than extract.

Evaluating representation quality for transfer learning and model selection. When choosing which pretrained model to use as a starting point for fine-tuning on a new task, practitioners often rely on probing benchmarks (e.g., SentEval, GLUE diagnostic sets) that measure which linguistic properties are encoded in the model's representations. The paper's framework refines this model selection process in two ways. First, capacity-controlled probing (selectivity, MDL) provides a more reliable ranking of representation quality than raw accuracy — a model that achieves high probing accuracy via a non-linear probe with low selectivity may have worse representations (from the perspective of linear readout, which is what downstream fine-tuning typically uses) than a model with lower raw accuracy but higher selectivity. Second, the dataset confound analysis (Section 4.4) warns that probing benchmarks are only informative to the extent that the probing dataset D_P is representative of the target transfer task — a model that excels at probing POS tagging on newswire text may not transfer well to POS tagging on biomedical text, and probing results on one dataset should not be assumed to generalize. The paper's formal decomposition makes these caveats operational: a practitioner selecting a model should check not just probing accuracy but whether the probing dataset, probe architecture, and control methodology are appropriate for their transfer scenario.