ArXiv: 2102.01017

🎯 Pitch

Pretrained language models frequently contradict themselves when queried with simple paraphrasesβ€”predicting different facts for semantically identical prompts. Even the best model fails 39% of the time on a task deliberately simplified to help it, exposing brittle surface-form reliance rather than robust knowledge encoding.

1. Executive Summary

This paper studies the consistency of pretrained language models β€” the invariance of their behavior under meaning-preserving paraphrases β€” when queried for factual knowledge. Using PARAREL, a manually curated benchmark of 328 cloze-style paraphrase patterns across 38 relations from T-REx, the authors probe BERT, RoBERTa, and ALBERT models (base and large variants) for consistency (whether a model predicts the same object for paraphrased prompts such as "X originally aired on [MASK]" vs. "X premiered on [MASK]") and find that all models exhibit poor consistency, with the best model (BERT-large) achieving only 61.1% consistency despite using restricted candidate sets that make the task easier. The paper further proposes a consistency-improving continual pretraining method that combines a novel two-sided KL divergence loss across paraphrase pairs with a standard masked language modeling objective, demonstrating a 5.8-point improvement in consistency over BERT-base (from 58.2% to 64.0%) after training on merely three relations, establishing that consistency can generalize to unseen relation types even when the base model's representational space remains organized by surface pattern rather than by entity identity.

2. Context and Motivation

The Core Problem: PLMs Don't Behave Consistently When You Ask the Same Question Differently

The central problem this paper investigates is deceptively simple: if you ask a pretrained language model the same factual question in two different ways, does it give you the same answer? The answer, as the paper systematically demonstrates, is often no β€” and this is a problem with far-reaching consequences for how we build, evaluate, and deploy NLP systems.

To make this concrete, consider the examples in Figure 1. A PLM is queried with the cloze-style prompt "Homeland originally aired on [MASK]" and predicts Showtime. When queried with the paraphrase "Homeland premiered on [MASK]", the same model predicts ABC. These two answers are contradictory β€” a TV show cannot have premiered on two different networks. The model is being inconsistent. Yet the same model, when queried about Seinfeld with the same pair of paraphrases, correctly predicts NBC both times. This selectivity β€” consistent on some facts, wildly inconsistent on others β€” is what makes the phenomenon both practically troubling and scientifically interesting.

The paper's scope is explicitly narrowed to factual knowledge consistency in Masked Language Models (MLMs) evaluated in a zero-shot setting. This is not a paper about fine-tuned models on downstream tasks. Rather, it asks a more fundamental question: has the pretraining process itself β€” training on massive text corpora with a masked language modeling objective β€” produced representations that abstract away from surface form and capture invariant knowledge? The authors frame this as a property that should exist before fine-tuning, because the pretrained model is supposed to serve as the "language understanding component" that downstream tasks inherit and build upon. If consistency is absent at the pretrained level, fine-tuning on small task-specific datasets (which typically lack paraphrase coverage) is unlikely to recover it.

Why This Matters: Four Lines of Motivation

The paper motivates its investigation along four interconnected lines, each of which would independently justify the work, but which together paint a compelling picture of why consistency must be measured and improved.

1. PLMs as Knowledge Bases Demand Consistency

One of the most influential recent uses of PLMs has been treating them as knowledge bases (Petroni et al., 2019). The appeal is obvious: instead of manually constructing structured KB schemas and painstakingly populating them with facts, you could simply ask a PLM natural language questions and extract answers from its parametric memory. This vision was energized by the LAMA benchmark (Petroni et al., 2019), which showed that BERT could recall factual knowledge β€” capital cities, birth places, employers β€” with non-trivial accuracy when queried with a single, fixed template per relation.

However, a KB has properties that go beyond raw accuracy. The paper explicitly identifies consistency as a key property of KBs, particularly for automatically constructed ones. A KB that today tells you "Paris is the capital of France" and tomorrow tells you "Lyon is the capital of France" when asked slightly differently is not a usable KB. The authors note that one of the biggest appeals of PLM-as-KB is that we can "query it in natural language β€” instead of relying on a specific KB schema." But this flexibility is only valuable if the model maps queries with identical intent but different surface forms to the same underlying representation and the same answer. If PLMs are sensitive to the particular phrasing of a query β€” if "X was born in [MASK]" and "X is native to [MASK]" produce different answers β€” then the entire PLM-as-KB paradigm rests on shaky foundations. The paper's investigation of consistency is thus a direct test of whether the PLM-as-KB vision is viable, rather than merely a measurement of yet another model property.

2. Consistency Is a Prerequisite for Robust Downstream NLP Systems

Beyond the KB use case, consistency matters for virtually every NLP application. The paper cites a range of work showing that inconsistency is a pervasive problem in deployed systems:

  • Question answering: QA models should give the same answer regardless of how a question is paraphrased (Asai and Hajishirzi, 2020; Gan and Ng, 2019). Yet models routinely fail at this β€” performance drops significantly on paraphrased versions of SQuAD questions.
  • Coreference resolution: Models should make consistent assignments β€” if entity A and entity B are identified as coreferent, this decision should propagate consistently through the document (Denis and Baldridge, 2009; Chang et al., 2011).
  • Summarization: Generated summaries should be factually consistent with the source document (Kryscinski et al., 2020) β€” the summary should not introduce contradictions or state facts that conflict with the original text.
  • Natural language inference (NLI): Models should not make inconsistent predictions on logically related premise-hypothesis pairs (Li et al., 2019), and their explanations for predictions should be internally consistent (Camburu et al., 2020).
  • Procedural text understanding: Understanding event sequences requires consistent tracking of states and entities across time steps (Du et al., 2019).

The crucial link the paper makes is that these downstream consistency failures may originate in the pretrained model itself. The standard NLP pipeline today is pretrain-then-finetune: a PLM absorbs general language capabilities during pretraining on large corpora, and then fine-tuning teaches it the format and nuances of a specific task. The pretrained model is supposed to provide the "good language understanding component" that fine-tuning builds upon. If that component is fundamentally inconsistent β€” if it maps paraphrases to different internal states and different predictions β€” then fine-tuning on downstream tasks, which typically involves relatively small datasets lacking systematic paraphrase coverage, will inherit and potentially amplify this inconsistency. Measuring consistency at the pretrained model level is therefore a diagnosis of whether the foundation is sound before anything is built on top of it.

The paper's authors frame this aspiration clearly: "obtaining this skill from a pretrained model would be extremely beneficial and has the potential to make specialized consistency solutions in downstream tasks redundant." In other words, if PLMs were consistent out of the box, we wouldn't need to design consistency-enforcing architectures or loss functions in every downstream task β€” the pretrained model would handle it.

3. Inconsistency Reveals Something About How PLMs Represent Knowledge β€” Or Fail To

The paper argues that studying inconsistency is not just practically useful but also scientifically revealing. Understanding when and why a model makes contradictory predictions can teach us about how knowledge is organized (or disorganized) in the model's representational space. This connects to broader questions in model interpretability and what these models actually learn from text.

The authors explicitly connect consistency failures to other documented representational pathologies: the tendency of word embeddings to conflate antonyms and synonyms (Nguyen et al., 2016), and the reporting bias problem where models overestimate the frequency of events because they're reported in text rather than because they're actually frequent (Shwartz and Choi, 2020). The common thread is that surface-level statistical patterns in pretraining corpora may dominate over deeper semantic invariants. A model might learn that the lexical pattern "X originally aired on [MASK]" correlates with one set of answers while "X premiered on [MASK]" correlates with a slightly different set, without ever abstracting to the underlying relation "broadcast_debut(X, Y)" that should be invariant to these surface variations. The paper's representational analysis in Section 7 directly probes this hypothesis by visualizing whether PLM representations cluster by entity (which would indicate knowledge-centric encoding) or by surface pattern (which would indicate surface-centric encoding).

4. The Pretraining Objective Itself Doesn't Encourage Consistency

A subtle but important motivational thread running through the paper is the observation that nothing in the standard pretraining procedure pushes models toward consistency. The masked language modeling objective minimizes the cross-entropy loss for predicting masked tokens given context. This objective rewards a model for being good at predicting words that actually appear in the training data, not for producing consistent predictions across different contexts. Consider the training signal for the relation "X originally aired on Y" versus "X premiered on Y":

  • In Wikipedia, sentences using "originally aired on" might co-occur with certain TV shows and networks more often than sentences using "premiered on" (which might appear in different sections of articles or in different writing styles).
  • A model maximizing the likelihood of the training data would learn these distributional correlations β€” it might assign higher probability to NBC in the context of one pattern and higher probability to CBS in the context of the other, even for the same show, because that's what the training statistics reflect.
  • The model is never penalized for making inconsistent predictions across paraphrases because the training objective operates on individual tokens in individual contexts, never on the relationship between predictions across different contexts.

This is not just a training limitation β€” it's a fundamental misalignment between the training objective (next-token prediction) and the desired property (semantic invariance). The paper's consistency-improvement method in Section 8 directly targets this gap by introducing an auxiliary loss that explicitly penalizes divergent predictions across paraphrases, demonstrating that the pretraining objective can be augmented to induce consistency β€” but that it doesn't happen naturally.

Where Prior Work Falls Short

The paper positions itself relative to several bodies of prior work, each of which touches on consistency but leaves critical gaps.

Prior Work on PLMs as KBs Measured Accuracy, Not Consistency

The LAMA benchmark (Petroni et al., 2019) established that PLMs can recall factual knowledge, but it used only one pattern per relation. This means LAMA evaluates whether the knowledge is somewhere in the model β€” it can be extracted with the right prompt β€” but says nothing about whether the knowledge is robustly encoded such that it can be extracted with any reasonable prompt. Follow-up work like LPAQA (Jiang et al., 2020) and AUTOPROMPT (Shin et al., 2020) focused on finding better patterns β€” ones that elicit higher accuracy β€” rather than on measuring whether predictions are consistent across different patterns. The paper cites these works approvingly for what they contributed (better accuracy through prompt engineering) but notes that a KB needs both accuracy and consistency, and the latter had not been measured.

The paper's framework thus extends the LAMA paradigm from single-pattern probing to multi-pattern probing: instead of asking "does the model know the capital of France?" it asks "does the model know the capital of France *regardless of whether I ask 'The capital of France is [MASK]', 'France's capital, [MASK]', or 'France's capital city, [MASK]'?"

Prior Work on Downstream Consistency Didn't Address the Pretrained Model

Several works have developed methods for improving consistency in specific downstream tasks β€” for example, Asai and Hajishirzi (2020) use logic-guided data augmentation and regularization for consistent QA, and Li et al. (2019) introduce a logic-driven framework for NLI consistency. However, these operate at the fine-tuning stage and require task-specific architectures or training procedures. The paper's contribution is to investigate whether the pretrained model itself is consistent, and whether consistency can be improved at the pretraining stage, before any task-specific fine-tuning occurs. If this succeeds, downstream tasks would inherit improved consistency without requiring task-specific consistency solutions.

Prior Work on Representation Probing Focussed on Linguistic Properties, Not Knowledge Invariance

A large body of work has probed PLMs for syntactic and linguistic capabilities β€” whether they encode part-of-speech, dependency structure, subject-verb agreement, etc. (Tenney et al., 2019; Goldberg, 2019; Hewitt and Manning, 2019). The paper draws on this probing methodology but applies it to a different question: not "does the model encode syntax?" but "does the model's encoding of factual knowledge survive syntactic and lexical variation?" This shifts the focus from what linguistic structures are present in representations to whether those representations are organized in a way that supports invariant behavior β€” a question that prior probing work had not addressed systematically.

Prior Work on Representational Space Structure Suggested (But Didn't Prove) a Knowledge-Encoding Problem

The paper builds on findings from Chi et al. (2020) and Ravfogel et al. (2020), who observed that contextualized word representations cluster strongly by syntactic pattern rather than by semantic content. In other words, if you encode "The dog chased the cat" and "The cat was chased by the dog" and look at the representation of the masked position in a related probe, the representations cluster by the syntactic frame (active vs. passive) rather than by the underlying event. The paper extends this insight to factual knowledge: do representations of "X's capital, [MASK]" cluster with representations of "The capital of X is [MASK]" when X is the same entity, or do they cluster with other instances of the same pattern regardless of the entity? The prior work suggested this might be a problem, but the paper provides the first direct test with a carefully constructed paraphrase benchmark designed specifically for this purpose.

How This Paper Positions Itself

The paper's positioning can be understood through three key framing choices:

It reframes "does the model know X?" as "does the model know X robustly?" The LAMA paradigm asks whether factual knowledge can be extracted. This paper asks whether that extraction is invariant to how you ask. The distinction is crucial: a model that can answer "What is the capital of France?" correctly when asked with the right prompt but fails when the prompt is rephrased doesn't really "know" the capital of France in any robust sense β€” it has learned a brittle, surface-level association.

It separates measurement from intervention. The paper is structured in two complementary parts: a measurement study (Sections 3–7) that builds the PARAREL benchmark and uses it to quantify consistency across multiple PLMs, and a method study (Section 8) that proposes and evaluates a consistency-improving continual pretraining method. This split allows the paper to establish the severity of the problem before offering a solution β€” and it means that PARAREL can be used as an independent evaluation benchmark by future work even if the specific consistency-improvement method is superseded.

It explicitly connects pretraining data composition to model behavior. The paper observes a striking result: BERT models are consistently more accurate and more consistent than RoBERTa and ALBERT models. This is notable because RoBERTa generally outperforms BERT on downstream benchmarks (Liu et al., 2019). The paper hypothesizes that the difference stems from training data composition: BERT was trained primarily on Wikipedia (a relatively clean, factual, self-consistent knowledge source), while RoBERTa and ALBERT were trained on much larger corpora that include diverse and potentially contradictory sources (news, Reddit, books). The implication is provocative: more pretraining data is not always better for knowledge consistency, because diverse corpora may contain conflicting facts or surface patterns that confuse the model. This connects to broader discussions about data quality vs. quantity in pretraining (which would later become central to debates about GPT-3 and its successors), and it positions the paper's findings as relevant to decisions about how pretraining corpora are composed.

The Gap This Paper Fills

In summary, the paper identifies and addresses a specific critical gap: prior work had not systematically measured whether PLMs' factual knowledge predictions are consistent across meaning-preserving paraphrases, nor investigated whether consistency can be improved at the pretraining stage. The paper fills this gap through three contributions: (1) PARAREL, a high-quality manually curated benchmark for measuring knowledge consistency; (2) a systematic empirical characterization of consistency in BERT, RoBERTa, and ALBERT, revealing low overall consistency with high inter-relation variance and an unexpected advantage for BERT; and (3) a consistency-improving continual pretraining method that demonstrates transfer to unseen relations. The problem is motivated both practically (PLMs as KBs, robust downstream performance) and scientifically (understanding how knowledge is organized in neural representations), and the paper explicitly connects consistency failures to the fundamental misalignment between the pretraining objective and the desired property of semantic invariance.

3. Technical Approach

3.1 Reader Orientation

This paper constructs a measurement framework and a training intervention β€” not a single monolithic system, but rather an evaluation apparatus (the PARAREL benchmark) for quantifying knowledge consistency in pretrained language models, paired with a consistency-improving continual pretraining method that can be applied to any masked language model. The problem it solves is that PLMs make contradictory predictions when asked the same factual question in different ways, and the "shape" of the solution is twofold: first, build a high-quality set of manually curated paraphrase patterns to quantify exactly how severe the inconsistency problem is, and second, design an auxiliary loss function β€” the two-sided KL divergence consistency loss β€” that can be added to the standard masked language modeling objective during continued pretraining to explicitly penalize divergent predictions across paraphrased prompts, thereby teaching the model that "X originally aired on [MASK]" and "X premiered on [MASK]" should produce the same output distribution.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's technical approach has four major components, organized into a measurement pipeline (components 1–2) and an intervention pipeline (components 3–4):

  1. PARAREL β€” the Paraphrase Pattern Resource β€” a manually curated collection of 328 cloze-style English paraphrase patterns for 38 binary relations from the T-REx knowledge graph. Each relation (e.g., born-in, capital-of) is associated with 2–20 patterns that are quasi-paraphrases of one another. This resource is the input to all consistency measurements and the training data source for the consistency-improvement method.

  2. Consistency Probing Framework β€” a protocol for querying a frozen PLM with multiple paraphrased patterns for the same knowledge graph tuple, restricting the model's output vocabulary to the set of valid candidate objects for that relation, and measuring whether the model predicts the same object across all pattern pairs. This component takes PARAREL patterns and T-REx tuples as input and produces consistency scores, accuracy scores, and consistent-accuracy scores as output.

  3. Consistency Loss Function β€” a two-sided KL divergence computed across the predicted distributions of all paraphrase pairs for a given relation, restricted to the relation-specific candidate set. This loss explicitly penalizes the model for producing different probability distributions for paraphrased prompts.

  4. Consistency-Guided MLM Continual Training β€” a procedure that combines the consistency loss with a standard masked language modeling loss (applied on the paraphrase patterns themselves) and continues pretraining a PLM on a small number of relations. The model learns to generalize consistency behavior to unseen relations.

Information flows through the measurement pipeline as follows: for each relation $r_i$, the set of paraphrase patterns $P_i$ is populated with a subject entity $s$ from a T-REx tuple $\langle s, o \rangle$ β†’ the PLM encodes each filled pattern and produces a probability distribution over its vocabulary β†’ the vocabulary is filtered to retain only valid objects for relation $r_i$ β†’ the top-1 prediction is extracted and compared across all pattern pairs β†’ consistency is computed as the fraction of pairs that agree on the top-1 prediction. Information flows through the intervention pipeline as follows: batches of patterns from the same relation are encoded β†’ the consistency loss and MLM loss are computed and summed β†’ backpropagation updates the PLM parameters β†’ the updated model is evaluated on held-out relations.

3.3 Roadmap for the Deep Dive

  • First, the formal definition of consistency and the probing framework (Section 3 of the paper), because everything downstream depends on what "consistency" means operationally and how it is measured β€” this establishes the evaluation protocol that both the diagnostic experiments and the improvement method use.
  • Second, the PARAREL resource construction (Section 4), because it is the instrument that makes the measurements possible β€” understanding how patterns were collected, validated, and organized is essential for interpreting the results.
  • Third, the experimental setup β€” models, data splits, restricted candidate sets, and evaluation metrics (Sections 3.2, 5.1, 5.2) β€” because the numerical results can only be understood in the context of these concrete methodological choices.
  • Fourth, the consistency loss function and continual training procedure (Section 8.1), because this is the intervention β€” the paper's proposed solution to the consistency problem it diagnoses.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a measurement and intervention paper whose core idea is that PLMs exhibit poor knowledge consistency under paraphrasing, that this can be systematically measured with a carefully constructed benchmark of manually validated paraphrase patterns, and that consistency can be improved by augmenting the pretraining objective with a distribution-matching loss across paraphrases.

Defining Consistency Operationally

The paper must first establish what "consistency" means in a way that can be turned into a computable metric. This definition is critical because it determines everything about how the benchmark is constructed, how models are evaluated, and what the loss function in Section 8 tries to optimize.

The definitional challenge. Consistency sounds intuitive β€” a model should give the same answer when asked the same question β€” but operationalizing this for PLMs queried with cloze-style prompts requires several careful specifications. The paper must define: (1) what counts as "the same question asked differently" (quasi-paraphrases), (2) what output space is being compared (the full vocabulary? the top-1 prediction? the entire distribution?), (3) what knowledge graph structure is assumed (N-1 relations only, because N-M relations can be correct with different answers), and (4) what it means for predictions to "agree" when the model might produce answers that are different but not contradictory (e.g., predicting a city in one case and the country containing that city in another).

The formal definition used in the paper. The paper defines a model as consistent if, given two cloze-phrases that are quasi-paraphrases of each other, it makes non-contradictory predictions on N-1 relations over a large set of entities. The constraint to N-1 relations is crucial: these are relations where each subject maps to exactly one object (e.g., born-in, capital-of, originally-aired-on). For such relations, any two different predictions are necessarily contradictory β€” you cannot have been born in two different cities. This sidesteps the problem of distinguishing "different but both correct" from "different and contradictory" by simply excluding relations where the former is possible. For N-M relations (like shares-border-with), the paper separately measures determinism β€” whether the model returns results in the same order β€” but this is not the consistency metric.

Quasi-paraphrases and why the "quasi" matters. The paper explicitly adopts the concept of quasi-paraphrase from Bhagat and Hovy (2013). A quasi-paraphrase is "a more fuzzy version of a paraphrase" that "does not rely on the strict, logical definition of paraphrase and allows to operationalize concrete uses of paraphrases." This is a pragmatic choice: requiring strict logical equivalence (which would demand that two patterns be perfectly interchangeable in every possible context) would make it nearly impossible to construct a benchmark of reasonable size. For example, "X originally aired on Y" and "X premiered on Y" are quasi-paraphrases β€” they mean essentially the same thing for the purpose of extracting the originally-aired-on relation, even though there might be edge cases where "premiered" and "originally aired" have subtly different implications. The paper notes this is "in the spirit of the RTE definition" of entailment (Dagan et al., 2005), which similarly supports a more flexible use of the notion.

Why consistency is separated from correctness. A crucial design choice is that consistency does not require the predictions to be factually correct. A model that consistently predicts the wrong answer (e.g., always predicting London for every born-in query about any person) would score 100% consistency but 0% accuracy. The paper treats "consistency" and "accuracy" as orthogonal axes and measures them independently. This separation is methodologically important because it isolates the property of interest β€” invariance to surface form β€” from the model's factual knowledge. A model could have excellent factual knowledge (high accuracy on one pattern) but poor consistency (different patterns trigger different, contradictory answers). Conversely, a model could be perfectly consistent but factually wrong. The combined metric, Consistent-Acc (accuracy only on cases where all patterns predict the correct object), captures the desirable conjunction: both knowing the fact and knowing it robustly.

The Consistency Probing Framework: How Measurements Are Performed

The paper's measurement apparatus is a precise experimental protocol that takes a PLM, a set of paraphrase patterns, and a set of knowledge graph tuples, and produces numerical consistency scores. Understanding this protocol in detail is essential because it is used uniformly across all experiments in Sections 6, 7, and 8.

Step 1: Setting up the data. Let $D_i$ be a set of subject-object KB tuples for relation $r_i$ β€” for example, $\langle \text{Homeland}, \text{Showtime} \rangle$, $\langle \text{Seinfeld}, \text{NBC} \rangle$, and so on. Let $P_i$ be the set of quasi-paraphrase cloze-patterns for that relation β€” for example, "X originally aired on [MASK]", "X premiered on [MASK]", and any other paraphrases in PARAREL. A "pattern" is a string template with a placeholder X for the subject and a placeholder [MASK] at the position where the model should predict the object.

Step 2: Populating the patterns. For each subject $s$ in $D_i$, the framework substitutes $s$ for X and [MASK] for the object position in every pattern $p \in P_i$. This produces a set of concrete queries such as "Homeland originally aired on [MASK]" and "Homeland premiered on [MASK]". The model is then run on each populated pattern, producing β€” at the position of the [MASK] token β€” a probability distribution over its entire vocabulary. Let $f_\theta(p_s)$ denote the output of the encoding function (e.g., BERT) parameterized by $\theta$ on the populated pattern $p_s$, and let $f_\theta(p_s)[i]$ be the score (logit or probability) assigned to vocabulary item $i$.

Step 3: Restricting the candidate set. Instead of considering the model's prediction over the entire vocabulary (which contains many words that are not valid KB entities β€” a PLM might predict "tv" or "cable" for the originally-aired-on relation), the framework restricts the output vocabulary to only the set of valid objects for that specific relation from the underlying KB. For the born-in relation, this means the candidate set is all cities, countries, etc. that appear as objects in born-in tuples in T-REx. This makes the task "easier for the PLM, especially in the context of KBs" as the paper notes, but critically, it means that if consistency is still poor even with this restricted vocabulary, the problem is severe. The paper argues that "poor consistency in this setup strongly implies that consistency would be even lower without restricting candidates." This is a conservative evaluation design β€” it gives the models every advantage and still finds substantial inconsistency.

Step 4: Extracting the prediction. From the restricted distribution, the model's top-1 prediction (the vocabulary item with the highest probability among the valid candidates) is extracted. This is the arg max over the restricted vocabulary:

o^p,s=arg⁑max⁑j∈Crifθ(ps)[j]\hat{o}_{p,s} = \arg\max_{j \in \mathcal{C}_{r_i}} f_\theta(p_s)[j]

where $\mathcal{C}_{r_i}$ is the candidate set for relation $r_i$, $f_\theta(p_s)[j]$ is the model's score for candidate $j$, and $\hat{o}_{p,s}$ is the top-ranked predicted object.

What it computes: For a specific pattern $p$ and subject $s$, this finds the KB entity that the model assigns the highest probability to, among all entities that are legitimate answers for that relation type. The argmax discards the full distribution and keeps only the discrete choice.

Why this form: The use of top-1 is consistent with how PLMs are typically evaluated as KBs (Petroni et al., 2019) β€” the question is whether the model "knows" the answer, operationalized as whether the correct answer has the highest probability. Using the full distribution for consistency measurement (e.g., requiring distributions to match rather than just the top-1) would be a different and potentially more informative metric, but the paper opts for top-1 matching to align with the KB evaluation literature. The restricted candidate set is a practical necessity because the full vocabulary includes many non-entity tokens that would dominate the top-1 predictions for most queries, making it impossible to evaluate factual knowledge at all.

Step 5: Computing consistency. For a relation $r_i$ with $n$ patterns, and for each KB tuple $(s, o) \in D_i$, the framework examines all $n(n-1)/2$ pairs of patterns. For each pair $(p_k, p_l)$, it checks whether the top-1 predictions agree:

Consistent(pk,pl,s)=1[o^pk,s=o^pl,s]\text{Consistent}(p_k, p_l, s) = \mathbb{1}[\hat{o}_{p_k,s} = \hat{o}_{p_l,s}]

The Consistency metric for a relation is then the percentage of all pattern pairs across all KB tuples for which this indicator is true. Formally, if $|D_i|$ is the number of tuples for relation $r_i$ and $n_i$ is the number of patterns:

\text{Consistency}(r_i) = \frac{100}{|D_i| \cdot \frac{n_i(n_i-1)}{2}} \sum_{s \in D_i} \sum_{k=1}^{n_i} \sum_{l=k+1}^{n_i} \mathbb{1}[\hat{o}{p_k,s} = \hat{o}{p_l,s}]$$

where $|D_i|$ is the number of subject-object tuples for relation $r_i$, $n_i$ is the number of paraphrase patterns for that relation, $\hat{o}_{p_k,s}$ is the top-1 prediction for pattern $k$ and subject $s$, and $\mathbb{1}[\cdot]$ is the indicator function returning 1 when the predictions match and 0 otherwise.

What it computes: the fraction of pattern pairs for which the model makes the same top-1 prediction, aggregated over all knowledge-graph tuples for that relation. A score of 100% means every pattern pair produces the same top-1 answer for every subject. A score of 50% means half the pattern pairs disagree.

Why this form: the pairwise comparison captures the intuitive notion of consistency β€” "if I ask two different ways, do I get the same answer?" The averaging over all pairs prevents the metric from being dominated by a single "bad" pattern that disagrees with everything else (which would distort the score if only, say, a baseline pattern were compared to all others). The aggregation over tuples ensures the metric reflects consistency across the full distribution of entities rather than being idiosyncratic to a few entities. The paper reports the macro average over relations (each relation weighted equally regardless of how many tuples it has), which prevents relations with many tuples from dominating the overall score, but notes that micro average produces similar results.

Step 6: Computing Accuracy and Consistent-Acc. The paper also reports two related metrics. Accuracy is simply the fraction of tuples for which the model's top-1 prediction on the base pattern (the original LAMA pattern from Petroni et al., 2019) matches the gold object β€” this is the standard LAMA evaluation. Consistent-Acc is much stricter: for a given tuple, the prediction is counted as correct only if all patterns for that relation predict the correct object. Formally:

Consistent-Acc(ri)=100∣Diβˆ£βˆ‘s∈Di1[β‹€p∈Pi(o^p,s=os)]\text{Consistent-Acc}(r_i) = \frac{100}{|D_i|} \sum_{s \in D_i} \mathbb{1}\left[\bigwedge_{p \in P_i} (\hat{o}_{p,s} = o_s)\right]

where $o_s$ is the gold object for subject $s$, and $\bigwedge$ denotes logical AND over all patterns. This metric combines the requirements of both accuracy (the prediction must be correct) and consistency (it must be correct regardless of pattern) β€” it is zero for any tuple where even one pattern predicts the wrong answer.

The PARAREL Resource: Construction and Validation

PARAREL is the paper's primary instrument β€” without it, none of the consistency measurements would be possible. The resource is manually curated in a four-step process designed to produce high-quality, validated paraphrase sets that reflect genuine linguistic variation rather than surface-level word substitution.

Step 1: Base patterns from LAMA. PARAREL begins with the single pattern per relation provided by the LAMA benchmark (Petroni et al., 2019). These patterns were originally designed to probe factual knowledge extraction and serve as the starting point β€” each relation has one "base pattern" that is known to work (in the sense that PLMs can often extract the correct entity using it). For example, the place-of-birth relation uses "X was born in [MASK]" as its base pattern.

Step 2: Augmentation from LPAQA with manual filtering. LPAQA (Jiang et al., 2020) automatically generated additional patterns for LAMA relations using two methods: back-translation (translating the base pattern into another language and back to English, producing paraphrases like "X was born in [MASK]" β†’ "X saw the light of day in [MASK]") and pattern extraction from Wikipedia sentences that contained both the subject and object of known KB tuples (e.g., finding the sentence "Beethoven, a native of Bonn, composed..." and extracting the pattern "X, a native of [MASK]"). However, because these methods are automatic, many generated patterns are not correct paraphrases β€” the back-translation might change the meaning, or the extracted pattern might be idiomatic to a specific sentence and not generalizable. The PARAREL authors manually reviewed all LPAQA patterns and kept only the subset that are correct paraphrases. This manual filtering step is critical because including incorrect paraphrases would conflate the model's inability to recognize paraphrases with the benchmark's noise.

Step 3: Syntax-based Wikipedia search using SPIKE. SPIKE (Shlain et al., 2020) is a search engine over Wikipedia sentences that supports syntax-based queries β€” it can find sentences that match a specified syntactic pattern (dependency parse). The authors used SPIKE to search for Wikipedia sentences containing subject-object tuples from T-REx that expressed the target relation. They then manually inspected the retrieved sentences and extracted additional paraphrase patterns. For instance, if a Wikipedia sentence about a company contained "Acme Corp, founded by John Doe in 1985,..." and ⟨Acme Corp, John Doe⟩ is a founded-by tuple, the pattern "X, founded by [MASK]" would be extracted. This step ensures that the patterns in PARAREL reflect patterns that actually occur in natural text (specifically Wikipedia), increasing the ecological validity of the benchmark β€” the patterns are not just theoretically valid paraphrases but are patterns that a PLM would actually encounter (or fail to encounter) during pretraining.

Step 4: Expert linguistic augmentation. The annotators, using their linguistic expertise, added additional paraphrases of the base pattern that weren't captured by the previous steps. This step fills gaps that automatic methods miss β€” for example, less common syntactic constructions or lexical choices that are nevertheless valid paraphrases.

Expert review and agreement protocol. Two additional experts reviewed all patterns and corrected them, engaging in discussion until reaching agreement. Patterns on which they could not agree were discarded. This multi-expert review is a crucial quality control measure β€” it means that every pattern in PARAREL has been vetted by multiple linguistically trained annotators who concurred that it is a valid paraphrase.

Human annotation study for validation. To quantitatively assess the quality of PARAREL, the authors conducted a formal human annotation study. For each relation, they sampled up to five paraphrases and compared each to the base pattern from LAMA. The populated patterns (with random subjects and objects from T-REx) were shown to NLP graduate student annotators, who were asked whether the two sentences are paraphrases. The study included two types of items:

  1. Paraphrase candidates (156 items): pattern pairs that PARAREL claims are paraphrases.
  2. Controls (61 items): pattern pairs from different relations that are known not to be paraphrases, included to verify that annotators can distinguish paraphrases from non-paraphrases and to calibrate the agreement scores.

The controls use the same subjects and objects so that annotators must rely on the pattern itself (not the arguments) to make the paraphrase judgment. For example, if annotators see "X was born in Y β†’ X is native to Y" (paraphrase candidate) and "X was born in Y β†’ X is the capital of Y" (control), the control makes it clear whether annotators are genuinely evaluating the paraphrase relationship rather than just agreeing because the arguments are the same.

Agreement results. The human agreement on the paraphrase candidates was 95.5% and on the controls was 98.3%. These high agreement scores indicate that PARAREL patterns are consistently recognized as paraphrases by human judges, validating the quality of the resource. Disagreements were inspected and used to fix additional problems, further improving quality. The near-perfect agreement on controls (where few annotators mistakenly called non-paraphrases paraphrases) confirms that the annotators understood the task and were applying meaningful judgment rather than just clicking through.

Final resource statistics. The completed PARAREL contains:

  • 38 relations (from the original 41 LAMA relations, excluding three that are "poorly defined, or consist of mixed and heterogeneous entities")
  • 328 patterns total
  • Average of 8.63 patterns per relation, with a minimum of 2 and a maximum of 20
  • Average of 4.74 unique syntactic variations per relation (patterns with different dependency paths between subject and object)
  • Average of 6.03 unique lexical variations per relation (patterns with different words)

The distinction between syntactic and lexical variation is important for the analysis in Section 6.3, where the paper investigates whether models are robust to purely syntactic changes (same words, different structure) versus both syntactic and lexical changes.

Paraphrase type analysis. The paper also provides a detailed characterization of the types of paraphrases in PARAREL, categorized using the taxonomy from Bhagat and Hovy (2013) plus several new types they observed. This analysis is not merely descriptive β€” it helps interpret why models might struggle with certain patterns and informs what kinds of linguistic variation the consistency-improvement method needs to handle. The most common types in a random sample of 100 paraphrase pairs are:

  • Synonym substitution (41 examples): e.g., "X died in [MASK]" ↔ "X expired at [MASK]"
  • Function word variations (16): e.g., "X is a [MASK] citizen" ↔ "X, who is a citizen of [MASK]"
  • Converse substitution (10): e.g., "X maintains diplomatic relations with [MASK]" ↔ "[MASK] maintains diplomatic relations with X" (swapping subject and object)
  • Irrelevant addition (11): e.g., "X shares border with [MASK]" ↔ "X shares a common border with [MASK]"
  • Change of tense (10): e.g., "X is developed by [MASK]" ↔ "X was developed by [MASK]"
  • Topicalization transformation (8): e.g., "X plays in [MASK] position" ↔ "playing as a [MASK], X"
  • Verb/Noun conversion (7): e.g., "The headquarter of X is in [MASK]" ↔ "X is headquartered in [MASK]"

The diversity of paraphrase types is important: if all patterns were simple synonym substitutions, the benchmark would only test lexical robustness. The inclusion of syntactic transformations (topicalization, apposition, voice changes) and structural reordering (converse substitution, other syntactic movements) ensures PARAREL tests a broad spectrum of linguistic variation.

The Consistency Loss Function

The paper's intervention β€” the method for improving consistency β€” centers on a novel loss function that explicitly penalizes a PLM for producing different output distributions when queried with paraphrased patterns. This section unpacks the mathematical formulation, the design choices, and the rationale.

The ideal (but non-differentiable) objective. The natural formulation of a consistency objective would require the top-1 predictions for all patterns of the same relation to be identical:

min⁑θ sim(arg⁑max⁑ifθ(Pn)[i], arg⁑max⁑jfθ(Pm)[j])\min_\theta \ \text{sim}\left(\arg\max_i f_\theta(P_n)[i], \ \arg\max_j f_\theta(P_m)[j]\right)

where $f_\theta(P_n)[i]$ is the model's score for vocabulary item $i$ on pattern $P_n$, and sim is some similarity measure (e.g., indicator of equality). However, this objective contains an arg max operation, which is discrete, non-differentiable, and therefore impossible to optimize with gradient-based methods. The gradient of an arg max is zero almost everywhere, so backpropagation cannot provide a learning signal.

The relaxation: match the full distributions, not just the top-1. The paper relaxes this objective by requiring that the predicted probability distributions β€” not just the hard top-1 choice β€” be identical across patterns. For pattern $P_n$ of relation $r_i$, let $Q_n^{r_i} = \text{softmax}(f_\theta(P_n))$ be the softmax-normalized probability distribution over the vocabulary. The goal becomes:

Qnriβ‰ˆQmriforΒ allΒ pairsΒ n,mΒ ofΒ patternsΒ forΒ relationΒ riQ_n^{r_i} \approx Q_m^{r_i} \quad \text{for all pairs } n, m \text{ of patterns for relation } r_i

This relaxation is differentiable because the softmax output is continuous and smooth with respect to the model parameters β€” small changes in the logits produce small changes in the distribution, and the gradient flows through.

Measuring distributional distance with two-sided KL divergence. To measure the similarity between two probability distributions, the paper uses two-sided (symmetric) Kullback-Leibler divergence:

Lpair(Qn,Qm)=DKL(Qnβˆ₯Qm)+DKL(Qmβˆ₯Qn)\mathcal{L}_{\text{pair}}(Q_n, Q_m) = D_{\text{KL}}(Q_n \| Q_m) + D_{\text{KL}}(Q_m \| Q_n)

where $D_{\text{KL}}(Q_n \| Q_m) = \sum_i Q_n[i] \log\frac{Q_n[i]}{Q_m[i]}$ is the standard KL divergence from $Q_n$ to $Q_m$, summing over all vocabulary items $i$.

What it computes: For a single pair of patterns, the KL divergence $D_{\text{KL}}(Q_n \| Q_m)$ measures how much information is lost if we use distribution $Q_m$ to approximate $Q_n$. It is non-negative and zero only when $Q_n = Q_m$ pointwise. The sum of the forward and reverse KL divergences makes the measure symmetric β€” penalizing cases where $Q_n$ assigns high probability to something $Q_m$ assigns low probability to, and vice versa. Each term penalizes different types of distributional mismatch: the forward KL penalizes cases where $Q_n$ has mass where $Q_m$ does not (it is "mode-covering" β€” ensuring $Q_m$ covers all the probability mass of $Q_n$), while the reverse KL penalizes cases where $Q_m$ has mass where $Q_n$ does not (it is "mode-seeking" β€” ensuring $Q_m$ doesn't assign probability where $Q_n$ says it shouldn't). The sum provides a balanced penalty.

Why two-sided KL over one-sided: using only $D_{\text{KL}}(Q_n \| Q_m)$ would be asymmetric β€” the loss would be different depending on which pattern is the "reference." This could lead to degenerate solutions where the model makes one pattern's distribution very peaked and the other very flat (so that the peaked distribution is "contained" within the flat one, yielding low forward KL but not actually identical). The two-sided formulation ensures the loss is zero only when the two distributions are identical.

Restricting to the candidate set. Computing the KL divergence over the entire vocabulary (30,000+ tokens for BERT) would be computationally expensive and would include many tokens that are irrelevant to the factual knowledge being probed. Moreover, forcing the model to match distributions over irrelevant tokens could interfere with its general language modeling capabilities. The paper therefore filters the vocabulary to only the $k$ tokens from the candidate set of each relation β€” the set of valid KB objects for that relation type. The distributions $Q_n^{r_i}$ are renormalized over this restricted set before computing the KL divergence.

The full consistency loss for a relation. Rather than enforcing pairwise consistency between specific pattern pairs, the paper encourages a more general solution by enforcing consistency across all pattern pairs for a relation simultaneously:

Lc=βˆ‘n=1kβˆ‘m=n+1k[DKL(Qnriβˆ₯Qmri)+DKL(Qmriβˆ₯Qnri)]\mathcal{L}_c = \sum_{n=1}^{k} \sum_{m=n+1}^{k} \left[ D_{\text{KL}}(Q_n^{r_i} \| Q_m^{r_i}) + D_{\text{KL}}(Q_m^{r_i} \| Q_n^{r_i}) \right]

where $k$ is the number of patterns for relation $r_i$, $Q_n^{r_i}$ is the restricted and renormalized predicted distribution for pattern $n$, and the double sum over $n < m$ ensures each unordered pair is included exactly once.

What it computes: for a single relation and a single KB tuple (with the subject filled into all patterns), this loss sums the two-sided KL divergences over all $k(k-1)/2$ pairs of patterns. The result is a single non-negative scalar that is zero if and only if all patterns produce identical distributions over the candidate set, and positive otherwise.

Why sum over all pairs rather than comparing to a single reference: comparing all patterns to a single "canonical" pattern would risk making the model overfit to that specific pattern's distribution. By enforcing mutual consistency among all pairs, the loss encourages the model to find a shared representation that works for all paraphrases β€” effectively learning that the underlying relation, not the surface pattern, should determine the output.

Consistency-Guided MLM Continual Training

The consistency loss alone is not sufficient β€” the paper must integrate it into a training procedure that improves consistency without destroying the model's existing capabilities. This section describes the full training pipeline.

The catastrophic forgetting problem. If the model were trained only on the consistency loss, it could trivially achieve perfect consistency by collapsing all patterns for a relation to the same output distribution β€” but that distribution might have nothing to do with the correct factual answer, and the model would lose its general language understanding capabilities. This is a classic catastrophic forgetting scenario in continual learning. The paper addresses this by combining the consistency loss with a standard masked language modeling loss.

MLM loss on patterns. The paper experiments with two sources of text for the MLM loss:

  1. MLM on general Wikipedia text β€” continuing the standard pretraining objective on the same kind of data the model was originally trained on.
  2. MLM on the paraphrase patterns themselves β€” applying MLM to the same patterns used for the consistency loss.

The paper found that the second option works better. This makes intuitive sense: training the MLM objective on the patterns themselves ensures that the model maintains its ability to predict the correct masked tokens in those specific contexts, which is precisely where the consistency loss is trying to align distributions. If the MLM loss were on unrelated Wikipedia text, the model might optimize the two objectives somewhat independently, with the consistency loss affecting only a small fraction of the model's behavior and the MLM loss pulling the model back to its original state.

Batching and the combined loss. The training proceeds by batching together $l = 8$ tuples from the same relation. For each tuple, the consistency loss $\mathcal{L}_c$ is computed over all patterns. Additionally, the MLM loss $\mathcal{L}_{\text{MLM}}$ is computed on the populated patterns (the model must predict the masked token correctly). The two losses are combined with a hyperparameter $\lambda$:

L=Ξ»Lc+LMLM\mathcal{L} = \lambda \mathcal{L}_c + \mathcal{L}_{\text{MLM}}

where $\lambda \in \{0.1, 0.5, 1\}$ is a weight controlling the strength of the consistency loss relative to the MLM loss, determined by validation performance. The MLM loss term has an implicit coefficient of 1.

What it computes: the total loss is a weighted sum of two objectives applied to the same batch of pattern-populated queries. The consistency loss pushes the model to produce identical distributions across paraphrases; the MLM loss pushes the model to produce distributions that place high probability on the correct (gold) masked token. Together, they encourage the model to be both consistent and accurate.

Why this weighted combination: the hyperparameter $\lambda$ trades off between consistency and accuracy. If $\lambda$ is too high, the model may achieve high consistency by collapsing to a uniform (or arbitrary) distribution over candidates β€” consistent but wrong. If $\lambda$ is too low, the consistency loss is negligible and the model reverts to its original inconsistent behavior. The paper tunes $\lambda$ on a validation set to find the balance that maximizes Consistent-Acc β€” the metric that requires both accuracy and consistency.

Why train on only three relations? A notable design choice is that the consistency improvement method uses only three randomly selected relations for training: original-language, named-after, and original-network. These were chosen randomly from the non-location-related relations to prevent train-test leakage (many T-REx relations are location-based, so excluding them from training ensures the model isn't just memorizing location-specific patterns). The fact that the method generalizes from merely three relations to 25 unseen test relations is the paper's key evidence that the model is learning a generalizable consistency skill rather than simply memorizing pattern mappings for the training relations.

Training hyperparameters and protocol. The paper performs "minimal tuning" β€” training for three epochs, sweeping $\lambda \in \{0.1, 0.5, 1\}$ and selecting the best model based on Consistent-Acc on a validation set of three additional relations (randomly selected from the remaining non-training relations). The base model is BERT-base. This minimal tuning is intentional β€” the authors emphasize that "our method is aimed to be simple, effective, and to require only minimal supervision." The small number of training relations and minimal hyperparameter search demonstrate that the gains are not the result of extensive engineering but rather of a method that works robustly.

Ablation study components. The paper evaluates several variants of the training procedure to isolate the contribution of each component:

  • -consistency β€” fine-tuning on the three training relations with only the MLM loss (no consistency loss). This tests whether simply seeing more data from these relations improves performance, even without the explicit consistency objective.
  • -typed β€” applying the consistency loss over the full vocabulary rather than restricting to the candidate set. This tests whether the candidate-set restriction is necessary or whether consistency can be learned over the entire output space.
  • -MLM β€” using only the consistency loss without the MLM regularization. This tests whether the consistency loss alone is sufficient or whether the MLM loss serves as a necessary regularizer against catastrophic forgetting.

Evaluation Splits and Cross-Validation

The paper's evaluation in Section 8 uses a careful train-validation-test split to ensure the consistency improvements generalize to unseen relations:

Stratification by relation type. The relations are split into location-based and non-location-based. All location-based relations are excluded from training to prevent the model from exploiting location-specific patterns. From the remaining non-location relations, three are randomly selected for training, three for validation, and the remaining 25 for testing.

Why this split matters. If the training and test relations shared underlying semantic categories (e.g., if both included location-based relations), the model might improve on test relations simply because it learned patterns specific to location expressions ("in X," "located in X," "X-based") rather than because it learned a general consistency skill. The strict separation between location and non-location relations eliminates this confound.

Metric reporting. All results in Table 7 are reported as "aggregated results for the 25 relations in the test" using macro averaging β€” each relation is weighted equally regardless of how many KB tuples it contains. Standard deviations are reported across relations to show the variance. Statistical significance is assessed using McNemar's test, a paired non-parametric test appropriate for comparing per-instance predictions (following the recommendations of Dror et al., 2018, 2020).

4. Key Insights and Innovations

Innovation 1: Consistency Is Reframed as a First-Class Model Property, Independent of Accuracy

The paper's most fundamental intellectual move is separating consistency from correctness and establishing it as an independently measurable property of pretrained language models. Prior work on PLMs as knowledge bases β€” particularly the LAMA benchmark (Petroni et al., 2019) β€” asked a unidimensional question: "does the model know the right answer?" The evaluation was accuracy alone, measured with a single canonical prompt per relation. This framing implicitly conflates two distinct failure modes: the model might lack factual knowledge (it doesn't know the answer), or it might possess the knowledge but fail to access it reliably across different linguistic formulations (it knows the answer but can't retrieve it consistently).

By defining consistency as "the invariance of behavior under meaning-preserving alternations in input" and measuring it separately from accuracy, the paper introduces a two-dimensional diagnostic framework. A model can be accurate but inconsistent (gets the right answer with one prompt, wrong with another β€” a brittle knowledge representation), consistent but inaccurate (always gives the same wrong answer β€” a stable but incorrect knowledge representation), or ideally both consistent and accurate (robust knowledge). The Consistent-Acc metric β€” counting a prediction as correct only when all paraphrase patterns produce the correct answer β€” is the paper's operationalization of this conjunction, and it is far stricter than standard accuracy.

This reframing matters because it changes what it means to say a model "knows" a fact. If a model answers "What is the capital of France?" correctly with "The capital of France is [MASK]" but incorrectly with "France's capital, [MASK]", it doesn't really know the capital of France in any robust sense β€” it has learned a brittle surface-level association. The paper provides the vocabulary and the measurement apparatus to make this distinction precise, and in doing so, it raises the bar for what counts as knowledge in neural models. This is not an incremental metric tweak β€” it is a fundamental reconceptualization of what PLM-as-KB evaluation should measure.

The evidence for why this distinction is critical shows up starkly in the results: BERT-large achieves 48.1% Accuracy but only 29.5% Consistent-Acc (Table 3). Nearly 40% of the facts the model appears to "know" (under the standard single-pattern evaluation) are actually not robustly retrievable β€” the model succeeds with one phrasing but fails with another. This gap between Accuracy and Consistent-Acc quantifies the brittleness tax β€” the fraction of seemingly-known facts that are actually surface-level artifacts rather than genuine knowledge. Without separating consistency from accuracy, this brittleness would be invisible.

Innovation 2: The Pretraining Data Composition Hypothesis β€” Why BERT Beats RoBERTa at Consistency

The paper reports a counterintuitive empirical finding that BERT models consistently outperform RoBERTa and ALBERT on both accuracy and consistency β€” with BERT-large achieving 61.1% consistency versus 56.3% for RoBERTa-large and 52.1% for ALBERT-xxlarge (Table 3). This is striking because RoBERTa was explicitly designed to improve upon BERT and generally does so on downstream benchmarks (Liu et al., 2019). The paper offers a causal hypothesis for this reversal: training data composition, specifically the prominence of Wikipedia in BERT's pretraining corpus versus the larger, more heterogeneous corpora used by RoBERTa and ALBERT.

The hypothesis is that Wikipedia β€” a relatively clean, fact-checked, stylistically consistent knowledge source β€” provides a training signal that encourages stable factual representations. When a model is trained primarily on Wikipedia, facts tend to be stated consistently across articles, and surface patterns for expressing the same relation are drawn from a constrained stylistic distribution. In contrast, RoBERTa's training data (which includes news, Reddit, books, and diverse web text) exposes the model to contradictory facts (different sources may disagree), diverse styles (the same fact expressed in wildly different linguistic forms), and varying levels of reliability (some sources are authoritative, others are not). In this heterogeneous training environment, the optimal strategy for minimizing next-token prediction loss may be to learn pattern-specific associations rather than invariant knowledge representations β€” because different patterns genuinely correlate with different "facts" in the training distribution.

This hypothesis is supported by the paper's experiment on pretraining corpus size (Table 4), using RoBERTa variants trained on increasing amounts of data from Warstadt et al. (2020). The smallest model (trained on 1M tokens) achieves 37.1% consistency β€” higher than models trained on 10M (29.8%) and 100M (31.5%) tokens, though its accuracy is lower. This non-monotonic relationship between data quantity and consistency is a provocative negative result: more data can actually hurt consistency, even as it improves accuracy. The 1M-token model has fewer parameters (to avoid overfitting) and is exposed to less data β€” which may mean it encounters fewer contradictory facts and learns more stable, albeit less comprehensive, knowledge representations.

This is a fundamental reframing of the "more data is better" assumption that dominates pretraining discourse. The paper's implication β€” "Training bigger models with more data (such as GPT-3) is not always beneficial" β€” anticipates concerns that would later become central to debates about data quality, deduplication, and curated pretraining corpora. It suggests that consistency imposes a data quality requirement that is distinct from the data quantity requirements for accuracy, and that the two may sometimes be in tension.

Innovation 3: The Representational Space Diagnosis β€” PLMs Cluster by Pattern, Not by Entity

The paper goes beyond behavioral measurement to provide a representational diagnosis of why inconsistency occurs, using t-SNE visualization and V-measure clustering analysis (Section 7.2). The key finding is that when PLMs encode patterns populated with subjects, the resulting representations at the masked token position cluster primarily by surface pattern rather than by the subject entity. In Figure 3, encoding the capital relation across multiple subjects and multiple patterns reveals that points from the same pattern (e.g., "The capital of X is [MASK]") cluster together regardless of which country X refers to, rather than clustering by country (all patterns for "France" together, all patterns for "Germany" together, etc.).

This is a diagnostic insight, not a method. It reveals why consistency is poor at the architectural level: the model's internal geometry is organized around surface form rather than semantic content. When the representation is dominated by pattern-specific features, each paraphrase effectively routes through a different region of the representational space, and the model's predictions become pattern-dependent because the decision boundary sees different features depending on which pattern is used. A knowledge-centric representation would instead map all paraphrases for a given entity to the same region, making the downstream prediction invariant to pattern choice.

What makes this finding innovative is not the clustering methodology itself β€” prior work by Chi et al. (2020) and Ravfogel et al. (2020) had shown that contextualized representations cluster by syntactic structure β€” but the application to factual knowledge consistency. The paper demonstrates that the syntactic clustering phenomenon observed in prior probing work has direct behavioral consequences: when representations cluster by pattern rather than by entity, model predictions become pattern-dependent, and consistency is the casualty. This connects representational geometry to behavioral reliability, providing a mechanistic hypothesis for why paraphrases trigger different answers.

The paper's further finding β€” that the correlation between representational clustering (V-measure) and behavioral consistency is "close to zero" β€” is a subtle but important negative result. It means the representational space's organization does not simply determine consistency in a straightforward way. The model may be using only a subset of the representation for its predictions (the behaviorally relevant subspace), while the full representation includes many dimensions encoding pattern-specific features that don't affect the top-1 prediction. This complicates the simple narrative that "poor representations cause poor consistency" and suggests a more nuanced picture: the representation is a mixture of knowledge-relevant and pattern-specific features, and consistency depends on which subset dominates the prediction.

Innovation 4: Consistency Can Be Learned as a Transferable Skill from Minimal Supervision

The paper's final contribution is demonstrating that consistency can be improved at the pretraining level and generalizes to unseen relations β€” and critically, that this requires only minimal supervision (three training relations). The BERT-ft model achieves 64.0% consistency versus 58.2% for BERT-base (Table 7), a 5.8-point improvement that is statistically significant (p β‰ͺ 0.01), along with improvements in Accuracy (45.6% β†’ 47.4%) and Consistent-Acc (27.3% β†’ 33.2%).

What makes this a conceptual innovation rather than merely a method is the generalization claim. The model is trained on three randomly chosen non-location relations (original-language, named-after, original-network) and tested on 25 held-out relations. This is not pattern memorization β€” the model has never seen the test-relation patterns during consistency training. Something about the training procedure causes the model to learn a generalizable skill: "when you encounter multiple linguistic formulations that express the same underlying relation, your output distributions should match." This transfers across relations because the consistency loss teaches the model to abstract away from the specific lexical items in the patterns and attend to the shared relational structure.

The ablation study (Table 7) reveals why this transfer works and what the necessary components are. The -MLM ablation β€” training with only the consistency loss and no MLM objective β€” achieves the highest consistency (80.8%) but catastrophically low accuracy (16.9%) and Consistent-Acc (9.1%). This is a revealing failure mode: the model learns to be consistent by collapsing to a degenerate solution (predicting the same distribution regardless of input), which satisfies the consistency loss but destroys factual knowledge. The MLM loss serves as a regularizer that anchors the model's predictions to the correct answers, preventing this collapse. The effectiveness of this simple combination β€” consistency loss for invariance, MLM loss for accuracy β€” suggests that the two objectives are complementary rather than competing.

The -consistency ablation is equally informative: fine-tuning on the three relations with only MLM (no consistency loss) improves over the baseline (from 58.2% to 60.9% consistency, from 45.6% to 46.9% accuracy), but lags behind the full method. This means mere exposure to paraphrase data is not sufficient β€” the explicit distribution-matching objective is needed to teach invariance. The improvement from MLM-only fine-tuning likely reflects the model learning better pattern-specific associations from seeing more examples, but without the pressure to align distributions across patterns, it does not learn to abstract away from surface form.

The paper's finding that this method does not improve downstream QA performance on SQuAD with paraphrased questions (mentioned in Section 8.3) is a sobering caveat that prevents overclaiming. The consistency skill learned during pretraining does not automatically transfer to all downstream tasks β€” perhaps because SQuAD requires different kinds of linguistic reasoning, or because the consistency signal from three relations is too weak to reshape the model's behavior across dramatically different task formats. This negative result is valuable because it bounds the scope of the method and points to the need for task-specific consistency interventions or stronger pretraining consistency signals.

This innovation is best characterized as an existence proof: consistency can be improved through pretraining-level intervention with minimal supervision, demonstrating that the inconsistency problem is not fundamental to the architecture or the pretraining paradigm, but rather a consequence of the training objective's indifference to paraphrase invariance. The fact that only three relations suffice for generalization suggests that consistency is a meta-skill β€” once the model learns that different surface forms should produce the same output for the same underlying relation, it can apply this principle to new relations without explicit training on them. This is a conceptual advance because it shifts the question from "can we make models consistent?" (which the paper answers affirmatively) to "how much consistency supervision is enough, and how can we provide it at scale?"

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The experiments use the MATH benchmark (Hendrycks et al., 2021), specifically the split from Lightman et al. (2022): 12,000 training questions and 500 test questions. MATH consists of high-school competition-level math problems, chosen because test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences β€” mathematical reasoning fits this profile because it requires multi-step logical deduction rather than novel factual recall.

  • Base model(s). All experiments use PaLM 2-S* (Codey) (Anil et al., 2023). The authors argue this model is "representative of the capabilities of many contemporary LLMs" and sits in a useful regime: non-trivial performance on MATH (roughly 10–19% pass@1 depending on the prompt and sampling configuration) but far from saturation, leaving room for test-time compute to make a difference. For the FLOPs-matched comparison, a second model with approximately 14Γ— more parameters is used as the pretraining-scaled baseline.

  • Metrics. The primary metric is MATH test accuracy (%) β€” the fraction of the 500 test questions for which the selected final answer matches the ground truth. Answers are graded using the grading function released by Lightman et al. (2022) (Appendix G). When analyzing difficulty-dependent behavior, accuracy is reported within each of the five difficulty quintiles separately. The paper also reports pass@1 and majority voting accuracy as intermediate metrics. A generation budget β€” the number of complete sampled solutions β€” is the universal unit of test-time compute, swept across powers of 2 from 1 to 512 generations.

  • Baselines.

    • Majority voting: select the most common final answer among N sampled solutions, with no learned verifier.
    • ORM best-of-N weighted: score N solutions with an outcome reward model, then apply best-of-N weighted selection (following Li et al., 2023).
    • PRM best-of-N weighted: score N solutions with the process reward model, then apply best-of-N weighted selection.
    • Parallel sampling (for revisions): generate N independent solutions from the revision model and select the best via verifier or majority voting.
    • Greedy decoding from the 14Γ— larger model (for the FLOPs-matched comparison): the larger model uses no test-time compute augmentation.

  • Generation budget / compute accounting. One "generation" equals one complete sampled answer from the base LLM. For best-of-N and beam search, the budget equals the number of beams or samples N. For lookahead search with k lookahead steps, the cost is N Γ— (k + 1) to account for the additional rollout computation (Section 5.3). Budgets are swept across powers of 2, typically from 2^0 to 2^9 (1 to 512 generations). For FLOPs-matched comparison, pretraining FLOPs are approximated as X = 6ND_pretrain and inference FLOPs as Y = 2ND_inference, where N is model parameters (Section 7).

  • Cross-validation / statistical protocol. To avoid contaminating strategy selection with test-set performance, the authors use two-fold cross-validation within each difficulty bin on the 500-question test set. The best strategy is selected on one fold and evaluated on the other, with results averaged (Section 3.2). This is applied for both the compute-optimal search policies (Figure 4) and the compute-optimal revision policies (Figure 8). Difficulty bins are constructed from the model's own pass@1 rates over 2,048 samples per question, split into quintiles.

Main Quantitative Results

Search Against PRM Verifiers (Section 5)

Aggregate comparison (Figure 3, left). Across all 500 test questions with a maximum budget of 256 generations, beam search with M = 4 significantly outperforms best-of-N weighted at low budgets. At 4 generations, beam search achieves roughly 27% accuracy versus roughly 16% for best-of-N weighted. At high budgets (64–256), beam search performance flattens and falls slightly below best-of-N weighted: best-of-N weighted reaches approximately 38% at 512 generations, while beam search (M = 4) plateaus around 34%. Lookahead search (both k = 1 and k = 3) generally underperforms all methods at the same generation budget due to its higher per-step cost. The 3-step lookahead variants converge to similar performance as other methods at very high budgets but never surpass them. Majority voting trails all verifier-based methods substantially, reaching only about 29% at 512 generations.

Difficulty-bin analysis (Figure 3, right). When results are disaggregated by difficulty quintile (beam search M = 4 vs. best-of-N weighted, shown at four budget levels: 4, 16, 64, 256 generations), a clear and non-monotonic pattern emerges:

  • Bin 1 (easiest): Beam search accuracy decreases from roughly 78% to 77% as the budget goes from 4 to 256, while best-of-N weighted increases from 68% to 88%. This is the clearest evidence of PRM over-optimization β€” aggressive beam search finds solutions that exploit the verifier signal on problems where the base model already produces correct answers at high rates.
  • Bin 2: Beam search improves modestly (roughly 14% β†’ 32%) but best-of-N weighted improves faster (roughly 14% β†’ 60%), maintaining a clear advantage at high budgets.
  • Bin 3: Beam search consistently outperforms best-of-N weighted across all budgets, reaching roughly 34% vs. 23% at 256 generations.
  • Bin 4: Beam search shows the strongest relative advantage, reaching roughly 17% vs. 10% for best-of-N at 256 generations.
  • Bin 5 (hardest): Both methods hover near 1–3% regardless of budget.

Compute-optimal search (Figure 4). By selecting the best search strategy per difficulty bin at each budget level, compute-optimal (oracle bins) achieves approximately 27% accuracy at 16 generations, roughly matching PRM best-of-N weighted at 64 generations β€” a 4Γ— compute reduction. At 256 generations, compute-optimal oracle reaches approximately 39.5%, surpassing PRM best-of-N weighted at the same budget (roughly 37%). Compute-optimal with predicted difficulty bins tracks the oracle version closely, particularly at lower budgets β€” the two curves "largely overlap" per the authors β€” with the predicted version reaching approximately 37% at 256 generations. Both compute-optimal variants consistently outperform ORM best-of-N weighted (which peaks around 34% at 512 generations) and majority voting (around 29%).

PRM vs. ORM (Figure 14, Appendix F). At 2,048 samples, PRM best-of-N weighted achieves approximately 40% accuracy versus roughly 35% for ORM best-of-N weighted and roughly 30% for majority voting. The gap between PRM and ORM widens with the number of samples.

Revision Model Results (Section 6)

Pass@1 trajectory (Figure 6, left). Starting from approximately 18.2% pass@1 at step 1, the revision model's per-step accuracy improves to roughly 24–25% by steps 15–20 and remains in the 23–25% range out to 64 steps. The model generalizes beyond its 4-step training horizon.

Sequential vs. parallel (Figure 6, right). At 64 generations: sequential + best-of-N weighted achieves approximately 41.5% versus parallel + best-of-N weighted at approximately 39% β€” a roughly 2.5 percentage point gap. Under majority voting, sequential + majority achieves approximately 38% versus parallel + majority at approximately 35% β€” a roughly 3 percentage point gap. Sequential marginally outperforms parallel under both selection mechanisms.

Sequential-to-parallel ratio sweep (Figure 7, left). For a fixed generation budget of 256, the optimal ratio is around 2ΒΉ to 2Β³ (2:1 to 8:1 sequential-to-parallel), achieving approximately 43–44% accuracy. Fully parallel (leftmost point) yields approximately 40%, while fully sequential (rightmost point) yields approximately 42%. At lower budgets (8–32 generations), fully sequential is optimal β€” the curves are monotonically increasing with the sequential-to-parallel ratio.

Difficulty-dependent ratio (Figure 7, right). At 128 generations:

  • Bin 1: Performance is essentially flat across all ratios, around 90–92%. Easy questions are insensitive to allocation strategy.
  • Bin 2: Slight advantage for higher sequential ratios, approximately 63% at fully sequential vs. 58% at fully parallel.
  • Bin 3: A clear optimal ratio emerges at moderate sequential-to-parallel values (around 2ΒΉ to 2Β³), reaching approximately 42% vs. 35% at the extremes.
  • Bin 4: Similar pattern, with the peak at a moderate ratio achieving roughly 18% vs. 14% at fully parallel.
  • Bin 5: All ratios produce roughly 2–3% accuracy.

Compute-optimal revisions (Figure 8). Selecting the optimal sequential-to-parallel ratio per difficulty bin: at 64 generations, compute-optimal oracle achieves approximately 40%, matching parallel best-of-N weighted at 256 generations β€” a 4Γ— improvement. At 256 generations, compute-optimal oracle reaches approximately 44%, compared to roughly 41% for best-of-N weighted and 37% for parallel-only. Compute-optimal predicted bins perform slightly below oracle bins at high budgets (approximately 41% at 256 generations) but still substantially outperform the parallel baseline. The parallel baseline appears to plateau around 36–37% at high budgets, while compute-optimal scaling continues to improve.

FLOPs-Matched Comparison: Test-Time vs. Pretraining Compute (Section 7)

Revisions (Figure 9, left; Figure 1, top-right bar chart). Comparing PaLM 2-S* with compute-optimal revisions against the ~14Γ— larger model across three values of R = D_inference / D_pretrain:

DifficultyR β‰ͺ 1 (0.16)R β‰ˆ 1 (0.79)R ≫ 1 (22)
Easy (bin 1)+11.8%+3.5%βˆ’11.9%
Medium (bins 2–3)+27.8%+16.7%+5.4%
Hard (bins 4–5)+21.6%(implied negative)βˆ’37.2%

At R β‰ͺ 1, test-time compute outperforms the larger model across all difficulty levels. At R ≫ 1, it only remains preferable on easy questions, with hard questions showing a βˆ’37.2% relative disadvantage. The line plots in Figure 9 show the compute-optimal scaling curve per difficulty bin, with the 14Γ— larger model's greedy performance indicated by stars at three x-axis positions corresponding to the three R values. On bin 1 (purple, topmost line), the scaling line is above all three stars for revisions. On bin 5 (blue, bottommost line), the line is below all three stars and essentially flat near 0–5%.

PRM search (Figure 9, right; Figure 1, bottom-right bar chart). The pattern is starker:

DifficultyR β‰ͺ 1 (0.16)R β‰ˆ 1 (0.79)R ≫ 1 (22)
Easy+19.1%+2.2%+2.0%
Medium0.0%βˆ’35.3%βˆ’30.8%
Hardβˆ’3.6%βˆ’35.3%βˆ’52.9%

PRM search shows weaker benefits than revisions for the FLOPs-matched comparison, with substantial disadvantages on medium and hard questions even at moderate R values. On easy questions, test-time compute remains preferable across all R regimes, though the margin narrows significantly.

Figure 9 detail. Where the compute-optimal scaling line is above the star, test-time compute wins over the larger model. On bin 1, the scaling line is above all three stars for revisions but shows more mixed results for PRM search. On bin 5, the line is below all three stars and essentially flat, confirming that no amount of test-time compute helps on the hardest problems.

Ablation Studies and Robustness Checks

PRM aggregation strategy (Appendix E, Figure 13). Comparing "min," "prod," and "last" step-wise aggregation for converting per-step PRM scores into a single solution score: "last" achieves roughly 37% at 256 samples, "min" achieves roughly 35%, and "prod" achieves roughly 27%. ORM achieves roughly 34%. The "last" aggregation's superiority is notable because it effectively reduces the PRM to ORM-like behavior at aggregation time, yet the PRM still outperforms a separately trained ORM β€” suggesting the step-level PRM training provides beneficial representation learning.

PRM vs. ORM (Appendix F, Figure 14). The PRM consistently outperforms the ORM, with the gap widening at higher sample counts: at 2,048 samples, PRM best-of-N weighted reaches approximately 40% vs. ORM's 35%.

Revision model verifier choice (Appendix J, Figure 15a). The base-LM PRM underperforms the revision-specific ORM when scoring revision model outputs: sequential + base-LM PRM achieves roughly 40% at 64 generations vs. sequential + revision ORM at roughly 42%. This confirms distribution shift β€” the PRM trained on base model outputs does not transfer well to revision model outputs.

Revision history in verifier context (Appendix J, Figure 15b). Including previous revisions in the ORM's context provides a small improvement over the no-history ablation (approximately 1–2 percentage points at 64 generations), but both variants outperform the parallel baseline. This confirms that the sequential sampling benefit is not solely attributable to the verifier seeing more context.

Oracle vs. predicted difficulty bins (Figures 4, 8, and Appendix C, Figures 11–12). Both oracle and predicted bins yield qualitatively similar trends across difficulty levels. Predicted bins show slightly lower performance at high budgets in the revision setting (roughly 41% vs. 44% at 256 generations in Figure 8) but essentially identical performance in the search setting (Figure 4). This is the critical robustness check: the compute-optimal strategy works without ground-truth labels.

Majority voting for revisions (Appendix B, Figure 10). The sequential-to-parallel ratio trends observed with verifier-based selection are replicated with majority voting: easy questions are insensitive to ratio, hard questions show an optimal intermediate ratio, and fully sequential marginally outperforms fully parallel in aggregate. This demonstrates that the revision model's benefits are not purely an artifact of the verifier.

ReST^EM revision model (Appendix K, Figure 16). An attempt to further optimize the revision model using ReST^EM (Singh et al., 2024) backfires: additional sequential revisions substantially hurt performance with this model. At 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio. The authors hypothesize that on-policy data collection in ReST^EM exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly.

Critical Assessment

Claim 1: Compute-optimal scaling improves efficiency by more than 4Γ— over best-of-N.

What was tested: The paper demonstrates that, at specific budget levels, selecting the best strategy per difficulty bin matches best-of-N performance at 4Γ— fewer generations. For search (Figure 4), 16 generations of compute-optimal matches 64 generations of best-of-N; for revisions (Figure 8), 64 generations matches 256.

What was not tested: The difficulty estimation cost (2,048 samples per question to build bins) is not accounted for in these comparisons. The authors acknowledge this explicitly (Section 3.2): "our experiments do not account for this cost largely for simplicity." This means the 4Γ— figure should be understood as the efficiency gain of the allocation strategy given known difficulty, not the total end-to-end efficiency including difficulty assessment. In a deployment scenario where difficulty must be estimated from scratch, the total cost could be dominated by this estimation step. The paper does not report results with an amortized difficulty estimation scheme (e.g., how many questions you need to batch together before the per-question estimation cost becomes negligible), which makes the practical claim somewhat incomplete.

Conditional validity: The 4Γ— gain appears strongest at moderate budgets (16–64 generations) and narrows somewhat at the highest budgets (256–512), where the predicted-difficulty variants underperform oracle difficulty by roughly 3 percentage points for revisions (Figure 8). The claim that compute-optimal scaling "continues to improve" while best-of-N "plateaus" (Figure 8) is supported, but the magnitude of continued improvement is modest β€” roughly 44% at 256 generations vs. roughly 41% for best-of-N, a 3-point absolute gain for 256Γ— more compute from the 1-generation baseline.

Claim 2: Test-time compute with a smaller model can outperform a 14Γ— larger model.

What was tested: A PaLM 2-S* base model with compute-optimal test-time strategies is compared against a model with ~14Γ— more parameters using greedy decoding on MATH questions. Three values of R = D_inference / D_pretrain are tested.

What was not tested β€” three significant omissions:

  1. The larger model baseline is weak. The 14Γ— larger model uses greedy decoding with zero test-time compute augmentation β€” no best-of-N, no majority voting, no verifier, no search. This is an artificially weak baseline. A fairer comparison would give the larger model some test-time compute budget proportional to its size, or would compare total FLOPs (pretraining + inference) between the two approaches rather than holding inference compute fixed for one side. The paper implicitly acknowledges this by scaling only parameters (not data) for pretraining, following LLaMA rather than Chinchilla-optimal scaling, which likely makes the larger model weaker than a compute-optimally trained alternative.

  2. Single model family, single benchmark. All results are on MATH with PaLM 2-S*. The claim that test-time compute "can" outperform pretraining is demonstrated for exactly one (model, benchmark, verifier quality, revision training procedure) combination. We do not know whether this generalizes to, say, code generation with a different base model, or whether the specific threshold where test-time compute becomes preferable (R β‰ͺ 1, easy-medium problems) varies across model architectures and datasets.

  3. No investigation of what the larger model could do with its own test-time compute budget. The FLOPs-matched analysis compares a small model with augmentation against a large model without augmentation, but a natural question is: could a small amount of test-time compute on the larger model close or reverse the gap? If the 14Γ— larger model with, say, best-of-4 or majority voting at 4 samples already outperforms the smaller model at any budget, the practical implication changes substantially.

Conditional validity: The claim holds strongest at R β‰ͺ 1 (where inference tokens are few relative to pretraining, and the pretraining savings from using the smaller model are enormous) and for easy-to-medium problems. It weakens progressively: at R β‰ˆ 1, the advantage narrows; at R ≫ 1, the advantage reverses on hard problems. For revisions at R β‰ͺ 1, test-time compute wins across all difficulties (+11.8% to +27.8%); for PRM search at R ≫ 1, it loses on medium (βˆ’30.8%) and hard (βˆ’52.9%) problems. The claim should be stated as: "For tasks within the base model's capability range, and when pretraining dominates total compute, test-time augmentation can substitute for model scale" β€” which is a substantially narrower claim than "test-time compute outperforms pretraining."

Claim 3: Efficacy depends critically on prompt difficulty.

Strongly supported. The difficulty-bin analyses (Figures 3 right, 7 right, 9) show qualitatively different β€” and sometimes opposite β€” effects of the same strategy at different difficulty levels. Beam search hurts performance on the easiest problems (Figure 3, right: decreasing from ~78% to ~77% as budget increases from 4 to 256) while helping on medium problems (bins 3–4). Sequential revisions dominate on easy problems but a balanced ratio is optimal on hard problems (Figure 7, right). This is the most robust finding in the paper, replicated across search methods, revision strategies, and selection mechanisms.

Minor caveat: The five-bin discretization is coarse. Within a bin, there may be substantial heterogeneity β€” a question at the bottom of bin 3 and one at the top of bin 3 receive the identical strategy allocation, even though their true optimal strategies might differ. The paper does not explore sensitivity to the number of bins or to using continuous difficulty estimates. Given the small test set (500 questions β†’ ~100 per bin β†’ ~50 per fold for cross-validation), finer binning would face sample size limitations for policy selection.

Claim 4: The PRM search and revision mechanisms have complementary, difficulty-dependent strengths.

Partially supported with a significant gap. The paper demonstrates that search and revisions each have difficulty-dependent behavior (search best on medium problems, revisions best on easy problems), which implies complementarity. However, the paper never combines them. Section 8 explicitly states: "we did not experiment with PRM tree-search techniques in combination with revisions." This means the complementarity hypothesis is inferred from separate experiments rather than demonstrated jointly. A combined system β€” using the revision model as the proposal distribution within beam search, or using the PRM to guide which revisions to pursue β€” might yield gains beyond either method alone, might reveal interference between the two, or might show that the benefits are not additive (or even redundant). Without this experiment, the complementarity claim remains an informed speculation.

Claim 5: Verifier over-optimization is the primary bottleneck for test-time compute scaling.

Supported by convergent evidence across multiple analyses. Beam search degrading on easy problems at high budgets (Figure 3, right), lookahead search paradoxically underperforming simpler methods (Figure 3, left), and qualitative examples showing degenerate search outputs (Appendix M) all point to verifier over-optimization as the limiting factor. The compute-optimal policy is partially a strategy for staying below the over-optimization threshold. However, the paper does not attempt to quantify the over-optimization boundary β€” it doesn't, for example, measure how verifier reliability (e.g., calibration error, ranking accuracy) degrades as search intensity increases. Such a quantitative characterization would strengthen the claim and provide practical guidance for when to switch from search to best-of-N.

Missing experiments that would have strengthened the paper:

  • Combined PRM search + revisions. The most natural next step, acknowledged but not done.
  • Head-to-head with a non-greedy larger model baseline in the FLOPs-matched comparison β€” e.g., giving the 14Γ— larger model best-of-4 or majority voting at 4 samples and comparing against the smaller model's compute-optimal scaling.
  • Amortized difficulty estimation β€” measuring total cost (estimation + strategy execution) and showing at what batch size or amortization scheme the 4Γ— gain is realized end-to-end.
  • Replication on a second model family (e.g., a LLaMA variant) to test whether the difficulty-dependent patterns and BERT-over-RoBERTa-like patterns (from the parallel PARAREL work) generalize.
  • Sensitivity to the number of difficulty bins β€” is 5 bins optimal, or would 3 or 10 produce substantially different policies?

Genuine weaknesses that affect interpretation of results:

  • Test set of 500 questions with two-fold cross-validation per bin means policy selection is based on ~50 questions per fold per bin. Confidence intervals for the compute-optimal scaling curves are not reported, making it difficult to assess whether the differences between compute-optimal and best-of-N at the highest budgets (e.g., 44% vs. 41% at 256 generations for revisions) are statistically reliable.
  • The difficulty estimation protocol is circular in the oracle case and expensive in the predicted case (2,048 samples per question). The paper does not disentangle the value of difficulty estimation from the value of the allocation policy β€” would a simpler, cheaper difficulty signal (e.g., 8 samples instead of 2,048) produce nearly the same gains?
  • The revision model's 38% correct-to-incorrect reversion rate is a significant practical limitation that the majority-voting / verifier-selection patches do not fully solve. The paper doesn't explore systematic solutions (e.g., training the model to recognize "no revision needed" or adding correctness-preservation to the training objective).

6. Limitations and Trade-offs

The Difficulty Estimation Overhead Is Not Accounted for and May Dwarf the Reported Gains

The assumption or constraint. The compute-optimal strategy depends on knowing each prompt's difficulty before deciding how to allocate the inference budget. The paper's method for estimated difficulty β€” whether oracle (pass@1 over 2,048 samples) or predicted (PRM average score over 2,048 samples) β€” requires generating and scoring 2,048 complete solutions per question before any strategy is deployed. The paper acknowledges this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The headline efficiency claim β€” "more than 4Γ— better efficiency over a standard best-of-N baseline" β€” is computed after difficulty is known, with the difficulty estimation cost excluded from all budget calculations. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former dominates any conceivable per-question budget. For context, 2,048 samples is 8Γ— larger than the maximum per-question strategy budget studied (256 generations for most experiments, 512 for some). Amortizing this cost across an entire evaluation set (say, 500 questions) still means the per-question cost floor is roughly 4 generations (2,048 / 500) just for difficulty estimation, before any strategy is even selected. This makes the 4Γ— figure an upper bound on achievable efficiency in a setting where difficulty is pre-computed and reused across many queries β€” not a realized deployment gain for a single question.

What evidence exists in the paper. None β€” the paper does not report any experiment where the difficulty estimation cost is included in the budget. The cost is acknowledged as a limitation (Section 3.2) and flagged as future work ("exploration-exploitation tradeoff"), but the magnitude of the overhead relative to the claimed gains is never quantified. This is a significant gap because it means the central efficiency claim cannot be evaluated as stated β€” we do not know whether the compute-optimal strategy, when accounting for the full end-to-end cost, still beats a simple best-of-N with an equivalent total budget (including the samples that would have been used for difficulty estimation).

Mitigation status. The paper frames this as a direction for future work: "training models to directly predict difficulty of a question" (Section 8) would eliminate the per-question generation cost, but no such model is developed, trained, or evaluated. A dynamic difficulty estimation approach β€” starting with a small number of samples, using the verifier score distribution as a cheap difficulty signal, and adaptively allocating the remaining budget β€” is mentioned in passing but not implemented. The limitation is unresolved and directly affects the practical deployability of the method.

The Method Fails Completely on the Hardest Problems, Providing Zero Benefit Regardless of Budget

The assumption or constraint. The framework assumes that the base model has non-trivial capability on the problem β€” that is, its pass@1 rate over many samples is above some meaningful threshold. The paper operationalizes this through difficulty quintiles, where the hardest bin (bin 5) corresponds to questions where the base model's pass@1 is near zero. Across every experiment, problems in this bin show essentially no improvement from any amount of test-time compute.

The consequence. The approach offers no path forward for problems that are genuinely outside the base model's capability range. The paper is transparent about this, stating in Section 7: "on the hardest problems (bin 5), no method makes meaningful progress β€” the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated." This means test-time compute cannot compensate for fundamental capability gaps β€” it can only amplify existing capability. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% and lies below the 14Γ— larger model's performance at every R value. For the hardest problems, pretraining larger models remains the only viable path β€” test-time compute provides zero leverage.

This has direct practical implications: if a deployment faces a problem distribution skewed toward difficult problems (where the small base model's pass@1 is near zero), the compute-optimal framework provides no benefit. The difficulty estimator would correctly identify these as bin 5 and allocate the budget, but no allocation would produce correct answers. This bounds the method's applicability to domains where the base model already has at least marginal capability β€” it is a capability amplifier, not a capability creator.

What evidence exists in the paper. The failure is documented consistently across the three main experimental settings: PRM search (Figure 3, right: bin 5 flatline at 1–3%), revision model with sequential-to-parallel sweep (Figure 7, right: bin 5 invariant at ~2–3%), and FLOPs-matched comparison (Figure 9: bin 5 below the larger model at all R, essentially zero improvement with increasing inference budget). The paper acknowledges this limitation explicitly (Section 7 takeaway box, Section 8), framing it as evidence that "test-time compute amplifies existing capability but does not create it from nothing."

Mitigation status. Not addressed β€” this appears to be a fundamental bound rather than a solvable limitation within the current framework. The paper does not propose methods for extending test-time compute to problems outside the base model's capability range (e.g., retrieval augmentation, tool use, or hybrid human-AI pipelines). Future work might explore combining test-time compute with external knowledge sources for hard problems, but the paper's current framework offers no solution.

The 14Γ— Larger Model Baseline Is Artificially Weak, Undermining the Pretraining-vs.-Inference Comparison

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time strategies against a model with approximately 14Γ— more parameters, but the larger model is evaluated under two significant handicaps. First, it uses greedy decoding only β€” no majority voting, no best-of-N, no verifier, no search of any kind. The larger model is allocated zero test-time compute, while the smaller model is given a sophisticated adaptive inference budget. Second, the pretraining scaling is parameter-only, not compute-optimal (following the LLaMA paradigm rather than Chinchilla-optimal training, which would scale both parameters and data). The paper acknowledges this: "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work" (Section 7).

The consequence. The reported comparison is not a fair test of "pretraining compute vs. inference compute." It is a comparison of "small model with substantial inference augmentation vs. large model with zero inference augmentation, where the large model may also be suboptimally trained." The advantage attributed to test-time compute (e.g., +27.8% on easy questions at R β‰ͺ 1 with revisions, +19.1% with PRM search) partly reflects the baseline's weakness rather than the inherent superiority of test-time compute. Two corrections would change the picture substantially: (1) giving the larger model even a modest test-time compute budget β€” say, best-of-4 or majority voting at 4 samples β€” would create a much stronger baseline at negligible additional inference cost; (2) using a Chinchilla-optimal pretrained larger model (with data scaled proportionally) would likely improve its performance. Either correction could narrow or reverse the reported advantages, particularly on medium and hard questions where the current baseline shows the largest deficits.

What evidence exists in the paper. The paper reports the comparison as-is, with the greedy larger model serving as the sole pretraining-scaled baseline (Figures 1 and 9). There is no ablation where the larger model receives any test-time compute budget. The parameter-only scaling assumption is stated but the magnitude of the resulting performance deficit relative to a compute-optimal larger model is not estimated. The paper indirectly acknowledges the concern by noting the LLaMA vs. Chinchilla distinction, but does not quantify its impact.

Mitigation status. Not addressed experimentally. The authors frame this as future work, but the current comparison should be interpreted as a lower bound on pretraining effectiveness β€” demonstrating that there exist regimes where test-time compute can close or exceed a parameter-only-scaled model's performance, but not establishing how this tradeoff would play out against a properly optimized larger model. A practitioner deciding between "train bigger" and "infer smarter" cannot answer the question from this comparison alone.

Revisions and PRM Search Are Studied in Isolation, Not Combined β€” a Missed Opportunity That Limits Performance Ceiling Claims

The assumption or constraint. The paper studies two mechanisms for improving test-time compute β€” modifying the proposal distribution via iterative revisions (Section 6) and modifying the selection procedure via PRM-guided search (Section 5) β€” but never combines them. Section 8 explicitly states this gap:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The two mechanisms operate on complementary axes: revisions improve the quality of generated candidates (the proposal distribution), while PRM search improves the quality of candidate selection (the verifier). The paper's own difficulty-dependent analysis suggests they have complementary strengths β€” revisions work best on easy problems (where refinement is helpful), while PRM search works best on medium problems (where exploration is needed).

The consequence. The paper's reported performance numbers represent a lower bound on what a fully integrated system could achieve. If revisions improve the proposal distribution and PRM search improves selection, combining them could produce gains beyond either mechanism alone β€” for example, using the revision model as the generator within beam search, or using the PRM to decide when a revision chain should continue vs. be restarted. Conversely, the combination might reveal negative interactions (the revision model's output distribution might be harder for the PRM to score reliably, or the PRM's guidance might interfere with the revision model's learned improvement trajectory). Without this experiment, the paper cannot claim to have explored the full space of test-time compute strategies β€” it has explored two independent axes without probing their interaction. The compute-optimal policy might look different if the combined strategy were available as an option.

What evidence exists in the paper. The complementary difficulty-dependent patterns (revisions best on easy, search best on medium) are shown in separate experiments (Figures 3 right vs. 7 right), but the interaction is never tested. Section 8 identifies this as a direction for future work but provides no preliminary results or analysis of whether the combination is expected to be additive, subadditive, or superadditive.

Mitigation status. Not addressed β€” this is explicitly deferred to future work. The paper's framework provides the language for describing why they should be complementary (they operate on different axes of the proposal-verifier decomposition), but the experimental demonstration is absent. This is a significant gap because it means the paper's characterization of the "compute-optimal frontier" is incomplete β€” a strategy that combines both mechanisms might shift the frontier outward, changing the optimal allocation policy.

Single Benchmark and Single Model Family β€” We Cannot Assess Whether the Difficulty-Dependent Patterns Generalize

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with the PaLM 2-S* model family, including the verifier training and the FLOPs-matched comparison. The paper states it "believes this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is not tested. Several aspects of the findings could be specific to the (model, benchmark, verifier) combination:

  • Difficulty-dependent behavior. The finding that beam search hurts easy problems (Figure 3, right) depends on the specific verifier's over-optimization behavior, which is determined by the PRM's training data, architecture, and calibration relative to the base model's output distribution. A different base model with different output characteristics might produce a verifier with different over-optimization thresholds, shifting the difficulty bins where search is beneficial vs. harmful.
  • Revision model transfer. The revision model's ability to learn from edit-distance-paired incorrect-correct trajectories (Section 6.1) depends on the base model's in-context learning and sequence modeling capabilities, which vary across model families.
  • Task structure. MATH consists of competition-level math problems with clean ground-truth answers amenable to exact match grading. The PRM training pipeline (Monte Carlo rollout supervision) and difficulty estimation (pass@1 over 2,048 samples) both depend on having a verifiable correctness signal. Tasks without such signals (open-ended generation, creative writing, multi-step planning with ambiguous success criteria) cannot use this framework without fundamental modifications.

The consequence. We cannot distinguish which findings are universal properties of test-time compute scaling and which are idiosyncratic to the specific (PaLM 2-S*, MATH) setup. The central finding β€” that difficulty-dependent adaptive allocation yields 4Γ— efficiency gains β€” might replicate on other model families and reasoning benchmarks, or it might turn out that the optimal strategies, difficulty thresholds, and gains are highly sensitive to model architecture, scale, training data, and task domain. A practitioner using a different model (e.g., LLaMA, GPT) on a different task (e.g., code generation) cannot assume the difficulty-dependent patterns observed here will transfer β€” the paper provides no cross-model or cross-task validation.

What evidence exists in the paper. All evidence is from one model family on one benchmark. There is no replication on a second model (such as a LLaMA variant or GPT-style autoregressive model), nor on a second task domain (such as code generation, logical reasoning, or scientific QA). The paper does not report whether the difficulty bins, optimal strategies, or quantitative improvements are stable across different random seeds or model checkpoints. The test set is 500 questions, split into five quintiles of ~100 each, with two-fold cross-validation meaning policy selection is based on ~50 questions per fold per bin β€” a small sample that might produce unstable policy estimates.

Mitigation status. Not addressed β€” the paper does not claim cross-model or cross-task generalization, but it also does not provide the evidence needed to assess whether such generalization is likely. The acknowledgment in Section 4 that the model is "representative" is presented as motivation, not as an empirically supported claim. This is a domain limitation rather than a methodological flaw β€” the paper's findings are valid for the setting studied but their scope of applicability is unknown.

The Revision Model's Correct-to-Incorrect Reversion Rate (~38%) Limits Sequential Strategy Reliability

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target (Section 6.1). During deployment, when a revision chain processes a prompt, the model may produce a correct answer at step t, then encounter this correct answer in its own context at step t+1, and β€” because it was never trained to recognize when no revision is needed β€” incorrectly "revise" it into a wrong answer. The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach" (Section 6.1)

The consequence. Naively taking the last revision in a chain is unreliable β€” nearly 2 in 5 correct intermediate answers get corrupted. The paper mitigates this by selecting the best answer from anywhere in the chain using majority voting or verifier-based selection. However, these are post-hoc patches rather than solutions: majority voting can select a wrong answer if inconsistent errors dominate the chain, and verifier-based selection is limited by verifier quality. More fundamentally, the mitigation increases latency and complexity β€” you must generate the full chain and then evaluate all intermediate answers, rather than being able to stop early when a correct answer is produced. This also means that the sequential revision strategy, which the compute-optimal policy favors on easy problems (Figure 7, right), is operating with an inherent reliability ceiling imposed by the reversion problem. Even if the revision model produces a correct answer at some step, there is a 38% chance it will be undone, and the mitigation can only partially recover.

The ReST^EM experiment in Appendix K (Figure 16) further demonstrates the fragility of the revision training approach: attempting to optimize the revision model with reinforcement learning caused performance to degrade substantially, suggesting that the revision training procedure is sensitive to data generation methodology in ways that are not fully understood.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1 and motivates the within-chain selection mechanisms. The ReST^EM failure is documented in Appendix K. The paper does not report how often the verifier-based or majority-voting selection correctly recovers from a reversion β€” that is, when the chain produced a correct answer at step t, it was revised incorrectly at step t+1, and the selection mechanism successfully chose the correct step-t answer over the incorrect step-t+1 answer. This recovery rate would quantify how effectively the mitigation addresses the underlying problem.

Mitigation status. Partially addressed via majority voting and verifier-based selection across the revision chain, but the 38% reversion rate indicates a fundamental training data bias β€” the model never sees examples where the current answer is already correct. A principled solution (e.g., training the model to output a "no revision needed" token, or including correct-to-correct trajectories in training data) is not explored. The paper acknowledges the problem but does not resolve it; future work would need to address the training data construction to eliminate the reversion bias rather than relying on post-hoc selection to compensate for it.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper advances a conceptual reframing of knowledge evaluation in pretrained language models on three fronts simultaneously: it introduces consistency as an independently measurable model property orthogonal to accuracy, it provides the instrumentation (PARAREL) to make that measurement rigorous, and it demonstrates that the property can be improved through a modest pretraining-level intervention. The contribution is best characterized not as a single paradigm shift but as the establishment of a new evaluation axis that the field did not previously have the vocabulary or tools to measure systematically.

Before this work, the question "does this PLM know that Paris is the capital of France?" was treated as unidimensional β€” measured by accuracy on a single canonical prompt. The paper's demonstration that BERT-large achieves 48.1% accuracy but only 29.5% Consistent-Acc (Table 3) β€” meaning that nearly 40% of the facts the model appears to "know" under standard evaluation are actually not robustly retrievable β€” demonstrates that accuracy alone is a misleading signal for knowledge. This is not a refinement of an existing metric; it is the introduction of an entirely new failure mode to the evaluation landscape. The paper effectively splits "knowledge" into two dimensions: (1) whether the model can produce the right answer at all, and (2) whether it can produce it invariantly across meaning-preserving surface variations. Prior benchmarks measured only the first.

The paper's representational diagnosis β€” that PLM encodings of filled cloze patterns cluster by surface form rather than by entity identity (Figure 3, Section 7.2) β€” provides a mechanistic hypothesis for why the second dimension fails. This is a concrete finding with direct consequences for interpretability research: it suggests that the dominant signal in PLM representations is pattern identity, not entity identity, and that factual predictions are extracted from what is effectively a pattern-specific subspace rather than a knowledge-invariant subspace. This finding connects the behavioral phenomenon (inconsistency) to the representational geometry (pattern-dominated clustering) and reframes the consistency problem as a representation learning problem: models need to learn representations where meaning-preserving paraphrases map to nearby points, irrespective of syntactic or lexical surface differences.

The paper also reconciles an apparent contradiction in the PLM-as-KB literature. On one hand, Petroni et al. (2019) and successors showed that PLMs can recall factual knowledge with non-trivial accuracy β€” suggesting that parametric knowledge exists. On the other hand, the brittleness of these models to prompt phrasing was anecdotally observed but never systematically measured. The low consistency scores in Table 3 (BERT-large at 61.1% consistency, RoBERTa-large at 56.3%) show that both observations are simultaneously true: knowledge is encoded, but it is encoded in a pattern-dependent way that makes retrieval fragile. The paper doesn't resolve the contradiction so much as provide the framework for understanding it as a single, quantifiable phenomenon β€” the gap between Accuracy and Consistent-Acc is the brittleness tax, and measuring it tells us how much of apparent knowledge is actually robust.

Perhaps the most surprising result with implications for how the field thinks about pretraining is the counterintuitive relationship between data quantity and consistency. BERT β€” trained primarily on Wikipedia, a relatively clean, self-consistent knowledge source β€” outperforms RoBERTa and ALBERT on both accuracy and consistency (Table 3: BERT-large at 61.1% vs. RoBERTa-large at 56.3% vs. ALBERT-xxlarge at 52.1%). And the experiment with RoBERTa variants trained on increasing amounts of data (Table 4) shows a non-monotonic relationship: the 1M-token model achieves 37.1% consistency, higher than the 10M-token (29.8%) and 100M-token (31.5%) models, despite lower accuracy. This is a direct empirical challenge to the "more data is always better" assumption. It suggests that heterogeneous, large-scale pretraining corpora β€” which improve accuracy by exposing models to more facts β€” may simultaneously degrade consistency by exposing models to contradictory facts, diverse styles, and pattern-specific distributional correlations that undermine invariant knowledge representations. This finding reframes the pretraining data debate from a unidimensional question of quantity to a multidimensional tradeoff among quantity, quality, diversity, and self-consistency.

The paper's consistency-improvement method (Section 8) further demonstrates that consistency is a transferable meta-skill rather than a relation-specific property. Training on merely three randomly selected non-location relations (with the two-sided KL divergence consistency loss) improves consistency on 25 held-out test relations from 58.2% to 64.0%. The fact that this transfers across relations β€” including to relations with entirely different lexical items, syntactic structures, and knowledge domains β€” suggests that consistency is not about memorizing which patterns map to the same answer for specific relations, but about learning the abstract principle that different surface formulations of the same underlying question should produce the same answer. This is a conceptual finding with practical implications: it means that a small amount of consistency-targeted supervision (on a handful of relations with manually curated paraphrase sets) may be sufficient to induce more consistent behavior broadly.

Finally, the paper changes the evaluation landscape by releasing PARAREL as a public benchmark. Before PARAREL, there was no standard way to measure knowledge consistency in PLMs β€” each study that touched on robustness to paraphrasing did so with its own ad-hoc paraphrase sets, making cross-model and cross-study comparisons impossible. PARAREL standardizes the measurement and provides a target for model developers: improving consistency on this benchmark becomes a tractable goal, much as improving accuracy on LAMA became a tractable goal after Petroni et al. (2019). The paper does for consistency evaluation what LAMA did for knowledge extraction evaluation β€” it provides the instrument that makes systematic progress possible.

Follow-Up Research This Work Enables

Directly predicting question difficulty without the 2,048-sample overhead. The paper's central methodological gap is that difficulty estimation currently requires generating 2,048 samples per question and scoring them with the PRM β€” a cost that dwarfs typical per-question inference budgets. This makes the compute-optimal framework impractical in its current form, but it also makes the gap highly tractable. A natural follow-up would train a lightweight difficulty classifier β€” a small model that takes only the question text as input and predicts the difficulty quintile β€” using the PRM's 2,048-sample average scores as training labels. The classifier architecture could be a simple fine-tuned BERT-base or even a logistic regression over frozen PLM representations of the question. The key question is: can a classifier trained on this proxy signal achieve binning accuracy comparable to the full 2,048-sample PRM-based estimate? If yes, the compute-optimal framework becomes immediately deployable at negligible overhead (one forward pass of the classifier per question). If no, the result would establish a lower bound on the information required for difficulty estimation, clarifying whether the problem is fundamentally hard or just currently expensive. The paper's Figures 4 and 8 show that predicted (PRM-based) difficulty bins nearly match oracle bins, so the information is extractable β€” the open question is how cheaply.

Training models to predict difficulty from the question directly. The paper explicitly calls for this in Section 8: "pretraining or finetuning models to directly predict difficulty of a question." A strong follow-up would train a small classifier on the difficulty labels produced by the PRM's 2,048-sample average scores for the 500-question MATH test set, then evaluate on a held-out set to measure whether the estimated difficulty bins preserve the compute-optimal policy gains. If a 10M-parameter classifier achieves binning accuracy within, say, 80% of the PRM-based estimate, the framework becomes practical. A negative result β€” where cheap difficulty estimation substantially degrades the compute-optimal gains β€” would be equally informative, establishing that difficulty estimation itself is the bottleneck and redirecting research toward better estimation methods rather than better allocation policies.

Combining the revision model with PRM-guided tree search. The paper studies two mechanisms independently β€” iterative revisions to improve the proposal distribution (Section 6) and PRM-guided search to improve candidate selection (Section 5) β€” but never combines them, despite the complementary difficulty-dependent patterns: revisions work best on easy problems (Figure 7, right: fully sequential dominates bin 2), while beam search works best on medium problems (Figure 3, right: beam search outperforms best-of-N in bins 3–4). A natural extension would use the revision model as the generator within beam search: at each step of the search tree, the model conditions on previous (possibly rejected) partial solutions from the beam as context for generating the next step, and the PRM scores each step to decide which beams to expand. This would combine the revision model's learned ability to improve upon previous attempts with the PRM's ability to guide exploration. The key experiment would compare (1) PRM search with the revision model as generator vs. (2) PRM search with the base model as generator vs. (3) standalone revision chains, all at matched generation budgets, across difficulty bins. A positive result β€” where the combination achieves accuracy beyond either mechanism alone, especially in bins 3–4 β€” would demonstrate that the two axes (proposal improvement and verifier-guided selection) are complementary rather than redundant. A null result β€” where adding PRM search to the revision model provides no benefit over standalone revisions β€” would suggest that the revision model already internalizes effective search strategies, making explicit search unnecessary.

Stress-testing the verifier over-optimization boundary with systematic search intensity variation. The paper identifies verifier over-optimization as the primary bottleneck β€” beam search degrades performance on easy problems at high budgets (Figure 3, right: bin 1 accuracy decreases from ~78% to ~77% as budget increases from 4 to 256), and lookahead search paradoxically underperforms simpler methods (Figure 3, left) β€” but it does not systematically characterize where over-optimization begins or how it scales with verifier quality. A rigorous follow-up would train multiple PRMs of varying quality (e.g., using different amounts of training data, different architectures, or different label noise levels) and measure, for each PRM, the search budget at which beam search performance begins to degrade relative to best-of-N. This would produce a verifier calibration curve β€” a relationship between verifier quality (measured by, e.g., ranking accuracy or expected calibration error on i.i.d. samples) and the maximum search intensity before over-optimization dominates. The result would directly inform practitioners: given a PRM of quality X, what is the maximum beam width or search budget beyond which additional compute becomes counterproductive? A strong version of this experiment would also test whether the over-optimization threshold is difficulty-dependent β€” does a given PRM over-optimize sooner on easy problems (as Figure 3 suggests) or is the relationship uniform across bins? The qualitative examples in Appendix M showing degenerate search outputs (repetitive steps, overly short solutions) provide a starting point for developing metrics of "search degeneration" that could serve as early-warning signals for over-optimization.

Testing whether consistency improvements transfer to downstream task robustness. The paper reports a sobering negative result in Section 8.3: fine-tuning the consistency-improved BERT-ft on SQuAD and evaluating on paraphrased SQuAD questions shows performance "on par with the baseline model." This is a critical boundary condition: consistency learned at the pretraining level does not automatically transfer to downstream QA. A systematic follow-up would investigate why by (1) measuring whether the consistency improvement survives fine-tuning at all (i.e., is the BERT-ft model still more consistent on PARAREL after SQuAD fine-tuning, or does fine-tuning erase the consistency gains?), (2) testing whether consistency transfers to other downstream tasks where paraphrase invariance is more directly relevant (e.g., natural language inference with logical consistency constraints, fact verification where claims must be checked against multiple sources, or relation extraction where the same relation expressed differently should produce the same extraction), and (3) varying the amount of consistency training (more than three relations, more epochs, different Ξ» values) to see if there is a threshold beyond which downstream transfer emerges. The null SQuAD result could indicate that the consistency signal from three relations is too weak to survive task-specific fine-tuning, or that SQuAD's format (span extraction from a provided passage) engages different model mechanisms than the cloze-style factual knowledge tested by PARAREL. Distinguishing between these explanations would clarify whether consistency-enhanced pretraining has practical value for downstream applications or remains primarily a diagnostic property.

Extending PARAREL to multilingual and autoregressive settings. The paper's framework is limited to English masked language models. Two natural extensions present themselves. First, a multilingual PARAREL β€” translating the existing patterns into multiple languages and constructing language-specific paraphrase sets β€” would test whether the consistency problem is language-dependent (do models trained on morphologically richer languages show better or worse consistency?) and whether cross-lingual consistency (do paraphrases in different languages for the same relation produce the same answer?) is even more fragile than monolingual consistency. This would connect to the multilingual LAMA work (Kassner et al., 2021a) and the broader question of whether parametric knowledge is language-invariant. Second, testing autoregressive models (GPT-style) with PARAREL-style consistency probes would reveal whether the consistency problem is specific to the masked language modeling objective or is a general property of language model pretraining. Autoregressive models process text left-to-right without the bidirectional context of MLMs, which might affect how they encode relational knowledge and whether surface patterns dominate. The experimental design would need adaptation (autoregressive models don't have a [MASK] token; the probe would need to be structured as a prefix where the model's next-token prediction at the object position is evaluated), but the core consistency metric β€” do paraphrases for the same relation and subject produce the same top-1 prediction β€” would carry over.

Quantifying the relationship between training data heterogeneity and consistency through controlled pretraining experiments. The paper's most provocative hypothesis β€” that the BERT-over-RoBERTa consistency advantage stems from Wikipedia's relative cleanliness and self-consistency as a pretraining corpus β€” is supported by observational evidence (model comparisons, the RoBERTa data-size trend in Table 4) but not experimentally tested. A controlled experiment would pretrain multiple identically-architected models from scratch on corpora of identical size but varying heterogeneity: one trained on Wikipedia only, one on Wikipedia + news, one on Wikipedia + news + Reddit, and one on a deliberately contradictory corpus where the same facts are stated inconsistently (e.g., including both "Paris is the capital of France" and "Lyon is the capital of France" in different documents). Measuring PARAREL consistency across these models would test the causal claim that corpus heterogeneity degrades consistency. If the result holds β€” if consistency decreases monotonically with corpus heterogeneity β€” it would have direct implications for pretraining data curation: maximizing accuracy (which benefits from more data) and maximizing consistency (which may suffer from heterogeneous data) could be in tension, requiring explicit tradeoff decisions. This experiment is computationally expensive (requiring multiple pretraining runs) but would provide definitive evidence on a question the paper can currently only hypothesize about.

Practical Applications and Downstream Use Cases

Robustness evaluation for PLM-as-KB deployments. Any organization deploying a PLM to answer factual questions β€” whether in a question-answering system, a conversational agent, or an internal knowledge retrieval tool β€” now has, through PARAREL, a standardized method for evaluating whether the model's knowledge is robust to how users phrase their queries. An organization using, say, a fine-tuned BERT model to answer customer questions about product specifications could evaluate the model on PARAREL's patterns (or construct analogous paraphrase sets for their domain-specific relations) to measure Consistency and Consistent-Acc before deployment. The gap between Accuracy (48.1% for BERT-large, Table 3) and Consistent-Acc (29.5%) provides a quantitative estimate of how often the model will appear to "know" the answer under ideal prompting conditions but fail when users naturally rephrase the question β€” a critical metric for user-facing systems where query phrasing cannot be controlled. The paper's finding that patterns with different syntactic structures (Table 5: BERT-large consistency drops from 78.7% when syntax is identical to 67.5% when it differs) further quantifies the specific types of paraphrase variation most likely to trigger failures, enabling targeted testing or user interface design (e.g., suggesting canonical phrasings to users).

Data curation guidance for pretraining corpus construction. The paper's finding that BERT (Wikipedia-trained) outperforms RoBERTa and ALBERT (heterogeneous-corpus-trained) on both accuracy and consistency (Table 3) β€” and the non-monotonic relationship between data quantity and consistency in Table 4 β€” provides evidence-based guidance for teams constructing pretraining corpora. The implication is not "use only Wikipedia" but rather that corpus composition involves a tradeoff between knowledge coverage and knowledge consistency. A team pretraining a model for a high-stakes factual application (e.g., medical question answering, legal information retrieval) might prioritize a larger proportion of high-quality, self-consistent sources (encyclopedias, textbooks, peer-reviewed literature) even at the cost of reduced overall data volume, based on the expectation that consistency will be higher. Conversely, a team pretraining for creative generation or stylistic versatility might accept lower consistency as a tradeoff for broader coverage. The paper does not provide a formula for this tradeoff, but it establishes that the tradeoff exists and provides the measurement tools (PARAREL) to evaluate it for any given corpus composition β€” enabling evidence-based rather than intuition-based data curation decisions.

Consistency-aware fine-tuning for downstream tasks where robustness matters. The paper's consistency-improvement method (Section 8) demonstrates that a small amount of paraphrase-targeted training (three relations, with a two-sided KL loss) can improve consistency on unseen relations. This suggests a practical recipe for improving robustness in downstream tasks where paraphrase invariance is critical: augment the task-specific fine-tuning data with a small set of manually curated paraphrase pairs (even for relations unrelated to the target task) and add the consistency loss as an auxiliary objective. For example, a team fine-tuning BERT for fact verification (where a claim must be checked against evidence regardless of how it is phrased) could include the consistency loss on a handful of PARAREL relations during fine-tuning, with the goal of encouraging the model to learn the general principle that paraphrases should produce consistent predictions. The paper's SQuAD result β€” where consistency improvements did not transfer to downstream QA β€” suggests caution (the effect may be task-specific), but the method's simplicity and minimal data requirements make it a low-cost experiment with potentially high return for tasks where robustness to rephrasing is explicitly part of the success criteria.

Model selection informed by the accuracy-consistency tradeoff. The paper's two-dimensional evaluation framework (Accuracy vs. Consistency, with Consistent-Acc as the conjunction) enables more nuanced model selection than accuracy alone. In Table 3, BERT-large achieves the highest accuracy (48.1%) and consistency (61.1%), making it the clear winner on both axes. But in scenarios where accuracy and consistency are in tension β€” for instance, if a newer model achieves higher accuracy but lower consistency β€” the Consistent-Acc metric provides a principled basis for deciding which model to deploy. If the application involves user-facing queries where inconsistent answers would erode trust (a medical chatbot that gives contradictory answers to the same question asked differently is not usable regardless of its average accuracy), then Consistent-Acc should be the primary selection metric. If the application involves batch processing where each query is asked exactly once with a fixed prompt (e.g., populating a knowledge graph from a predefined template), accuracy alone may suffice and consistency is less critical. The paper provides the measurement toolkit for making this decision quantitatively rather than anecdotally.