ArXiv: 2504.12459

🎯 Pitch

Linear representations of facts in LMs don’t just reflect model capability—they reliably snap into place once the underlying facts co-occur in pretraining data above a sharp, model-specific frequency threshold. This threshold is surprisingly consistent across training (e.g., ~1k-4k co-occurrences), and you can even read out how often a model saw a term from the quality of its internal linear encoding.

1. Executive Summary

This work studies the relationship between pretraining data frequency and the formation of linear representations in language models, analyzing factual subject-relation-object triplets (e.g., mapping "France" to "Paris" in a capital prediction task) using OLMo-7B, OLMo-1B, and GPT-J-6B models. The paper measures a particular class of linear representations called Linear Relational Embeddings (LREs) (affine transformations that approximate a model's internal computation for completing factual relations) and connects their quality to pretraining term frequencies via custom batch-level co-occurrence counting. The central finding is that LRE quality—measured by causality—correlates strongly with subject-object co-occurrence frequency (r = 0.82), and that high-quality LREs consistently form once a relation's average co-occurrence count surpasses a model-specific threshold (~1k for GPT-J, ~2k for OLMo-7B, ~4k for OLMo-1B), regardless of when during pretraining that threshold is reached. The paper further demonstrates that a regression model trained on LRE quality metrics can predict term frequencies in a held-out model's pretraining corpus with roughly 65% within-order-of-magnitude accuracy, establishing that linear representations carry recoverable signal about training data statistics even when the pretraining corpus is unknown.

2. Context and Motivation

The Core Problem: Why Do Linear Representations Form for Some Concepts and Not Others?

The fundamental question driving this paper is deceptively simple: given that language models sometimes encode semantic relationships as simple linear transformations in their representation space, what determines when this happens? The Linear Representation Hypothesis (Park et al., 2024) proposes that LMs will represent features linearly, but it provides no mechanism for why or when. This gap is significant because linear representations have become a central tool in interpretability research—they allow researchers to localize model capabilities to specific directions in activation space, enabling interventions like steering (Todd et al., 2024; Subramani et al., 2022; Hendel et al., 2023; Rimsky et al., 2024) and editing. If we don't understand what causes these representations to form, we can't predict when our interpretability tools will work, nor can we design training procedures that encourage or discourage their emergence.

The paper documents this unpredictability concretely through prior findings: Hernandez et al. (2024) showed that over 80% of entities in the "country-largest-city" relation could be approximated linearly, but less than 30% of entities in the "star-in-constellation" relation exhibited the same structure. Similarly, Chanin et al. (2024) found wide variation in linearity across different relation types. These findings complicate the clean picture suggested by the Linear Representation Hypothesis—linearity is not a universal property of model representations but instead depends on something about the specific concept being encoded.

The authors trace this open question through a broader intellectual history. The puzzle of when linear structure emerges has been present since the era of static word embeddings, where Mikolov et al. (2013b) famously showed that vector arithmetic (e.g., France − Paris + Spain = Madrid) could capture semantic relationships. That property was subject to extensive debate precisely because it worked for some relations and not others (Köper et al., 2015; Karpinska et al., 2018; Gladkova et al., 2016). Ethayarajh et al. (2019) showed that for static embeddings, linguistic regularities formed under specific dataset frequency constraints for relevant terms—but that finding was specific to skipgram models and did not clearly transfer to modern transformer LMs trained with different objectives. The present paper explicitly picks up this thread, asking whether a frequency-based explanation also holds in the context of large autoregressive transformer LMs where representations are contextual rather than static.

Why This Problem Matters

The practical importance of answering this question stems from two converging trends in LM research.

First, linear representations are the backbone of many interpretability and control methods. If researchers want to use linear probes to detect whether a model "knows" a fact, steer a model's outputs away from undesirable behaviors, or edit a model's knowledge by modifying its representations, they need to know in advance whether the relevant concept is encoded linearly. Deploying these techniques without understanding when they work risks drawing false conclusions—a failed steering attempt might mean the concept isn't linearly encoded, not that the intervention approach is flawed. Conversely, knowing that linearity is tied to pretraining frequency would let practitioners predict when linear methods are appropriate and when they need to reach for more complex tools.

Second, and more broadly, the relationship between pretraining data and model internals remains poorly understood in general. While there is extensive work on how pretraining data affects downstream task performance (Ma et al., 2024; Xie et al., 2023; Longpre et al., 2024; Razeghi et al., 2022; Mallen et al., 2023; McCoy et al., 2024), there is far less work connecting data statistics to the internal representations that produce that performance. The paper identifies this as a critical missing link:

"the ways in which frequency affects the internal representations of LMs to cause this difference in performance remain unclear"

Establishing that a simple, measurable property of pretraining data (co-occurrence frequency) directly predicts the structure of internal representations (linearity) would provide a concrete bridge between data composition and model internals. This bridge has downstream implications for data curation—if practitioners want models that form linear representations for certain concepts (because they want to steer or edit those concepts), they should ensure those concepts meet frequency thresholds during training.

Prior Approaches and Where They Fall Short

The paper identifies several bodies of prior work that touch on related questions but leave the central gap unaddressed.

Theoretical work on linear representations. Jiang et al. (2024) provide both theoretical and empirical evidence that the training objectives of LMs implicitly encourage linear representations. They prove a strong frequency-based condition (based on matched log-odds between subjects and objects) and an implicit bias of gradient descent that together encourage linearity. However, this work operates in simplified theoretical settings and does not explain the empirical variation observed across relations in real LMs—why does "country-largest-city" become highly linear while "star-in-constellation" does not? The present paper positions itself as providing the empirical grounding for when the theoretical conditions identified by Jiang et al. actually translate into linear representations in practice.

Interpretability methods for finding linear structure. The recent interpretability literature has produced many methods for detecting and leveraging linear representations in transformer LMs (Huben et al., 2024; Gao et al., 2025; Templeton et al., 2024; Rimsky et al., 2024; Todd et al., 2024; Hendel et al., 2023; Hernandez et al., 2024; Chanin et al., 2024). These works focus on how to find linear representations and what can be done with them, but they do not address why they form in the first place. The paper explicitly states:

"While previous work has shown that the training objective encourages this type of representation (Jiang et al., 2024), our results suggest that the reason why some concepts form a linear representation while others do not is strongly related to the pretraining frequency."

In other words, the training objective may create the possibility of linearity, but frequency determines when that possibility is realized for specific concepts.

Work on factual recall and frequency. Separate from the linear representations literature, several studies have established that LMs perform better on factual recall for higher-frequency terms (Chang et al., 2024; Mallen et al., 2023; Razeghi et al., 2022). Chang et al. (2024) specifically showed that repeated exposure during pretraining encourages higher retention of facts. However, these works treat accuracy as the dependent variable of interest and measure it as a behavioral outcome. The present paper goes a step deeper, asking whether accuracy and linearity are independent phenomena or whether frequency drives them both through a common mechanism. The paper finds they are highly correlated but not identical—there are cases where accuracy is high but linearity is low (e.g., "star-constellation-name" in OLMo-7B has 84% 5-shot accuracy but only 44% causality), suggesting that linearity captures something distinct from behavioral performance.

Training data inference. A separate line of work studies whether it is possible to infer properties of a model's training data from the trained model itself—primarily through membership inference attacks (Shokri et al., 2017; Carlini et al., 2022), memorization analysis (Carlini et al., 2023; Oren et al., 2024; Shi et al., 2024), or distribution inference (Hayase et al., 2024; Ateniese et al., 2015; Suri & Evans, 2022). The present paper connects to this work obliquely but from a novel angle: rather than asking whether a specific example was in the training data (membership inference) or what the distribution of data sources was (distribution inference), it asks whether the strength of linear representations can reveal how often particular terms appeared in pretraining. This is a fundamentally different signal source—one derived from interpretability research rather than from output probabilities or memorization—and it opens a new avenue for training data inference that does not require ground truth counts from the target model.

How This Paper Positions Itself

The paper positions itself at the intersection of three previously disconnected literatures: (1) interpretability research on linear representations, (2) data-centric work on how pretraining data affects model behavior, and (3) privacy-oriented work on inferring training data from models. The novel synthesis is the claim that linear representations are the mechanism through which pretraining frequency manifests in model internals, and that this mechanism is sufficiently stable to serve as a signal for training data inference.

The paper's framing is explicitly empirical and correlational—it does not claim to prove causation between frequency and linearity (this would require counterfactual pretraining experiments, which the authors acknowledge as prohibitively expensive). Instead, it marshals evidence from multiple models (OLMo-7B, OLMo-1B, GPT-J-6B), multiple training checkpoints, and both factual and commonsense relations to establish that the frequency-linearity relationship is robust, consistent across pretraining stages, and strong enough to be practically useful for prediction tasks. The regression experiments in Section 5 serve both as a practical demonstration (training data inference) and as an additional form of evidence that the frequency-linearity link is not spurious—if linearity metrics can predict frequencies on held-out relations and held-out models, the relationship must reflect something genuine about how pretraining shapes representations.

A key subtlety in the paper's positioning is its relationship to the Linear Representation Hypothesis. The paper does not argue against the hypothesis but rather specifies its boundary conditions. It suggests that linearity is not an inevitable consequence of LM training objectives (as Jiang et al. (2024) might be interpreted as suggesting) but rather a property that emerges when sufficient training signal exists—that is, when the relevant terms co-occur frequently enough in the pretraining data. Below some model-specific frequency threshold, the training objective's implicit bias toward linearity is not enough to overcome the scarcity of signal. This framing refines rather than refutes existing theory, and it provides actionable guidance: if you want a linear representation for a concept, ensure its terms co-occur above the threshold for your model scale.

3. Technical Approach

3.1 Reader Orientation

This paper builds a measurement and analysis pipeline rather than a novel machine learning system; it is an empirical study that systematically connects two observable quantities: (1) how often concepts appear in an LM's pretraining data, and (2) how strongly those concepts form linear representations inside the model. The problem it solves is understanding when and why linear representations emerge—specifically, it shows that if you can measure the quality of a linear relation in a model's representations and count how often the relevant terms co-occurred during training, you discover that the first is a predictable function of the second, and that this relationship is stable enough that you can reverse the direction and predict training data frequencies from the model's internal representations alone.

3.2 Big-Picture Architecture (Diagram in Words)

The analysis framework has four major components that operate in two complementary directions:

Direction 1 (Sections 3 and 4) — From pretraining data to representations:

  1. Batch Search Counter: A custom Cython tool that scans tokenized pretraining batches (sequences of [batch_size, sequence_length]) to count exact subject-object co-occurrences at arbitrary training steps. Input: a list of ~10k query terms and the tokenized training corpus. Output: per-batch co-occurrence counts for every subject-object pair.
  2. LRE Fitting and Evaluation: For each factual relation (e.g., "capital-city"), an affine transformation is fitted using 8 few-shot examples from the relation and two hidden states from the LM (the subject representation at some middle layer and the object representation at the final layer). Output: a learned linear map $W, b$ and two quality scores (faithfulness and causality).
  3. Correlation Analysis: The co-occurrence counts from component 1 are correlated against the LRE quality scores from component 2, both at final checkpoints and across intermediate training checkpoints, establishing the frequency-linearity relationship.

Direction 2 (Section 5) — From representations back to pretraining data: 4. Frequency Regression Model: A random forest regressor trained on LRE quality features (faithfulness, causality, and variants) plus log-probability features to predict term frequencies. The model is evaluated on held-out relations (within-model) and on held-out models (cross-model, to simulate closed-data inference).

Information flows as follows: pretraining data → tokenized batches → batch search → co-occurrence counts → correlated with LRE quality scores derived from model hidden states (direction 1). Then: LRE quality scores + log probabilities → random forest regressor → predicted term frequencies → compared against ground-truth counts (direction 2).

3.3 Roadmap for the Deep Dive

  • First, the Linear Relational Embedding machinery — how LREs are defined mathematically, fitted from model activations, and evaluated with faithfulness and causality. This is the measurement apparatus for the dependent variable.
  • Second, the frequency counting pipeline — the Batch Search tool, why it operates on tokenized sequences rather than documents, and how co-occurrence is defined. This is the measurement apparatus for the independent variable.
  • Third, the cross-checkpoint analysis framework — how intermediate model checkpoints are used to study LRE emergence over pretraining time, and why the paper uses the same 8 examples to fit LREs across all checkpoints regardless of whether the model gets them correct.
  • Fourth, the frequency regression setup — the random forest model, the feature set (LRE metrics, log probabilities, accuracy), the held-out relation evaluation protocol, and the cross-model generalization experiment.
  • Fifth, design choices and their justifications — why the paper makes specific methodological decisions (per-relation β, incorrect examples in LRE fitting, within-order-of-magnitude accuracy as the evaluation metric) and what alternatives were considered.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical analysis paper driven by two measurement instruments (LRE fitting and batch-level co-occurrence counting) whose relationship forms the core finding. The technical machinery serves to operationalize a specific hypothesis: that the quality of a linear representation for a factual relation is a function of how often the subjects and objects of that relation appear together during pretraining.

Linear Relational Embeddings (LREs): Definition, Fitting, and Evaluation

What is an LRE? An LRE is an affine transformation (a matrix multiply plus a bias vector) that approximates the computation a language model performs internally when mapping a subject representation to an object representation within a factual relation. For example, given the input "Miles Davis plays the", an LM processes this through many transformer layers to produce a final hidden state that decodes "trumpet." The LRE hypothesis is that there exists a single matrix $W$ and bias $b$ such that, for any subject $s$ in the relation, the model's internal object representation $o$ is approximately $Ws + b$, where $s$ is extracted from an intermediate layer (not the embedding layer) and $o$ is extracted from the final layer at the last token position.

The formal definition is a first-order Taylor approximation. Let $F(s, c) = o$ be the forward pass through the model that maps a subject representation $s$ and few-shot context $c$ to an object representation $o$ (the hidden state that will decode the answer token). Around a specific reference subject $s_i$, the computation is approximated as:

F(s,c)≈F(si,c)+∂F∂s∣(si,ci)(s−si)F(s, c) \approx F(s_i, c) + \frac{\partial F}{\partial s}\bigg|_{(s_i, c_i)}(s - s_i)

where $F(s_i, c)$ is the model's output for the reference subject, $\frac{\partial F}{\partial s}$ is the Jacobian matrix of the forward pass with respect to the subject representation, and $(s - s_i)$ is the deviation of a new subject from the reference.

To make this a single transformation usable across many subjects, Hernandez et al. (2024) propose averaging the Jacobian and the bias term over $n$ examples from the relation (here, $n = 8$):

W=Esi,ci[∂F∂s∣(si,ci)]W = \mathbb{E}_{s_i, c_i}\left[\frac{\partial F}{\partial s}\bigg|_{(s_i, c_i)}\right]

b=Esi,ci[F(s,c)−∂F∂ss∣(si,ci)]b = \mathbb{E}_{s_i, c_i}\left[F(s, c) - \frac{\partial F}{\partial s}s\bigg|_{(s_i, c_i)}\right]

where $W$ is the average Jacobian matrix across the $n$ examples, $b$ is the average bias term computed as the model's object representation minus the Jacobian-multiplied subject representation, $s_i$ is the subject representation for the $i$-th example, $c_i$ is the few-shot context for that example, and the expectations are empirical means over the $n$ pairs.

What it computes: For a given relation (e.g., "capital-city"), you take 8 subject-object pairs where the model has been prompted with a few-shot template. For each pair, you extract the subject representation $s$ from a chosen intermediate layer at the subject token position and the object representation $o$ from the final layer at the last token position. You compute the Jacobian $\partial F / \partial s$ at each of the 8 points (measuring how small perturbations to $s$ would change $o$). You average these 8 Jacobians to get $W$, and you compute 8 bias vectors and average them to get $b$. Once fitted, the LRE can be applied to any new subject in the relation: the predicted object representation is $\hat{o} = Ws_{\text{new}} + b$.

Why this form: The first-order Taylor expansion is used because it is the simplest differentiable approximation to a nonlinear function. Averaging over multiple examples produces a linear map that captures the shared structure of the relation (the consistent transformation from subjects to objects) while averaging out example-specific noise. Using the Jacobian rather than, say, a difference vector between $o$ and $s$ accounts for the fact that the transformation may involve rotation and scaling, not just translation. The choice of $n = 8$ follows Hernandez et al. (2024) and represents a tradeoff: too few examples give high variance in the Jacobian estimate; too many become computationally expensive (each requires a forward and backward pass).

β-scaling modification. Hernandez et al. (2024) found that Equation 1 systematically underestimates the optimal slope of the linear transformation, so they scale each relation's $W$ matrix by a scalar hyperparameter $\beta$. Crucially, this paper uses one $\beta$ per relation rather than one per model:

"Unlike the original work, which finds one β per model, we use one β per relation, as this avoids disadvantaging specific relations."

The intuition is that different relations have different inherent linearity, and a single global β would force all relations to share the same scaling regardless of whether they actually benefit from rescaling to different degrees. The specific β sweep range is $[0, 5]$, tested at varying intervals described in Appendix C.

LRE quality metrics. The paper uses two metrics from Hernandez et al. (2024) to evaluate how well an LRE captures a relation:

  • Faithfulness: Measures whether the LRE-predicted object representation $\hat{o}$, when fed through the remainder of the model (the layers after the extraction point for $s$), produces the same token prediction as the original model does for the true object. It is the fraction of test examples where $\text{argmax}(\text{LM}(\hat{o})) = \text{argmax}(\text{LM}(o_{\text{true}}))$.
  • Causality: Measures whether editing the subject representation to produce the LRE's predicted representation for a different subject actually changes the model's prediction to that different subject's object. More precisely, for a subject $s_A$ whose true object is $o_A$, the LRE is used to compute what the representation would be if the subject were $s_B$ (whose true object is $o_B$): we compute $\hat{o}_B = o_A + W(s_B - s_A)$. Causality is the fraction of times that injecting $\hat{o}_B$ into the model causes it to predict $o_B$ instead of $o_A$. A pseudoinverse of the LRE weight matrix is used to perform this edit (the "rank" hyperparameter controls the rank of this pseudoinverse).

The paper prefers causality as the primary metric because:

"faithfulness can be high when LMs predict the same token very often (like in early checkpoints)."

In other words, if a model has a strong default prediction (e.g., in early training it almost always predicts the most frequent token for a position), faithfulness can be artificially high because the LRE doesn't need to do much to match the model's output. Causality is a stricter test: it requires the LRE to actively redirect the model's prediction to a specific alternative.

Hyperparameter sweeps (Appendix C). Three hyperparameters are tuned per relation:

  • Layer at which to extract the subject representation $s$. The paper sweeps all layers and selects the layer that maximizes causality (not faithfulness, to avoid the early-checkpoint issue).
  • $\beta$ for scaling $W$, swept over $[0, 5]$ at varying intervals.
  • Rank of the pseudoinverse used for causality edits, swept over $[0, \text{full rank}]$ at varying intervals (every 2 from 0–100, every 5 from 100–200, every 25 from 200–500, every 50 from 500–1000, every 250 from 1000 to the full hidden size).

A critical design choice: fitting on incorrect examples. In Hernandez et al. (2024), the 8 examples used to fit the LRE are filtered to only those where the model predicts the correct answer, under the assumption that an LRE will only exist once the model has acquired the corresponding knowledge. This paper removes that constraint:

"Interestingly, using examples that models predict incorrectly to fit Equation 1 works as well as using only correct examples."

This is important for the cross-checkpoint analysis (Section 4), because early checkpoints may not get any examples correct for low-frequency relations, making it impossible to fit LREs for those relations using the original approach. By allowing incorrect examples, the paper can fit the same LRE on the same 8 examples across all checkpoints, enabling direct comparison of how LRE quality evolves through training. Appendix B (Figures 4 and 5) validates that this choice does not significantly affect LRE quality: the average causality and faithfulness are similar whether the LRE is fit on all-correct, half-correct, or zero-correct examples.

The few-shot prompt structure. Each relation is tested with a 5-shot prompt. For a relation like "instrument-played-by-musician," the model sees four examples in the form "[X] plays the [Y]" and on the fifth example, when predicting e.g., "trumpet" from "Miles Davis plays the," the subject representation $s$ (at the subject token position in the chosen layer) and object representation $o$ (at the last token position in the final layer) are extracted. The few-shot context $c$ provides the model with in-context demonstrations of the relation pattern.

The Batch Search Tool: Counting Co-Occurrences Throughout Pretraining

Why a custom tool is needed. The paper needs to count how often subjects and objects co-occur within the exact sequences seen by the model during training, not within documents. Existing data counting tools like WIMBD (Elazar et al., 2024) provide document-level counts but cannot give counts for arbitrary training steps or for the specific tokenized sequences that constitute a batch. Since LMs are trained on fixed-length sequences that often split documents across multiple batches, document-level counts can be inaccurate: two terms that appear in the same document but different batches would be counted as co-occurring by WIMBD but would not actually appear together in any single training example.

How Batch Search works. The tool operates on tokenized batches of shape [batch_size, sequence_length]. It searches for occurrences of ~10k query terms (the unique subjects and objects from the RELATIONS dataset) throughout the entire Dolma corpus (approximately 2 trillion tokens) used to pretrain OLMo models. A co-occurrence is counted whenever a subject and object both appear in the same sequence within a batch—that is, within the same sequence_length-length window that the model sees as a single training example. Per-batch counts are accumulated over training steps to compute cumulative co-occurrence frequencies at each checkpoint.

Implementation and scale. The tool is implemented as Cython bindings that integrate with existing data processing libraries, released to support future research. Running on 900 CPUs, it completes the full search across ~2T tokens in about one day. The paper explicitly releases two code repositories: one for the frequency analysis and one for the efficient batch search tool ("batchsearch").

Definition of co-occurrence. Following Elsahar et al. (2018) and Elazar et al. (2022), the paper uses subject-object co-occurrence as a proxy for the frequency of the entire factual triplet appearing in text. The justification, from prior work, is that this approximation is "quite accurate"—if a document mentions both a subject and its relation object, it very likely mentions them in the context of that relation. This is more reliable for factual relations (e.g., "France" and "Paris" co-occurring strongly implies the capital relation is being discussed) than for commonsense or abstract relations.

What "average co-occurrence" means in the analysis. For a given relation, the paper computes the mean co-occurrence count across all its subject-object pairs. For example, for "capital-city," it counts how many times each (country, capital) pair co-occurs in Dolma and averages these counts. This relation-level average is what appears on the x-axis of Figure 2 and what correlates with causality at $r = 0.82$. The authors also report correlations with subject-only frequencies ($r = 0.66$) and object-only frequencies ($r = 0.59$), but these are notably lower, suggesting that co-occurrence specifically captures the relational signal better than individual term frequency.

WIMBD as an approximation for GPT-J. For GPT-J, which was trained on the Pile (Gao et al., 2020) rather than Dolma, the paper does not have access to the exact training batches. Instead, it uses WIMBD (Elazar et al., 2024) to count co-occurrences at the document level in the full Pile dataset. Appendix D (Figure 10) validates that this is a reasonable approximation: for the final OLMo checkpoint, Batch Search counts and WIMBD counts have a correlation of $r = 0.99$ with a slope of $0.94$. The slope being less than 1 confirms that WIMBD overestimates co-occurrences (because document-level windows are larger than sequence-level windows), but the near-perfect rank correlation means the relative ordering of relations by frequency is preserved.

Cross-Checkpoint Analysis: Tracking LRE Emergence Over Training

Which checkpoints are used. The OLMo model family (Groeneveld et al., 2024) releases intermediate checkpoints throughout training. The paper uses 8 checkpoints for OLMo-7B (v. 0424) and OLMo-1B (v. 0724), corresponding to models that have seen {41B, 104B, 209B, 419B, 628B, 838B, 1T, and 2T} tokens. For OLMo-7B, these correspond to {10k, 25k, 50k, 100k, 150k, 200k, 250k, 409k} pretraining steps respectively.

Why training tokens rather than steps. The paper reports results in terms of training tokens seen rather than step count because the 7B and 1B models use different batch sizes and thus see different numbers of tokens per step. Using tokens normalizes the x-axis across model sizes, allowing direct comparison of whether the frequency threshold depends on model scale.

How LREs are fitted at each checkpoint. For each relation, the same 8 examples are used to fit the LRE at every checkpoint, regardless of whether the model at that checkpoint predicts those examples correctly. As discussed above, this is enabled by the design choice to fit on incorrect examples and is critical for comparability: if different examples were used at different checkpoints, changes in LRE quality could be due to example selection rather than genuine improvement in the linear representation.

The frequency threshold analysis. The key visualization in Figure 2 plots log average subject-object co-occurrence (x-axis) against causality (y-axis). Each point is a relation at a specific checkpoint. The critical observation is that the relationship between frequency and causality is stable across checkpoints: for a given relation, the causality score at early checkpoints (after only 41B tokens) is already comparable to the final checkpoint if the relation has high co-occurrence frequency. Conversely, low-frequency relations never develop high-quality LREs even after the full 2T tokens of training.

The paper operationalizes the threshold as the co-occurrence count above which the mean causality across relations exceeds 0.9. This threshold is:

  • GPT-J (6B): 1,097 co-occurrences
  • OLMo-7B: 1,998 co-occurrences
  • OLMo-1B: 4,447 co-occurrences

These numbers suggest (though the paper is cautious given only three data points) that larger models require fewer exposures to form linear representations—the threshold decreases as model size increases from 1B to 7B parameters. The paper notes:

"it is possible that scale also affects this threshold"

The Frequency Regression Model: Predicting Pretraining Data from Representations

The prediction task. Given features derived from a model's representations and output probabilities for a specific subject-relation-object triplet, predict how often the object term (or the subject-object pair) appeared in the model's pretraining corpus. The target is the log-transformed frequency count.

Model choice: random forest regressor. The paper uses a random forest regression model with 100 decision tree estimators. This is a non-parametric, non-linear model that can capture complex interactions between features and does not require the feature-target relationship to be linear. The choice is pragmatic: given the relatively small number of data points (10,488 unique subject-object pairs across 24 retained relations after dropping "landmark-on-continent"), a random forest is less prone to overfitting than a deep neural network while being more expressive than linear regression.

Feature sets. Two sets of features are compared:

  1. LM-only features (baseline):

    • Log probability of the correct answer given the few-shot prompt (e.g., the log probability of "trumpet" given "Miles Davis plays the").
    • Average accuracy across 5 trials on the same prompt. The intuition is that models will be more confident about higher-frequency terms.

  2. LRE + LM features (full set):

    • All LM-only features.
    • Faithfulness: as defined above; measures whether the LRE reproduces the model's original prediction.
    • Causality ("soft causality"): as defined above; measures whether the LRE can redirect the model's prediction to a specific alternative.
    • Faith Probability: the log probability of the correct answer as produced specifically by the LRE (i.e., when the object representation is the LRE-predicted $\hat{o}$ rather than the model's natural representation).
    • Hard Causality: identical to soft causality but with a stricter success criterion—the edit is counted as successful only if the target answer becomes the number one predicted token (rather than just appearing in the top predictions or having its probability increased).

The distinction between soft and hard causality matters for the prediction task: soft causality measures whether the LRE can shift probability mass toward the target, while hard causality measures whether it can make it the top prediction—a stronger signal that the LRE captures the relation precisely.

Training and evaluation protocol. The paper fits 24 regression models (one per relation held out, across 4 random seeds: $24 = 24 / 1 \times 4$? Actually, the paper states "24 models such that each relation is held out once per random seed across 4 seeds," meaning 24 relations × 4 seeds = 96 total fits, with results averaged). For each model, one relation is held out entirely from training and used for evaluation. This ensures that the regression model is tested on relations it has never seen, avoiding leakage from the fact that subjects/objects within the same relation tend to have similar frequencies. The held-out set's objects and relations are guaranteed to not have appeared in the training set.

Evaluation metric: within-order-of-magnitude accuracy. Because predicting exact occurrence counts is extremely difficult (counts vary from single digits to millions), the paper reports the proportion of predictions that are within one order of magnitude (a factor of 10) of the ground truth count. This measures whether the regression can place a term in the right approximate frequency bucket—distinguishing "seen ~10 times" from "seen ~1,000 times" rather than distinguishing "seen 100 times" from "seen 101 times." Mean absolute error in natural log space is also reported as a complementary metric.

Cross-model generalization experiment (Section 5.3). This is the most ambitious experiment. A regression model is trained on features extracted from one LM (e.g., OLMo-7B, where ground-truth Dolma counts are known) and evaluated on features extracted from a different LM (e.g., GPT-J, where ground-truth Pile counts are known). The features are scaled by the ratio of total training tokens between the two models to account for the different dataset sizes. Critically, no supervision from the target model's training data is provided—the regression weights are fit entirely on the source model. This simulates the scenario of analyzing a closed-data but open-weights model.

Feature importance analysis (Section 5.2 and Appendix E). To determine which features contribute most to prediction accuracy, the paper uses permutation importance: each feature is randomly shuffled (breaking its relationship with the target while preserving its marginal distribution), and the drop in prediction accuracy is measured. Because faithfulness and faith probability are strongly correlated (Figure 11, $r \approx 0.76$ for GPT-J), the paper first applies PCA to reduce these two features to a single component capturing 89% of their variance, preventing the permutation test from being confounded by redundancy. The result (Figure 12) shows that hard causality is "by far the most important feature for generalization performance, causing a difference of about 15% accuracy," followed by the faithfulness PCA component at about 5%.

Design Choices and Their Justifications

Per-relation β vs. per-model β. The original Hernandez et al. (2024) work finds a single optimal β per model. This paper uses per-relation β because different relations have different inherent linearity, and a shared β would force all relations to use the same scaling, potentially masking differences in LRE quality. This is important for the frequency analysis: if a single β were used, relations that are genuinely less linear at any β would not be distinguishable from relations that just happened to need a different β than the global optimum.

Incorrect examples in LRE fitting. As discussed above, this enables cross-checkpoint comparison with identical examples. A secondary justification (Appendix B) is empirical: LRE quality is similar regardless of whether examples are correct or incorrect. This is surprising because one might expect that a model needs to "know" a relation for it to have a linear representation, but the result suggests that linear structure can exist even when the model's predictions are wrong—the representation space may encode the relation even if the decoding process doesn't produce the correct answer.

Causality as the primary metric over faithfulness. Faithfulness is unreliable in early checkpoints because models often have strong default predictions (e.g., predicting the most common token), making faithfulness artificially high. Causality is more stringent: it requires the LRE to actively redirect the model's prediction to a specified alternative. The authors explicitly state: "In general, we prefer to use causality in our analysis, as faithfulness can be high when LMs predict the same token very often."

Co-occurrence rather than individual frequency. The paper tests correlations with subject-only frequency ($r = 0.66$), object-only frequency ($r = 0.59$), and subject-object co-occurrence ($r = 0.82$). The substantially higher correlation with co-occurrence supports the theoretical intuition: a linear representation for a relation requires the model to have seen the two terms together in relational contexts, not just individually. This also aligns with the finding that the formation of linear representations is about relations (mappings between subjects and objects) rather than isolated concepts.

Why 8 examples for LRE fitting. Following Hernandez et al. (2024), $n = 8$ is chosen as a tradeoff. More examples give a lower-variance estimate of the average Jacobian but cost more computation (each requires forward and backward passes). Eight is sufficient to average out noise while keeping the fitting procedure tractable across 24 relations, 8 checkpoints, and 3 models.

Why within-order-of-magnitude accuracy. Exact count prediction from model internals is extremely challenging—counts range from single digits (e.g., "Caroline Bright" appearing 48 times) to millions (e.g., "Australia" appearing ~3.6 million times). An exact-prediction accuracy metric would be near zero and would not distinguish between a prediction that is close (e.g., predicting 30 for a true count of 48) and one that is wildly off (e.g., predicting 1,000,000). Within-order-of-magnitude accuracy captures whether the method provides practically useful approximate information about data frequency.

Why a random forest over a linear model or deep network. The relationship between LRE features and frequency is not necessarily linear—the paper's own results show threshold effects (LREs form abruptly above certain co-occurrence counts) and non-monotonic behavior. A random forest can capture these non-linearities without requiring the large amounts of data that a neural network would need. With 10,488 data points (subject-object pairs), a 100-tree random forest is a standard choice that balances expressiveness with regularization.

Dropping the "landmark-on-continent" relation. The paper drops this relation from the 25 factual relations in the RELATIONS dataset because "74% of the answers are Antarctica, making it potentially confounding for extracting a representation for the underlying relation." That is, if almost all landmarks are on Antarctica, the model can achieve high accuracy by memorizing the single answer "Antarctica" rather than learning a genuine continent-mapping relation, and any LRE for this relation would likely just encode the "Antarctica" constant rather than a meaningful transform.

4. Key Insights and Innovations

Innovation 1: Pretraining Frequency as the Boundary Condition for the Linear Representation Hypothesis

The paper's most conceptually significant move is recasting the Linear Representation Hypothesis from an unconditional claim ("LMs represent features linearly") into a conditional one ("LMs represent features linearly, provided those features meet a frequency threshold during pretraining"). This is not merely an empirical correlation—it is a specification of the hypothesis's scope conditions that resolves a tension that had been accumulating in the interpretability literature.

What the field assumed before. The Linear Representation Hypothesis (Park et al., 2024) proposes that LMs will represent features linearly, and Jiang et al. (2024) provided theoretical justification by showing that the training objective of next-token prediction encourages linear structure. This created an implicit expectation: if linearity is encouraged by the objective, it should be a near-universal property of trained LMs. Yet empirical results consistently contradicted this—Hernandez et al. (2024) found that over 80% of entities in "country-largest-city" exhibited linear structure but less than 30% in "star-in-constellation," and Chanin et al. (2024) documented similarly wide variation. The field lacked an explanation for this heterogeneity; it was unclear whether the variation reflected genuine differences in how concepts were represented, limitations of measurement methods, or something else entirely.

What this paper changes. By demonstrating that LRE quality (measured by causality) correlates at $r = 0.82$ with subject-object co-occurrence frequency and that high-quality LREs consistently form only above model-specific frequency thresholds (~1k for GPT-J, ~2k for OLMo-7B, ~4k for OLMo-1B), the paper identifies frequency as the missing variable that reconciles theory with observation. The training objective does encourage linearity, but that encouragement only translates into actual linear representations when there is sufficient training signal—sufficient co-occurrences of the relevant terms in the pretraining data. Below the threshold, the gradient signal is too sparse or noisy for the implicit bias to overcome.

The cross-checkpoint analysis in Figure 2 makes this point with unusual clarity: relations that have surpassed their frequency threshold by 41B tokens (only ~2% of total training) already exhibit high-quality LREs, while relations that never reach the threshold fail to develop them even after 2T tokens. This strongly suggests that frequency is the dominant factor, not training duration or model maturity—the model doesn't gradually "figure out" linear structure for rare concepts given enough training time; it simply never acquires it.

Significance beyond the specific finding. This insight constitutes a fundamental refinement rather than an incremental finding because it changes how researchers should think about both the Linear Representation Hypothesis and interpretability methods more broadly. It implies that the hypothesis is correct but incomplete without a frequency condition, and that interpretability tools relying on linear structure (steering vectors, linear probes, representation editing) should be expected to work only for concepts whose training frequency exceeds the model's threshold. This is actionable: before attempting to use linear methods to intervene on a concept in a model, practitioners can now estimate—based on the concept's likely frequency in the training data and the model's scale—whether linear structure is likely to exist.

Evidence anchor. Figure 2 and its embedded table, showing that mean causality exceeds 0.9 only above model-specific co-occurrence thresholds, with the relationship holding across all intermediate checkpoints.

Innovation 2: Linear Representations as a Signal Source for Training Data Inference

The paper's second conceptual contribution is demonstrating that the internal structure of representations—specifically, the degree to which a relation is encoded linearly—carries recoverable information about pretraining corpus statistics that is distinct from and stronger than the signal in output probabilities alone. This reframes linear representations from a purely interpretability concern into a potential tool for auditing and understanding models whose training data is unknown.

What the field assumed before. Prior work on training data inference has focused almost exclusively on behavioral signals: log probabilities of specific sequences (membership inference; Shokri et al., 2017; Carlini et al., 2022), memorization patterns (Carlini et al., 2023; Shi et al., 2024), or tokenizer-level artifacts (Hayase et al., 2024). The assumption—implicit in the choice of these signals—is that what a model outputs is the primary window into what it was trained on. Representations, when considered at all, were studied for what they encode about the world (facts, concepts, relationships), not about the training process that produced them.

What this paper changes. By showing that a regression model trained on LRE quality metrics (faithfulness, causality, hard causality, faith probability) achieves ~70% within-order-of-magnitude accuracy at predicting object frequencies—compared to ~40% for an LM-only baseline using log probabilities alone (Figure 3)—the paper establishes that linear representations encode frequency information that output probabilities do not. The feature permutation analysis in Figure 12 confirms this: hard causality alone accounts for a ~15 percentage point accuracy difference, dwarfing the contributions of log probability and task accuracy.

This is conceptually significant because it means that interpretability research has accidentally produced measurement tools (LREs) that double as training data forensics. The causal mechanism is plausible: the quality of a linear approximation to a relation depends on how much training signal the model received for that relation, and the training signal depends on frequency. But the key insight is that this signal survives in the representations even when it is not fully expressed in the model's output behavior—the regression generalizes across held-out relations and, crucially, across models trained on different datasets (Section 5.3; Table 1; ~65% accuracy when training on OLMo and evaluating on GPT-J).

The cross-model generalization is the linchpin. If the regression only worked within the same model it was trained on, the contribution would be primarily confirmatory—additional evidence for the frequency-linearity link. But the fact that it transfers from OLMo (trained on Dolma) to GPT-J (trained on the Pile) without any supervision from GPT-J's training data demonstrates that the relationship between LRE quality and frequency is not an artifact of a specific model or dataset. It reflects something general about how transformer LMs encode relational knowledge as a function of exposure. This is what transforms the finding from a post-hoc analysis of known data into a potentially useful method for analyzing closed-data but open-weights models.

Incremental vs. fundamental. This is a fundamental advance for the training data inference literature because it introduces an entirely new class of signals—representation-geometric properties—that had not previously been considered relevant. It is incremental in its practical impact (the predictions are approximate, with order-of-magnitude-level precision, and subject-object co-occurrence prediction remains difficult even with LRE features), but the conceptual opening it creates—that interpretability can serve dataset forensics—is new.

Evidence anchor. Figure 3 (within-magnitude accuracy, LRE+LM vs. LM-only) and Table 1 (cross-model generalization, LRE features achieving ~65% vs. ~42% for log-probability-only baseline).

The paper makes a subtle but important diagnostic contribution by showing that linear representation quality and task accuracy are correlated but not identical consequences of pretraining frequency. This matters because it clarifies what linear representations actually reflect about a model's knowledge, and it suggests that accuracy and linearity may be driven by frequency through partially distinct mechanisms.

What the field assumed before. There is a natural—and largely untested—assumption that linear representations and behavioral performance go hand in hand: if a model can reliably complete "France is the capital of [Paris]," then the capital relation must be encoded linearly somewhere in the model, and conversely, if the relation is linearly encoded, the model should perform well on it. Research on in-context learning has reinforced this by showing that ICL can be understood as the model identifying and applying linear mappings between input-output pairs (Hendel et al., 2023; Garg et al., 2022). This suggests a tight coupling.

What this paper changes. The paper documents cases where this coupling breaks. In OLMo-7B, the "star-constellation-name" relation achieves 84% 5-shot accuracy but only 44% causality—the model can answer correctly in a few-shot setting, but the underlying relation is not well-approximated by a single linear transform. Conversely, for "food-from-country" in intermediate OLMo checkpoints, causality reaches 65% while 5-shot accuracy is only 42%—linear structure emerges before reliable behavioral performance. By the final checkpoint, the gap narrows to an average of ~11% across relations, but it does not close entirely.

The relationship between frequency and the two measures also differs. Frequency correlates more strongly with causality ($r = 0.82$) than with 5-shot accuracy ($r = 0.74$ in OLMo-7B), suggesting that frequency is more directly linked to the formation of linear representational structure than to behavioral accuracy per se. Accuracy may be achievable through other mechanisms (e.g., memorization of specific examples, non-linear computation, or reliance on in-context demonstrations to compensate for weak internal representations) even when the underlying representation is not linear.

Significance. This finding is important for two reasons. First, it provides a diagnostic distinction: if a model achieves high accuracy on a task, you cannot automatically assume there exists a clean linear representation underlying it. Interpretability researchers using linear methods need to verify linearity rather than inferring it from performance. Second, it suggests that frequency may act on model internals in two ways: by strengthening linear structure (as evidenced by the causality correlation) and by improving overall task competence through other pathways (as evidenced by the remaining accuracy that is not explained by causality). Teasing apart these pathways is left to future work, but the paper's documentation of their partial independence is a valuable empirical contribution.

The finding is incremental rather than fundamental—it refines understanding of an existing relationship rather than introducing a new one—but it has practical implications for interpretability methodology and for how researchers should reason about the connection between model internals and model behavior.

Evidence anchor. Section 4.3, reporting that the "star-constellation-name" relation has 84% 5-shot accuracy but only 44% causality in OLMo-7B, and that causality reaches 65% before 5-shot accuracy (42%) for "food-from-country" in intermediate checkpoints. Appendix F (Figures 13 and 14) provides the full accuracy-vs-causality comparison across all relations and checkpoints.

Innovation 4: Model Scale Reduces the Frequency Threshold for Linear Representation Formation

The paper observes a pattern across its three tested models that, while tentative given the sample size, carries significant implications for how we understand the interaction between model capacity and data requirements for representational structure.

What the field assumed before. Prior work has established that larger models are more sample-efficient at learning—they achieve higher performance with fewer training examples (Kaplan et al., 2020; Hoffmann et al., 2022). However, this efficiency is typically measured in terms of behavioral outcomes (loss, accuracy). Whether larger models also form structured internal representations with less data—and specifically, whether they form linear representations with fewer co-occurrences of relevant terms—was not known.

What this paper changes. The frequency thresholds above which mean causality exceeds 0.9 show a clear monotonic relationship with model scale among the three tested models: GPT-J (6B parameters) requires ~1,097 average co-occurrences, OLMo-7B requires ~1,998, and OLMo-1B requires ~4,447. While the paper is appropriately cautious—"it is possible that scale also affects this threshold"—the pattern is consistent with the hypothesis that larger models extract usable linear structure from sparser training signals. A 1B parameter model needs to see a subject-object pair roughly twice as often as a 7B model to form a high-quality linear representation of their relation.

Significance. If this pattern generalizes to larger models (which the paper does not test), it has direct implications for data curation: as models scale up, the frequency bar for forming interpretable linear representations drops, meaning more concepts become accessible to linear interpretability methods without requiring massive increases in training data. This also provides a new lens on the sample efficiency of larger models—not just that they learn faster behaviorally, but that they form cleaner, more geometrically structured internal representations from less data. This is a distinct claim from standard scaling law analyses and connects the scaling literature to the interpretability literature.

The finding is suggestive rather than conclusive given only three models, and the paper acknowledges this limitation. It is best understood as an empirical observation that opens a research question rather than a settled finding. If confirmed across more model scales and architectures, it would constitute a fundamental insight about how model capacity interacts with the geometry of learned representations.

Evidence anchor. The table in Figure 2 listing co-occurrence thresholds for GPT-J (1,097), OLMo-7B (1,998), and OLMo-1B (4,447), with the authors' explicit caveat about the limited sample size.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper uses the RELATIONS dataset from Hernandez et al. (2024), focusing on 25 factual relations (e.g., capital-city, person-mother) containing 10,488 total unique subjects and objects. One relation, "landmark-on-continent," is dropped because 74% of its answers are "Antarctica," making it "potentially confounding for extracting a representation for the underlying relation" (Section 3.1), leaving 24 relations for analysis.

  • Base model(s). The paper evaluates three models: OLMo-7B (v. 0424), OLMo-1B (v. 0724) from Groeneveld et al. (2024), and GPT-J-6B (Wang & Komatsuzaki, 2021). OLMo models are chosen because they release intermediate checkpoints and provide tooling for accurately reconstructing pretraining batches from Dolma (Soldaini et al., 2024), which is essential for the per-batch co-occurrence counting. GPT-J is included as a testbed for the cross-model generalization experiments, where it simulates a "closed-data" model since its training data (the Pile; Gao et al., 2020) is known to the researchers but treated as unknown during regression evaluation. For the cross-checkpoint analysis, 8 intermediate checkpoints are used for OLMo models, corresponding to models that have seen {41B, 104B, 209B, 419B, 628B, 838B, 1T, and 2T} tokens.

  • Metrics. Two primary metrics from Hernandez et al. (2024) measure LRE quality. Faithfulness measures the proportion of test examples where the LRE-predicted object representation, when fed through the remainder of the model, produces the same token prediction as the original model for the true object. Causality measures whether injecting an edited subject representation (computed via the LRE to redirect prediction to a different subject's object) actually causes the model to predict that alternative object. Causality is preferred as the primary metric because "faithfulness can be high when LMs predict the same token very often (like in early checkpoints)" (Section 3.1). A causality score above 0.9 is considered "nearly perfectly linear." For the regression experiments, the evaluation metric is within-order-of-magnitude accuracy (the proportion of predictions within one factor of 10 of the ground truth count), and mean absolute error in natural log space is reported as a secondary metric. Task accuracy is measured as 5-shot accuracy on the factual recall task (the same prompt structure used for LRE fitting).

  • Baselines. The primary baselines in the regression experiments (Section 5) are:

    • LM-only features: log probability of the correct answer and average accuracy across 5 trials on the few-shot prompt. The intuition is that models will be more confident about higher-frequency terms.
    • Mean frequency baseline: always predicting the average training data frequency from the training set.
    • Random baseline: predicting random frequencies drawn from the training distribution. For the correlation analyses in Section 4, the baseline for comparison is the strength of correlation with individual term frequencies (subject-only, $r = 0.66$; object-only, $r = 0.59$) versus subject-object co-occurrence ($r = 0.82$). The paper also compares against 5-shot accuracy ($r = 0.74$) as a behavioral baseline for how strongly frequency predicts performance versus representational structure.

  • Generation budget / compute accounting. This paper does not use generation budgets in the conventional sense of inference-time compute scaling. Compute is relevant in two places: (1) the cost of Batch Search, which processes ~2 trillion tokens of Dolma across 900 CPUs in about one day; (2) the cost of LRE fitting, which requires forward and backward passes for 8 examples per relation to compute Jacobians, plus hyperparameter sweeps across layers, β values, and pseudoinverse ranks (described in Appendix C). The paper explicitly notes that the difficulty estimation cost—counting co-occurrences across all checkpoints—is substantial but is a one-time analysis cost rather than a per-query inference cost. The cross-checkpoint analysis uses identical LRE fitting examples across all checkpoints to keep the measurement cost fixed per relation-model pair.

  • Cross-validation / statistical protocol. For the regression experiments (Section 5), the paper fits 24 models such that each relation is held out once per random seed across 4 seeds (totaling 96 model fits, with results averaged). The held-out set's objects and relations are guaranteed to not have been in the training set, preventing leakage from the fact that subjects/objects within a relation tend to have similar frequencies. For the cross-model generalization experiment, a regression model trained entirely on one model's features (e.g., OLMo-7B) is evaluated on another model's features (e.g., GPT-J) with no access to the target model's ground-truth counts. The main correlation analyses use the Pearson correlation coefficient on the full set of relations and checkpoints. The paper does not report confidence intervals or statistical significance tests for most comparisons.

Main Quantitative Results

Correlation Between Co-Occurrence Frequency and LRE Quality

The paper's central quantitative finding is the correlation between pretraining subject-object co-occurrence frequency and LRE quality. Figure 2 presents scatter plots of log average subj-obj co-occurrence (x-axis) against causality (y-axis) for OLMo-7B, OLMo-1B, and GPT-J. The headline result: co-occurrence frequencies highly correlate with causality at $r = 0.82$ (Figure 2, OLMo-7B). This is substantially higher than correlations with subject-only frequencies ($r = 0.66$) and object-only frequencies ($r = 0.59$), and also higher than the correlation between frequency and 5-shot accuracy ($r = 0.74$ in OLMo-7B).

The scatter plots in Figure 2 reveal the threshold behavior: across all checkpoints (gray dots show intermediate, red dots show 41B tokens, blue dots show final), the relationship between log co-occurrence and causality takes an approximate sigmoid shape—relations with very low co-occurrence counts (left side of the plots) have uniformly low causality, relations with very high counts (right side) have uniformly high causality, and there is a transition zone in the middle where causality varies. The dashed black lines indicate the co-occurrence count at which mean causality exceeds 0.9. The embedded table in Figure 2 reports these thresholds:

"GPT-J (6B): 1,097 co-occurrences; OLMo-7B: 1,998 co-occurrences; OLMo-1B: 4,447 co-occurrences."

The key observation is that this relationship holds regardless of pretraining stage. The gray dots (intermediate checkpoints) are interleaved with the red and blue dots (41B and final checkpoints) rather than systematically lower, meaning that at 41B tokens, relations that have already surpassed their frequency threshold exhibit causality scores comparable to the final checkpoint. In the paper's own words (Section 4.2):

"Regardless of pretraining step, models that surpass this threshold have very high causality scores."

Figure 2 highlights five example relations with darker lines, ordered from best to worst performing at the final checkpoint: "country largest city," "country currency," "company hq," "company CEO," and "star constellation name." The "star constellation name" relation is the notable outlier—it has the lowest causality despite having non-trivial co-occurrence counts, and it is one of the few relations where 5-shot accuracy (84%) far exceeds causality (44%) in OLMo-7B (Section 4.3).

Relationship Between Causality and Accuracy

Section 4.3 examines the relationship between linear representation quality and behavioral performance. The correlation between causality and subject-object frequency ($r = 0.82$) exceeds the correlation between 5-shot accuracy and frequency ($r = 0.74$ in OLMo-7B), suggesting that frequency is more directly linked to linear structure than to behavioral accuracy. However, both correlations are high, and in general, few-shot accuracy and causality track each other closely—the paper notes they are "within 11% on average" in the final model.

Two specific cases illustrate divergence. The "star-constellation-name" relation in OLMo-7B achieves 84% 5-shot accuracy but only 44% causality (Section 4.3), with subjects and objects co-occurring only about 21 times on average across the full dataset. This is a high-accuracy, low-linearity case: the model can answer correctly in a few-shot setting, but the underlying relation is not well-approximated by a single affine transform. Conversely, "food-from-country" in intermediate checkpoints shows causality (65%) exceeding 5-shot accuracy (42%)—linear structure emerges before reliable behavioral performance, with the gap closing as training progresses. Across all 24 relations, only the "star-constellation-name" and "product-by-company" relations have few-shot accuracies that "far exceed their causality scores (and both are low frequency)" (Section 6).

Appendix F provides the full breakdown across all relations and checkpoints for both OLMo-1B (Figure 13) and OLMo-7B (Figure 14), showing how zero-shot accuracy, few-shot accuracy, and causality evolve in parallel through training for most relations, with the ordering of relations by performance remaining largely stable.

Regression Results: Predicting Frequency from LRE Quality

Figure 3 presents the within-magnitude accuracy for regression models predicting object frequencies and subject-object co-occurrence frequencies from LM features alone versus LRE + LM features. The results show a substantial gap between feature sets:

  • Object frequency prediction: LRE + LM features achieve approximately 70% within-order-of-magnitude accuracy, compared to approximately 42% for LM-only features. The mean baseline achieves approximately 31%, and the random baseline is near 0%. The paper reports:

"Mean absolute error of the predictions (in natural log space) for LRE features (LM-only features) are 2.1, (4.2)... on object prediction."

This is a factor-of-2 improvement in log-space error when LRE features are included.

  • Subject-object co-occurrence prediction: Both feature sets perform poorly relative to the mean baseline. LRE + LM features achieve approximately 67% accuracy, LM-only features achieve approximately 60%, and the mean baseline achieves approximately 57%. The paper acknowledges this difficulty:

"We find that subject-object co-occurrence frequency is likely too difficult to predict given the signals that we have here, as our predictions are higher than, but within one standard deviation of the mean baseline."

The corresponding mean absolute errors are 1.9 for LRE features and 2.3 for LM-only features—a smaller but still present advantage for LRE features.

Feature importance (Figure 12): The permutation importance test on held-out relations shows that hard causality "is by far the most important feature for generalization performance, causing a difference of about 15% accuracy," followed by the faithfulness PCA component (capturing 89% of shared variance between faithfulness and faith probability) at about 5% accuracy change. The LM-only features (log probability and accuracy) show near-zero permutation importance, consistent with their poor standalone performance in Figure 3.

Cross-Model Generalization (Table 1)

Table 1 presents the cross-model regression results, where a regression fitted on one model's features is evaluated on features extracted from a different model. The two settings are:

  • Train on OLMo, evaluate on GPT-J (simulating GPT-J as a closed-data model)
  • Train on GPT-J, evaluate on OLMo (the reverse)

The LRE features are scaled by the ratio of total training tokens between the two models to account for different dataset sizes.

For predicting object occurrences:

  • Eval on GPT-J (train on OLMo): LRE features achieve $0.65 \pm 0.12$, LogProb features achieve $0.42 \pm 0.10$, mean frequency baseline achieves $0.31 \pm 0.15$.
  • Eval on OLMo (train on GPT-J): LRE features achieve $0.49 \pm 0.12$, LogProb features achieve $0.41 \pm 0.09$, mean frequency baseline achieves $0.41 \pm 0.17$.

For predicting subject-object co-occurrences:

  • Eval on GPT-J: LRE features achieve $0.76 \pm 0.12$, LogProb features achieve $0.66 \pm 0.09$, mean frequency baseline achieves $0.57 \pm 0.15$.
  • Eval on OLMo: LRE features achieve $0.68 \pm 0.08$, LogProb features achieve $0.60 \pm 0.07$, mean frequency baseline achieves $0.67 \pm 0.16$.

The key finding is that LRE features consistently outperform log-probability-only features, often by a wide margin, even when evaluated on a completely different model with a different pretraining dataset. For object frequency prediction on GPT-J (the most practical use case: inferring frequencies for a closed-data model using a regression trained on an open-data model), LRE features achieve 65% accuracy vs. 42% for log-probability features—a ~55% relative improvement. The authors state:

"We are able to fit a much better generalizable model when using LRE features as opposed to the LM probabilities alone" (Section 5.3).

However, the paper notes that for subject-object co-occurrence prediction, the gain over the mean baseline is minimal (76% vs. 57% for GPT-J, but 68% vs. 67% for OLMo)—within one standard deviation in the OLMo evaluation case.

Error Analysis (Table 2)

Table 2 illustrates example predictions from the regression fitted on OLMo and evaluated on GPT-J for held-out relations. The examples reveal several patterns:

  • Good transfer for geo-political relations: For "landmark-in-country" with subject "Menangle Park" and object "Australia," the prediction (2,986,989) is within 1.2× of the ground truth (3,582,602). For "country-language" with "Brazil" → "Portuguese," the prediction (845,406) is within 1× of the ground truth (561,005)—a factor of 1.5.

  • Catastrophic failure for low-frequency relations: For "star-constellation-name" with "Arcturus" → "Boötes," the prediction (974,550) is 346× the ground truth (2,817). This is the relation with the weakest LREs across all models (as established in Section 4).

  • Subject sensitivity: For "person-mother," the prediction for "Prince William" → "Princess Diana" (5,826) is within 4.6× of the ground truth (27,094), but the prediction for "Prince Harry" → "Princess Diana" (131) is off by 207× (ground truth 27,094). The paper notes this demonstrates that "the regression model can be sensitive to the choice of subject... telling us the choice of data to measure LREs for is important for predictions" (Table 2 caption).

The regression is shown to use the full prediction range: the same model that predicts millions for "Australia" also predicts 59 occurrences for "Caroline Bright" (Will Smith's mother), close to the ground truth of 48 (Section 5.4).

Ablation Studies and Robustness Checks

  • LRE fitting on correct vs. incorrect examples (Appendix B, Figures 4 and 5): The paper finds "no notable difference in the choice of examples" for fitting LREs. Figure 4 shows average causality and faithfulness across relations when LREs are fitted with all-correct, half-correct, or zero-correct examples—all three settings produce similar averages. Figure 5 breaks this down by relation, showing that the choice of examples matters little for most relations. This justifies the paper's use of uniform examples across checkpoints (many of which produce incorrect predictions at early stages) without biasing the LRE quality measurements. A practical constraint: some relations do not have enough incorrect examples for the "none correct" setting, resulting in missing bars for those relations in Figure 5.

  • Batch Search vs. WIMBD co-occurrence counts (Appendix D, Figure 10): To validate that WIMBD document-level counts are a reasonable proxy for sequence-level counts when per-batch data is unavailable (as with GPT-J and the Pile), the paper compares Batch Search and WIMBD counts for the final OLMo checkpoint. The slope of the best-fit line is 0.94 with $r = 0.99$. The slope being less than 1 means WIMBD overestimates co-occurrences (because document windows are larger than sequence windows), but the near-perfect rank correlation means the relative ordering of relations by frequency is preserved. This justifies using WIMBD for GPT-J in the absence of per-batch data.

  • Commonsense relations (Appendix G, Figure 15): The paper extends the analysis to 8 commonsense relations (e.g., "fruit-inside-color," "task-person-type," "word-sentiment"). For OLMo-7B, the correlation between causality and co-occurrence frequency is $r = 0.42$—substantially lower than the $r = 0.82$ for factual relations. The paper attributes this to the fact that subject-object co-occurrences "do not accurately track occurrences of the relation being mentioned" for commonsense relations. For example, "researching history" co-occurring with "historian" does not capture all the ways the "historian" concept is defined during pretraining. This serves as a robustness check that validates the co-occurrence proxy specifically for factual relations (where Elsahar et al. (2018) established its accuracy) while cautioning against extending the method to abstract or commonsense relations without more careful frequency measurement.

  • Feature correlations and PCA dimensionality reduction (Appendix E, Figure 11): The paper reports pairwise correlations between all features used in the regression. Faithfulness and faith probability are strongly correlated ($r = 0.76$ for GPT-J, $r = 0.73$ for OLMo-7B), which could confound permutation importance tests (if two features encode the same signal, permuting one leaves the signal present in the other). To address this, the paper applies PCA to reduce faithfulness and faith probability to a single component capturing 89% of the variance, used only for the feature importance test. Other feature pairs show low to moderate correlations, confirming that the LRE metrics carry complementary information.

  • Hyperparameter sweeps for LRE fitting (Appendix C, Figures 6–9): The paper reports per-layer faithfulness (Figure 6) and causality (Figure 7) sweeps for OLMo-7B across all 24 relations at four checkpoints (41B, 419B, 1.05T, 2.05T tokens). The optimal layer varies by relation and checkpoint, but the paper selects the layer based on causality (not faithfulness) to avoid the early-checkpoint bias where faithfulness is inflated by default predictions. Figures 8 and 9 show the β and rank sweeps at the best-performing layer. The paper uses per-relation β (unlike Hernandez et al. (2024), which uses one β per model) because "this avoids disadvantaging specific relations" (Section 3.1).

  • Zero-shot and few-shot accuracy across checkpoints (Appendix F, Figures 13–14): Full breakdowns for OLMo-1B (Figure 13) and OLMo-7B (Figure 14) show zero-shot accuracy, 5-shot accuracy, and causality for every relation at all 8 checkpoints. These figures confirm that for most relations, the three metrics rise together through training, with causality sometimes leading (as with "food-from-country"). The ordering of relations by performance is largely stable across checkpoints—relations that perform well early continue to perform well, and low-frequency relations never catch up.

  • Replication across model sizes and families: The paper demonstrates the frequency-linearity relationship across three models spanning different scales (1B, 6B, 7B parameters) and different training datasets (Dolma for OLMo models, the Pile for GPT-J). The relationship holds qualitatively in all cases, with the threshold varying by model scale. The cross-model regression generalization (Table 1) further demonstrates that the relationship is not an artifact of a specific model or dataset. The paper does not, however, test models beyond the 1B–7B range (e.g., 13B, 70B, or larger), which would be needed to establish whether the frequency threshold continues to decrease with scale and whether the relationship remains linear.

Critical Assessment

Claim 1: The development of linear representations for factual recall relations in LMs is related to frequency as well as model size.

This claim is well-supported for the models tested but with important scope limitations. The correlation $r = 0.82$ between co-occurrence frequency and causality in OLMo-7B (Figure 2) is robust and is replicated across OLMo-1B and GPT-J (though exact correlation coefficients for those models are not reported separately). The cross-checkpoint analysis in the same figure shows that this correlation is not an artifact of training duration—it holds at 41B tokens as strongly as at 2T tokens.

However, the claim is established for exactly three models in the 1B–7B parameter range, all of similar architecture. There is no evidence about whether the relationship holds for models at substantially different scales (e.g., 70B, 405B parameters), different architectures (e.g., mixture-of-experts, state-space models), or models trained with different objectives (e.g., instruction-tuned rather than base models). The "model size" component of the claim is supported by exactly three data points (the thresholds in Figure 2's table), which the paper correctly identifies as suggestive rather than conclusive: "it is possible that scale also affects this threshold." The monotonic decrease in threshold with model size (4,447 → 1,998 → 1,097 for 1B → 7B → 6B) is suggestive but could be coincidental, especially since GPT-J's threshold is lower than OLMo-7B's despite having fewer parameters (6B vs. 7B), which could reflect dataset differences (Dolma vs. the Pile) rather than a pure scale effect.

Claim 2: Linear representations form at predictable frequency thresholds during training, regardless of when this frequency threshold is met. The formation of these representations also correlates strongly with recall accuracy.

The first sentence is well-supported by Figure 2. The scatter plots clearly show that causality scores are determined by co-occurrence count, not by training step—relations that have surpassed their threshold by 41B tokens already exhibit high-quality LREs, and relations that never reach it fail to form them even after 2T tokens. The gray dots (intermediate checkpoints) are interleaved with the final checkpoint, not systematically below them, which is strong evidence that the threshold effect dominates any gradual improvement with training time.

The second sentence—that formation correlates strongly with recall accuracy—is supported ($r = 0.74$ correlation between frequency and 5-shot accuracy), but the paper's own analysis shows this correlation is weaker than the frequency-linearity correlation ($r = 0.82$). More importantly, the paper documents counterexamples where accuracy is high but linearity is low ("star-constellation-name" at 84% accuracy, 44% causality), which means the claim as stated is an oversimplification. A more precise statement would be: frequency correlates with both linearity and accuracy, but they are distinct consequences, and frequency is more directly linked to linearity than to accuracy.

A stronger test of this claim would require demonstrating that the threshold is truly about frequency rather than some correlated variable (e.g., relation "naturalness," template diversity, or semantic complexity). The paper does not control for these confounds. For instance, low-frequency relations like "star-constellation-name" may also be inherently more complex or diverse (stars belong to many constellations, constellations have many stars) in ways that make linear encoding harder independent of frequency. Without ablating these alternative explanations, the causal claim that frequency causes linearity remains correlational.

Claim 3: Measuring the extent to which a relation is represented linearly allows prediction of approximate frequencies of individual terms in the pretraining corpus, even without access to the model's training data.

This claim is supported with clear evidence but also clear limitations that the paper mostly acknowledges. For object frequency prediction within the same model (Figure 3), LRE features achieve ~70% within-order-of-magnitude accuracy vs. ~42% for log-probability features—a substantial improvement. For cross-model prediction (Table 1), the LRE features maintain an advantage (~65% for GPT-J) even when the regression is trained only on OLMo data, which is the key demonstration that the method works for closed-data models.

However, the practical utility of this method is significantly constrained:

  • Within-order-of-magnitude is coarse. Predicting that a term appeared ~1,000 times when it actually appeared ~5,000 times counts as "correct." For many forensic or auditing purposes, this level of precision is insufficient—distinguishing between a term appearing 100 times vs. 1,000 times vs. 10,000 times matters for questions about memorization, contamination, or fair use.
  • Subject-object co-occurrence prediction is near-baseline. Figure 3 shows that even with LRE features, subject-object co-occurrence prediction is only marginally above the mean baseline ($0.67 \pm 0.08$ vs. $0.67 \pm 0.16$ for OLMo in Table 1). Since co-occurrence is arguably more informative than individual object frequency (it captures relational knowledge rather than mere term presence), this limits forensic applications.
  • The method requires measuring LREs for specific relations. To predict frequencies, one must have a dataset of factual relations, fit LREs for those relations on the target model, and then apply a regression trained on a source model. This is non-trivial setup that limits ad-hoc use.
  • Performance varies by relation type (Table 2). The regression transfers well for geo-political relations but catastrophically for low-frequency ones like star-constellation. Without knowing a priori which relations will transfer well, the method's reliability on arbitrary held-out relations is uncertain.
  • The method only predicts frequency for terms that appear in the relations dataset. It cannot predict the frequency of arbitrary terms—only those that participate in the measured relations.

Claim 4 (implicit): The LRE-frequency relationship is causal—frequency determines whether linear representations form.

The paper is appropriately cautious about this claim, explicitly stating in Appendix A: "we can not draw causal claims about how exposure affects individual representations, due to the cost of counterfactual pretraining." The evidence is purely correlational. The strongest causal-leaning evidence is the cross-checkpoint analysis (LRE quality is high at 41B tokens if frequency is high, not just at the end of training), but this only rules out the alternative explanation that training time alone drives linearity—it does not rule out the possibility that some third variable (e.g., semantic simplicity, template consistency) drives both frequency and linearity.

The paper would be strengthened by any of the following (none of which are present): (1) controlled experiments where specific relation terms are artificially upsampled or downsampled in a training corpus and the effect on LRE quality is measured; (2) analysis showing that the frequency-linearity relationship holds after controlling for other measurable properties of relations (e.g., number of unique subjects/objects, average subject-object edit distance, template diversity); (3) evidence that LRE quality for a relation changes when its frequency in the training data changes (e.g., comparing models trained on different data mixtures). Without such evidence, the paper demonstrates a strong and practically useful correlation but not a causal mechanism.

Overall assessment of weak points in experimental design:

  • Single dataset (RELATIONS), single task type (factual recall). All LRE measurements are on 24 factual relations from Hernandez et al. (2024). The commonsense extension (Appendix G) produces a much weaker correlation ($r = 0.42$), suggesting the findings may be specific to factual relations where co-occurrence is a good proxy for relation mention. The paper does not test on other relation types (e.g., syntactic relations, temporal relations, comparative relations) or other domains (e.g., code, scientific reasoning).

  • Test set size for regression evaluation. With 10,488 subject-object pairs across 24 relations, the held-out relation evaluation uses approximately $\sim$10,488/24 ≈ 437 examples per fold. The standard deviations in Table 1 (ranging from ±0.07 to ±0.17) suggest non-trivial variance across folds. The paper does not report whether performance varies systematically by which relation is held out (beyond the anecdotal error analysis in Table 2).

  • The β hyperparameter introduces a confound. Each relation has its own optimal β, found through a sweep. If a relation can achieve high causality only with a specific β, but that β is found through search, the reported causality might be optimistic relative to what would be obtained with a fixed β. The paper does not compare per-relation β to a fixed β in terms of the frequency-linearity correlation, so it is unclear whether the per-relation tuning inflates the correlation.

  • Batch Search is validated against WIMBD but only for the final checkpoint. Figure 10 shows the comparison for the full Dolma dataset. However, the paper's core contribution involves per-checkpoint counts, where WIMBD cannot serve as a reference. There is no independent validation that the per-batch counting is correct at intermediate checkpoints—the paper trusts the OLMo batch reconstruction tooling.

  • No confidence intervals on the main correlation. The paper reports $r = 0.82$ without confidence intervals, making it difficult to assess whether the difference from $r = 0.74$ (frequency-accuracy correlation) is statistically significant. Given that the data points are relations (n = 24), the effective sample size for the correlation is small.

  • The regression model choice (random forest) is not ablated. The paper uses 100-tree random forests without comparing to simpler models (linear regression, ridge regression) or other non-linear models (gradient boosting, small neural networks). It is possible that a simpler model would achieve similar performance with LRE features, which would strengthen the claim that the signal is genuinely present in the features rather than extracted through the model's capacity.

6. Limitations and Trade-offs

Limitation 1: The Causal Claim Is Not Established — The Frequency-Linearity Relationship Is Purely Correlational

The assumption or constraint. The paper's central narrative—that pretraining frequency causes linear representations to form—is supported exclusively by correlational evidence. Throughout the paper, the authors are careful to note this: "we can not draw causal claims about how exposure affects individual representations, due to the cost of counterfactual pretraining" (Appendix A). The evidence consists of: (1) a strong Pearson correlation between average co-occurrence frequency and causality ($r = 0.82$, Figure 2), (2) the observation that this correlation holds across checkpoints, and (3) a regression model that can predict frequency from LRE quality (Section 5). None of these establish causation.

The consequence. Without a causal link, the paper's practical recommendations are weaker than they appear. The finding that linear representations form above a certain frequency threshold (~1k–4k co-occurrences depending on model scale) could be actionable—practitioners might curate training data to ensure target relations exceed these thresholds, expecting linear structure to follow. But if frequency is merely correlated with some third variable that actually drives linearity (e.g., semantic simplicity of the relation, consistency of the templates in which it appears, number of distinct subject-object pairs), then manipulating frequency alone may have no effect. The paper cannot distinguish between "frequency causes linearity" and "both frequency and linearity are driven by the same underlying property of a relation (e.g., how formulaically it is expressed in text)." Low-frequency relations like "star-constellation-name" may also be inherently more complex—stars belong to many constellations, constellations have many stars, and the mapping may be expressed in more diverse and indirect ways in text—making linear encoding harder independent of raw co-occurrence count. The paper does not control for any measure of relation complexity, template diversity, or semantic regularity.

What evidence exists in the paper. The cross-checkpoint analysis (Figure 2) constitutes the strongest causal-leaning evidence: if training time alone drove linearity, relations at early checkpoints should have lower causality than relations at late checkpoints with the same frequency, but they do not. However, this only rules out training duration as an alternative explanation—it does not rule out confounding by relation properties. The paper acknowledges this gap in Appendix A but does not propose or conduct any analysis that would strengthen the causal interpretation, such as controlling for measurable confounds (number of unique subjects/objects, average diversity of templatic expressions, relation arity) or conducting a targeted data ablation (upsampling/downsampling specific relations in a small-scale training run and measuring LRE quality).

Mitigation status. The paper explicitly flags this as a limitation and motivates the regression experiments (Section 5) as partial evidence for the relationship's robustness: "We motivate this approach as a possible way to detect the training data of closed-data LMs; however, we are not able to make any guarantees on its efficacy in settings not shown here, and would caution drawing strong conclusions without additional information" (Appendix A). The regression results show the relationship is stable enough to be practically useful, but this is still a demonstration of correlation, not causation. Counterfactual pretraining is identified as the gold standard but is acknowledged as prohibitively expensive. No surrogate causal analysis (e.g., instrumental variables, difference-in-differences across data mixtures, or small-scale controlled training runs) is attempted.

Limitation 2: The Difficulty Estimation Protocol (Batch Search) Is Computationally Prohibitive for Deployment and for Replication

The assumption or constraint. The paper's entire analysis depends on counting subject-object co-occurrences in the exact tokenized batches seen during pretraining—this is the independent variable that everything else correlates with. However, performing Batch Search across the ~2 trillion tokens of Dolma required 900 CPUs running for approximately one day (Section 3.2 and 3.4). This cost is substantial: the paper states they "searched 10k terms in the approximately 2T tokens of Dolma" (Section 3.2), which at 900 CPU-days represents a significant computational investment. For models beyond the OLMo family where batch reconstruction tooling is not available, even this approach is infeasible—the paper falls back to WIMBD document-level counts for GPT-J (Section 3.2), which they validate as a reasonable approximation (Appendix D, Figure 10, slope = 0.94, $r = 0.99$) but which they acknowledge overestimates co-occurrence counts because document windows are larger than sequence windows.

The consequence. This limitation operates on two levels. First, it makes the paper's own methodology difficult to replicate for other models and datasets. The Batch Search tool is released as open source, and the paper states "we release our code as Cython bindings that integrate out of the box with existing libraries" (Section 3.2). However, the cost is high enough that few research groups will perform comparable analyses on other models—the paper effectively reports on exactly three models (OLMo-7B, OLMo-1B, GPT-J) with no pathway for the community to easily extend the analysis to, say, Llama-3, Mistral, or Qwen models where training data and batch reconstruction may be unavailable. Second, it means the frequency-linearity relationship cannot be used as an inexpensive diagnostic by practitioners. The paper's framing suggests that knowing the frequency threshold could guide data curation (ensure target concepts meet the threshold), but actually measuring whether a concept meets the threshold for a given model requires exactly the expensive counting infrastructure the paper built.

What evidence exists in the paper. Section 3.2 describes the computational cost (900 CPUs, ~1 day), and Appendix D validates the cheaper WIMBD approximation for document-level counts. However, the paper does not provide scaling estimates for smaller or larger corpora, nor does it explore whether co-occurrence counts from a small random subsample of the corpus (say, 1% of Dolma) would produce sufficiently correlated counts. This last point is important: if 1% subsampling preserved the rank ordering of relations by frequency (as WIMBD does), the cost would drop 100×, making the analysis far more accessible. The paper does not test this.

Mitigation status. The paper partially addresses this through the WIMBD comparison—for final checkpoints, document-level counts are shown to be a "good approximation" (Figure 10)—and by releasing the Batch Search code. However, WIMBD does not solve the per-checkpoint counting problem, which is where the paper's most interesting result lives (the cross-checkpoint analysis in Figure 2 showing that frequency thresholds matter regardless of training stage). The paper does not suggest lower-cost alternatives for per-checkpoint counting (e.g., extrapolating from final counts using a simplified model of data ordering) or estimate how the cost scales with corpus size, making it difficult for practitioners to assess whether they could replicate the analysis on their own models and corpora.

Limitation 3: Single Domain (Factual Relations) — Generality to Other Relation Types and Tasks Is Weakly Supported and Likely Breaks Down

The assumption or constraint. The paper's entire analysis is on 24 factual relations from the RELATIONS dataset (Hernandez et al., 2024), such as "capital-city," "person-mother," and "country-currency." These relations have a specific property that makes the co-occurrence proxy work: when a subject and object co-occur in text, it very likely reflects the relation being described. The paper acknowledges this property explicitly: "Factual relations are much easier to get accurate counts for, so we leave non-factual relations for future work" (Section 3.1 footnote 2). The choice is deliberate—the paper builds on Elsahar et al. (2018), who "show that this approximation is quite accurate" for factual triplets (Section 3.2). But this means the entire framework rests on a domain where the frequency measurement is unusually clean.

The consequence. The paper does test commonsense relations as an extension (Appendix G, Figure 15), and the result is sobering: the correlation between co-occurrence frequency and causality drops from $r = 0.82$ (factual) to $r = 0.42$ (commonsense). The paper's own diagnosis is that "subject-object frequencies do not accurately track occurrences of the relation being mentioned" for these relations. For example, the co-occurrence of "researching history" and "historian" does not capture all the ways the concept of a historian is defined during pretraining—someone could be described as a historian without ever being adjacent to the phrase "researching history." This means that for the vast majority of interesting concepts in LMs—commonsense knowledge, abstract relations, social stereotypes, procedural knowledge—the paper's method either doesn't apply or would require a fundamentally different kind of frequency measurement that the paper does not provide.

More broadly, the paper studies only relations that can be expressed as subject-relation-object triplets with discrete, tokenizable answers (most of which are single-token or short phrases—capital cities, names, currencies). This excludes continuous-answer tasks, generative tasks, relational tasks where the output is not a named entity, and any concept that doesn't fit cleanly into a relational triplet format. The generalizability of the frequency-linearity link to, say, syntactic phenomena, emotional valence, or factual knowledge about procedures (e.g., "how to bake a cake") is completely unaddressed.

What evidence exists in the paper. Appendix G (Figure 15) presents the commonsense extension: 8 relations evaluated on OLMo-7B, showing $r = 0.42$ correlation between log co-occurrence and causality. This is a significant drop from $r = 0.82$ for factual relations. The paper acknowledges that "it is possible subject-object frequencies do not accurately track occurrences of the relation being mentioned" in these cases and that "we caution treating these under the same lens as the factual relations." The full set of factual relations (24) is listed in Appendix B and Figure 5. All are entity-centric mapping relations—no syntactic relations, no event-participant relations beyond simple attribution, no comparative relations.

Mitigation status. The paper is transparent about this scope limitation ("we leave non-factual relations for future work") but does not propose how the method could be extended. The commonsense experiment (Appendix G) is presented as an exploratory extension rather than a systematic attempt to generalize. The paper does not test on any dataset other than RELATIONS, nor does it test on any task type other than factual recall (e.g., sentiment classification, natural language inference, coreference resolution), even though LREs have been applied to some of these in other work (Hernandez et al., 2024; Chanin et al., 2024). This means the main result—$r = 0.82$—should be understood as applying specifically to factual entity-mapping relations with clean co-occurrence signals, not to relations or representations in general.

Limitation 4: The Regression-Based Frequency Prediction Produces Only Order-of-Magnitude Estimates and Fails for Subject-Object Co-Occurrence

The assumption or constraint. Section 5 presents a regression model that predicts pretraining term frequencies from LRE quality metrics. The evaluation metric is within-order-of-magnitude accuracy—a prediction is counted as correct if it is within a factor of 10 of the true count (e.g., predicting 500 for a true count of 2,000 is correct; predicting 50 or 20,000 is not). The paper acknowledges that "predicting the exact number of occurrences" is "difficult" (Section 5.2) and reports mean absolute error in log space as a complementary metric. But the practical question is whether order-of-magnitude precision is useful for the applications the paper motivates—specifically, "analyzing the pretraining corpora of closed-data models with open weights" (Abstract) and detecting "whether a model was trained on specific domains" (Section 5 intro).

The consequence. Order-of-magnitude accuracy is extremely coarse. For a term appearing 50,000 times in pretraining, a prediction anywhere from 5,000 to 500,000 is considered correct—a 100× range. For many forensic or auditing purposes, this resolution is insufficient. Consider the motivating example from Section 5: testing "whether a model was trained on specific domains (e.g., Java code) by measuring the presence of relevant LREs." If the question is "was this model trained on Java code?" and the method predicts that Java-related terms appeared between ~10,000 and ~100,000 times when they actually appeared ~30,000 times, that's correct by the metric—but it doesn't distinguish between a model that saw Java heavily and one that saw it tangentially. Conversely, if the regression predicts a term appeared in the millions when it appeared only in the thousands (as in the "star-constellation-name" example in Table 2, where "Arcturus" → "Boötes" is predicted at 974,550 vs. ground truth 2,817, a 346× error), the estimate is not just imprecise but catastrophically wrong.

The subject-object co-occurrence prediction results are even weaker. In Table 1, the mean frequency baseline achieves $0.67 \pm 0.16$ on OLMo evaluation, and LRE features achieve $0.68 \pm 0.08$—essentially identical within error. The paper acknowledges: "We find that subject-object co-occurrence frequency is likely too difficult to predict given the signals that we have here, as our predictions are higher than, but within one standard deviation of the mean baseline" (Section 5.2). Since subject-object co-occurrence is arguably the more informative quantity (it captures relational knowledge rather than mere term frequency), the failure to predict it well limits the forensic value of the method.

What evidence exists in the paper. Figure 3 shows that for object frequency prediction, LRE + LM features achieve ~70% within-magnitude accuracy vs. ~42% for LM-only and ~31% for the mean baseline—a genuine improvement. However, the absolute ~70% figure means ~30% of predictions are wrong even by the generous order-of-magnitude standard. Table 2 provides a qualitative error analysis showing predictions ranging from remarkably accurate (1× ground truth for "Brazil" → "Portuguese") to catastrophically wrong (346× for "Arcturus" → "Boötes," 207× for "Prince Harry" → "Princess Diana"). The subject sensitivity (Prince William vs. Prince Harry, both predicting Princess Diana with 4.6× vs. 207× error) shows the regression is fragile to which specific examples are chosen to measure LREs for. The paper reports mean absolute error in log space as LRE features: 2.1 (vs. 4.2 for LM-only) for object prediction, and 1.9 (vs. 2.3 for LM-only) for subject-object prediction (Section 5.2). A log-space error of 2.1 corresponds to about an 8× multiplicative error on average—not catastrophic but far from precise.

Mitigation status. The paper acknowledges the difficulty of the prediction task and the limitations of within-order-of-magnitude accuracy, but does not attempt to improve resolution through additional features, different model architectures, or ensemble methods. The error analysis in Table 2 is provided as a qualitative illustration rather than a systematic study of failure modes. The paper suggests that "future work could expand on this tool by incorporating it with other data inference methods for greater confidence" (Appendix A), implying that the standalone method's precision is insufficient for most practical purposes and would need to be combined with other signals.

Limitation 5: The Method Requires Access to Model Internals (Hidden States and Gradients) and a Predefined Set of Relations — Limiting Practical Applicability to Open-Weights Models and Known Relation Types

The assumption or constraint. The entire LRE methodology requires extracting hidden states from intermediate layers of the model and computing Jacobians of the forward pass—operations that are only possible with full access to the model's weights and computational graph. This means the paper's frequency analysis and regression method are restricted to open-weights models. The paper explicitly targets this regime: the abstract states the regression provides "a new method for estimating properties of the otherwise-unknown training data of closed-data models," but critically, this means closed-data models with open weights (like GPT-J, Llama, OLMo)—not truly closed models like GPT-4 or Claude, where neither weights nor data are accessible. The paper does not discuss this distinction, but it is fundamental: the method cannot be applied to the most important targets of training data inference (proprietary models) because their internal representations are inaccessible.

Furthermore, the method requires a predefined dataset of subject-relation-object triplets (the RELATIONS dataset) against which LREs can be measured. To predict the frequency of a term, that term must appear as a subject or object in at least one of the 24 factual relations in the dataset. The paper uses 10,488 unique subjects and objects across these relations (Section 3.1), which is a fixed vocabulary. To query the frequency of an arbitrary term not in this set, one would need to first construct a relation containing that term and then fit LREs for that relation—a significant expansion of the methodology that the paper does not address.

The consequence. The practical applicability is doubly constrained. First, the method is limited to the small and shrinking set of models for which full weights are released (primarily research models like OLMo, Llama, and Mistral, plus a few older commercial models like GPT-J). The most important targets for training data forensics—frontier proprietary models—are completely inaccessible. Second, even for open-weights models, the method can only assess frequency for the specific set of terms in the RELATIONS dataset. A practitioner wanting to know whether a model was trained on a particular codebase, book, or domain-specific terminology would need those terms to serendipitously appear in the factual relations data, which is unlikely for most terms of interest.

The paper's motivating example—"testing whether a model was trained on specific domains (e.g., Java code) by measuring the presence of relevant LREs" (Section 5)—illustrates this gap. The RELATIONS dataset does not contain Java-specific relations. One would need to construct a new relation (e.g., "Java-class-extends" mapping classes to their parent classes), verify that co-occurrence is a reasonable proxy in that domain, fit LREs, and then apply the regression. This is a non-trivial extension that the paper gestures toward but does not validate.

What evidence exists in the paper. The requirements are implicit in the methodology (Sections 3.1 and 3.4): LRE fitting requires "hidden states from LMs during the processing of the test example in a few-shot setup" and Jacobians computed via backpropagation (Equation 1). The reliance on the RELATIONS dataset is stated in Section 3.1: "We use a subset of the RELATIONS dataset Hernandez et al. (2024), focusing on the 25 factual relations of the dataset." The paper does not experiment with extending the method to new relations, nor does it discuss the cost or feasibility of constructing new relation datasets for terms of interest.

Mitigation status. The paper does not explicitly acknowledge this as a limitation. The "closed-data models with open weights" framing in the abstract is technically accurate but could be misleading—it applies to a narrow class of models. The paper's cross-model generalization experiment (Table 1) is impressive within this constrained setting (OLMo → GPT-J), but it does not demonstrate that the method could work in a setting where model internals are unavailable (e.g., through API-accessible representations or black-box probing). No discussion is provided of how the method could be adapted to truly closed models or to arbitrary query terms.

Limitation 6: Model Scale Dependency Is Suggested but Unvalidated — Only Three Models Across a Narrow Range (1B–7B) Are Tested

The assumption or constraint. The paper reports frequency thresholds for the emergence of high-quality LREs across three models: GPT-J (6B) requires ~1,097 average co-occurrences, OLMo-7B requires ~1,998, and OLMo-1B requires ~4,447 (Figure 2 table). The paper states that "it is possible that scale also affects this threshold" (Section 4.2) but frames this as an open question rather than a settled finding. The implication is clear: larger models may require fewer co-occurrences to form linear representations. If this trend extrapolates, a 70B or 405B model might form high-quality LREs with only a few hundred co-occurrences, which would have significant practical implications for data curation and interpretability tool applicability.

The consequence. With only three data points across a narrow scale range (1B to 7B, less than one order of magnitude), the relationship between model scale and frequency threshold cannot be reliably characterized. Worse, the ordering is not even monotonic within this narrow range: GPT-J (6B) has a lower threshold (~1,097) than OLMo-7B (~1,998), which could reflect model architecture differences (GPT-J vs. OLMo), training data differences (the Pile vs. Dolma), or measurement differences (WIMBD vs. Batch Search for counting) rather than a pure scale effect. The paper correctly notes that it "cannot draw conclusions from only three models" (Section 4.2), but this caveat is easy to overlook given the prominence of the threshold table in Figure 2.

Without testing at larger scales, the paper cannot answer the most practically important question: do the frequency thresholds continue to decrease with scale, and do they eventually become low enough that most naturally occurring concepts in web-scale corpora will exceed them? If the threshold drops to, say, 100 co-occurrences at 70B parameters, then linear representations become near-universal for factual relations in large models, and the paper's central finding (that frequency explains variation in linearity) would actually imply that linearity is guaranteed for all but the very rarest concepts in large models. Conversely, if the threshold plateaus, then even very large models may never form linear representations for low-frequency relations, and the practical boundaries the paper identifies remain stable. The paper provides no evidence either way.

What evidence exists in the paper. The three data points in the Figure 2 table are the entirety of the scale analysis. The paper does not test OLMo-13B, Llama-2-70B, or any model outside the 1B–7B range. The OLMo family (Groeneveld et al., 2024) includes a 7B model and a 1B model but not intermediate sizes, so the paper cannot even establish whether the threshold scales smoothly with parameter count or exhibits discontinuities. GPT-J is from a different model family entirely (Wang & Komatsuzaki, 2021), trained on different data (the Pile vs. Dolma), and measured with different frequency counting tools (WIMBD vs. Batch Search), making it a poor point of comparison for isolating the effect of scale.

Mitigation status. The paper explicitly flags this: "it is possible that scale also affects this threshold. Although we cannot draw conclusions from only three models" (Section 4.2). The hedging is appropriate, but the paper does not discuss why additional model scales were not tested, nor does it propose specific future experiments to characterize the scale-threshold relationship. Given the OLMo team also released a 7B model (tested) and the paper uses 1B and 7B variants, the absence of a mid-range model (e.g., 3B) or a larger model (e.g., 13B from another family with open training data) is notable. The threshold finding is presented as a key result (headline table in Figure 2) but the caveat about sample size is buried in discussion text, creating a risk that readers will overinterpret the scale-threshold relationship.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new method, architecture, or training procedure. Its contribution is conceptual and diagnostic: it specifies the boundary conditions under which the Linear Representation Hypothesis holds, converting an unconditional claim ("LMs represent features linearly") into a conditional one ("LMs represent features linearly, provided the relevant terms co-occur above a model-specific frequency threshold during pretraining"). This is best understood as a refinement of existing theory with practical teeth—it does not overturn the Linear Representation Hypothesis or the theoretical work by Jiang et al. (2024) showing that the training objective encourages linearity, but it identifies why that encouragement succeeds for some concepts and fails for others in actual trained models.

The magnitude of this shift is moderate but consequential for several subfields. For interpretability researchers, the paper provides a diagnostic tool: before investing effort in finding or using linear representations for a concept, estimate its likely pretraining frequency. If it falls below the model's threshold (approximately 1k–4k co-occurrences for models in the 1B–7B range, and possibly lower for larger models), linear methods may simply not work, and alternative approaches (non-linear probes, mechanistic analysis of circuits) should be pursued instead. This is actionable and can prevent wasted effort on hunting for representations that the training data never supported.

For the data-centric ML community, the paper provides a new bridge between pretraining data composition and model internals—a connection that has been largely absent from the literature. Prior work showed that frequency affects behavioral performance (Razeghi et al., 2022; Mallen et al., 2023; Chang et al., 2024), but this paper shows that frequency also shapes the geometric structure of representations in predictable ways. This suggests that data curation for interpretability—designing pretraining mixtures to ensure that target concepts form linear representations amenable to steering or editing—is a coherent goal, not just a post-hoc hope.

For the training data inference community, the paper introduces a novel signal source: representation-geometric properties (specifically, LRE quality) that carry recoverable information about pretraining corpus statistics. The fact that this signal transfers across models trained on different datasets (Table 1: 65% within-order-of-magnitude accuracy when training regression on OLMo and evaluating on GPT-J, vs. 42% for log-probability-only features) demonstrates that the relationship between linearity and frequency is not an artifact of a specific model or corpus. This opens a genuinely new direction for training data forensics—one that leverages interpretability tools for a purpose their designers did not anticipate.

The paper also reconciles a tension that had been accumulating in the interpretability literature. Hernandez et al. (2024) found wide variation in LRE quality across relations (>80% linear for "country-largest-city," <30% for "star-in-constellation"), and Chanin et al. (2024) documented similar heterogeneity. These findings were puzzling under the Linear Representation Hypothesis: if linearity is encouraged by the training objective, why such stark differences? The paper's answer—that subject-object co-occurrence frequency correlates at $r = 0.82$ with LRE quality (Figure 2), and that relations below a model-specific threshold consistently fail to form linear representations—provides a unified, empirically grounded explanation. The heterogeneity is not noise or measurement failure; it reflects genuine differences in training signal strength.

Finally, the paper redirects attention within interpretability research. The finding that some relations achieve high accuracy without high linearity (e.g., "star-constellation-name" at 84% 5-shot accuracy but only 44% causality in OLMo-7B; Section 4.3) implies that linear representations are not the only mechanism by which LMs perform factual recall. This makes it less attractive to assume that any task a model performs well must have a clean linear encoding somewhere in its representations. Conversely, it makes it more attractive to investigate what non-linear mechanisms models use for low-frequency but accurately-recalled facts—a direction the paper explicitly flags (Section 6: "it is still an open question how LMs are able to recall these tasks").

Follow-Up Research This Work Enables

Controlled counterfactual pretraining experiments to establish causation between frequency and linearity. The paper's central finding is correlational, as the authors acknowledge (Appendix A). A strong follow-up would train multiple small LMs (e.g., 100M–300M parameters) from scratch on synthetic or semi-synthetic corpora where the frequency of specific subject-object pairs is experimentally controlled—some relations are artificially upsampled, others downsampled—and then measure whether LRE quality for those relations moves in the predicted direction. This would test whether frequency causes linearity or whether both are driven by an unmeasured confound (e.g., relation complexity, template diversity, semantic regularity). A positive result would upgrade the paper's central claim from correlational to causal and would provide direct guidance for data curation. A negative result (frequency manipulation fails to change LRE quality, or changes it only when other factors co-vary) would substantially refine our understanding of what drives linear structure and would redirect attention toward those confounds. The paper's Batch Search tool and LRE fitting pipeline provide the measurement infrastructure needed for such experiments, and the threshold estimates from Figure 2 provide target frequencies to test.

Characterizing the frequency-linearity relationship across model scales, architectures, and training objectives. The paper tests three models in the 1B–7B parameter range and observes a suggestive monotonic decrease in frequency threshold with model size (4,447 → 1,998 → 1,097 co-occurrences for 1B → 7B → 6B parameters; Figure 2 table). A systematic follow-up would measure LRE quality on the same set of factual relations across a wide range of openly-available model scales—OLMo-7B, Llama-2-7B, Llama-2-13B, Llama-2-70B, OLMo-1B, Pythia-1B through Pythia-12B (Biderman et al., 2023, which also releases intermediate checkpoints)—and compute frequency thresholds for each. This would answer whether the threshold continues to decrease with scale (potentially reaching levels where most naturally-occurring concepts in web-scale corpora exceed it, making linearity near-universal for factual relations in large models) or plateaus. Testing across architectures (e.g., mixture-of-experts vs. dense, rotary vs. learned position embeddings) and training objectives (base models vs. instruction-tuned variants) would reveal whether the relationship is architecture- or objective-dependent. A negative result—finding that the threshold is unstable across architectures or does not monotonically decrease with scale—would caution against overgeneralizing from the three-model sample in the current paper. The paper provides the LRE fitting and batch counting methodology that makes this extension straightforward; the primary cost is per-model counting, which the released Batch Search tool accelerates.

Extending the analysis to non-factual relations and alternative frequency metrics. The paper's main results are on factual entity-mapping relations where subject-object co-occurrence is a good proxy for relation mention ($r = 0.82$; Figure 2). The commonsense extension (Appendix G, Figure 15) shows the correlation drops to $r = 0.42$, likely because co-occurrence does not accurately track relation mentions for abstract concepts. A strong follow-up would develop alternative frequency metrics that better capture training signal for non-factual relations—for instance, counting not just subject-object co-occurrence but co-occurrence within specific syntactic patterns (e.g., dependency paths connecting subject and object tokens), or using an auxiliary classifier to identify sentences that express the target relation and counting only those. The hypothesis is that a more accurate frequency metric would recover the high correlation with LRE quality for commonsense and abstract relations. A positive result would extend the paper's framework to the much broader class of concepts LMs encode, dramatically increasing its practical relevance. A negative result (no frequency metric recovers the correlation) would imply that linear representations for non-factual relations are governed by fundamentally different principles than those for factual ones, an important boundary condition on the paper's claims.

Using LRE-based frequency prediction as one signal in a multi-modal training data inference system. The paper demonstrates that LRE features carry frequency information distinct from and stronger than output probabilities (Figure 3: ~70% vs. ~42% within-order-of-magnitude accuracy for object frequency prediction). However, the precision is coarse (order-of-magnitude) and the method requires a predefined relations dataset, limiting standalone practical use. A natural follow-up would combine LRE-based frequency predictions with other training data inference signals—tokenizer-based distribution inference (Hayase et al., 2024), membership inference attacks (Carlini et al., 2022), memorization-based counts (Carlini et al., 2023), and direct log-probability features—into an ensemble model that predicts term frequencies with higher precision and for arbitrary query terms (not just those in the RELATIONS dataset). The specific experiment would be: construct a dataset of models with known training data (OLMo, Pythia, GPT-J, and any other fully-open models), extract all available signals for a large vocabulary of terms, train a meta-regressor to predict actual frequencies, and evaluate held-out model generalization. The paper's cross-model generalization result (Table 1) provides initial evidence that the LRE signal transfers across models, making it a viable component of such an ensemble. The key metric would be whether the ensemble achieves substantially better precision (e.g., within-factor-of-2 or within-factor-of-3 accuracy) than any single signal alone. A positive result would produce a practically useful forensic tool for analyzing open-weights, closed-data models; a negative result (LRE features add negligible improvement in ensemble) would clarify the limits of representation-based inference.

Investigating whether the frequency threshold can be lowered through training interventions. The paper shows that linear representations consistently form only above model-specific frequency thresholds (~1k–4k co-occurrences for 1B–7B models; Figure 2). If these thresholds are a consequence of the standard pretraining objective (causal language modeling), it may be possible to lower them through explicit interventions—for example, by adding an auxiliary loss that encourages linear structure in the representation space for relations of interest, by using data augmentation to artificially increase co-occurrence counts during training without collecting more raw data, or by fine-tuning on carefully constructed relation datasets after pretraining. A specific experiment: take a pretrained model where a target relation falls below the frequency threshold and has low LRE quality, fine-tune it on a small dataset of that relation's subject-object pairs with varying sizes (10, 100, 1,000 examples), and measure whether LRE quality rises to match what would be expected from a naturally high-frequency relation. This would test whether the frequency threshold is a training-data-imposed constraint that can be overcome with targeted fine-tuning (a practical finding for model editors) or whether it reflects a deeper limitation of the model architecture or optimization process. The paper's LRE measurement pipeline provides the pre- and post-intervention evaluation tool.

Stress-testing the cross-model generalization of frequency regression under distribution shift. The paper's cross-model experiment (Table 1) evaluates generalization from OLMo (trained on Dolma) to GPT-J (trained on the Pile)—two English-heavy web-crawl corpora that likely share substantial content overlap. A critical stress test would be: train the regression on a model trained primarily on English text and evaluate on a model trained primarily on, say, code or non-English languages (e.g., StarCoder for code, or a multilingual model like BLOOM). The hypothesis is that the LRE-frequency relationship should remain stable (it reflects a general property of transformer training dynamics), but this could break if the target domain's statistical properties differ substantially from the source domain—for instance, if code tokens have fundamentally different co-occurrence patterns than natural language tokens, or if the relationship between co-occurrence and LRE quality is language-specific. A negative result (cross-domain generalization fails) would establish an important boundary on the method's applicability: LRE-based frequency prediction only works when source and target models are trained on similar data distributions. A positive result (generalization holds across domains) would substantially strengthen the case that the frequency-linearity relationship is a universal property of autoregressive transformer training, not an artifact of English factual text.

Practical Applications and Downstream Use Cases

Data curation for interpretability and model control. The paper provides concrete frequency thresholds above which linear representations reliably form (~1,997 co-occurrences for OLMo-7B, ~4,447 for OLMo-1B; Figure 2 table). A practitioner building a language model intended to be steerable or editable for specific concepts (e.g., a customer-support model where toxicity or factual accuracy needs to be controlled via representation editing) can use these thresholds to guide data curation: ensure that the target concepts' subject-object pairs appear together in training sequences at least as often as the threshold for the model scale being used. If the threshold is not met for a concept of interest, the practitioner knows in advance that linear steering methods are unlikely to work and can either (a) invest in additional training data for those concepts, (b) accept that alternative control methods (prompting, fine-tuning, guard models) will be needed, or (c) plan post-hoc fine-tuning specifically to induce linear structure (if follow-up work validates that this is possible). The paper's Batch Search tool (released as open-source Cython bindings, Section 3.2) provides the counting infrastructure to audit whether a given corpus meets frequency targets, though at significant computational cost for large corpora.

Triaging interpretability effort based on estimated concept frequency. For researchers and engineers applying linear interpretability methods (probing, steering, representation editing, activation patching) to a new model, a common failure mode is spending significant effort trying to find linear representations for concepts that do not have them. The paper's finding—that high-quality LREs are extremely likely above the frequency threshold and extremely unlikely below it—provides a pre-screen: before attempting to find or use a linear representation for a concept in an open-weights model, estimate the concept's likely co-occurrence frequency in the model's training data (using document-level counts from tools like WIMBD as a cheaper proxy, validated in Appendix D, Figure 10, $r = 0.99$ with Batch Search). If the estimated frequency is well below the model-specific threshold, expect linear methods to fail and redirect effort toward alternative approaches (non-linear probes, mechanistic circuit analysis) or toward data augmentation for the concept. If the frequency is well above the threshold, linear methods are likely to succeed. For the models tested, the thresholds are approximately 1k co-occurrences for 6B–7B models and 4k for 1B models (Figure 2 table). For larger models, the paper suggests but does not confirm lower thresholds, so conservative planning would use the 7B threshold as an upper bound. This triage approach is low-cost (document-level counting is far cheaper than the full batch search) and can prevent wasted effort in interpretability projects.

Inference of pretraining data composition for open-weights, closed-data models. The paper's regression model, trained on OLMo LRE features and evaluated on GPT-J, achieves 65% within-order-of-magnitude accuracy at predicting object frequencies without access to GPT-J's training data (Table 1). This provides a new—though approximate—tool for auditing models whose weights are public but whose training data is proprietary or undocumented. Specific scenarios include: (a) Contamination detection: if a benchmark's test-set answers appear as objects in the factual relations dataset, their predicted frequencies from the regression model can indicate whether the model was likely exposed to them frequently enough to raise contamination concerns. For example, if a benchmark question's answer term is predicted to appear millions of times when it should appear only thousands of times in a clean corpus, this flags potential data leakage. (b) Domain auditing: by measuring LREs for relations whose terms are associated with specific domains (e.g., legal terminology, medical facts, code-related entities), practitioners can estimate whether and how heavily those domains appeared in training, even when the training data is not disclosed. The paper's error analysis (Table 2) shows that predictions are better for "country related relations" and worse for low-frequency relations like "star-constellation-name" (346× error), so users should calibrate expectations by domain. (c) Compliance verification: for models claiming to exclude certain types of data (e.g., personal information, copyrighted material), the regression can provide approximate frequency estimates for relevant terms to check for unexpected high-frequency exposures. The method's primary limitation for these applications is the coarse precision (order-of-magnitude) and the requirement that target terms participate in one of the 24 factual relations in the RELATIONS dataset (or that users construct new relations for their terms of interest, which the paper does not validate). The method is therefore best used as a screening tool—identifying terms that are anomalously high-frequency and warrant deeper investigation via other methods—rather than as a standalone audit.