On the Acquisition of Shared Grammatical Representations in Bilingual Language Models

ArXiv: 2503.03962

🎯 Pitch

Models primed by a matching grammatical structure in Dutch predict English targets better than the reverse, but surprisingly this advantage for English holds even when English is the model's second language—upending the standard L1-to-L2 account of asymmetric priming. This target-language effect disappears, however, when the language pair is typologically distant, revealing that shared representations only crystallize when languages overlap enough to support them.

1. Executive Summary

This paper studies how bilingual language models develop shared grammatical representations across languages by training small (124M-parameter) GPT-2 Transformers from scratch with controlled language exposure—manipulating language pair, exposure order, and whether the second language is introduced sequentially or simultaneously—then probing those models using crosslingual structural priming (measuring whether a target sentence is more probable following a prime with matching grammatical structure, e.g., an English double-object dative primed by a Dutch double-object dative). The central finding is an asymmetrical effect: structural priming is more robust when English is the target language, independent of whether English is the first (L1) or second (L2) language and after controlling for training data quantity, establishing that the asymmetry previously attributed in human experiments to L1→L2 priming may instead be driven by features of the target language itself. A second boundary finding shows that priming effects are substantially stronger and more resilient to catastrophic forgetting for more similar language pairs (Dutch-English, Spanish-English) than for typologically more distant pairs (Polish-English, Greek-English), demonstrating that shared grammatical representations emerge robustly only when languages share sufficient structural overlap.

2. Context and Motivation

The Core Problem: We Don't Know How Multilingual Models Build Shared Grammatical Representations

The fundamental question this paper tackles is deceptively simple: what happens inside a language model when it starts learning a second language? Specifically, do bilingual language models develop shared abstract grammatical representations that span both languages, or do they maintain separate, language-specific representations that happen to produce similar outputs? And if representations are shared, what factors determine how easily they emerge?

This matters for several reasons the authors highlight throughout Section 1 and the related work discussion:

• Crosslingual transfer depends on shared representations. The dominant paradigm in multilingual NLP assumes that training on high-resource languages (primarily English) transfers benefits to lower-resource languages because models learn representations that are language-agnostic at some level of abstraction (Artetxe et al., 2020; Conneau et al., 2020; Wu and Dredze, 2019). If we don't understand how these shared representations form, we are essentially relying on an empirical phenomenon without mechanistic insight — which makes it impossible to predict when transfer will succeed or fail, or how to design training procedures that maximize it.

• Psycholinguistics has a confounded empirical record. Human experiments on bilingual structural priming (e.g., Schoonbaert et al., 2007; Bernolet et al., 2013) have found that priming effects are generally stronger when the prime is in the participant's first language (L1) and the target is in their second language (L2) — what the field calls L1→L2 priming. This has been interpreted as evidence about how bilinguals organize their grammatical knowledge. But there is a critical confound: in nearly all of these experiments, English is the L2. This is an artifact of the populations typically sampled — university students in countries like the Netherlands where L1 Dutch / L2 English bilinguals are plentiful. It has therefore been impossible to determine whether the observed asymmetries are due to the order of language acquisition (L1 vs. L2 status) or due to something about English specifically as the target language. Disentangling these factors would require training human participants to have controlled bilingual experiences from birth — clearly unethical and infeasible.

• Language similarity and typological diversity are underexplored constraints. Previous crosslingual transfer research has shown that transfer works better between more similar languages (Lin et al., 2019; Ogueji et al., 2021; Chang et al., 2024a), but this has primarily been studied through downstream task performance rather than through direct probes of representational sharing. Prior crosslingual structural priming work (Michaelov et al., 2023) found more robust priming for English-Dutch and English-Spanish than for English-Polish and English-Greek, but couldn't determine whether this was due to language similarity or differences in the amount of training data for each language in the pretrained models they studied — a fundamental confound when working with existing multilingual models whose training data compositions are often opaque or uncontrolled.

• There is no temporal account of shared representation formation. Previous structural priming studies involving language models have evaluated only the final checkpoint of pretrained models (Prasad et al., 2019; Sinclair et al., 2022; Michaelov et al., 2023). This provides a static snapshot: either the model exhibits priming or it doesn't. But it tells us nothing about when during training these shared representations emerge, whether their emergence is temporally linked to the acquisition of grammatical competence in the second language, or what happens to them when models experience catastrophic forgetting of the first language. Understanding the training dynamics would connect structural priming behavior more tightly to the learning process itself, rather than treating it as an endpoint phenomenon.

The Gap: No Controlled Causal Manipulations Exist

The abstract of Michaelov et al. (2023) provided the first evidence of crosslingual structural priming in Transformer language models, arguing this demonstrated shared abstract grammatical representations. But that study — like all prior work — evaluated existing pretrained models (XGLM, GPT-2, etc.) without manipulating training conditions. The authors could observe that priming existed, and that it was asymmetric across language pairs, but they could not isolate why. Was the asymmetry due to training data quantity imbalances? Order of language exposure? Inherent linguistic properties of the languages involved? These questions cannot be answered by probing a single frozen model; they require training models from scratch under systematically varied conditions.

This is where the paper's "controlled rearing" approach (the term from Misra and Mahowald, 2024) becomes essential. By training small models where the researchers control exactly what data the model sees, in what order, and in what proportions, they can perform causal manipulations that would be impossible with either human participants or with large pretrained models. If you train one model with English as L1 and Dutch as L2, and another with Dutch as L1 and English as L2, and you still see stronger priming when English is the target language regardless of L1/L2 status, you have directly demonstrated that the asymmetry is not about acquisition order — it's about English.

Where Prior Approaches Fall Short

The paper identifies specific limitations in prior work along several axes (Section 2):

Large pretrained models obscure causal relationships. Models like XGLM or mBERT are trained on undocumented, uncontrolled mixtures of web data across dozens of languages. When Michaelov et al. (2023) observe that English-Polish priming is weaker than English-Dutch priming, they cannot determine whether this is because Polish is typologically more distant from English, or because the model saw less Polish data during pretraining, or because the Polish data was lower quality, or some combination. The inability to control for data quantity is a fundamental limitation when trying to isolate the effects of linguistic similarity.

Human experiments have an irremediable confound. As discussed above, the L1/L2 vs. English-as-target confound in the psycholinguistics literature cannot be disentangled with human participants. The populations needed — for example, native English speakers who are highly proficient L2 speakers of Greek or Polish — are extremely difficult to recruit at scale, and even if recruited, their language experience would be uncontrolled (varying ages of acquisition, varying proficiency levels, varying immersion experiences). Structural priming experiments are also sensitive to lexical overlap between prime and target (Pickering and Ferreira, 2008), adding another layer of uncontrolled variation in naturalistic human data.

Static checkpoint analysis misses the learning process. Previous LM structural priming studies evaluate only the final trained model. But shared representations don't appear magically at the end of training — they are built gradually as the model is exposed to data. Without analyzing intermediate checkpoints, we can't answer questions like: Do crosslingual priming effects appear before the model has non-trivial grammatical knowledge in the second language? (If so, priming might not actually reflect shared grammatical representations — it might be driven by surface-level lexical overlap or statistical artifact.) Do priming effects emerge concurrently with L2 grammatical competence? Do they survive catastrophic forgetting when training transitions entirely to L2? These temporal questions require checkpoint-level analysis.

No controlled manipulation of bilingual experience type. Real bilinguals have different types of language experience: some learn both languages simultaneously from birth, others learn an L2 sequentially after L1 is established, and many have experience that falls between these extremes. Prior work on "bilingual" language models typically used simultaneous training (mixing languages throughout) or continued pretraining without systematically comparing these regimes (Aoyama and Schneider, 2024; Constantinescu et al., 2025). Without manipulating the timing and nature of L2 introduction, we can't understand how different patterns of language exposure affect the formation of shared representations.

How This Paper Positions Itself

The paper positions itself at the intersection of three research traditions: multilingual NLP (where shared representations are assumed to drive crosslingual transfer), computational psycholinguistics (which uses language models as testbeds for hypotheses about human language processing), and the "model organism" approach to cognitive science (where language models serve as manipulable analogs to human participants). The paper's core methodological contribution is bringing the controlled rearing paradigm to bear on the question of shared grammatical representations — training models from scratch with systematically varied language exposure conditions, then using structural priming (a behavioral probe that doesn't require access to internal representations) to measure representational sharing.

The paper explicitly draws on the model organism framing (Section 2, citing Müller, 2024 and Jain et al., 2024): just as fruit flies allow geneticists to perform manipulations that would be impossible in humans, small controlled language models allow linguists and cognitive scientists to test hypotheses about bilingual representation that are confounded or impossible in human experiments. The authors don't claim that language models are humans or that they learn language the same way — rather, they argue that LMs are sufficiently language-like to serve as informative model systems where causal manipulations can refine hypotheses that can then be tested (or re-interpreted) in human studies.

Crucially, the paper doesn't propose a new probing method, a new training objective, or a new model architecture. Instead, it takes an established behavioral paradigm from psycholinguistics (structural priming), applies it to a carefully constructed set of controlled models, and uses the resulting data to reinterpret conflicting findings in the human literature while simultaneously characterizing the boundary conditions on shared representation formation in language models. The dual contribution — informing both our understanding of language models and our interpretation of human experiments — is what distinguishes this work from a standard model probing study.

The paper also positions its findings as having practical implications for multilingual model development. The observation that language similarity constrains the formation of shared representations (Section 5.2) aligns with work showing that continued pretraining from a model in a very different language is less effective than training on a model in a more similar language (Ogueji et al., 2021). If shared grammatical representations only form robustly when languages share sufficient structural overlap, then the common practice of starting with an English-heavy model and doing vocabulary adaptation + continued pretraining for a typologically distant target language may be fundamentally limited — a point the paper raises directly in the Discussion.

3. Technical Approach

3.1 Reader Orientation

This paper constructs a controlled experimental framework for studying how bilingual grammatical representations form: the authors train small (124M-parameter) GPT-2 Transformer language models from scratch under systematically varied bilingual exposure conditions — manipulating which languages are paired, which is learned first, and whether the second language is introduced gradually alongside the first or replaces it entirely — then probe these models at every training checkpoint using crosslingual structural priming, a behavioral test borrowed from psycholinguistics that measures whether a grammatical structure in one language (the prime) makes the same structure more probable in another language (the target), without requiring any access to the model's internal representations.

The problem being solved is a causal inference challenge that cannot be addressed with either human participants or large pretrained models: is the asymmetry in crosslingual structural priming (stronger when English is the target language) caused by the order of language acquisition (L1 vs. L2 status) or by inherent properties of English as a target language? The "shape" of the solution is a factorial experimental design — train models that differ only in the manipulated variables (language pair, exposure order, bilingual condition) while holding all other factors (model architecture, total training tokens, data source, tokenizer construction) constant, then compare priming effects across these models to isolate the causal contribution of each factor.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental pipeline has four major stages, each producing artefacts that feed into the next:

1. Data preparation and tokenizer training — For each language in each language pair, the first 128M lines of the deduplicated OSCAR corpus are extracted. Separate SentencePiece tokenizers are trained for each model using the same language proportions as that model's planned training data, ensuring the tokenizer's subword inventory reflects the actual distribution the model will see.

2. Controlled model pretraining — 16 distinct 124M-parameter GPT-2 Transformer models are trained from scratch for 128,000 steps (2B tokens per language in the training set, plus 1M tokens per language for evaluation). The models differ along three manipulated dimensions: language pair (English-Dutch, English-Spanish, English-Polish, English-Greek), language exposure order (which language is L1 vs. L2), and bilingual condition (simultaneous — L1 only for first half, then equal mix of L1 and L2 for second half; or sequential — L1 only for first half, then exclusively L2 for second half). Checkpoints are saved at high temporal resolution, especially just after the introduction of L2 at step 64,000.

3. Structural priming evaluation — At each checkpoint, for each model, the authors measure crosslingual structural priming using experimental stimuli from five published human studies covering three grammatical alternations (dative alternation, genitive alternation, active/passive alternation) across the four language pairs. The measure is the difference in normalized target sentence probability between match and mismatch prime conditions, assessed via linear mixed-effects models with false discovery rate correction across multiple comparisons.

4. Grammatical competence measurement — In parallel, at each checkpoint, the authors evaluate the model's English grammatical knowledge using BLiMP (Benchmark of Linguistic Minimal Pairs), which measures whether the model assigns higher probability to grammatical sentences than to minimally different ungrammatical ones. This provides an independent measure of L2 grammatical acquisition that can be temporally aligned with the emergence of structural priming effects.

Information flows from stage 1 (data) → stage 2 (training, producing checkpoints) → stage 3 (priming measurement at each checkpoint) and stage 4 (BLiMP measurement at each checkpoint), with stage 3 and stage 4 operating on the same checkpoints in parallel. The key analytical move is comparing the time course of priming effects (stage 3) with the time course of grammatical competence (stage 4) within each model, and comparing priming patterns across models that differ only in the manipulated variables.

3.3 Roadmap for the Deep Dive

• First, the model architecture and training configuration — the GPT-2 Transformer specification, hyperparameters, optimizer settings, and the rationale for training at this scale rather than using larger models. This establishes what is held constant across all 16 models.

• Second, the data pipeline and tokenizer construction — how OSCAR data is extracted, sequenced, and split; how tokenizers are trained per-model using language proportions matched to training data; and the critical detail of how batches are constructed to alternate L1 and L2 sequences in the simultaneous condition. This explains how the "controlled rearing" is implemented at the data level.

• Third, the bilingual exposure conditions — the precise definitions of the simultaneous and sequential conditions, the training schedules, the token counts and proportions, and the rationale for choosing these specific regimes rather than other possible bilingual training paradigms. This is the core experimental manipulation.

• Fourth, the checkpointing strategy — the schedule of when checkpoints are saved, why the temporal resolution is increased just after L2 introduction, and how this enables the training dynamics analysis that distinguishes this work from prior static evaluations.

• Fifth, the structural priming calculation — how normalized target probabilities are computed from raw model log-probabilities, how linear mixed-effects models are fit to test for priming effects, how multiple comparisons are corrected, and why normalized probabilities (rather than raw probabilities or log-probabilities) are the appropriate dependent measure. This includes the handling of the experimental stimuli and the augmentation procedure for low-N datasets.

• Sixth, the BLiMP evaluation — how grammatical competence is measured, why BLiMP is chosen over other syntactic evaluation benchmarks, and how its time course is aligned with the priming time course for each model.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a controlled experimental design paper whose core contribution is a factorial manipulation of bilingual training conditions, combined with behavioral probing via structural priming, to disambiguate causal factors that are confounded in both human experiments and prior model-based work.

Model Architecture and Training Configuration

All 16 models share an identical architecture: an autoregressive GPT-2 Transformer language model with 124 million parameters (Radford et al., 2018, 2019). The specific hyperparameters, as reported in Appendix C (Table C.2), are:

• 12 layers (transformer blocks), each with 768-dimensional embeddings and 768-dimensional hidden states.
• 12 attention heads, each with 64-dimensional attention head size (standard for GPT-2 small: 12 × 64 = 768).
• Intermediate feed-forward hidden size of 3072 (4× the embedding dimension, following the standard Transformer expansion ratio).
• GELU activation function (Gaussian Error Linear Unit), the standard activation for GPT-2.
• Vocabulary size of 50,004 tokens (a standard SentencePiece vocabulary size, accommodating multi-language subword units).
• Maximum sequence length of 128 tokens — deliberately short, following the BabyLM challenge paradigm (Warstadt et al., 2023) where models are trained on human-scaled data quantities and learn robust syntactic generalizations from relatively short contexts.
• Absolute position embeddings (the standard learned positional embedding of the original GPT-2, not rotary or relative position embeddings).
• Batch size of 128 sequences.
• Total training steps: 128,000.
• Learning rate schedule: linear decay from an initial learning rate of $1 \times 10^{-4}$, with 10,000 warmup steps (linear increase from 0 to the initial learning rate over the first 10,000 steps).
• Adam optimizer with $\beta_1 = 0.9$, $\beta_2 = 0.999$, and $\epsilon = 1 \times 10^{-6}$.
• Dropout of 0.1 (applied to attention weights and feed-forward hidden states; attention dropout also set to 0.1).

The choice of 124M parameters is explicitly motivated by compute constraints (see Limitations in Section 6), but the authors also argue it is methodologically appropriate. Small models trained on human-scaled data quantities (around 100M words) have been shown to learn robust syntactic generalizations (Warstadt et al., 2023), and syntax is learned very early in training (Blevins et al., 2022; Chang et al., 2024b). Training at this scale makes it feasible to train 16 separate models, each with dense checkpointing, on the authors' available hardware (one NVIDIA RTX A6000, approximately 512 GPU hours total). Larger models would have reduced the number of experimental conditions the authors could explore given their fixed compute budget, sacrificing the factorial design that is essential for causal inference.

The training code is based on that from Chang and Bergen (2022), which itself implements a standard autoregressive language modeling objective: the model is trained to predict the next token given all previous tokens in the sequence, minimizing the cross-entropy between the predicted token distribution and the ground-truth next token. Formally, for a sequence of tokens $x_1, x_2, ..., x_T$, the loss at each position $t$ is:

$\mathcal{L}_t = -\log P_\theta(x_t \mid x_1, ..., x_{t-1})$

where $P_\theta$ is the model's predicted probability distribution over the vocabulary given the prefix $x_1, ..., x_{t-1}$, and $x_t$ is the actual token at position $t$.

What it computes: the negative log-probability the model assigns to the correct next token. A low value means the model assigns high probability to the correct token; a high value means it assigns low probability. The total loss for a sequence is the average of these position-wise losses (or equivalently, the sum, depending on implementation), and the training objective is to minimize this average across all sequences in the training set.

Why this form: this is the standard maximum-likelihood objective for autoregressive language modeling. It is the only objective that directly optimizes the model to generate text matching the training distribution, and it has well-understood theoretical properties (minimizing cross-entropy is equivalent to minimizing KL divergence between the true data distribution and the model's distribution). No alternative objective (e.g., masked language modeling, contrastive learning) is used because the structural priming evaluation requires the model to compute conditional probabilities of entire target sentences given prime sentences, which is straightforward with an autoregressive model but would require additional machinery with a bidirectional encoder.

The total training data budget is fixed across models: 2 billion tokens per language in the training set, plus 1 million tokens per language for evaluation. Because models differ in the proportions of L1 and L2 data they see (see the bilingual condition descriptions below), the total training tokens vary slightly across conditions, but the per-language token counts are matched. This ensures that differences in priming effects cannot be attributed to one language simply having less training data than another — a critical control that was impossible with the large pretrained models studied by Michaelov et al. (2023).

Data Pipeline and Tokenizer Construction

OSCAR corpus extraction. For each of the five languages (English, Dutch, Spanish, Polish, Greek), the authors take the first 128 million lines of the deduplicated OSCAR corpus (Abadji et al., 2021). OSCAR is a multilingual web corpus derived from Common Crawl, processed through language identification, deduplication, and filtering. The choice of the "first 128M lines" (rather than random sampling) is a practical convenience — it ensures deterministic reproducibility and avoids the complexity of shuffling a multi-terabyte corpus — but it does mean that the data reflects whatever distributional properties happen to appear early in the OSCAR ordering, which could introduce uncontrolled variation if, for example, the first 128M lines of Dutch OSCAR are systematically different in domain or register from the first 128M lines of Greek OSCAR. The authors do not characterize these potential distributional differences.

The 128M-line threshold is chosen to provide approximately 2B tokens of training data per language after tokenization and sequence construction. Following Chang et al. (2024b), who also used this threshold for studying syntactic learning dynamics, this quantity is sufficient for 124M-parameter models to learn robust syntax while remaining within the authors' compute budget.

Sequence construction. Sequences are constructed as contiguous spans of 128 tokens from the extracted text. Each sequence is monolingual — no single sequence contains tokens from multiple languages. However, in the simultaneous bilingual condition (described below), batches alternate between L1 and L2 sequences: a batch of 128 sequences contains 64 L1 sequences and 64 L2 sequences, interleaved. This ensures the model receives gradient updates from both languages in each training step, which is important for preventing the model from "forgetting" one language between steps (since the optimizer state would otherwise adapt exclusively to whichever language appears in the current batch).

The sequences are shuffled before training. Shuffling at the sequence level (rather than, say, shuffling documents or paragraphs) means that adjacent sequences in the same batch have no necessary topical or narrative relationship — they are independent spans from different parts of the corpus. This is standard practice in language model pretraining and reduces the risk of the model learning spurious cross-sequence dependencies.

Per-model tokenizer training. A distinct SentencePiece tokenizer (Kudo and Richardson, 2018) is trained for each of the 16 models, using the unigram language model subword segmentation algorithm (the default for SentencePiece). The key design choice is that the tokenizer training data proportions match the model's training data proportions. For a simultaneous bilingual model with Dutch as L1 and English as L2, the training data is 75% Dutch and 25% English (see bilingual conditions below); the tokenizer for that model is therefore trained on a 75/25 mix of Dutch and English text. This ensures that the subword vocabulary appropriately represents the morphological and orthographic properties of both languages in proportion to their prevalence in training. If, instead, a single tokenizer were used for all models (e.g., a multilingual tokenizer trained on equal proportions of all five languages), models that see predominantly one language would have many "wasted" vocabulary entries for subwords from languages they rarely encounter, and would need to use more tokens to represent common words in their training languages.

The vocabulary size is fixed at 50,004 tokens for all tokenizers. This is a standard SentencePiece size (50,000 plus 4 special tokens: <pad>, <s> for sequence start, </s> for sequence end, and <unk> for unknown tokens). The vocabulary includes both language-specific subwords (e.g., Dutch morphological suffixes, Greek characters) and subwords shared across languages (e.g., numbers, punctuation, Latin-script words that appear in multiple languages due to borrowing or code-switching). The extent of vocabulary overlap between languages depends on orthographic similarity: English and Dutch (both using Latin script with substantial Germanic cognate vocabulary) will share many subword tokens, while English and Greek (using different scripts) will share essentially no subword tokens beyond numbers and punctuation. This is one mechanism through which language similarity may affect shared representation formation — languages with more token overlap have more opportunities for the model to observe analogous surface patterns and generalize grammatical abstractions.

Bilingual Exposure Conditions

This is the core experimental manipulation. The authors define two bilingual training conditions — simultaneous and sequential — that operationalize different patterns of bilingual language experience. Both conditions follow the same two-phase training schedule (128,000 total steps, with the transition from phase 1 to phase 2 occurring at step 64,000), but they differ in what language data the model sees during the second phase.

Simultaneous bilingual condition. This condition is designed to approximate the language experience of a simultaneous bilingual: someone who acquires one language first, then begins learning a second language while continuing to use and be exposed to the first. The training schedule is:

• Steps 0–63,999 (first half of training): the model is trained exclusively on L1 data. It sees only monolingual L1 sequences, learning the vocabulary, syntax, and semantics of the first language.
• Steps 64,000–128,000 (second half of training): the model is trained on an equal mix of L1 and L2 data. Specifically, each batch contains 64 L1 sequences and 64 L2 sequences, interleaved. The L2 is introduced suddenly at step 64,000, but the L1 does not disappear — the model continues to receive reinforcement of its first language throughout the second half of training.

The overall training data proportions in the simultaneous condition are therefore 75% L1, 25% L2 (because L1 is present for all 128,000 steps, while L2 is present only for the last 64,000 steps, and during those steps the batch is equally split). The total tokens per language are 2B for both L1 and L2 (since each gets 2B tokens in the training set, and the model uses all 2B of L1 but sees L1 data only when it appears in the batch — actually, let me recalculate: the training set contains 2B tokens per language. In the simultaneous condition, during steps 0–63,999, the model trains on L1 sequences drawn from the 2B L1 training tokens. During steps 64,000–128,000, each batch is 50% L1 and 50% L2, so the model consumes approximately 1B L1 tokens and 1B L2 tokens in this phase. Total L1 consumption: approximately 3B tokens (2B in phase 1 + 1B in phase 2). Total L2 consumption: approximately 1B tokens. This means the model sees L1 and L2 in a roughly 3:1 ratio overall, but L2 is introduced only after L1 is well-established.

Sequential bilingual condition. This condition is designed to approximate the language experience of a sequential bilingual: someone who acquires an L1 to fluency, then at some point moves to an environment where they are immersed exclusively in an L2 and cease using their L1 regularly. The training schedule is:

• Steps 0–63,999 (first half of training): identical to the simultaneous condition — the model is trained exclusively on L1 data.
• Steps 64,000–128,000 (second half of training): the model is trained exclusively on L2 data. All L1 data is removed from training. Every sequence in every batch is monolingual L2.

The overall training data proportions in the sequential condition are 50% L1, 50% L2 (each language is present for exactly half of the training steps). The total tokens consumed are 2B for L1 (all during phase 1) and 2B for L2 (all during phase 2). This is an extreme version of sequential bilingualism — a complete and abrupt switch — which is intended to induce catastrophic forgetting of L1, a well-documented phenomenon in neural networks where training on a new distribution causes rapid degradation of performance on the previous distribution (McCloskey and Cohen, 1989). The authors explicitly expect this to happen and treat it as a feature, not a bug: by observing whether structural priming effects survive catastrophic forgetting, they can test whether shared grammatical representations are robust to the loss of surface-level L1 knowledge.

Why these specific conditions? The simultaneous condition operationalizes the most common form of bilingualism in the human populations studied in structural priming experiments — individuals who learned an L1 from birth, then acquired an L2 (typically English) later but continue to use both languages regularly. The sequential condition, while extreme, serves as a stress test: if structural priming effects are driven by surface-level statistical associations between lexical items across languages, they should disappear when the model forgets the L1 lexical items. If they are driven by genuinely abstract, shared grammatical representations that are encoded in parameters that are not overwritten by L2 training, they should persist despite catastrophic forgetting. The comparison between the two conditions therefore helps distinguish between representational and surface-level accounts of structural priming.

Why not other possible bilingual conditions? The authors acknowledge other paradigms exist — for example, Constantinescu et al. (2025) included an "interleaved" condition where L1 and L2 are mixed from the start of training, and Aoyama and Schneider (2024) froze some model parameters after L1 training before L2 exposure. The choice of these two specific conditions is driven by the research questions: the simultaneous condition establishes whether priming effects require L2 exposure (by showing they are absent before L2 is introduced) and whether they depend on continued L1 reinforcement; the sequential condition tests whether they survive L1 forgetting. Other conditions (e.g., fully interleaved from the start, or L2-only from the start) would address different questions (e.g., whether priming requires L1 to be well-established first, or whether priming can emerge from simultaneous bilingual exposure) and are left to future work.

Multi-Factor Experimental Design

The full experimental design crosses three factors, producing 16 models:

Factor 1: Language pair (4 levels). The four language pairs are English-Dutch, English-Spanish, English-Polish, and English-Greek. These are chosen to span a range of typological similarity to English:

• English-Dutch: both West Germanic languages, sharing substantial vocabulary, similar word order patterns (SVO, similar dative and genitive alternations), and the same Latin-based writing system. Dutch is the most similar to English of the four non-English languages.
• English-Spanish: Spanish is an Italic (Romance) language, while English is Germanic, but both are Indo-European, share substantial Latinate vocabulary (via Norman French in English), use the Latin script, and have broadly similar word order patterns (SVO, similar active/passive alternations). Spanish is intermediate in similarity.
• English-Polish: Polish is a Slavic language, still Indo-European but from a different branch than Germanic or Romance. It uses the Latin script but has rich morphological case marking that encodes grammatical relations (subject/object) that English expresses through word order, and has more flexible word order. Polish is typologically more distant.
• English-Greek: Greek is an Indo-European language but forms its own branch (Hellenic), uses a non-Latin writing system (the Greek alphabet), and has rich verbal morphology (including synthetic passive voice marking via verb inflection, unlike the periphrastic passive of English, Dutch, Spanish, and Polish). Greek is the most distant from English in this set, with essentially no vocabulary overlap (different script) and substantial typological differences in how grammatical alternations are encoded.

The choice of these four pairs is constrained by the availability of psycholinguistic stimuli for crosslingual structural priming experiments. The stimuli for each pair come from published human studies (see §4.2 of the paper), and the authors are limited to language pairs where such stimuli exist. This is acknowledged as a limitation (§6, "Language Sample"): all languages are Indo-European, and a more diverse sample (including non-Indo-European languages) would be needed for fully general conclusions.

Factor 2: Language exposure order (2 levels per pair). For each language pair, two models are trained: one where English is L1 and the non-English language is L2 (e.g., English-Dutch, meaning English L1, Dutch L2), and one where the order is reversed (e.g., Dutch-English, meaning Dutch L1, English L2). This factor is the critical manipulation for disambiguating the L1/L2 confound: if priming asymmetries are due to L1/L2 status, then reversing the order should reverse the asymmetry (Dutch L1 → English L2 should show stronger priming than English L1 → Dutch L2, for example). If the asymmetry persists regardless of order (always stronger when English is the target), then the asymmetry is about English, not about acquisition order.

Factor 3: Bilingual condition (2 levels). As described above: simultaneous vs. sequential. This factor tests whether shared representations require continued exposure to both languages (simultaneous) or can survive even when L1 is completely removed from training (sequential).

The complete grid: 4 language pairs × 2 exposure orders × 2 bilingual conditions = 16 models.

Checkpointing Strategy

The authors save model checkpoints at 45 specific training steps, with the temporal density varying across training to provide high resolution during periods of rapid change:

• Steps 0, 10000, 20000, 30000, 40000, 50000, 60000: checkpoints every 10,000 steps during phase 1 (L1-only training). This coarse sampling is sufficient because learning during this phase is gradual and monotonic (see the loss curves in Figure 2, where L1 surprisal decreases smoothly).
• Step 64,000: the checkpoint immediately before L2 introduction. This serves as the baseline for measuring the onset of any L2-related effects.
• Steps 64010, 64020, 64030, 64040, 64050, 64060, 64070, 64080, 64090, 64100, 64110, 64120, 64130, 64140, 64150, 64160, 64170, 64180, 64190, 64200: checkpoints every 10 steps for the first 200 steps after L2 introduction. This extremely dense sampling (19 checkpoints in the first 200 steps) is motivated by the expectation that structural priming effects may emerge very rapidly after L2 exposure begins — within the first few hundred training steps, corresponding to only a few million L2 tokens. The authors want to capture the precise moment when priming first appears.
• Steps 64300, 64400, 64500, 64600, 64700, 64800, 64900, 65000, 66000, 67000, 68000, 69000, 70000, 80000, 90000, 100000, 110000, 120000, 128000: gradually increasing intervals after the initial burst, returning to coarse sampling (every 10,000 steps) for the remainder of training.

This checkpointing scheme is a deliberate tradeoff: the authors want to minimize storage (45 checkpoints × 16 models × ~500MB per checkpoint ≈ 360GB of model files) while maximizing temporal resolution during the period of interest (immediately after L2 introduction). The dense sampling near step 64,000 is what enables the training dynamics analysis that distinguishes this work from prior static checkpoint evaluations — without it, the authors could not observe that priming emerges as early as 600 steps after L2 exposure (4.9M tokens) and stabilizes shortly thereafter.

Structural Priming Calculation

The core behavioral probe is crosslingual structural priming, adapted from the human psycholinguistics paradigm. The measurement procedure has several stages:

Prime-target pairs. For each grammatical alternation in each language pair, the experimental stimuli consist of sets of sentences that instantiate the two alternative constructions. For the dative alternation (English-Dutch, stimuli from Schoonbaert et al., 2007), each item is a semantically matched pair: a Prepositional Object (PO) sentence like "The cook shows a hat to the boxer" and a Double Object (DO) sentence like "The cook shows the boxer a hat." These sentences differ in syntactic structure (word order, presence/absence of the preposition "to") while expressing the same event. The Dutch equivalents follow the same structural pattern: "De kok toont een hoed aan de bokser" (PO) vs. "De kok toont de bokser een hoed" (DO).

For each experiment, the authors construct all possible prime-target pairs: for each target sentence (say, an English PO target), there are matching primes (an English PO prime, or a Dutch PO prime, depending on the priming direction) and mismatching primes (an English DO prime, or a Dutch DO prime). This produces a factorial design: prime language × prime construction × target language × target construction. The key comparison is between the match condition (prime and target share the same construction, e.g., Dutch PO → English PO) and the mismatch condition (prime and target have different constructions, e.g., Dutch DO → English PO).

Stimulus augmentation for statistical power. The original Spanish (Hartsuiker et al., 2004), Greek (Kotzochampou and Chondrogianni, 2022), and Polish (Fleischer et al., 2012) experiments have substantially fewer stimuli pairs than the Dutch experiments. To match statistical power across language pairs, the authors create new prime-target pairs by combining every possible prime sentence with every possible target sentence (the Cartesian product of the prime set and the target set), then randomly sample 144 pairs. This matches the number of pairs in the Dutch materials, ensuring that the statistical tests for different language pairs have comparable sensitivity. The random sampling introduces some stochasticity, but the authors do not report confidence intervals across different random samples.

Normalized probability computation. For each prime-target pair, the structural priming effect is measured as the normalized probability of the target sentence given the prime. The normalized probability is needed because the two alternative constructions (e.g., PO and DO) may have very different baseline probabilities — in English, PO datives are generally more frequent than DO datives, so a raw probability comparison would be confounded by baseline frequency differences. Normalization removes this confound.

For a target construction $T_{\text{PO}}$ (a PO target sentence) and a prime construction $P$ (which could be PO or DO), the normalized probability is:

$P_N(T_{\text{PO}} \mid P) = \frac{P(T_{\text{PO}} \mid P)}{P(T_{\text{PO}} \mid P) + P(T_{\text{DO}} \mid P)}$

where $P(T_{\text{PO}} \mid P)$ is the model's raw probability of the PO target sentence given the prime sentence, and $P(T_{\text{DO}} \mid P)$ is the model's raw probability of the DO target sentence given the same prime.

What it computes: the relative probability of the PO target compared to both possible targets (PO and DO), conditional on a specific prime. If the prime has no effect, $P(T_{\text{PO}} \mid P_{\text{PO}})$ and $P(T_{\text{PO}} \mid P_{\text{DO}})$ should both be close to the baseline PO probability, so $P_N(T_{\text{PO}} \mid P_{\text{PO}}) \approx P_N(T_{\text{PO}} \mid P_{\text{DO}})$. If priming exists, $P_N(T_{\text{PO}} \mid P_{\text{PO}}) > P_N(T_{\text{PO}} \mid P_{\text{DO}})$ — a matching prime makes the PO target relatively more probable than a mismatching prime does.

Why this form: the normalization converts raw probabilities into a relative measure that is bounded between 0 and 1 and is invariant to the overall frequency of the target construction. Without normalization, a model might show higher raw probability for a PO target after a PO prime simply because PO constructions are more common in general, not because the prime specifically facilitated PO structure. By dividing by the sum of probabilities for both alternative targets, the normalized measure isolates the relative preference for one construction over the other, which is what structural priming theory predicts should be affected by the prime. Alternative normalizations (e.g., subtracting log-probabilities, or using a softmax over multiple target constructions) would also remove baseline frequency effects, but the ratio form used here is standard in the structural priming literature (both human and computational) and has the advantage of being directly interpretable as a probability.

Raw probability from the language model. The model's raw probability of a target sentence given a prime sentence is computed using the standard autoregressive factorization. For a prime sequence $x_1, ..., x_M$ followed by a target sequence $y_1, ..., y_N$:

$P(T \mid P) = \prod_{t=1}^{N} P_\theta(y_t \mid x_1, ..., x_M, y_1, ..., y_{t-1})$

where $P_\theta(y_t \mid \text{context})$ is the model's predicted probability for token $y_t$ given the full preceding context (prime tokens plus already-generated target tokens). In practice, this is computed by feeding the concatenated prime + target sequence through the model and extracting the predicted probabilities at each target token position.

What it computes: the joint probability of the target sequence conditional on the prime, which is the product of the conditional probabilities of each target token given all preceding tokens (both prime and earlier target tokens). This is the standard definition of sequence probability in autoregressive models.

Why this form: the autoregressive factorization is the natural way to compute sequence probabilities in a causal (left-to-right) language model. An alternative would be to compute pseudo-log-likelihood (masking individual tokens and predicting them from bidirectional context), but this would require a different model architecture and is less straightforward. The product form is exactly the quantity the model is trained to maximize (minimize negative log-likelihood of), so it is the most directly grounded probability measure available.

Linear mixed-effects modeling. For each model checkpoint, language pair, and priming direction (L1→L2, L2→L1), the authors fit a linear mixed-effects model predicting the normalized probability of the target. The model specification (in R/lme4 notation) is:

normalized_probability ~ prime_type + (1 | item)

where:

• normalized_probability is the dependent variable ($P_N$ as computed above),
• prime_type is a fixed effect with two levels (match vs. mismatch), and
• (1 | item) is a random intercept for each experimental item (i.e., each distinct prime-target pair identity, accounting for the fact that some items may have systematically higher or lower baseline probabilities independent of priming).

The fixed effect of prime_type is the parameter of interest: a positive coefficient for the match condition (relative to mismatch as the reference level) indicates a structural priming effect — targets following matching primes have higher normalized probability than targets following mismatching primes.

Why linear mixed-effects models? This is the same statistical approach used in human structural priming experiments (e.g., Schoonbaert et al., 2007; Bernolet et al., 2013), and using it enables direct comparison with the human literature. Mixed-effects models account for both fixed effects (the experimental manipulation) and random effects (item-level variability that is not of theoretical interest but could inflate Type I error if ignored). Alternative approaches — simple t-tests on aggregated means, or fixed-effects-only regression — would either lose statistical power (by ignoring item-level structure) or risk anti-conservative inference (by treating observations as independent when they are clustered within items). The mixed-effects framework is the field-standard approach for this type of data.

Multiple comparison correction. Because the authors fit many linear mixed-effects models (one per checkpoint per model per priming direction, with up to 45 checkpoints × 16 models × 2 priming directions = potentially 1440 models, though in practice they report a subset), they correct for multiple comparisons by controlling the false discovery rate (FDR) using the Benjamini-Hochberg procedure (Benjamini and Hochberg, 1995). FDR control is less conservative than family-wise error rate control (e.g., Bonferroni), which is appropriate here because the goal is exploratory and hypothesis-generating rather than confirmatory — the authors want to identify patterns without being so stringent that they miss genuine effects. The FDR threshold is not explicitly stated in the paper but is presumably the standard 0.05.

BLiMP Evaluation for Grammatical Competence

To temporally align structural priming effects with the acquisition of grammatical knowledge in the second language, the authors evaluate each model checkpoint on BLiMP (Benchmark of Linguistic Minimal Pairs; Warstadt et al., 2020). BLiMP is a targeted syntactic evaluation benchmark consisting of 67 individual datasets, each testing a specific grammatical phenomenon in English. Each dataset contains 1,000 minimal pairs: a grammatical sentence and a minimally different ungrammatical sentence that differ only in the grammatical feature being tested. For example, a minimal pair testing subject-verb agreement might contrast "The cat chases the mouse" (grammatical) with "The cat chase the mouse" (ungrammatical: the singular subject "cat" requires "chases," not "chase").

The model's performance on each minimal pair is scored as 1 if it assigns higher probability to the grammatical sentence than to the ungrammatical sentence, and 0 otherwise. The overall BLiMP score for a checkpoint is the average across all 67 datasets (equivalently, across all 67,000 minimal pairs). A score of 1.0 would indicate perfect grammatical knowledge (always preferring the grammatical variant), 0.5 indicates chance-level performance, and below 0.5 indicates a systematic preference for ungrammatical forms.

Why BLiMP? Three reasons: (1) it is a well-validated, widely-used benchmark for measuring syntactic knowledge in language models, with clear psycholinguistic motivation (the minimal pair design directly tests whether the model has acquired the abstract grammatical rule, not just statistical frequency patterns); (2) it exists only for English, which is the L2 for all models where the authors track grammatical competence (they focus primarily on models where English is the L2, since the research question concerns L2 acquisition); and (3) it is lightweight enough to evaluate at every checkpoint (67,000 judgments per checkpoint) within reasonable compute time. A more comprehensive syntactic evaluation (e.g., SyntaxGym, full parsing evaluations) would be too expensive to run at 45 checkpoints × all relevant models.

Limitation acknowledged by the authors (Section 4.4, footnote 3): BLiMP does not exist for all the other languages in the sample (Dutch, Spanish, Polish, Greek). Therefore, the analysis of the temporal link between priming and grammatical competence is restricted to English as L2, and the authors cannot make the symmetric claim for non-English languages as L2 targets. This is a practical limitation of available evaluation resources, not a conceptual limitation of the approach — if BLiMP equivalents existed for Dutch, Spanish, Polish, and Greek, the same analysis could be performed.

The evaluation is performed using the LM Evaluation Harness (Biderman et al., 2024), a standard framework for reproducible LM evaluation. The harness handles the mechanics of feeding minimal pairs to the model, extracting log-probabilities, and computing accuracy scores.

Alignment with priming time course. For each model checkpoint, the authors have two measurements: the structural priming effect size (the difference in normalized probability between match and mismatch conditions, extracted from the linear mixed-effects model) and the BLiMP accuracy. By plotting these two quantities against training step on the same axes (as in Figure 4 and Figures E.8–E.12 in the Appendix), the authors can visually assess whether priming effects emerge concurrently with or subsequent to the acquisition of grammatical competence. If priming appears before BLiMP scores rise above chance, this would suggest that priming is not actually driven by grammatical knowledge — it might be a surface-level statistical artifact. If priming appears at the same time as or after BLiMP scores increase, this strengthens the interpretation that priming reflects genuinely abstract grammatical representations that are learned through L2 exposure.

Contamination Analysis

A potential confound in any language model training study is data contamination: if the experimental stimuli (the prime and target sentences used to measure structural priming) appear verbatim in the model's training data, then the model might exhibit apparent "priming" simply because it has memorized the specific sentence pairs, not because it has acquired abstract grammatical representations that generalize.

The authors address this by checking whether their stimuli appear in web-scale text corpora. Since the original OSCAR corpus indexed by the "What's in My Big Data?" tool was inaccessible to them, they use Infini-gram (Liu et al., 2024), which indexes C4 (Colossal Clean Crawled Corpus) — a different but related multilingual web corpus that is much larger than the portion of OSCAR used for training. If stimuli appear in C4, they likely also appear in OSCAR; if they do not appear in C4, this provides some evidence (though not proof) that they are not in training data.

The results (Appendix B, Table B.1) show that contamination is minimal:

• Schoonbaert et al. (2007) stimuli (English-Dutch datives): 0 contaminated items (0%).
• Bernolet et al. (2013) stimuli (English-Dutch genitives): 0 contaminated items (0%).
• Hartsuiker et al. (2004) stimuli (English-Spanish active/passive): 3 contaminated items out of approximately 384 items (0.78%).
• Fleischer et al. (2012) stimuli (English-Polish active/passive): 0 contaminated items (0%).
• Kotzochampou and Chondrogianni (2022) stimuli (English-Greek active/passive): 4 contaminated items out of approximately 192 items (2.08%).

The contamination rates are extremely low (at most ~2% of items, and 0% for the two Dutch experiments that form the core of the training dynamics analysis). The authors do not remove contaminated items from their analyses, presumably because the rates are so low that removal would have negligible impact on the statistical results. However, they also do not report sensitivity analyses (e.g., checking whether results change when contaminated items are excluded), which would strengthen the contamination argument.

Why check contamination at all? In structural priming experiments with language models, the concern is not just that the model might "know" the sentences (which would be true even without contamination, since the model learns the statistical properties of the language). The concern is specifically that the model might have memorized the exact prime-target pairings from training data. If the training data contained, for example, a passage where a Dutch PO dative sentence was immediately followed by an English PO dative sentence, the model could learn this specific sequential pattern and exhibit apparent priming without having formed any abstract grammatical representation. The near-zero contamination rates suggest this is unlikely to be driving the observed effects.

Summary of Design Choices and Their Justifications

• Small models (124M parameters) trained from scratch over large pretrained models: enables complete control over training data composition, language exposure order, and bilingual condition; enables dense checkpointing and within-training dynamics analysis; makes the full factorial design (16 models) computationally feasible.
• Per-model tokenizers with training-data-matched proportions over a single shared multilingual tokenizer: ensures the subword vocabulary is optimized for the specific language distribution each model sees, avoiding wasted vocabulary capacity and ensuring fair comparison across language pairs with different orthographic overlap patterns.
• OSCAR corpus (first 128M lines per language) over other multilingual corpora: provides deduplicated, language-identified web text across all five target languages; the "first 128M lines" choice is driven by reproducibility and practicality, though it introduces potential ordering confounds.
• Two bilingual conditions (simultaneous and sequential) over a single training regime: enables testing both whether priming requires continued L1 exposure (by comparing conditions) and whether priming survives catastrophic forgetting (via the sequential condition specifically).
• Dense checkpointing near L2 introduction over uniform checkpointing: enables the training dynamics analysis that distinguishes this work from prior static evaluations; captures the rapid emergence of priming effects that would be missed with coarser sampling.
• Structural priming with normalized probabilities over raw probability comparisons: eliminates baseline frequency confounds; matches the analysis approach used in human psycholinguistics, enabling direct comparison.
• Linear mixed-effects models with FDR correction over simpler statistical tests: accounts for item-level random effects; controls for multiple comparisons without being overly conservative.
• BLiMP for grammatical competence measurement over other syntactic benchmarks: lightweight enough for per-checkpoint evaluation; psycholinguistically motivated minimal-pair design that directly tests abstract grammatical knowledge.
• Infini-gram / C4 contamination check (acknowledged as imperfect proxy for OSCAR): provides evidence that memorization of specific prime-target pairs is unlikely to explain observed priming effects, though not definitive proof since C4 and OSCAR are different corpora.

4. Key Insights and Innovations

Innovation 1: Language Models as Causal Testbeds That Disambiguate Confounded Human Experiments

The paper's most intellectually distinctive contribution is not the priming results themselves but rather the methodological argument that controlled language model training can resolve confounds that are structurally impossible to address with human participants. This represents a fundamental shift in how language models relate to psycholinguistics — not merely as objects of study (do LMs exhibit human-like behavior?) or as engineering artifacts, but as experimental platforms where causal manipulations of language experience enable hypothesis testing that can reinterpret the human literature.

What was the prior assumption? The dominant framework in computational psycholinguistics has been to probe existing pretrained models for human-like behaviors and then draw inferences about whether models "learn like humans" (Prasad et al., 2019; Sinclair et al., 2022; Michaelov et al., 2023). This is intrinsically correlational: you observe that a model exhibits some effect, but you cannot manipulate its "language experience" to determine why the effect exists. In parallel, the human psycholinguistics literature on bilingual structural priming (Schoonbaert et al., 2007; Bernolet et al., 2013; Shin and Christianson, 2009, 2011) has been stuck with an irremediable confound: in nearly all experiments, English is the L2, making it impossible to determine whether observed asymmetries (stronger priming from L1 to L2) are due to the order of language acquisition or due to English specifically being the target language. No ethical experiment with human participants can randomize which language a child learns first and at what age — the very manipulation needed to answer the question.

What makes this innovation distinctive? The paper doesn't just train bilingual models — it uses them to perform a causal decomposition that would be impossible otherwise. By training models where English is the L1 and models where English is the L2, while holding all other factors constant (architecture, training data quantity, data source), the authors isolate the effect of language exposure order from the effect of target language identity. The result — that priming asymmetries persist even when English is the L1 (i.e., stronger priming when English is the target, regardless of L1/L2 status) — directly implies that the human psycholinguistics literature may have been misinterpreting its central finding for nearly two decades. The asymmetry is not about L1 vs. L2; it's about English as a target language, potentially because of English's relatively fixed word order making structural repetition more detectable than in morphologically rich languages like Polish, Greek, or Korean where grammatical relations are marked by case inflection rather than linear position.

This is not an incremental refinement. It reframes the role of language models in cognitive science from descriptive (models exhibit phenomenon X, which tells us something about models) to interventional (models allow us to perform experiments that inform our interpretation of human data). The "model organism" framing (Section 2) is not just rhetorical window-dressing — it is the intellectual engine of the paper. The authors don't claim LMs learn like humans; they claim LMs are useful for generating and refining hypotheses about humans that can then be tested (or re-analyzed) in human studies. This is a fundamentally different epistemic relationship than the typical "LMs as models of human cognition" paradigm.

Evidence anchor: The key result appears in Figures 3 and D.6–D.7, where structural priming effects are consistently more robust when English is the target language regardless of whether English is L1 or L2. In the simultaneous condition (Figure 3, top row), English as target produces significant priming effects for all language pairs (English-Dutch, English-Spanish, English-Polish, English-Greek), while the reverse direction produces significant effects only for English-Dutch and English-Greek (and numerical but non-significant effects for the others). This pattern holds even when the models are trained with English as L1 (so the priming direction is L1→L2 but with English as prime, not target), which is the condition that would produce strong priming if the L1→L2 hypothesis were correct. The fact that priming is weaker when English is the prime (L1→L2 priming with English as L1) than when English is the target (L2→L1 priming with English as L1) is the exact opposite of what the L1/L2 account predicts.

Significance beyond performance: This innovation doesn't produce a "better" model or a metric improvement. Its value is entirely in conceptual reframing and hypothesis generation. It provides the first empirical evidence from a causally controlled setting that the L1/L2 asymmetry in human structural priming may be a target-language effect, which should prompt re-examination of two decades of psycholinguistic findings and potentially new human experiments designed to control for target language identity (e.g., testing L1-English/L2-Korean speakers to see if the asymmetry reverses).

Innovation 2: Language Similarity as a Hard Constraint on Shared Representation Formation — Not a Soft Preference

A second fundamental contribution is the demonstration that language similarity is not merely a facilitator of crosslingual transfer but a boundary condition that determines whether shared abstract grammatical representations form at all, even when training data quantity is perfectly controlled. This moves the language similarity effect from a correlational observation (prior work showed transfer works better between similar languages; Lin et al., 2019; Ogueji et al., 2021) to a causal constraint that operates independently of data quantity, language exposure order, and bilingual training condition.

What was the prior assumption? The field has known for some time that crosslingual transfer is more effective between more similar languages, but the mechanism was ambiguous. It could be that transfer works better because similar languages share more vocabulary (increasing the number of tokens the model can reuse), or because similar languages tend to appear together in training data more often (creating more opportunities for the model to observe crosslingual patterns), or because multilingual models are typically trained on imbalanced data (with more data in higher-resource languages, which happen to be more similar to each other). Michaelov et al. (2023) observed weaker crosslingual structural priming for English-Polish and English-Greek than for English-Dutch and English-Spanish, but could not rule out the data-quantity confound — the models they studied were pretrained on undocumented data mixtures, and Polish and Greek may simply have been underrepresented. Without controlling for data quantity, the claim that language similarity causally constrains representation sharing was unsubstantiated.

What makes this innovation distinctive? The paper controls for data quantity directly: every language in every model receives exactly 2B tokens of training data, extracted from the same corpus (OSCAR) using the same procedure. Any remaining differences in priming effects must be driven by properties of the languages themselves. The finding — that priming effects are substantially more robust for Dutch-English and Spanish-English than for Polish-English and Greek-English, across both bilingual conditions and both exposure orders — is therefore a clean demonstration that linguistic similarity (typological and orthographic) is a causal constraint on the formation of shared representations. The model simply does not form strong shared grammatical abstractions between two languages if they are too structurally different, even when trained on equal amounts of data from both.

This is not an incremental contribution because it rules out the most plausible alternative explanation (data quantity) and establishes a boundary condition that directly constrains how we think about crosslingual transfer. The implication — stated explicitly in the Discussion — is that the common practice of taking a large English-pretrained model and doing continued pretraining for a typologically distant target language may be fundamentally limited. If shared grammatical representations only form when languages share sufficient structural overlap, then crosslingual transfer from English to, say, Japanese or Turkish may never achieve the level of representational integration that English-to-Dutch transfer achieves, regardless of how much continued pretraining data is used. This is a sharp, falsifiable prediction that goes beyond the standard "transfer works better for similar languages" observation.

Evidence anchor: The language similarity effect appears across multiple results:

• In the sequential bilingual condition (Figure 3, bottom row), structural priming effects survive catastrophic forgetting for Dutch-English and Spanish-English models (significant priming effects in both directions for Dutch, and for Spanish→English) but disappear for Polish-English and Greek-English models (no significant effects in either direction for Greek, and non-significant for Polish→English).
• In the loss curves (Figure 2), when English is the L2 in the simultaneous condition (second column), the gap between English and the non-English language's mean surprisal at the end of training is much smaller for Dutch (most similar) and Spanish than for Polish and Greek (least similar), indicating greater crosslingual transfer benefit for more similar languages.
• In the BLiMP comparisons (Appendix F, Figure F.13), final BLiMP accuracy for English-L1 models in the simultaneous condition is highest for Dutch and Spanish L2s and lowest for Greek and Polish L2s, despite identical training data quantities.

Significance beyond performance: This innovation establishes a causal boundary condition on a phenomenon (crosslingual representational sharing) that the field has largely treated as universally available given sufficient data. It suggests that the architecture of multilingual models — which typically share parameters across all languages — may be mismatched to the goal of achieving deep grammatical integration for typologically diverse language pairs. If shared representations require structural similarity, then the one-model-fits-all-languages paradigm may need to be replaced with architectures that selectively share parameters based on linguistic typology.

Innovation 3: Dynamic Training Analysis That Links Priming Emergence to L2 Acquisition, Not Surface Statistics

A third conceptual contribution is the temporal alignment of structural priming emergence with grammatical competence acquisition, which provides evidence that crosslingual priming reflects genuinely abstract grammatical representations rather than surface-level statistical associations between lexical items across languages. This is the first demonstration that priming effects in language models appear only after the model has acquired non-trivial grammatical knowledge in the second language, and that they emerge concurrently with (rather than preceding) that knowledge.

What was the prior assumption? Prior structural priming studies in language models (Prasad et al., 2019; Sinclair et al., 2022; Michaelov et al., 2023) evaluated only the final model checkpoint. This static approach cannot distinguish between two competing accounts of what structural priming measures: (1) the representational account, where priming reflects the model having learned abstract grammatical schemas that are shared across languages, and (2) the surface-statistical account, where priming reflects the model having learned that certain lexical co-occurrence patterns (e.g., the Dutch word "aan" [to] in a PO dative context predicts the English word "to" in the subsequent sentence) are associated across languages, without any abstract grammatical representation. If priming effects were present at the very beginning of L2 exposure — before the model could possibly have learned L2 grammar — this would favor the surface-statistical account. If priming emerges only as L2 grammatical knowledge develops, this supports the representational account.

What makes this innovation distinctive? The dense checkpointing strategy (saving model states every 10 steps for the first 200 steps after L2 introduction) enables the authors to precisely locate the moment when priming first appears and compare it with an independent measure of grammatical competence (BLiMP accuracy). The result — visually clear in Figure 4 for the English-Dutch simultaneous model — is that priming effects and BLiMP scores rise together. Priming does not appear before grammatical competence; it appears at the same time. Specifically, the first significant priming effects appear around 600 steps after L2 exposure (approximately 4.9M L2 tokens), which coincides with a sharp increase in BLiMP accuracy. Before L2 introduction (at the step 64,000 checkpoint), despite the model having been exposed to extensive L1 data and having learned L1 grammar thoroughly, there is no crosslingual priming effect.

This temporal coincidence rules out the simplest version of the surface-statistical account. If priming were driven purely by cross-language token co-occurrence patterns, the model could potentially exhibit priming from the very first L2 exposure, since it already knows the L1 patterns and could map them onto L2 tokens via shared subword units or orthographic overlap. The fact that priming requires L2 grammatical knowledge to develop first suggests that the model is forming genuinely abstract representations of grammatical constructions that generalize across languages, not just learning surface-level translation equivalents.

Evidence anchor: Figure 4 (left panel) shows the structural priming effect over training for the English-Dutch simultaneous bilingual model, with significant priming effects (triangles) emerging shortly after L2 introduction and persisting thereafter. Figure 5 (top) superimposes the priming effect size (black) with BLiMP accuracy (pink) for the Dutch-English simultaneous model, showing correlated increases in both measures after L2 introduction. The text reports that priming appears "as early as 600 steps after exposure to L2, or 4.9M L2 tokens." Appendices E.1–E.5 show the same pattern across all language pairs, though the effect is weaker and less temporally crisp for Polish and Greek, consistent with the language similarity constraint.

Significance beyond performance: This innovation strengthens the construct validity of structural priming as a measure of abstract grammatical representation in language models. If priming could emerge without grammatical knowledge, it would be a less interesting phenomenon — one of many surface-level statistical patterns that LMs learn, with no necessary connection to representations that generalize. By demonstrating that priming is temporally contingent on grammatical acquisition, the paper builds a stronger case that structural priming in LMs genuinely reflects the kind of abstract structural knowledge that it is taken to reflect in humans. This matters not just for interpreting the current results, but for the broader research program of using structural priming to study language model representations — it validates the probe itself.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The experiments use experimental stimuli from five published human psycholinguistic studies, covering four language pairs and three grammatical alternations. The stimuli come from Schoonbaert et al. (2007) for English-Dutch dative alternation (DO/PO), Bernolet et al. (2013) for English-Dutch genitive alternation (s-genitive/of-genitive), Hartsuiker et al. (2004) for English-Spanish active/passive alternation, Fleischer et al. (2012) for English-Polish active/passive alternation, and Kotzochampou and Chondrogianni (2022) for English-Greek active/passive alternation. The original Spanish, Greek, and Polish experiments have substantially fewer stimuli pairs than the Dutch experiments; to match statistical power, the authors create new prime-target pairs by considering every possible combination of prime and target sentences, then randomly sample 144 pairs for each of these three language pairs, matching the number of pairs in the Dutch materials. Contamination analysis using Infini-gram against C4 shows minimal overlap: 0% for Schoonbaert and Bernolet stimuli, 0.78% for Hartsuiker stimuli (3 items), 0% for Fleischer stimuli, and 2.08% for Kotzochampou stimuli (4 items; Appendix B, Table B.1).

• Base model(s). All experiments use 16 GPT-2 Transformer language models with 124M parameters, trained from scratch by the authors. Each model has 12 layers, 768-dimensional embeddings and hidden states, 12 attention heads with 64-dimensional head size, 3072-dimensional feed-forward hidden size, GELU activation, vocabulary size of 50,004 tokens, and maximum sequence length of 128 tokens (Appendix C, Table C.2). The models are trained for 128,000 steps with batch size 128, linear learning rate decay from 1e-4 with 10,000 warmup steps, Adam optimizer with β₁=0.9, β₂=0.999, ε=1e-6, and dropout of 0.1. The choice of this scale is explicitly motivated by compute constraints (16 models × dense checkpointing) and by prior evidence that models of this size trained on human-scaled data quantities (~100M words) learn robust syntactic generalizations (Warstadt et al., 2023), with syntax learned very early in training (Blevins et al., 2022; Chang et al., 2024b).

• Metrics. The primary metric is the structural priming effect, measured as the difference in normalized target sentence probability between match and mismatch prime conditions. For a target construction T_PO and a prime P, the normalized probability is computed as P_N(T_PO | P) = P(T_PO | P) / (P(T_PO | P) + P(T_DO | P)), where raw sequence probabilities are computed using the standard autoregressive factorization of the language model. Structural priming is operationalized as a statistically significant positive coefficient for the match condition (relative to mismatch as reference) in a linear mixed-effects model predicting normalized target probability, with prime type as a fixed effect and experimental item as a random intercept (normalized_probability ~ prime_type + (1 | item)). Multiple comparisons across models, checkpoints, and priming directions are corrected by controlling the false discovery rate using the Benjamini-Hochberg procedure. A secondary metric is BLiMP accuracy (Warstadt et al., 2020), which measures the proportion of 67,000 English grammatical minimal pairs for which the model assigns higher probability to the grammatical sentence than to the ungrammatical one, averaged across 67 datasets. The training loss metric reported in Figure 2 is mean surprisal (log-perplexity or evaluation loss) on held-out evaluation data (1M tokens per language).

• Baselines. This paper does not use baseline models in the conventional sense — there is no "competing method" whose performance is compared. Rather, the experimental design is factorial: each model serves as a control for others that differ in exactly one manipulated factor. The key comparisons are within-language-pair (e.g., Dutch-English simultaneous vs. English-Dutch simultaneous, to isolate the effect of language exposure order) and across-language-pair (e.g., Dutch-English vs. Greek-English, to isolate the effect of language similarity), with all other factors (architecture, training data quantity, data source, training schedule) held constant. The "baseline" against which priming effects are evaluated is the within-model null hypothesis that match and mismatch conditions produce equal normalized target probabilities, assessed via the linear mixed-effects model's fixed effect for prime type.

• Generation budget / compute accounting. There is no "generation budget" in this paper — structural priming evaluation does not involve sampling from the model. Instead, it requires computing the log-probability of target sentences given prime sentences, which involves exactly one forward pass per prime-target pair. The total evaluation cost is therefore proportional to the number of prime-target pairs × the number of checkpoints × the number of models. The paper does not report total FLOPs or wall-clock time for evaluation, only for training: 16 small models trained for a total of approximately 512 GPU hours on one NVIDIA RTX A6000, with estimated carbon emissions of 66 kg CO₂ equivalents. The critical "compute" control is not about inference budget but about training data quantity: every language in every model receives exactly 2B tokens of training data, ensuring that priming differences cannot be attributed to data imbalance.

• Cross-validation / statistical protocol. The paper does not use train/validation/test splits in the standard ML sense — the models are evaluated exclusively on held-out psycholinguistic stimuli that were never part of training. The experimental items (prime-target pairs) are analyzed using linear mixed-effects models, which account for item-level variability through random intercepts, avoiding the pseudo-replication problem that would arise from treating individual prime-target pairs as independent observations. For the training dynamics analysis, the authors evaluate every checkpoint on the same set of stimuli, and multiple comparison correction (FDR) is applied across the full set of statistical tests (all checkpoints, all models, all priming directions). There is no separate validation set for hyperparameter tuning since all hyperparameters are fixed across models.

Main Quantitative Results

Overall Replication and Asymmetry Pattern

The central finding, shown in Figure 3 (top row, simultaneous bilingual condition), is that crosslingual structural priming effects replicate successfully, but with a persistent asymmetry: priming is more robust when English is the target language than when English is the prime language, regardless of whether English is L1 or L2.

In the simultaneous bilingual condition with L1→L2 priming (prime language = L1, target language = L2, the configuration that the human literature would predict to be strongest):

• When English is the target (i.e., the non-English language is L1 and English is L2): statistically significant priming effects are found for all four language pairs — Dutch→English (Schoonbaert stimuli: significant, indicated by *; Bernolet stimuli: significant), Spanish→English (Hartsuiker stimuli: significant), Greek→English (Kotzochampou stimuli: significant), and Polish→English (Fleischer stimuli: significant). The paper reports these as significant at the FDR-corrected threshold, with the specific effect magnitudes visible as the difference between orange (match) and purple (mismatch) bars in Figure 3.

• When English is the prime (i.e., English is L1 and the non-English language is L2, so priming direction is English→non-English): statistically significant priming effects are found only for English→Dutch (Schoonbaert stimuli: significant; Bernolet stimuli: significant) and English→Greek (Kotzochampou stimuli: significant). English→Spanish (Hartsuiker stimuli) and English→Polish (Fleischer stimuli) show non-significant effects, though the numerical pattern is in the expected direction (match > mismatch) for both.

The asymmetry is starkest for Spanish: Spanish→English priming is significant, but English→Spanish priming is not. This cannot be attributed to data quantity (both Spanish and English receive 2B tokens) or to L1/L2 status (the effect is present in one direction regardless of which language is L1). The paper argues this implicates properties of English as a target language — specifically, English's relatively fixed word order may make structural repetition more detectable than in Spanish, where word order is more flexible.

In the sequential bilingual condition (Figure 3, bottom row, again L1→L2 priming), the asymmetry persists but with a crucial additional pattern driven by language similarity:

• Dutch→English and Dutch←English priming both remain significant (Schoonbaert and Bernolet stimuli in both directions show significant effects).
• Spanish→English priming remains significant, but English→Spanish priming remains non-significant.
• Greek→English priming becomes non-significant (unlike the simultaneous condition where it was significant), and English→Greek priming becomes non-significant as well (it was significant in the simultaneous condition).
• Polish→English priming becomes non-significant (it was significant in the simultaneous condition), and English→Polish priming remains non-significant.

The sequential condition thus reveals a language similarity gradient in robustness to catastrophic forgetting: Dutch (most similar to English) shows fully preserved crosslingual priming in both directions despite the L1 forgetting evident in Figure 2; Spanish shows partially preserved priming (only the Spanish→English direction remains significant); Greek and Polish show essentially complete loss of crosslingual priming effects. The text in Section 4.3 explicitly notes that "the stronger effects for Dutch and Spanish, and less strong effects for Greek and Polish, are likely an effect of language similarity with English."

Training Dynamics: Priming Emerges Concurrently with L2 Grammatical Competence

The paper's second major quantitative result concerns the time course of structural priming emergence, evaluated by testing each checkpoint for priming effects. Figure 4 (left panel) shows the priming trajectory for the English-Dutch simultaneous bilingual model on Schoonbaert stimuli: significant priming effects (marked with triangles) first appear shortly after L2 introduction at step 64,000, with the first significant effects detectable "as early as 600 steps after exposure to L2, or 4.9M L2 tokens" (Section 4.4). Before L2 introduction (checkpoints 0 through 64,000), no priming effects are found (circles indicating non-significance).

The critical temporal alignment is shown in Figure 5 (and Appendices E.1–E.5 for all language pairs). In the Dutch-English simultaneous model (Figure 5, top), the structural priming effect size (black line) and English BLiMP accuracy (pink line) are plotted together over training. Both measures show a sharp increase immediately after L2 introduction at step 64,000, rising together over the next several thousand steps. The BLiMP accuracy jumps from near-chance levels before L2 exposure to substantially above chance within a few thousand steps, and the priming effect emerges in the same window. The paper argues this concurrent emergence "draws a stronger link between structural priming behavior and shared multilingual representations" because it rules out the possibility that priming could appear before any grammatical knowledge is acquired (which would suggest surface-level statistical mechanisms rather than abstract representation sharing).

In the sequential bilingual condition (Figure 5, bottom, showing the English-Dutch sequential model where English is L1 and Dutch is L2), the pattern is different but equally informative. English BLiMP accuracy (L1 grammatical competence) drops sharply after L2 introduction at step 64,000, consistent with the catastrophic forgetting evident in the loss curves (Figure 2). Despite this severe degradation of surface-level L1 knowledge, the structural priming effect (now measuring Dutch→English priming, where Dutch is L2 and English is the target) appears and remains stable through the second half of training. The paper interprets this as evidence that "even when the model experiences catastrophic forgetting, representations may still be shared between languages and allow for transfer learning." However, this resilience to forgetting is language-dependent: Appendices E.1–E.5 show that for Greek and Polish models in the sequential condition, priming effects do not persist after catastrophic forgetting, again consistent with the language similarity constraint.

L2→L1 Priming Results (Appendix D)

The main text focuses on L1→L2 priming, following the human psycholinguistics convention, but Appendix D reports the complementary L2→L1 results (Figures D.6 and D.7). In the simultaneous condition (Figure D.6), the pattern is broadly consistent: English as target (i.e., L2→L1 priming where English is L1 and the non-English language is L2 as prime) shows robust priming effects for Dutch (significant in both Schoonbaert and Bernolet experiments) and for Greek (significant in Kotzochampou experiment), with numerical but non-significant effects for Spanish and Polish. In the sequential condition (Figure D.7), essentially all effects become non-significant except for Dutch, which retains significant L2→L1 priming in both directions. This further reinforces the language similarity gradient: only the most similar language pair preserves bidirectional priming through catastrophic forgetting.

BLiMP Comparisons Across Language Pairs (Appendix F)

Figure F.13 provides an additional quantitative window into the language similarity effect by comparing English BLiMP accuracy for models where English is L1, across all four non-English L2 languages, in both simultaneous and sequential conditions. In the simultaneous condition (dotted lines), final-checkpoint BLiMP accuracy is highest for Dutch L2, followed by Spanish L2, then Polish and Greek L2 (which are similar to each other and substantially lower). The ordering is Dutch > Spanish > Polish ≈ Greek, which maps directly onto the typological similarity gradient: Germanic > Romance > Slavic ≈ Hellenic relative to English. In the sequential condition (solid lines), BLiMP accuracy drops after L2 introduction for all models (catastrophic forgetting), but the final accuracy level follows the same ordering, with Dutch maintaining the highest accuracy and Greek the lowest. This provides convergent evidence — using a completely independent syntactic evaluation benchmark rather than structural priming — that language similarity causally constrains how well grammatical knowledge transfers between languages.

Loss Curve Analysis (Section 3.1)

Figure 2 provides the training dynamics context for all models by plotting mean surprisal (evaluation loss) for both L1 and L2 at every checkpoint. Several quantitative patterns are notable:

• In the simultaneous condition (left four columns of Figure 2), L1 mean surprisal decreases smoothly throughout training, with a slight perturbation (a small temporary increase or plateau) at step 64,000 when L2 is introduced, then continuing to decrease. L2 mean surprisal is high during the first half of training (when the model has never seen L2 data) and drops sharply after L2 introduction at step 64,000, then continues to decrease slowly for the remainder of training.

• In the sequential condition (right four columns), L1 mean surprisal drops during the first half (L1-only training), then rises sharply after step 64,000 when training switches exclusively to L2. This rapid increase — often back to near-initial levels — is the hallmark of catastrophic forgetting. L2 mean surprisal drops rapidly after step 64,000, mirroring the simultaneous condition.

• There is a quantitative asymmetry in the simultaneous condition when comparing English as L1 vs. English as L2. When English is the L2 (second column from left in Figure 2), the gap between L1 and L2 mean surprisal at the end of training is substantially smaller for Dutch→English and Spanish→English than for Polish→English and Greek→English. For Dutch→English, the English and Dutch surprisal curves nearly converge by the end of training; for Greek→English, there remains a large gap. This is the loss-based counterpart of the language similarity effect: English benefits more from crosslingual transfer when the L1 is typologically similar.

• A corresponding asymmetry appears in the sequential condition when English is the L1 (fourth column from left). The rise in English (L1) mean surprisal after step 64,000 — i.e., the magnitude of catastrophic forgetting — is smallest for Dutch L2, intermediate for Spanish and Polish L2, and largest for Greek L2. This means that learning a similar L2 interferes less with existing L1 knowledge than learning a distant L2, which the paper attributes to greater parameter sharing when languages are similar.

The paper also notes an interesting asymmetry that it attributes to data contamination rather than linguistic factors: "in Figure 2, there is an asymmetry between the English-Dutch and Dutch-English simultaneous models, where the English L2 loss drops much more quickly in the first half of training than does the loss for Dutch as L2." When Dutch is the L1 and English is the L2 (the model supposedly not trained on English during the first half), English mean surprisal decreases somewhat even before step 64,000 — the model appears to be learning something about English from the Dutch data alone. The paper hypothesizes that this is due to English contamination in the Dutch OSCAR data, since "many Dutch people speak English" and web text in Dutch may contain English code-switching or untranslated English passages. This is a confound the paper acknowledges but does not control for beyond noting it.

Ablation Studies and Robustness Checks

Contamination analysis (Appendix B): The paper checks whether experimental stimuli appear in web-scale text using Infini-gram against the C4 corpus. As reported in Table B.1, contamination rates are 0% for Schoonbaert (2007), Bernolet (2013), and Fleischer (2012) stimuli; 0.78% (3 out of ~384 items) for Hartsuiker (2004); and 2.08% (4 out of ~192 items) for Kotzochampou and Chondrogianni (2022). The paper does not report sensitivity analyses removing contaminated items, but the rates are low enough that removal would not meaningfully affect statistical results.

Stimulus augmentation for statistical power (Section 4.2): The original Spanish (Hartsuiker et al., 2004), Greek (Kotzochampou and Chondrogianni, 2022), and Polish (Fleischer et al., 2012) experiments have fewer stimuli pairs than the Dutch experiments. To match statistical power, the authors create new pairs by taking the Cartesian product of all prime and target sentences, then randomly sample 144 pairs. This is not strictly an ablation but a methodological control ensuring that null results for these language pairs are not attributable to lower statistical power. The paper does not report whether results would differ with different random samples of 144 pairs.

Multiple comparison correction (Section 4.1): The paper applies FDR correction (Benjamini and Hochberg, 1995) across all linear mixed-effects models fitted for each checkpoint, model, and priming direction. Without this correction, some numerically positive but weak effects (e.g., English→Spanish priming in the simultaneous condition, which shows a visible match > mismatch difference in Figure 3 but is marked n.s.) might appear significant by chance given the large number of tests performed. The FDR correction makes the reported pattern more conservative, and the survival of key effects (particularly the Dutch and Spanish patterns) through this correction strengthens confidence in them.

L2→L1 priming as robustness check (Appendix D): The main text reports only L1→L2 priming results, following the human literature. Appendix D shows that the key qualitative patterns (English-as-target asymmetry, language similarity gradient) appear in the L2→L1 direction as well, though effects are generally weaker. This confirms that the findings are not an artifact of which priming direction is tested.

Cross-check with independent syntactic benchmark (Appendices E and F): The temporal alignment of priming effects with BLiMP accuracy is not a formal ablation but serves as a convergent validity check. If structural priming were measuring something unrelated to grammatical competence, there would be no reason for the two measures to rise together after L2 introduction, nor for their final-checkpoint ordering across language pairs to match (Dutch > Spanish > Polish ≈ Greek for both priming robustness and BLiMP accuracy). The consistency between the behavioral priming measure and the targeted grammatical evaluation strengthens the paper's interpretation that priming reflects abstract grammatical representations.

Catastrophic forgetting as an accidental ablation: The sequential bilingual condition was not designed as an ablation study per se, but it functions as one: it tests whether structural priming effects survive when the model's surface-level L1 knowledge is severely degraded (as measured by rising L1 mean surprisal and falling L1 BLiMP accuracy). The persistence of priming for Dutch and Spanish (but not Greek and Polish) in the sequential condition demonstrates that priming is not purely driven by current L1 proficiency — it can survive substantial forgetting, at least for similar language pairs. This effectively rules out an account where priming is simply a function of how well the model currently "knows" the L1, since Dutch models in the sequential condition show robust priming while having severely degraded L1 loss.

Checkpoint density justification (implicit ablation): The paper does not explicitly ablate checkpoint density, but the rationale for the dense sampling near L2 introduction (every 10 steps) is empirically justified by the results: significant priming effects appear within the first 600 steps of L2 exposure. With sparser checkpointing (e.g., every 10,000 steps), the authors would have missed the precise onset of priming and could not have established the temporal coincidence with BLiMP improvement. The dense sampling thus functions as an implicit demonstration that high temporal resolution is necessary to capture the rapid emergence of shared representations.

Critical Assessment

Claim 1: Priming asymmetries are due to English as target language, not L1/L2 status.

The experiments directly support this claim for the specific languages and conditions tested, but the support is narrower than a universal statement would require. The critical evidence comes from the within-language-pair comparison: for Spanish, Spanish→English priming is significant while English→Spanish priming is not (Figure 3, simultaneous condition), even though both directions are tested on models with identical architecture and training data quantities. Since Spanish is L1 in one model and L2 in the other, the L1/L2 account would predict symmetric effects (L1→L2 stronger in the Spanish→English model, L2→L1 weaker in the English→Spanish model), which is not what is observed. The finding that English-as-target consistently produces more robust priming across three of four language pairs (Dutch, Spanish, Polish — Greek is the exception, where both directions are significant in the simultaneous condition) strengthens the case.

However, several limitations qualify this claim. First, the sample of non-English languages is small (four) and exclusively Indo-European, all using the Latin alphabet except Greek. The hypothesis that English's relatively fixed word order drives the asymmetry would be more convincingly tested by including languages with even more flexible word order than Polish or Greek (e.g., Tagalog, which Garcia et al., 2023 showed does not exhibit word-order-based structural priming even within-language), or languages with fixed word order that are not English (e.g., Chinese, which also has relatively rigid SVO order). Without such a control, the claim remains that "something about English" drives the asymmetry, but the specific mechanism (word order rigidity, morphological richness, something else) is speculative.

Second, the data contamination confound that the paper itself identifies (Section 5.1) could partly explain the English-as-target advantage. If the Dutch OSCAR data contains English contamination (as the loss curves in Figure 2 suggest — English L2 loss drops during L1-only training when Dutch is L1), then models with English as L2 may have effectively seen more English data than intended, potentially boosting priming effects when English is the target. The contamination analysis in Appendix B only checks whether the specific experimental stimuli appear verbatim in training data, not whether the training data contains English text more generally. If Dutch training data contains 5-10% English code-switching (plausible for web text from a highly English-proficient population), then the Dutch→English model would have been exposed to substantially more English than the English→Dutch model was exposed to Dutch. This would create an asymmetric data quantity confound that, unlike the carefully controlled per-language token counts in the training set as designed, operates in practice. The paper acknowledges this possibility but does not quantify the extent of cross-language contamination in OSCAR, which is a genuine gap.

Third, the paper does not report the actual numerical effect sizes for the priming measurements — only significance (via asterisks) and visual bar plots. This makes it difficult to assess whether the English-as-target advantage is a large effect or a small one that happens to cross the significance threshold. The bar plots in Figure 3 show the match vs. mismatch difference visually, but without error bars or precise effect sizes, the magnitude of the asymmetry cannot be directly compared across language pairs. A quantitative asymmetry index (e.g., the difference in priming effect size between the two directions for each language pair) would have been informative.

Claim 2: Language similarity causally constrains the formation of shared grammatical representations.

This is the most strongly supported claim in the paper, with convergent evidence from multiple independent measures. The key experimental control — matching training data quantity across languages — eliminates the most plausible alternative explanation (that Polish and Greek priming is weaker because those languages are underrepresented in training data). The finding that similarity effects appear in (a) structural priming robustness (Figure 3), (b) resistance to catastrophic forgetting (Figure 3 bottom row, Appendices E.1–E.5), (c) L2 loss convergence (Figure 2), (d) BLiMP accuracy (Figure F.13), and (e) magnitude of catastrophic forgetting of L1 (Figure 2, sequential condition) provides strong convergent validity.

However, "language similarity" is treated as a unitary construct when in practice it confounds multiple dimensions. The languages differ along at least four potentially independent axes: (1) typological similarity (word order patterns, morphological richness, how grammatical alternations are encoded — Greek uses synthetic passive morphology while the others use periphrastic constructions), (2) orthographic similarity (Greek uses a non-Latin script; the others use Latin), (3) vocabulary overlap due to shared etymology (English-Dutch cognates are abundant; English-Greek cognates exist but are obscured by different scripts; English-Polish cognates are rarer), and (4) subword token overlap in the SentencePiece vocabulary (a direct consequence of orthography). The four-language sample is too small and too confounded along these dimensions to determine which aspect of "similarity" drives the effect. Is it that Greek fails because Greek uses a different script (so the model has no shared subword tokens to bootstrap structural generalization), or because Greek encodes passives through verbal morphology rather than word order (so the structural alternation is fundamentally different), or because Greek word order is more flexible (so the "same" construction may appear in different linear positions)? These are different causal mechanisms with different implications for model architecture and training, but they cannot be disentangled with the current language sample.

The paper acknowledges this limitation in Section 5.2: "As our sample of languages is small and comes from one language family, it is not possible to quantitatively analyze the impact of various typological features." However, it then proceeds to offer speculative explanations (orthography, morphological encoding of alternations) that treat the four-language sample as if it supports differentiation among these factors. A more rigorous test would require a larger typologically diverse sample with orthogonal manipulation of script, word order flexibility, and morphological richness — for example, including languages like Turkish (Latin script, agglutinative morphology, relatively flexible word order), Chinese (non-Latin script, isolating morphology, relatively fixed word order), or Finnish (Latin script, agglutinative morphology, relatively flexible word order) to begin disentangling these factors.

Additionally, there is a potential confound in the stimulus materials themselves. The Greek and Polish stimuli (active/passive alternations) and the Dutch/Spanish stimuli (dative and genitive alternations for Dutch, active/passive for Spanish) test different grammatical constructions. It is possible that crosslingual structural priming is inherently weaker for active/passive alternations than for dative or genitive alternations, independent of language similarity. The Spanish-English active/passive stimuli (Hartsuiker et al., 2004) do show robust priming in one direction, suggesting that active/passive priming can work, but without a within-language-pair comparison of different alternation types (e.g., testing active/passive priming for Dutch-English as well as dative priming), the confound between language pair and alternation type cannot be fully excluded.

Claim 3: Priming emergence is temporally linked to L2 grammatical acquisition, supporting the representational account over surface-statistical alternatives.

This claim is supported for English as L2 (the only case where BLiMP provides an independent grammatical measure), and the temporal coincidence in Figures 4 and 5 is visually compelling. However, the evidence has important limitations.

First, the claim rests on a temporal correlation, not a causal manipulation. The fact that priming and BLiMP accuracy rise together does not prove that priming requires grammatical knowledge; it could be that both are driven by a common cause (e.g., the model learning to represent L2 word order patterns, which simultaneously improves BLiMP scores and enables priming). A stronger test would involve an intervention: for example, training a model where L2 grammatical knowledge is specifically impaired (e.g., by training on scrambled L2 sentences that preserve lexical statistics but destroy syntax) and testing whether priming still emerges. This is not done.

Second, the "surface-statistical account" that the paper argues against is somewhat underspecified. What specific surface statistics would produce priming without grammatical representations? If the account is that priming reflects cross-language lexical co-occurrence (e.g., the Dutch word "aan" predicting the English word "to"), then the temporal argument is reasonable: such co-occurrences could be learned from the very first L2 exposures, before the model has organized them into grammatical abstractions. But a more sophisticated surface-statistical account — where priming requires the model to have learned language-specific word order regularities that happen to align with grammatical structure, but without forming abstract representations that generalize across constructions — would predict a similar temporal profile to the representational account. The temporal coincidence evidence distinguishes between the simplest surface-statistical account and the representational account, but not between all possible alternatives.

Third, the BLiMP analysis is restricted to English as L2 because BLiMP does not exist for Dutch, Spanish, Polish, or Greek. This means the symmetrical claim — that priming emergence for a non-English target language coincides with grammatical acquisition in that language — cannot be tested. Given the asymmetry finding (priming is weaker when English is not the target), this is not a minor omission: it is precisely the cases where the temporal link would be most informative (does Spanish→English priming emerge with English grammatical competence? does English→Spanish priming emerge only if Spanish grammatical competence is acquired? does the weak English→Spanish priming correspond to weak Spanish grammatical acquisition?) that cannot be evaluated.

Fourth, the temporal resolution, while much finer than prior work, may still be insufficient to distinguish truly concurrent emergence from closely sequential emergence. The first significant priming effect is detected at 600 steps, but this is the first checkpoint at which significance is achieved — the effect might have been present (but not statistically significant) at 590 steps, or might have required exactly 600 steps to cross the threshold. The claim of "concurrent" emergence would be more convincing with a formal test for temporal precedence (e.g., Granger causality or cross-correlation analysis) rather than visual inspection of overlaid time series.

Neglected Analyses and Missing Experiments

Several experiments would have substantially strengthened the paper's claims:

Cross-validation across random stimulus samples. The paper randomly samples 144 prime-target pairs for the Spanish, Greek, and Polish stimuli to match the Dutch statistical power. Results could vary across different random samples. Reporting results for multiple samples and showing stability would address concerns about sampling variability. The paper does not do this.

Removal of contaminated items. The contamination analysis identifies a small number of items that appear in C4. A sensitivity analysis showing that results are unchanged when these items are removed would strengthen the contamination argument. The paper does not report this.

Direct comparison of alternation types within a language pair. Testing whether dative priming and genitive priming and active/passive priming differ in magnitude for the same language pair (e.g., English-Dutch) would help determine whether the Greek/Polish weakness is about active/passive alternations specifically or about language similarity more generally. The paper has the Dutch stimuli for datives and genitives but does not test active/passive priming for Dutch-English, which would provide a within-language-pair comparison of alternation types.

Quantification of cross-language contamination in OSCAR. The paper notes evidence of English contamination in Dutch OSCAR data but does not attempt to quantify it (e.g., by running language identification on the actual training data and measuring the proportion of non-target-language text). This leaves the contamination confound as an acknowledged but unresolved concern.

Effect size reporting. The paper reports significance (asterisks) and shows bar plots, but does not report numerical effect sizes (the β coefficient for prime type from the linear mixed-effects models) with confidence intervals. This would enable quantitative comparison across language pairs and priming directions, and would allow assessment of whether non-significant effects reflect genuinely null effects or small effects that are underpowered.

Larger sample of typologically diverse languages. The four-language, all-Indo-European sample is acknowledged as a limitation but fundamentally constrains the generality of the language similarity findings. Including even one non-Indo-European language with a different script and different word order properties (e.g., Turkish, Chinese, Japanese, Korean, Finnish) would substantially strengthen the claim that "language similarity" rather than "Indo-European similarity" constrains shared representation formation. The authors explicitly note this as future work ("Training larger models and how these effects change with model and data scale would also be illuminating"), but the limitation is inherent to the current study's design, not a question of scale.

Investigation of tokenizer effects. The per-model tokenizers match training data proportions, which is a principled choice, but it also means that tokenizers differ across models in ways that could affect priming measurements. A model trained on Dutch-English has a tokenizer that allocates vocabulary capacity to both languages; a model trained only on Dutch during phase 1 and then only on English during phase 2 uses a tokenizer that was trained to represent both but is applied to a changing input distribution. The effect of tokenizer design on structural priming measurements is not explored — for example, whether priming effect sizes are sensitive to the granularity of subword segmentation (which affects how "abstract" the representations need to be to generalize across languages).

Overall Assessment

The experiments provide credible evidence for the paper's three main claims, but each claim comes with qualifications that constrain its scope. The English-as-target asymmetry is demonstrated for the specific languages studied but the mechanism (word order rigidity vs. other factors) is speculative. The language similarity constraint is the most robust finding, supported by convergent evidence across multiple measures, but "similarity" remains a confounded construct that cannot be decomposed with the current language sample. The temporal link between priming and grammatical competence is demonstrated for English as L2, but the evidence is correlational rather than causal, and the symmetrical claim for non-English L2s cannot be tested with available tools.

The paper's strongest contributions are conceptual and methodological — using controlled model training to address confounds that plague the human literature, and establishing that language similarity is a hard constraint on representation sharing even when data quantity is controlled — rather than quantitative precision. The experimental design is clever and the manipulations are well-motivated, but the paper would benefit from more rigorous quantification (effect sizes, confidence intervals, sensitivity analyses) and from a larger, more typologically diverse language sample to decompose the "similarity" construct into its constituent dimensions. These limitations are mostly acknowledged by the authors, and the paper's value lies more in the framework it establishes and the hypotheses it generates than in the specific numerical claims.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For

The assumption or constraint. The compute-optimal framework in this paper depends on knowing, before allocating test-time compute, how difficult each prompt is for the base model. The paper estimates difficulty by generating 2,048 complete solutions per question from the base LLM, then binning questions into quintiles based on either ground-truth pass@1 (oracle difficulty) or the PRM's average final-answer score (predicted difficulty). The paper explicitly states (Section 3.2):

"this still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The headline efficiency figure — "more than 4× better efficiency over a standard best-of-N baseline" — is computed after difficulty is known, without amortizing the cost of learning it. Generating 2,048 samples per question to estimate difficulty is extraordinarily expensive and may dominate the total inference budget. For instance, at a budget of 16 generations of beam search (which the paper reports matches best-of-N at 64 generations), the difficulty estimation cost of 2,048 samples is roughly 128× the actual problem-solving budget. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the difficulty estimation cost alone may exceed the budget of all but the most compute-intensive strategies studied. This means 4× is strictly an upper bound on achievable efficiency — practical gains with amortized difficulty estimation would be substantially smaller and perhaps negative (i.e., the approach could be less efficient than a uniform strategy once difficulty estimation is accounted for).

What evidence exists in the paper. Section 3.2 acknowledges the cost explicitly but provides no measurement of total amortized efficiency. Figure 4 and Figure 8 in the paper show compute-optimal scaling curves for both oracle and predicted difficulty bins, but these curves treat difficulty as a known input rather than an expensive quantity to be estimated. The paper never reports a plot where the x-axis includes difficulty estimation cost, nor does it report how results change if difficulty is estimated from fewer than 2,048 samples (e.g., 32, 64, or 128 samples).

Mitigation status. The paper flags this as a key avenue for future work (Section 3.2: "training models to directly predict the difficulty of a question"), but no such model is developed or evaluated. The predicted difficulty bins use the PRM's average score rather than ground-truth labels, but they still require the cost of generating and scoring 2,048 samples. No experiments vary the number of samples used for difficulty estimation. The exploration-exploitation tradeoff — spending some fraction of the budget to assess difficulty, then allocating the remainder optimally — is mentioned (Section 8) but not formalized, modeled, or tested.

Hard Problems Remain Essentially Unsolved — Test-Time Compute Cannot Create Capability

The assumption or constraint. The paper's approach assumes that the base model already produces correct solutions at some non-trivial rate for the prompts of interest. The compute-optimal framework selects among strategies that amplify existing capability — better search finds correct solutions the model would otherwise miss, revisions refine nearly-correct solutions — but neither mechanism creates correct solutions from scratch. The paper explicitly acknowledges this boundary in the Section 7 takeaway box:

"on the hardest problems (bin 5), test-time compute provides essentially zero benefit regardless of budget, meaning that some capabilities can only be acquired through pretraining"

The consequence. For any problem where the base model's pass@1 is near zero, the entire framework — search, revisions, and compute-optimal allocation — offers no path forward. As Figure 3 (right) shows, both best-of-N and beam search on bin 5 plateau at roughly 1–3% accuracy regardless of budget. Figure 7 (right) shows all sequential-to-parallel ratios for revisions on bin 5 produce roughly 2–3% accuracy. Figure 9 shows the compute-optimal scaling line for bin 5 is essentially flat near 0–5% across all budgets, never approaching the 14× larger model's performance. This is not a soft ceiling that more budget can push through — it is a hard boundary: test-time compute amplifies existing capability but does not create it. For deployments where the problem distribution includes a non-trivial fraction of genuinely hard prompts (outside the model's capability range), the approach provides no benefit and users must fall back on larger pretrained models. The framework also provides no mechanism for distinguishing "hard problems that can become solvable with more compute" from "hard problems that require fundamentally different model capabilities" — the difficulty bins conflate both types.

What evidence exists in the paper. The bin 5 results in Figure 3 (right), Figure 7 (right), and Figure 9 provide consistent evidence of near-zero improvement across all methods. The FLOPs-matched comparison in Section 7 quantifies this precisely: on hard problems at R ≫ 1, PRM search shows a −52.9% relative disadvantage compared to the 14× larger model, and revisions show a −37.2% disadvantage. The scaling curves for bin 5 are essentially flat from budget 4 to budget 256, with no upward trajectory suggesting that even larger budgets might eventually help.

Mitigation status. The authors are transparent about this limitation and treat it as a finding rather than a failure — establishing that test-time compute has a boundary condition is itself a contribution. The paper acknowledges (Section 8) that "improving the base model's capabilities through pretraining remains the only viable path" for hard problems. However, no practical guidance is provided for determining, before deploying the system, what proportion of a given problem distribution falls into the "unsolvable at any budget" category versus the "solvable with enough test-time compute" category. The difficulty estimation procedure (2,048 samples + PRM scoring) could in principle identify bin 5 problems, but the paper does not evaluate how reliably bin assignment predicts which problems are genuinely unsolvable.

Revisions and PRM Search Are Studied Independently, Never Combined

The assumption or constraint. The paper decomposes test-time compute into two axes — modifying the proposal distribution via revisions (Section 6) and optimizing the verifier via PRM-guided search (Section 5) — but studies them entirely independently. Section 8 explicitly acknowledges:

"we did not experiment with PRM tree-search techniques in combination with revisions"

Each axis is evaluated in isolation against a best-of-N baseline, and a compute-optimal policy is derived within each axis separately (Figure 4 for search, Figure 8 for revisions). The two axes are never combined in any experiment — there is no model that uses both a revision-trained proposal distribution and PRM-guided beam search, no comparison of combined strategies against either axis alone, and no compute-optimal policy that selects among strategies from both axes simultaneously.

The consequence. The reported results represent a lower bound on what a fully integrated system could achieve. The paper's own framework (Section 2) argues that the proposal distribution and the verifier are complementary, independent scaling axes, and the difficulty-dependent analysis shows they have complementary strengths: revisions work best on easy problems (local refinement), while PRM search works best on medium problems (guided exploration). A combined approach that uses the revision model as the proposal distribution within beam search — or uses the PRM to guide which revisions to pursue — could potentially outperform either axis alone, particularly on medium-difficulty problems where both mechanisms show partial effectiveness. Without this experiment, the reader cannot assess whether the axes are truly independent (in which case combining them would yield additive gains) or partially redundant (in which case the gains would be sub-additive). The paper's 4× efficiency claim is therefore measured against a best-of-N baseline but not against a combined revision + search baseline, which might be more efficient still.

What evidence exists in the paper. Figure 3 (search results) and Figure 6–8 (revision results) are reported in separate sections with no cross-comparison. The FLOPs-matched analysis in Figure 9 presents revision results and PRM search results as separate panels, with no combined condition. The compute-optimal policy in Figure 4 selects among search strategies only; the policy in Figure 8 selects among revision strategies only. Nowhere does a single model undergo both revision fine-tuning AND PRM-guided search at inference time.

Mitigation status. The paper explicitly flags this as future work (Section 8) but provides no preliminary evidence, no speculation about expected gains, and no analysis of whether the two approaches are compatible at the implementation level (e.g., does the revision model's output distribution differ sufficiently from the base model's that a PRM trained on base-model outputs would need to be retrained? — a concern raised in Appendix J, Figure 15a for revision-specific ORMs). The revision model experiments (Section 6) already use a separate ORM rather than the base-model PRM because of distribution shift (Figure 15a); combining PRM search with revisions would likely require retraining the PRM on revision-model outputs, which is not attempted.

Single Benchmark, Single Model Family — No Evidence of Generalization

The assumption or constraint. All experiments use a single model family (PaLM 2-S*) on a single benchmark (MATH). The paper states (Section 4):

"we believe this model is representative of the capabilities of many contemporary LLMs"

but provides no evidence for this belief. The experimental results may be specific to PaLM 2-S*'s particular output distribution, calibration properties, or error patterns; to MATH's specific difficulty distribution and problem types; or to the interaction between this model and this benchmark. Several aspects of the findings could be model-specific: the PRM's over-optimization behavior depends on the base model's propensity to produce solutions that exploit verifier weaknesses; the revision model's learnability depends on the base model's in-context learning and self-correction capabilities; the difficulty quintile boundaries depend on the model's overall MATH performance level. A model with different overall accuracy on MATH (e.g., 30% pass@1 instead of ~10–19%) would have different difficulty bin distributions, potentially shifting the optimal strategies per bin.

The consequence. None of the paper's quantitative claims — the 4× efficiency improvement, the strategy recommendations per difficulty bin (best-of-N for easy, beam search for medium), the FLOPs-matched comparison results — come with any guarantee that they would replicate on a different model, a different reasoning benchmark, or a different domain altogether. The difficulty-dependent strategy selection (the core compute-optimal policy) is a set of discrete choices (e.g., beam search M = 4 for bin 3, best-of-N weighted for bin 1) that are optimized specifically for PaLM 2-S* on MATH's test set via cross-validation. There is no reason to expect these specific choices — or even the qualitative pattern of which strategy works best at which difficulty — to transfer to a different model (e.g., Llama, Gemma, GPT-4) or a different benchmark (e.g., GSM8K, HumanEval, MMLU). A practitioner wanting to apply the compute-optimal framework to their own model and task distribution would need to replicate the entire experimental pipeline — train a PRM, train a revision model, sweep strategies, estimate difficulty bins, run cross-validation for strategy selection — without being able to rely on the paper's specific recommendations.

What evidence exists in the paper. Section 4 states the model choice explicitly. No experiments use any model other than PaLM 2-S*. No experiments use any benchmark other than MATH. The paper does not report — even as a small-scale validation — whether the key qualitative findings (beam search degrades on easy problems, revisions help on easy problems, hard problems are unsolved) hold for a second model or a second task. The revision results in Appendix K show that a different training procedure (ReST^EM) degrades revision model performance (Figure 16), demonstrating that results are sensitive to training methodology even within the same model family and benchmark — which suggests they may be sensitive to model choice as well.

Mitigation status. The paper acknowledges this implicitly by limiting its claims to PaLM 2-S* and MATH, but does not treat the single model/benchmark as a limitation requiring mitigation. There is no discussion of how the findings might generalize, no suggestion of model properties that would predict similar vs. different results, and no call for replication studies. The compute-optimal framework (the meta-strategy of difficulty-conditioned allocation) is presented as a general approach, but the specific instantiations (which strategies work for which difficulties) are entirely benchmark- and model-specific, and the paper does not disentangle the general framework from the specific instantiations.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales only model parameters while holding training data fixed, following the LLaMA paradigm rather than compute-optimal pretraining where both parameters and data are scaled equally (Hoffmann et al., 2022). The paper acknowledges (Section 7):

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the 14× larger model is evaluated with greedy decoding only — no majority voting, no best-of-N, no search, no revision chains. The smaller model with compute-optimal test-time scaling gets up to 256 generations of search or revisions, while the larger model gets exactly 1 generation.

The consequence. The comparison is systematically biased in favor of test-time compute. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both data and parameters) would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it should be. The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at R ≪ 1 for revisions) would likely shrink against a properly compute-optimal larger model. Furthermore, giving the larger model even a modest test-time compute budget — say, best-of-8 or best-of-32 with majority voting — would create a much stronger baseline. The paper is effectively comparing test-time compute with a small model against greedy decoding with a large model, which confounds the test-time vs. pretraining compute tradeoff with the test-time compute vs. no-test-time-compute tradeoff. A fair FLOPs-matched comparison would allocate compute optimally at both training and inference time for both models, which the paper does not do.

What evidence exists in the paper. Section 7 describes the FLOPs accounting and the 14× parameter scaling. The results in Figure 9 and the bar charts in Figure 1 show the comparison between PaLM 2-S* with compute-optimal test-time strategies and the 14× larger model with greedy decoding. There is no ablation where the larger model receives some test-time compute budget, and no comparison against a Chinchilla-optimal larger model. The paper does not report what performance the larger model would achieve with even a small test-time compute budget (e.g., best-of-8 majority voting), which would be a natural point on the tradeoff curve.

Mitigation status. The paper acknowledges the parameter-only scaling departure from compute-optimal pretraining but treats it as a reasonable representative of current practice (the LLaMA paradigm). The greedy-only decoding for the larger model is not discussed as a limitation — the paper presents the comparison as "test-time compute vs. pretraining compute," but structurally it is "small model with inference compute vs. large model without inference compute." No sensitivity analysis explores how results would change if the larger model received even a modest test-time compute budget. The paper suggests (Section 8) that future work should explore "jointly optimizing the pretraining recipe and the inference strategy under a total FLOPs constraint," implicitly acknowledging the current comparison's incompleteness but not treating it as a limitation of the reported claims.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the study of multilingual representations from observation to intervention — from probing frozen pretrained models for evidence of shared representations to building models from scratch under controlled conditions to causally test what factors enable or constrain those representations. This is not merely a methodological refinement; it fundamentally changes what kinds of claims the field can make. Before this work, a researcher observing crosslingual structural priming in a pretrained model could only report that priming exists and that it varies across language pairs, but could not determine why. Data quantity, language exposure order, typological similarity, orthographic overlap, and training data contamination were all hopelessly confounded. This paper demonstrates that small, controlled models — trained with a "controlled rearing" approach — can systematically isolate these factors, transforming the study of multilingual representations from a correlational to a causal enterprise.

The reframing extends beyond multilingual NLP into psycholinguistics. The finding that priming asymmetries persist even when English is the L1 directly challenges the dominant interpretation of human experiments spanning nearly two decades (Schoonbaert et al., 2007; Bernolet et al., 2013; Shin and Christianson, 2009, 2011). The field has attributed asymmetric crosslingual priming to the order of language acquisition (L1→L2 priming being stronger than L2→L1). But this paper shows that when data quantity and exposure order are controlled, the asymmetry is about English as a target language, not about L1/L2 status. This is a concrete, falsifiable reinterpretation of the human literature that would have been impossible to generate from human experiments alone — because human experiments cannot randomize which language a child learns first. The paper does not claim that language models are humans or that they learn identically, but rather that they serve as hypothesis-generation engines: they produce a candidate explanation (English-as-target drives the asymmetry) that can now be tested in human populations where English is the L1 and a typologically distant language is the L2 (e.g., English-dominant Korean learners). If human experiments replicate the reversed asymmetry, the paper's model-organism approach is validated as a discovery tool; if not, the boundary between LM and human learning becomes more precisely characterized. Either outcome advances understanding.

The paper also reframes language similarity from a soft preference ("transfer works better for similar languages") to a hard boundary condition on representation sharing. The finding that priming effects completely disappear for Polish-English and Greek-English in the sequential condition, while persisting robustly for Dutch-English and Spanish-English, establishes that language similarity is not merely a matter of degree — it determines whether shared abstract grammatical representations form at all, at least in models of this scale. This has immediate implications for how the field thinks about crosslingual transfer. The dominant practice of starting from a large English-pretrained model and doing continued pretraining for a distant target language implicitly assumes that shared representations can form given enough data, even across large typological distances. This paper suggests that assumption may be false: if the architectural capacity is limited and the languages share insufficient structural overlap, no amount of continued pretraining will create deep grammatical integration. Training on a model in a more similar language (e.g., a Hindi→Nepali transfer chain rather than English→Nepali) may be fundamentally more effective because similarity is a prerequisite for shared representation formation, not just a facilitator.

The paper's third reframing concerns what structural priming actually measures in language models. Prior work (Prasad et al., 2019; Sinclair et al., 2022; Michaelov et al., 2023) demonstrated that LMs exhibit structural priming, but left open whether this reflected genuine abstract grammatical representations or surface-level statistical associations between lexical items across languages. This paper provides the first evidence distinguishing these accounts: priming effects emerge only after the model has acquired non-trivial grammatical knowledge in the second language (measured by BLiMP), and they do not appear during L1-only training despite the model having perfectly good L1 grammar. If priming were a surface-level artifact — e.g., the model learning that certain L1 function words predict certain L2 function words — it could emerge immediately upon L2 exposure, before any abstract grammatical knowledge. The temporal coincidence with BLiMP improvement strengthens the construct validity of structural priming as a measure of abstract grammatical representation in LMs, which in turn strengthens the inference that shared representations genuinely exist and are not just an artifact of probing methodology.

Research directions that become more attractive:

• Controlled bilingual training as a standard methodology for causal claims about multilingual representations. The paper establishes a template — factorial manipulation of language pair × exposure order × training regime, dense checkpointing, behavioral probes — that can be applied to new questions (What role does script similarity play independent of typological similarity? Does word order flexibility alone explain the English-as-target advantage? Do shared representations form differently for morphology vs. syntax?).
• Re-evaluation of human bilingual priming experiments with English as L1 and a typologically distant L2. The paper generates a specific, falsifiable prediction that human experiments should show weaker priming when English is the prime and a morphologically rich, flexible-word-order language is the target.
• Typography-aware multilingual model architectures that selectively share parameters based on linguistic similarity rather than forcing all languages into a single shared representation space. If similarity is a hard constraint, architecture should respect it.

Research directions that become less attractive:

• Purely observational studies of shared representations in large pretrained models without controlled manipulations. This paper demonstrates that the confounds in pretrained models (unknown data mixtures, uncontrolled exposure order) make causal inference impossible, reducing the value of observational probing studies that cannot isolate mechanisms.
• The L1/L2 acquisition order hypothesis as an explanation for asymmetric structural priming in humans. The paper provides a clean alternative explanation (target language properties) that can be tested in humans and, if confirmed, would substantially reduce the theoretical importance of acquisition order in explaining priming asymmetries.
• Uniform continued pretraining from English for all target languages. If similarity constrains the formation of shared grammatical representations, continued pretraining from English for a language like Japanese may be fundamentally limited compared to transfer from a more similar high-resource language (e.g., Chinese→Japanese or Korean→Japanese).

Follow-Up Research This Work Enables

Disentangling script similarity from typological similarity. The current language sample confounds orthography and typology: Greek is the only language with a non-Latin script AND the most typologically distant from English, making it impossible to determine whether the Greek-English priming weakness is driven by the lack of shared subword tokens (orthography) or by genuine typological differences (synthetic passive morphology, flexible word order). A strong follow-up would replicate the paradigm with language pairs that decouple these factors: test Latin-script languages that are typologically distant from English (e.g., Turkish, which uses Latin script but has agglutinative morphology and flexible word order) and non-Latin-script languages that are typologically more similar (e.g., transliterated Russian, which is Indo-European but normally uses Cyrillic; or Serbian, which has both Latin and Cyrillic variants of the same language). If Turkish-English priming is as weak as Greek-English priming despite sharing the Latin script, the effect is genuinely typological. If Latin-script Turkish shows stronger priming than Greek, orthography plays an independent role. This experiment is directly enabled by the paper's controlled training pipeline and would clarify which dimension of "language similarity" actually matters.

Testing the word-order-flexibility hypothesis for the English-as-target asymmetry. The paper speculates that English's relatively fixed word order makes structural repetition more detectable as a priming cue, while languages with flexible word order (Polish, Greek) and rich case marking dilute the priming signal because the "same" construction can appear in multiple linear orders. A direct test would train models on an artificial language pair where word order flexibility is experimentally manipulated: create two "languages" that differ only in whether word order is fixed or free, train bilingual models with both exposure orders, and test whether the fixed-order language consistently shows stronger priming as a target. If the asymmetry reverses when the fixed-order language is not English, the hypothesis is confirmed. If the asymmetry persists with English-like properties removed, something else about English (e.g., its specific lexical statistics or the structure of the stimuli themselves) is driving the effect. This experiment is newly tractable because the paper provides a complete, reproducible training and evaluation pipeline that can be adapted to artificial languages with controlled properties.

Does the sequential-condition priming persistence reflect residual L1 knowledge or genuinely shared representations? The paper finds that Dutch-English and Spanish-English priming survive catastrophic forgetting in the sequential condition, while Polish-English and Greek-English do not. But the surviving priming could reflect either (a) genuinely abstract grammatical representations encoded in parameters that are not overwritten by L2 training, or (b) residual surface-level L1 knowledge that persists despite the rise in mean surprisal. A strong follow-up would use model editing or parameter ablation to distinguish these: identify which parameters are most important for L1 grammatical knowledge (e.g., via probing classifiers or activation patching), selectively ablate or reset them after sequential training, and test whether priming effects survive. If priming persists even when L1-specific parameters are ablated, the representations are genuinely shared across languages in the model's core grammatical circuitry. If priming disappears, it was being driven by residual L1 parameters that happened to survive the distributional shift. The paper's release of all 16 models with dense checkpoints makes this experiment feasible — researchers can compare parameter-level representations before and after the L1→L2 switch.

Scaling up: Do larger models overcome the language similarity constraint? The paper trains only 124M-parameter models and acknowledges that larger models might show different patterns. A critical stress test is whether the language similarity constraint weakens or disappears at larger model scales, where greater capacity might allow the model to represent typologically diverse grammars in ways that still permit crosslingual abstraction. A follow-up would replicate the Dutch-English and Greek-English conditions at 1B, 7B, or 13B parameters (training from scratch) and test whether Greek-English priming becomes more robust in the simultaneous condition, and whether it survives catastrophic forgetting in the sequential condition. If larger models overcome the constraint, the paper's findings are specific to small-scale models and the constraint is about capacity, not about similarity per se. If the constraint persists, it is a fundamental architectural limitation of the Transformer — the representational geometry simply cannot support shared abstractions across sufficiently different grammatical systems, regardless of capacity. This experiment is expensive but directly motivated by the paper's findings and limitations discussion.

What specific grammatical features transfer, and which don't? The paper tests priming for three alternation types (dative, genitive, active/passive) but these are confounded with language pairs (Dutch gets datives and genitives; Spanish, Polish, and Greek get active/passive). A systematic follow-up would test all three alternation types within a single language pair where all three exist — likely Dutch-English, which has comparable dative, genitive, and active/passive alternations — and compare priming magnitudes across alternation types. This would reveal whether some grammatical constructions are inherently more transferable than others (e.g., perhaps word-order-based alternations like datives transfer better than morphology-based alternations like the Greek synthetic passive), which would refine the "language similarity" construct into a more precise account of construction-level transferability. The paper's stimuli, evaluation pipeline, and model training code make this a straightforward extension.

Cross-language contamination in OSCAR as a confound that needs measurement. The paper identifies but does not quantify English contamination in non-English OSCAR data. A rigorous follow-up would run language identification on the actual training data for each language and measure the proportion of non-target-language text (e.g., what percentage of Dutch OSCAR tokens are actually English?). If contamination rates are substantial (e.g., 10-15% English in Dutch data), then the language exposure in the "controlled" training is less controlled than assumed, and some of the English-as-target advantage might be an artifact of models with English as L2 having effectively seen more English than intended. This would not invalidate the paper's core qualitative findings — the within-experiment comparisons would still identify causal effects of the manipulated factors — but it would temper the quantitative interpretation. Measuring and reporting contamination rates would also establish a baseline for future controlled training studies using web corpora.

Practical Applications and Downstream Use Cases

Informing crosslingual transfer strategies for low-resource languages. The paper's finding that language similarity causally constrains the formation of shared grammatical representations has direct implications for how practitioners approach crosslingual transfer. The current default — start from a large English-pretrained model, do vocabulary adaptation and continued pretraining for the target language — implicitly assumes that transfer benefit is proportional to data quantity and training steps, regardless of typological distance. This paper suggests that for a typologically distant target language (e.g., an agglutinative language with flexible word order, or a language with a non-Latin script), this approach may be fundamentally limited: shared grammatical representations simply may not form, regardless of how much continued pretraining is done. Instead, practitioners should consider transfer chains through more similar languages: for a target language like Nepali (Indo-Aryan), continued pretraining from Hindi (also Indo-Aryan, shared script, similar word order) may produce deeper grammatical integration than continued pretraining from English, even if the Hindi model has fewer total parameters or was trained on less data. This recommendation follows directly from the paper's quantitative finding that even within Indo-European, priming effects range from robust (Dutch-English) to essentially absent (Greek-English, sequential condition) depending on similarity, and that similarity effects appear in loss convergence (Figure 2), BLiMP accuracy (Figure F.13), and priming robustness (Figure 3) — converging evidence that similarity is a first-order constraint.

Guiding multilingual model architecture design. The paper implies that the standard multilingual Transformer architecture — a single shared parameter set across all languages — may be suboptimal for typologically diverse language collections. If similarity constrains representation sharing, then forcing all languages into the same representational space may cause interference: the model must simultaneously satisfy conflicting grammatical constraints (e.g., fixed SVO order for English and flexible case-marked order for Polish) within the same parameters, potentially degrading representations for all languages. A better architecture might selectively share parameters based on typological similarity: languages in the same family or with similar word order patterns share transformer layers, while typologically distant languages use separate parameters that interact through a shared bottleneck or cross-attention mechanism. This paper's controlled training paradigm provides a testbed for such architectures — train models with varying degrees of parameter sharing across language pairs of varying similarity, and measure whether selective sharing produces better priming (indicating deeper grammatical integration) than full sharing or full separation. The paper's finding that even closely related languages like Dutch and English show partial (not complete) representational sharing suggests the optimal sharing configuration may be more nuanced than current all-shared or all-separate extremes.

Refining psycholinguistic experimental design. The paper's primary "practical" application is not in engineering but in experimental psychology and linguistics. The finding that the L1/L2 asymmetry may be a target-language effect rather than an acquisition-order effect should prompt re-examination of two decades of human structural priming experiments. Specifically, researchers designing new bilingual priming studies should include conditions where English is the L1 and a morphologically rich, flexible-word-order language is the L2 (e.g., English-dominant learners of Korean, Turkish, or Russian), and test whether the priming asymmetry reverses (stronger when English is the prime and the flexible-order language is the target, or symmetric effects in both directions). If human experiments confirm the paper's prediction, the theoretical interpretation of crosslingual structural priming would need substantial revision — it would become a probe of target-language properties (specifically, how much the target language relies on word order to encode grammatical relations) rather than a probe of bilingual language organization (L1 vs. L2 representational strength). The paper thus serves as a hypothesis-generation engine for human experimentalists, providing specific, falsifiable predictions that can be tested with relatively modest human subject experiments (a few dozen bilingual participants per language pair).

When to Prefer This Method

The paper positions controlled bilingual model training with structural priming evaluation as a methodology for causal hypothesis testing about multilingual representations, explicitly in contrast to two alternatives: (1) probing large pretrained models without controlled training (the predominant approach in prior work, e.g., Michaelov et al., 2023; Conneau et al., 2020) and (2) human bilingual experiments with confounded language exposure histories (Schoonbaert et al., 2007; Bernolet et al., 2013). The choice between these methods depends on the research question:

Prefer controlled bilingual model training when:

• The research question requires causal manipulation of language exposure (order, quantity, timing, similarity) that is impossible or unethical with human participants.
• The goal is to generate or refine hypotheses about human bilingual representation that can later be tested in human experiments.
• The phenomenon of interest (e.g., shared grammatical representations) has a well-defined behavioral signature (structural priming) that can be measured without access to internal model states.
• Training data quantity and composition must be precisely controlled to eliminate confounds present in existing pretrained models.

Prefer probing large pretrained models when:

• The research question concerns whether a phenomenon exists in models at deployment scale, rather than why it exists or what factors causally influence it.
• The goal is to characterize the behavior of a specific production model that users interact with, without manipulating its training.
• The computational cost of training models from scratch under multiple conditions is prohibitive (this paper trained 16 models at 512 GPU-hours; scaling to larger models or more conditions would multiply this cost).
• The research question does not require causal inference — e.g., surveying whether structural priming is present across many language pairs in many existing models, without needing to isolate why some pairs show stronger effects.

Prefer human bilingual experiments when:

• The research question concerns actual human cognitive architecture, and model-based results serve only as a source of hypotheses to be validated or falsified in humans.
• The population of interest (e.g., simultaneous bilinguals from birth, heritage speakers with specific proficiency profiles) can be recruited and tested, and the confounds this paper identifies (English-as-target vs. L1/L2 status) can be controlled through participant selection (e.g., testing English L1 / Korean L2 speakers to reverse the standard confound).
• The richness of human behavior — including production latencies, error patterns, developmental trajectories, and interaction with non-linguistic cognition — is central to the question, and current language models cannot approximate these dimensions.

The paper's primary contribution is not a new model or a new evaluation metric, but a methodological template for using controlled language model training to perform causal experiments about multilingual representation. The choice to use this template depends on whether the research question demands causal inference with controlled language exposure — when it does, the template is uniquely powerful because no other approach (probing existing models, testing human participants) can simultaneously manipulate language experience and measure representational outcomes with the precision and control demonstrated here.