Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)

ArXiv: 1905.11833

🎯 Pitch

Brains can debug neural language models: giving shallow BERT layers a uniform attention pattern—a tweak inspired by fMRI recordings—actually boosts syntactic performance over the original. But apply the same change to deep layers and it backfires, revealing that the optimal inductive bias depends on layer depth.

1. Executive Summary

This paper proposes a novel interpretation approach that uses human brain imaging recordings—fMRI and MEG data from subjects reading natural text—to analyze the intermediate representations learned by NLP models, establishing a framework where brain activity serves as a "demultiplexer" that decomposes network representations into interpretable components corresponding to different language processing functions. Studying four recent models—ELMo, BERT, USE, and Transformer-XL—the work investigates how representations differ across layer depth, context length, and attention type, with the headline finding that replacing learned attention with uniform attention in BERT's shallow layers (layers 1–6) improves brain alignment, and that this brain-aligned BERT variant outperforms the original model on syntactic NLP probe tasks in 8 of 13 cases, establishing that insights from brain recordings can transfer to improve language understanding in machines—but only when the brain-derived modification is applied to the appropriate network layers, as the same uniform-attention alteration harms performance when applied to deep layers.

2. Context and Motivation

The Core Problem: We Don't Know What Deep NLP Models Actually Represent

The fundamental puzzle this paper addresses is deceptively simple: deep neural networks for NLP achieve remarkable performance without being programmed with any explicit knowledge of linguistic structure, yet we have limited understanding of what linguistic information they actually encode in their intermediate representations. This matters because—unlike traditional NLP systems where features were hand-engineered based on linguistic theory—modern neural models learn representations that are opaque. We can measure their downstream task accuracy, but that doesn't tell us what about language they've captured or how that information is organized across the network's layers.

This gap is significant for several practical and theoretical reasons the authors identify:

Engineering better models: If we don't understand what current models learn, we can't systematically improve them. The dominant approach is trial-and-error architecture search and hyperparameter tuning, guided only by downstream task metrics. A richer understanding of the representations themselves could provide more principled design guidance—for example, telling us which layers need what kind of attention, or how deep a network needs to be to capture long-range dependencies.
Trust and interpretability: As NLP models are deployed in higher-stakes applications (medical text processing, legal document analysis, content moderation), knowing what linguistic information they represent becomes a safety issue. Can a model distinguish syntactic structure from surface statistics? Does it represent semantic meaning or merely distributional co-occurrence? These questions cannot be answered by benchmark scores alone.
Scientific understanding of language: Neural networks that learn language from text alone provide a potential model system for studying what linguistic structure can be acquired from distributional statistics. Understanding their representations can inform debates in linguistics and cognitive science about the learnability of different aspects of language.
The pretraining-finetuning paradox: The authors note an intriguing observation: "word embeddings and language models trained on a large generic corpus and then optimized for downstream NLP tasks produce even better results than training the entire model only to solve this one task." This suggests these models capture something "generic about language" during pretraining. But what is that generic something? The question is not merely academic—if we knew which aspects of linguistic structure emerge during pretraining, we could design better pretraining objectives.

Where Existing Interpretation Approaches Fall Short

The paper identifies specific limitations in the interpretability tools available at the time of writing (2019), which create a need for the brain-based approach:

Synthetic NLP probe tasks capture only what we think to test for. A dominant paradigm for interpreting neural network representations involves designing "probe tasks"—simplified NLP classification problems that test whether specific linguistic properties (part-of-speech, syntactic depth, semantic role) are decodable from a layer's representations. The authors cite work by Conneau et al. (2018), Zhu et al. (2018), and Linzen et al. (2016) as exemplars of this approach. While informative, this methodology has a fundamental limitation: it can only detect linguistic properties that researchers explicitly design tasks for. Complex and "arguably more interesting aspects of language, such as high level meaning," are "difficult to capture in an NLP task or in behavioral measures." The probe task approach is inherently bounded by our imagination and by our ability to formalize linguistic phenomena into classification problems. If a network represents some aspect of language that we haven't thought to test for—or that we don't know how to formalize—probe tasks will miss it entirely.

Theoretical analyses are limited to specific architectures. A complementary line of work (Peng et al., 2018; Chen et al., 2017; Weiss et al., 2018) attempts theoretical characterization of what recurrent networks can represent—for instance, analyzing the formal language recognition capabilities of RNNs with finite precision. The authors acknowledge this work but note two limitations: first, it has been "centered around understanding the properties of sequential models such as LSTMs and RNNs, with considerably less work focused on non-sequential models such as transformers," which were the dominant architecture by 2019. Second, theoretical capacity results (what a network can represent) don't directly tell us what a trained network does represent in practice after gradient-based optimization on real data.

Behavioral measures are coarse. Some prior work attempted to evaluate whether word embeddings "contain cognition-relevant semantics" by measuring how well they predict eye-tracking data or fMRI recordings (Søgaard, 2016). However, these approaches typically evaluate the overall quality of a single word embedding—they don't provide the kind of layer-by-layer, context-length-by-context-length decomposition that would reveal how information is organized across a deep network, or which layers encode what aspects of language.

The fundamental limitation: no external ground truth for linguistic representation. All of the above approaches share an underlying constraint: they evaluate network representations against hypotheses that researchers formulate and operationalize. There is no external, independently validated benchmark for what counts as a "good" linguistic representation. Probe tasks can tell us whether information is present, but they cannot tell us whether the network's organization of that information is natural, efficient, or human-like. The paper's key insight is that the human brain provides an existence proof of a system that does represent complex linguistic information during language processing—and brain recordings offer a window into those representations without requiring us to fully specify in advance what we're looking for.

Why the Brain? The Demultiplexer Analogy

The paper's core motivation for using brain data is not that brain-like representations are inherently desirable (though Section 5 suggests they may be), but rather that brain recordings provide a uniquely structured interpretability signal. The authors articulate this through the "demultiplexer" analogy (Section 1.1):

"In the brain, different processes are assigned to specific locations as has been revealed by a large array of fMRI experiments. These processes have specific latencies and follow a certain order, which has been revealed by electrophysiology methods such as MEG. In contrast to the brain, a network-derived representation might encode information that is related to multiple of these processes without a specific organization. When we align that specific network representation with fMRI and MEG data, the result will be a decomposition of the representation into parts that correspond to different processes and should therefore be more interpretable."

This is the intellectual move that distinguishes the paper from prior work. Rather than designing a probe task (which requires specifying what to probe for), the authors propose using the brain's own functional organization as a "readout" that decomposes network representations according to biological language processing pathways. A network layer that predicts activity in visual cortex (areas 1a) is likely encoding orthographic or low-level features. A layer that predicts activity in anterior temporal regions thought to process multi-word composition (group 2) is likely encoding long-range contextual information. The brain provides what the authors call a "data-driven" prior—based on experimental results from naturalistic language studies, not on theoretical commitments to particular linguistic frameworks.

This is a departure from the dominant direction of interaction between NLP and neuroscience at the time. As the authors note, prior work typically used NLP models to study the brain (e.g., Wehbe et al., 2014a; Frank et al., 2015; Hale et al., 2018; Jain and Huth, 2018—using neural network representations as feature spaces to model brain activity), not to study the NLP models themselves. The paper explicitly reframes the relationship:

"Evaluating neural network representations with brain activity is a departure from existing studies that go the other way, using such an alignment to instead evaluate brain representations."

This is what the authors mean by "closing the loop"—using neuroscience to improve NLP, rather than only using NLP to study neuroscience.

The Proof-of-Concept Gap

The paper is motivated partly by a missing empirical demonstration. While the idea of using brain data to interpret NLP models has conceptual appeal, the authors need to establish that the approach can recover known linguistic information before using it to discover unknown information. This is why Section 4 of the supplementary materials (and Figure 8) serves as a "proof of concept": the authors demonstrate that their method can detect that ELMo's word embedding contains information about word length and part-of-speech—properties we already know word embeddings encode—by showing that the embedding and explicit word-length/POS features predict shared variance in brain activity at the expected times and locations (word length shared in visual cortex ~100ms after word onset; POS shared in frontal sensors ~200ms after word onset). This validation step is crucial because it establishes that the brain-alignment method produces interpretable, non-spurious results before the authors apply it to more novel questions about contextual representations, layer depth, and attention mechanisms.

How This Paper Positions Itself

The paper positions its contribution at the intersection of three research traditions:

NLP interpretability: It offers a new tool for probing network representations that doesn't require designing explicit linguistic probe tasks, addressing the "you only find what you look for" limitation of prior work.
Cognitive neuroscience of language: It provides a framework for comparing artificial and biological language processing, but inverts the typical direction of investigation—using the brain to evaluate models rather than using models to evaluate the brain.
Model improvement: It goes beyond interpretation to demonstrate that brain-alignment insights can directly inform model modification that improves NLP task performance (Section 5), establishing a practical pathway from interpretability to better models.

The paper does not claim that brain alignment is the only or the best way to interpret NLP models. Rather, it argues that brain activity provides complementary information to existing probe tasks and theoretical analyses—information that is particularly valuable for studying high-level semantic and contextual phenomena that are difficult to operationalize in classification tasks. The authors are careful to acknowledge limitations (the "Intricacies" box in Section 1.1 warns about reverse inference fallacies and the dangers of directly comparing performance across network architectures), positioning the work as an initial demonstration of feasibility and utility rather than a comprehensive solution.

3. Technical Approach

3.1 Reader Orientation

The paper constructs an evaluation framework, not a trainable system—it takes a frozen, pre-trained NLP model and passes the same natural text through both the model and human subjects, then learns linear mappings from the model's internal representations to the subjects' brain activity, using the quality of those mappings to interpret what linguistic information the model's layers encode. The core problem it solves is the opacity of neural network representations: because NLP models learn without explicit linguistic rules, we don't know what aspects of language (orthographic, syntactic, semantic, contextual) are encoded in which layers—and the "shape" of the solution is to use the brain's own functionally-specialized regions as a biological readout that decomposes the model's entangled representations into interpretable components, without requiring researchers to design explicit probe tasks.

3.2 Big-Picture Architecture (Diagram in Words)

The framework has five major components connected in a feed-forward evaluation pipeline:

Brain Data Acquisition (fMRI and MEG) — Pre-recorded brain activity from human subjects reading Chapter 9 of Harry Potter and the Sorcerer's Stone one word at a time (0.5s per word), producing voxel-level fMRI time series (8 subjects, 2s TR) and sensor-level MEG time series (3 subjects, 1kHz sampling, downsampled to 25ms bins). This component provides the ground-truth "labels" against which model representations are evaluated.
Network Representation Extraction — The same text is fed word-by-word into pre-trained NLP models (ELMo, BERT, USE, Transformer-XL), and for each word $w_n$ , the intermediate activation at each layer $\ell$ is extracted given a context window of $k$ preceding words $(w_{n-k+1}, ..., w_n)$ , producing a vector $\mathbf{x}^{\ell,k}$ per word per layer per context length. This component generates the "features" to be mapped to brain activity.
Encoding Model Training (Ridge Regression) — For each layer representation $\mathbf{x}^{\ell,k}$ and each brain measurement channel (voxel in fMRI, sensor/timepoint tuple in MEG), a separate linear model $\mathbf{w}$ is trained via ridge regression to predict the brain signal from the network representation, with the regularization parameter $\lambda$ chosen independently per output dimension via nested 10-fold cross-validation. This component provides a quantitative measure of how much each layer's information overlaps with each brain region's processing.
Classification-Based Evaluation — The trained encoding model's predictions are evaluated through a two-alternative forced-choice classification task: given a chunk of real brain data (20 consecutive TRs for fMRI, 20 randomly sampled words for MEG), the classifier must identify which of two candidate prediction chunks (the correct one vs. a randomly selected incorrect one) matches the real data, with accuracy averaged over many random trials per measurement channel. This component produces a spatially- and temporally-resolved map of prediction accuracy.
Statistical Significance and Interpretation — Voxels/sensors with above-chance classification accuracy are identified via a false-discovery-rate (FDR) controlled thresholding procedure, results are rendered on a standardized brain template (MNI space) for cross-subject averaging, and interpreted against a data-driven prior that partitions language-relevant brain regions into two groups: Group 1 (regions activated by both isolated words and connected text, covering inferior frontal gyrus and middle/superior temporal cortex) and Group 2 (regions activated only by connected multi-word text, covering lateral frontal, supramarginal/angular gyrus, precuneus, medial superior frontal, and medial orbito-frontal cortex).

Information flows linearly: Text → [Brain] produces brain recordings → [NLP Model] produces layer representations $\mathbf{x}^{\ell,k}$ → [Encoding Models] learn $\mathbf{w}$ maps → [Classification] produces accuracy maps → [FDR Thresholding] identifies significantly predicted regions → [Prior-Based Interpretation] decomposes what each layer encodes.

3.3 Roadmap for the Deep Dive

First, the formal encoding model setup—how exactly a layer representation $\mathbf{x}^{\ell,k}$ is mapped to brain activity, including the fMRI-specific handling of hemodynamic lag and the MEG-specific absence of temporal delay—because this mapping is the quantitative engine behind every subsequent analysis.
Second, the classification-based evaluation protocol and the statistical significance procedure—because the raw regression predictions are too noisy to interpret directly, and the way the authors handle multiple comparisons and cross-subject aggregation determines what claims can be made.
Third, the construction of the brain prior (Groups 1 and 2) and how it serves as a data-driven functional decomposition—because this is what replaces hand-designed probe tasks and enables interpretation without specifying linguistic properties in advance.
Fourth, the four NLP models under study (ELMo, BERT, USE, Transformer-XL) and how their representations are operationalized—because understanding what $\mathbf{x}^{\ell,k}$ means concretely for each architecture is necessary to interpret the results.
Fifth, the specific experimental manipulations—varying context length $k$ , varying layer depth $\ell$ , and replacing learned attention with uniform attention in single BERT layers—because these controlled interventions are what allow the paper to make causal claims about what information each architectural component contributes.
Sixth, the proof-of-concept validation using MEG and known word-level properties (length, part-of-speech)—because this establishes that the methodology recovers known information before it is applied to unknown aspects of contextual representations.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a framework and empirical analysis paper whose core idea is that brain recordings provide a biologically-grounded, spatially-resolved, and temporally-resolved decomposition of what NLP model representations encode—a decomposition that does not require researchers to specify in advance which linguistic properties to test for, and which can discover organizational principles (such as the benefit of uniform attention in shallow BERT layers) that would not emerge from standard probe tasks.

The Encoding Model: Mapping Network Representations to Brain Activity

The fundamental operation in the paper is learning a function $f$ such that $f(\mathbf{x}^{\ell,k}) = y$ , where $\mathbf{x}^{\ell,k}$ is the representation extracted from layer $\ell$ of an NLP model given the $k$ most recent words, and $y$ is the brain activity recorded when a human subject read those same $k$ words. This is a brain encoding model—a predictive model that estimates what brain activity a given stimulus should evoke, rather than a decoding model (which would go in the opposite direction, predicting the stimulus from brain activity).

The authors model $f$ as a linear function regularized by ridge penalty. This choice is not arbitrary—it reflects a deliberate tradeoff between expressiveness and interpretability. A linear mapping ensures that if a network representation predicts brain activity in a given region, the relationship is direct (a weighted combination of representation dimensions), making it interpretable: we can say the representation contains information that the brain region processes. A nonlinear mapping could achieve high prediction accuracy even if the relationship were complex and opaque, which would defeat the purpose of using the brain as an interpretability tool. The authors cite prior work showing that with proper per-voxel regularization, "different regularization techniques lead to similar results" for fMRI data (Wehbe et al., 2015), and ridge regression is "a common regularization technique used for building predictive fMRI" (citing Mitchell et al., 2008; Nishimoto et al., 2011; Wehbe et al., 2014b; Huth et al., 2016).

For fMRI data, the encoding model requires special handling because of the hemodynamic lag—fMRI measures blood-oxygen-level-dependent (BOLD) signal, which is a slow, indirect consequence of neural activity that peaks approximately 6 seconds after stimulus onset. The authors handle this by:

Grouping words by repetition time (TR): Words presented within the same 2-second fMRI acquisition window are grouped together, and their layer representations are averaged to form feature vectors $\mathbf{x}_t^\ell$ aligned with each TR index $t$ .
Concatenating temporal context: To account for the delayed and sustained hemodynamic response, the feature vector at time $t$ is not just $\mathbf{x}_t^\ell$ but the concatenation of the current and three previous TRs: $\mathbf{z}_t^\ell = [\mathbf{x}_{t-1}^\ell, \mathbf{x}_{t-2}^\ell, \mathbf{x}_{t-3}^\ell, \mathbf{x}_{t-4}^\ell]$ . This means that at each TR, the model predicts the BOLD signal using the network's representations of words presented over the preceding ~8 seconds (4 TRs × 2 seconds each), which approximately spans the hemodynamic response function.

The regression for each fMRI voxel $i$ estimates a weight vector $\mathbf{w}^{i,\ell}$ by solving:

$\arg \min_{\mathbf{w}^{i,\ell}} ||\mathbf{y}^{R,i} - \mathbf{Z}^{R,\ell}\mathbf{w}^{i,\ell}||_2^2 + \lambda_i ||\mathbf{w}^{i,\ell}||_2^2$

where $\mathbf{y}^{R,i} = [y_1^i, y_2^i, ..., y_{n_R}^i]$ is the fMRI signal time series for voxel $i$ in the training fold (a vector of length $n_R$ , the number of TRs in the training set), $\mathbf{Z}^{R,\ell} = [\mathbf{z}_1^\ell, \mathbf{z}_2^\ell, ..., \mathbf{z}_{n_R}^\ell]^\top$ is the design matrix of concatenated layer- $\ell$ features for the training fold (dimensions $n_R \times 4d$ , where $d$ is the dimensionality of the layer representation), and $\lambda_i \geq 0$ is a voxel-specific regularization parameter.

What it computes: This equation finds the linear combination of network features (across the current and three previous time windows) that best predicts the BOLD signal in a single voxel, with an L2 penalty on the weight magnitudes to prevent overfitting. The squared error term $||\mathbf{y}^{R,i} - \mathbf{Z}^{R,\ell}\mathbf{w}^{i,\ell}||_2^2$ measures how far the model's predictions are from the actual fMRI signal; the penalty term $\lambda_i||\mathbf{w}^{i,\ell}||_2^2$ shrinks all weights toward zero, with $\lambda_i$ controlling the shrinkage strength. The output is a vector $\mathbf{w}^{i,\ell}$ of learned coefficients (one per feature dimension × 4 temporal lags) that can be applied to new feature vectors to predict unseen brain activity.

Why this form: Ridge regression is preferable to ordinary least squares here because the number of features (768 for BERT × 4 lags = 3,072 dimensions) is large relative to the number of training examples (~1,100 TRs in a training fold, given ~45 minutes of data at 2s TR ≈ 1,350 total TRs, split into 4 folds). Without regularization, the model would overfit, producing weights that fit training noise rather than genuine brain-representation relationships—which would yield artificially inflated classification accuracy during evaluation. The L2 penalty is specifically appropriate because it distributes predictive power across many correlated features (rather than selecting a sparse subset as L1 would), which matches the expectation that many dimensions of a distributed neural network representation jointly contribute to predicting a given brain region's activity (brain regions themselves process information in a distributed, not sparse, manner). The per-voxel $\lambda_i$ is essential because different voxels have different signal-to-noise ratios—a voxel in highly vascularized tissue will have more physiological noise than one in well-isolated gray matter, and pooling a single $\lambda$ across voxels would under-regularize noisy voxels and over-regularize clean ones.

The choice of $\lambda_i$ for each voxel is made via 10-fold nested cross-validation on the training fold: the training data is further split into sub-folds, a range of $\lambda$ values is evaluated, and the value that minimizes validation error is selected before fitting the full training fold. This ensures that the regularization strength is optimized without touching the held-out test fold, which would contaminate the final evaluation.

For MEG data, the encoding model is structurally identical (ridge regression, per-channel $\lambda$ , nested CV) but differs in two key ways:

No temporal lag: MEG measures changes in magnetic fields outside the skull, which are direct, instantaneous consequences of neural currents. There is no hemodynamic lag to compensate for. The model therefore predicts the MEG signal at sensor $i$ and timepoint $\tau$ after word onset using only the current word's representation $\mathbf{x}_t^\ell$ , not a concatenation of previous words: $\arg \min_{\mathbf{w}^{(i,\tau),\ell}} ||\mathbf{y}^{(i,\tau),R} - \mathbf{X}^{R,\ell}\mathbf{w}^{(i,\tau),\ell}||_2^2 + \lambda_{(i,\tau)}||\mathbf{w}^{(i,\tau),\ell}||_2^2$ , where $\mathbf{X}^{R,\ell}$ is the design matrix of word-level layer- $\ell$ representations for the training fold.
Per-timepoint modeling: A separate encoding model is trained for each of the 20 timepoints (25ms bins spanning 0–500ms after word onset) at each sensor location, for a total of 306 sensors × 20 timepoints = 6,120 encoding models per layer. This high temporal resolution is what enables the proof-of-concept demonstration—the authors can show that a word embedding's shared variance with word length emerges ~100ms post-onset (when visual information reaches cortex) and its shared variance with part-of-speech emerges ~200ms post-onset in frontal sensors (matching known electrophysiology results).

The Classification-Based Evaluation and Statistical Significance Procedure

The raw squared error from an encoding model is not directly interpretable as a measure of "how well the layer predicts the brain." Different voxels have vastly different noise levels; a voxel in primary auditory cortex (which shouldn't be involved in reading) might have lower raw error than one in language cortex simply because its signal variance is lower, not because the model predicts it better. The authors therefore adopt a classification-based evaluation that normalizes performance relative to a chance baseline and pools information across neighboring measurement locations to increase signal-to-noise ratio.

fMRI classification task. For each voxel $i$ and layer $\ell$ , the encoding model (trained on training folds) produces predicted fMRI time series $\mathbf{p}^{i,\ell}$ for the held-out validation fold. The classification task operates on the validation fold as follows:

A cortical-searchlight neighborhood $\mathcal{N}^i$ of size $k_i$ is defined around voxel $i$ —but critically, this neighborhood is computed on the reconstructed cortical surface, not the 3D volumetric grid. The cortical sheet is highly folded (to fit a large surface area inside the skull), and voxels that are adjacent on the cortical surface may be far apart in 3D space (e.g., voxels on opposite banks of a sulcus). The authors reconstruct each subject's cortical surface using Freesurfer (Fischl, 2012) and define neighbors as voxels within a two-step adjacency on this surface (the immediate cortical neighbors, plus their immediate neighbors). This ensures that the neighborhood pools signals from a functionally coherent patch of cortex rather than an arbitrary 3D ball that might mix gray matter from different gyri or include white matter.
A chunk of 20 consecutive TRs is selected from the real validation data for voxels in $\mathcal{N}^i$ , producing a real data matrix of size $20 \times k_i$ (20 timepoints × $k_i$ neighboring voxels).
Two candidate prediction matrices of the same size are constructed: one is the encoding model's predictions for those exact 20 TRs in those voxels (the correct candidate), and the other is the encoding model's predictions for a randomly selected 20-TR chunk from elsewhere in the validation fold (the incorrect candidate).
The Euclidean distance is computed between the real data matrix and each candidate prediction matrix. The candidate with the smaller Euclidean distance is the classifier's guess. Average accuracy is computed over many random draws of chunks and incorrect candidates.

The authors choose 20 TRs (40 seconds of real time, covering roughly 80 words at 0.5s/word) because "classifying 20 TRs at a time" improves the signal-to-noise ratio: "fMRI data is noisy" and "performance using a single TR will be close to chance accuracy and will therefore have low power." They cite Wehbe et al. (2014b) showing that accuracy plateaus after ~15 TRs, so 20 TRs is a conservative choice.

Statistical significance via empirical FDR control. The classification task produces, for each voxel in each subject, an average accuracy between 0 and 1 (chance = 0.5 for the two-alternative task). The question is: which accuracies are "significantly above chance" such that we can conclude the layer $\ell$ genuinely carries information about that voxel's processing? The authors use a novel empirical method based on the Barber-Candès knockoff framework (Barber et al., 2015; Arias-Castro et al., 2017; Rabinovich et al., 2017) adapted to control the False Discovery Rate (FDR) at level $q = 0.05$ .

The key insight is that chance accuracies are symmetrically distributed around 0.5. The set of voxels with accuracy $\leq 0.5 - \delta$ (for some margin $\delta$ ) can be used to estimate the distribution of chance accuracies above $0.5 + \delta$ . The procedure finds the smallest $\delta$ such that:

$\widehat{\text{FDP}} = \frac{1 + \#\{ \text{voxels s.t. accuracy} \leq 0.5 - \delta\}}{1 \lor \#\{ \text{voxels s.t. accuracy} \geq 0.5 + \delta\}} \leq q$

where $\lor$ denotes the maximum operator (to prevent division by zero if no voxels exceed the threshold). The "1 +" in the numerator is a finite-sample correction. Starting at $\delta = 0.001$ , the procedure increments $\delta$ until the estimated False Discovery Proportion falls below $q = 0.05$ or a limit is reached. All voxels with accuracy $\geq 0.5 + \delta_{\text{final}}$ are declared significantly predicted.

Cross-subject aggregation. Each subject's brain has different anatomy, so individual-subject significance maps cannot be directly overlaid. The authors transform each subject's results to the Montreal Neurological Institute (MNI) template space—the standard coordinate system in human neuroimaging—using pycortex (Gao et al., 2015), then average the thresholded significance maps across subjects. For the qualitative comparisons in Figures 2 and 3, they report "only significantly predicted voxels for each of the 8 subjects" with FDR controlled at 0.05.

MEG classification task. The MEG evaluation follows the same logic but pools spatially across the 306 sensors to 102 sensor locations (each location has 3 sensor types measuring orthogonal magnetic field components) and pools across subjects to increase SNR. The classification uses chunks of 20 words (rather than 20 TRs), following the value from Wehbe et al. (2014a). The critical output is a classification accuracy map at 102 sensor locations × 20 timepoints for each layer $\ell$ , showing when and where the layer's information is relevant to brain processing.

The Data-Driven Brain Prior: Groups 1 and 2

The linchpin of the paper's interpretability framework is that specific brain regions have been functionally characterized through prior experiments using naturalistic language stimuli, providing a data-driven prior on what information those regions process. The authors deliberately avoid theory-driven priors ("Because we don't know which of the existing theories are correct, we abandon the theory-based approach") and instead rely on the results of Lerner et al. (2011), who presented subjects with stimuli at different levels of linguistic complexity—disconnected words, sentences, and paragraphs—and measured which brain regions showed consistent activation across levels.

The prior divides the language network into two groups:

Group 1: Regions that are "consistently activated across subjects when they listen to disconnected words or to complex fragments like sentences or paragraphs." These regions process information at the level of both isolated words (short-range context) and multi-word composition (long-range context). They include two subdivisions:
- Group 1a: Inferior Frontal Gyrus
- Group 1b: Middle/Superior Temporal Gyrus
Group 2: Regions that are "consistently activated only when they listen to complex fragments"—i.e., they require multi-word sequences and are not reliably activated by lists of disconnected words. These regions are specialized for long-range context processing (multi-word composition, syntactic integration, discourse-level meaning). They include five subdivisions:
- Group 2a: Lateral Middle/Superior Frontal
- Group 2b: Supramarginal Gyrus / Posterior Superior Temporal / Angular Gyrus
- Group 2c: Precuneus
- Group 2d: Medial Superior Frontal
- Group 2e: Medial Orbito-Frontal

The locations are "approximated" on the MNI template as shown in Figure 1 of the main text and detailed in Supplementary Table 2.

How the prior enables interpretation. If a network layer representation predicts brain activity predominantly in Group 1 regions but not Group 2 regions, we can infer that the layer encodes information relevant to individual word processing (orthographic, phonological, or isolated semantic features) but does not capture information specific to multi-word composition. If a layer predicts activity in Group 2 regions, we can infer it encodes information about relationships between words—syntactic structure, discourse coherence, compositional semantics, or other forms of context integration. The spatial resolution of fMRI (2–3mm) allows this functional segregation to be read out: different layers can be compared not just on "overall prediction accuracy" but on where in the brain they predict activity, providing a decomposition that probe tasks would require designing many separate experiments to achieve.

The authors note that inspection of Jain and Huth (2018) corroborates this division, providing convergent evidence from a different dataset and experimental paradigm.

Why data-driven rather than theory-driven. The authors explicitly note that "there doesn't yet exist a unique theory of how the brain processes language that researchers agree upon" (citing Hickok and Poeppel, 2007; Friederici, 2011; Hagoort, 2003). Competing theories make different claims about which regions perform syntactic vs. semantic processing, whether processing is strictly hierarchical or interactive, and how early vs. late processing stages are organized. By relying on a purely empirical division—based on which regions respond differently to different stimulus classes in a naturalistic experiment—the authors avoid committing to any particular theoretical framework. The cost is a coarser functional characterization (Group 1 = "processes isolated words and word sequences"; Group 2 = "processes only word sequences") but the benefit is that the prior is uncontroversial and robust across theoretical positions.

Operationalizing NLP Model Representations

The paper examines four models, each of which produces a token-level contextualized representation in a different way. For each word $w_n$ in the input text, the models produce a vector that depends on the preceding context $(w_{n-k+1}, ..., w_n)$ . The context length $k$ is an experimental variable the authors manipulate to study how much context each model integrates.

ELMo (Peters et al., 2018). ELMo is a bidirectional language model with 2 LSTM layers. The contextualized representation for a word is the concatenation of the input (non-contextualized) token embedding and the hidden states from both LSTM layers at that word position. The authors use the pretrained model from Gardner et al. (2017), with $\ell \in \{1, 2\}$ (the two LSTM layers). The non-contextualized token embedding is also used as a "word embedding" baseline.

BERT (Devlin et al., 2018). BERT is a bidirectional transformer pretrained on masked language modeling and next-sentence prediction. The authors use the "base" BERT model (12 layers, 12 attention heads per layer, 768 hidden units) from Hugging Face. Layer representations $\mathbf{x}^\ell$ for $\ell \in \{1, ..., 12\}$ are the hidden states of the transformer layer at the target word position. The token embeddings (layer 0, before any transformer layers) are also accessible.

USE (Universal Sentence Encoder; Cer et al., 2018). USE is a sentence-level encoder based on a deep averaging network that produces fixed-length embeddings for texts of arbitrary length (words, sentences, paragraphs). The authors use the version from TensorFlow Hub with 512 dimensions. Critically, USE has only a single output representation (no intermediate layers), and it operates at the full-sentence level—it does not expose token-level representations across different context windows. This architectural constraint is itself informative: the authors can compare USE's sentence-level representations against token-level representations from the other models.

Transformer-XL (T-XL; Dai et al., 2019). T-XL augments the standard transformer with segment-level recurrence: when processing long sequences, the hidden states from the previous segment are cached and reused as an extended context for the current segment (with gradients stopped across segments, making it a state-reuse mechanism rather than full backpropagation-through-time). The authors use a pretrained model from Hugging Face with 19 layers and 1024 hidden units. Layer representations $\mathbf{x}^\ell$ for $\ell \in \{1, ..., 19\}$ are computed similarly to BERT.

Context window manipulation. For each model, the authors feed the $k$ most recent words $(w_{n-k+1}, ..., w_n)$ through the network to obtain the representation for the current word $w_n$ . By varying $k$ (from 1—the current word only—up to 40 words for the longer-context experiments), the authors can study how much context each layer integrates. For BERT, the input is the full sequence of $k$ tokens, and the output at position $k$ (the last token) is taken as the representation. For T-XL, the internal recurrence mechanism means that even if $k$ is the explicit window size, the model may still retain information from earlier segments through its cached hidden states—a design feature the authors explicitly investigate.

Why these four models. The selection is deliberate: ELMo represents the LSTM-based recurrent paradigm (sequential processing, theoretically unlimited context but practical limitations from vanishing gradients), BERT represents the pure transformer paradigm (parallel processing, fixed context window), T-XL represents a hybrid approach (transformer with recurrence to extend effective context), and USE represents a sentence-embedding approach (no token-level processing, purely compositional). Together, they span the major architectural families in NLP at the time of writing and allow the authors to investigate whether findings (e.g., "middle layers are best for context integration") are architecture-specific or general.

Experimental Manipulations: Context Length, Layer Depth, and Attention

The paper's core analytical strategy is to perform controlled within-model experiments, varying one factor at a time while holding others fixed, to isolate how specific architectural components affect the information encoded in representations. The authors explicitly warn against cross-model comparisons: "Another possible fallacy is to directly compare the performance of layers from different networks and conclude that one network performs better than the other: information is likely organized differently across networks and such comparisons are misleading." All substantive claims come from within-model manipulations.

Manipulation 1: Context length. For ELMo, BERT, and T-XL, the authors sweep the number of context words $k$ provided to the model and measure how the encoding model's prediction performance changes. The context lengths tested range from 1 word (the current word only—a pure word embedding) to 40 words. This manipulation reveals how much context a given layer integrates. If a layer's brain prediction accuracy plateaus at $k = 5$ words, the layer only captures local context; if it continues to improve up to $k = 40$ , the layer integrates long-range information.

For the initial qualitative comparison (Figures 2 and 3), the authors define two specific representations per model:

Word embedding: The representation of the most recent word only ( $k = 1$ , or the input token embedding for models with explicit token embeddings).
10-word representation: The representation of the 10 most recent words, taken from a middle layer (layer 1 for ELMo's 2-layer architecture, layer 7 for BERT's 12-layer architecture, layer 11 for T-XL's 19-layer architecture). The choice of middle layers is hypothesis-driven: preliminary experiments presumably indicated that deeper layers behaved differently, and the paper's later results confirm that middle layers are optimal for context integration.

By comparing the brain regions predicted by the word embedding vs. the 10-word representation within each model, the authors can identify which brain areas are sensitive to multi-word context (Group 2) vs. which process information already present in single-word embeddings (Group 1b).

Manipulation 2: Layer depth. For ELMo, BERT, and T-XL, the authors train separate encoding models for every layer $\ell$ in the network and compare their brain prediction performance at different context lengths. This produces the layer-depth × context-length interaction plots (Figure 4 of the paper), which reveal:

Whether information is organized hierarchically (e.g., lower layers predict primary sensory regions, higher layers predict association cortex)
Whether there is an optimal layer depth for context integration (the authors find middle layers perform best for long contexts)
Whether the deepest layers behave differently from intermediate layers (the authors find a characteristic "sharp increase at short context followed by decrease" pattern in the deepest layers of all three models)

Manipulation 3: Attention type (BERT only). The authors perform a causal intervention on BERT's attention mechanism: for a single layer $\ell$ at a time (keeping all other layers at their pretrained weights), they replace the learned multi-head attention with uniform attention over the previous layer's outputs. This means that instead of computing attended values as weighted combinations where some positions are upweighted and others downweighted, every position in the previous layer contributes equally to the current layer's representation.

The technical implementation: For each attention head $i$ in layer $\ell$ , the standard attention computation is:

$\text{Attn}_i(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$

where $Q = XW_i^Q$ , $K = XW_i^K$ , $V = XW_i^V$ are linear projections of the input $X$ (the previous layer's outputs), and $d_k$ is the dimensionality of the key vectors. The learned parameters are the projection matrices $W_i^Q$ , $W_i^K$ , and $W_i^V$ .

To achieve uniform attention, the authors replace $W_i^Q$ and $W_i^K$ with zero-filled matrices and replace $W_i^V$ with the identity matrix. When $Q$ and $K$ are zero vectors, the dot products $QK^\top$ are all zero, and the softmax over zeroes produces a uniform distribution (each position gets equal probability $1/n$ , where $n$ is the sequence length). The value matrix $V$ is then simply the input $X$ (since $W_i^V = I$ ), and uniform attention over $V$ computes the element-wise mean of the previous layer's outputs across all positions. This is equivalent to a "bag-of-words" or "average pooling" operation over the context window for that layer.

Why this specific manipulation is informative. By replacing learned attention with uniform attention in a single layer while keeping all other layers intact, the authors can ask: does this layer's learned attention pattern capture information that is relevant for predicting brain activity? If brain prediction improves with uniform attention (as the authors find for layers 1–6), this suggests that the learned attention in those layers is actually detrimental to encoding the kind of linguistic information the brain processes—perhaps because it focuses on spurious statistical patterns that don't align with human language processing, or because it discards information that the brain integrates across all positions equally. If brain prediction worsens with uniform attention (as for layers 7–11), this suggests that those layers' learned attention patterns capture genuinely useful structure.

Controls and experimental rigor for the attention manipulation. The authors only alter a single layer at a time while keeping all other BERT parameters fixed, and they test the effect on brain prediction at the same layer (i.e., they alter layer $\ell$ 's attention and measure how well the output of layer $\ell$ predicts brain activity). This ensures that downstream effects are isolated to the direct consequence of the attention change, not to cascading effects through later layers. The manipulation is also tested across context lengths to determine whether the benefit of uniform attention depends on how much context is available (Figure 6 shows the benefit for shallow layers holds for context lengths up to ~25 words).

The Proof-of-Concept Validation: MEG and Known Word Properties

Before applying the framework to investigate contextual representations, the authors validate that it can recover known information from word embeddings. This is a critical step: if the method cannot detect that a word embedding encodes word length and part-of-speech—properties we independently know word embeddings capture—then its findings about contextual representations would be suspect.

Experimental design. The authors use the MEG data (high temporal resolution) and ELMo's non-contextualized word embedding. They construct two sets of explicit feature vectors:

Word length: encoded as a scalar (number of characters in the word), which can be expanded to a one-hot or continuous representation for the encoding model.
Part-of-speech (POS): encoded as a one-hot vector over POS tags for each word (e.g., noun, verb, adjective).

For each of these feature sets $A$ (e.g., POS one-hot) and $B$ (e.g., ELMo embedding), they train separate encoding models and evaluate classification accuracy at every MEG sensor/timepoint tuple. Then, to measure shared information, they define the shared accuracy $A \cap B$ as:

$\text{Acc}_{A \cap B} = \text{Acc}_A + \text{Acc}_B - \text{Acc}_{A \cup B}$

where $\text{Acc}_{A}$ is the classification accuracy using only features $A$ , $\text{Acc}_{B}$ is the accuracy using only features $B$ , and $\text{Acc}_{A \cup B}$ is the accuracy using both feature sets concatenated together.

What this computes: If $A$ and $B$ contain entirely non-overlapping information about brain activity, then $\text{Acc}_{A \cup B}$ should approximately equal $\text{Acc}_A + \text{Acc}_B - 0.5$ (the independent contributions sum, minus the chance baseline that was double-counted), and the shared accuracy $\text{Acc}_{A \cap B}$ would be near zero. If $A$ and $B$ contain identical information, then $\text{Acc}_{A \cup B} = \text{Acc}_A = \text{Acc}_B$ , and the shared accuracy $\text{Acc}_{A \cap B}$ would equal the individual accuracies. In practice, the shared accuracy measures the amount of predictive power that is redundant between the two feature sets—the extent to which they predict the same variance in the same brain locations at the same times.

Why this form (additive decomposition of shared variance): This is a linear approximation to the shared explained variance between two feature spaces, operationally defined through the concatenation model's performance. The intuition is that if $A$ and $B$ predict overlapping parts of brain activity, then adding $B$ to a model that already has $A$ will improve performance less than if they predicted complementary parts. The shared accuracy formula captures this intuition—it's the amount by which the union's accuracy is less than the sum of the individual accuracies (correcting for chance). This approach avoids the computational expense of formally decomposing variance components (which would require fitting models that partial out shared subspaces) while still providing a interpretable measure of overlap.

Expected results from neuroscience. The authors have clear temporal and spatial predictions:

Word length should share variance with the ELMo embedding in posterior/visual sensors at ~100ms after word onset, because the visual cortex processes orthographic information about word form (number of letters, visual complexity) at early latencies (Sudre et al., 2012).
Part-of-speech should share variance with the ELMo embedding in left frontal sensors at ~200ms after word onset, because electrophysiology studies have established that syntactic category violations elicit a left anterior negativity (LAN) or early left anterior negativity (ELAN) around 200ms post-stimulus (Frank et al., 2015).

The results (Figure 8 of the supplementary materials) confirm both predictions: the shared accuracy between ELMo embedding and word length peaks in posterior sensors around 100ms, and the shared accuracy between ELMo embedding and POS peaks in frontal sensors around 200–250ms, with the bottom-right comparison panel showing that at these frontal sensors, the embedding-POS overlap is much stronger than the embedding-length overlap. The authors conclude: "From these results we can hypothesize that the word-embedding contains both word length and POS information, as was expected." This confirmation—that the method finds known relationships—establishes credibility for the subsequent analyses of unknown relationships in contextual representations.

4. Key Insights and Innovations

Innovation 1: Reframing NLP Interpretability as a Brain-Alignment Problem Rather Than a Probe-Task Design Problem

The paper's most fundamental intellectual contribution is not any specific finding about BERT or ELMo, but rather the conceptual reframing of what it means to interpret a neural network's representations. Prior to this work, the dominant paradigm for understanding what NLP models encode was the probe task approach: design a simplified classification problem that tests for a specific linguistic property (part-of-speech, syntactic depth, named entity type), train a lightweight classifier on top of a frozen layer's representations, and use the classifier's accuracy as evidence that the property is—or isn't—encoded at that layer. This approach, exemplified by Conneau et al. (2018) and Linzen et al. (2016), is fundamentally detective-driven: the researcher must hypothesize in advance what linguistic information might be present, operationalize it as a labeled dataset, and run the probe. The approach can confirm or disconfirm specific hypotheses, but it is structurally incapable of discovering organizational principles the researcher hasn't thought to look for.

The brain-alignment framework inverts this logic. Instead of designing tasks to test for specific properties, the authors use the brain's own functional organization as a readout that decomposes network representations according to biologically-characterized processing pathways. The key intellectual move is captured in the "demultiplexer" analogy from Section 1.1: a network representation is an entangled mixture of information about orthography, syntax, semantics, and discourse structure; aligning that representation with brain activity separates it into components corresponding to different brain regions, each with independently-characterized functions from prior naturalistic neuroimaging experiments (Lerner et al., 2011). The researcher doesn't need to specify what to look for—the brain's spatial and temporal organization provides the decomposition. A layer that predicts activity in visual cortex (area 1a) is encoding orthographic features; a layer that predicts anterior temporal regions (group 2) is encoding multi-word compositional information. The brain serves as a biologically-grounded, data-driven prior that replaces the researcher's hypotheses.

This is not merely an alternative measurement technique—it is a fundamentally different epistemic stance toward interpretability. The probe-task approach asks: "Does the network represent property P (which I have formalized as task T)?" The brain-alignment approach asks: "What does the brain's response to this text tell us about what the network represents, without requiring me to pre-specify the categories?" The proof-of-concept validation (Figure 8, Supplementary Section 11) demonstrates that this approach recovers known properties (word length, part-of-speech) at the expected times and locations, which establishes that the brain is a reliable measurement instrument. But the real power of the framework is for discovering unknown organizational principles—such as the finding that uniform attention in shallow BERT layers improves brain alignment (Figure 6), a result that would never emerge from a probe task because no researcher would think to design a syntactic or semantic task to test for "does uniform attention encode more brain-relevant information than learned attention?"

This reframing is fundamental rather than incremental because it changes what counts as an interpretability result. In the probe-task paradigm, the output is a set of binary claims ("layer 7 encodes syntactic number agreement," "layer 3 does not encode semantic role"). In the brain-alignment paradigm, the output is a spatially- and temporally-resolved map of which aspects of linguistic processing a layer aligns with—a richer, more graded, and more open-ended characterization. The authors explicitly position this as extending interpretability beyond "complex and arguably more interesting aspects of language, such as high level meaning," which are "difficult to capture in an NLP task or in behavioral measures." The brain, by virtue of being a system that does process high-level meaning, provides a window into these phenomena that no hand-designed task can match.

The significance extends beyond NLP. The framework establishes a two-way bridge between artificial and biological language processing: prior work (Wehbe et al., 2014a; Jain and Huth, 2018) used NLP models to study the brain, but this paper uses the brain to study—and subsequently improve—NLP models. This closes the loop that the authors describe as enabling "the interaction between NLP and cognitive neuroscience to be a true cross-pollination." The intellectual contribution is not the encoding model mathematics (which follows standard practice from systems neuroscience), but the recognition that brain data can serve as an interpretability tool for models, not just the other way around.

Innovation 2: The Discovery That the Benefit of Learned Attention Is Layer-Dependent, With Shallow Layers Performing Better Under Uniform Attention

The paper's most counterintuitive empirical finding—and its most actionable insight for model design—is that replacing learned attention with uniform attention in BERT's shallow layers (1–6) improves brain alignment, while the same manipulation in deep layers (7–11) degrades it (Figure 6). This result is genuinely surprising because the dominant narrative around attention mechanisms, since Vaswani et al. (2017), has been that learned, content-dependent attention is uniformly beneficial—it's the core innovation that enables transformers to capture long-range dependencies and contextual nuance. The implicit assumption in the field was that more sophisticated attention patterns = better representations, at every layer. This paper provides evidence that this assumption is false for the early layers of a pretrained transformer, at least with respect to the kind of linguistic information that the human brain processes during reading.

What makes this finding fundamental rather than incremental is that it reveals a layer-specific functional specialization in transformers that was not predicted by prior theoretical or empirical work. The shallow layers of BERT, under learned attention, appear to discard or distort information that is actually useful for representing language in a brain-like way—when forced to average uniformly over the context window, these layers produce representations that better predict brain activity, particularly for context lengths up to ~25 words. The deeper layers show the opposite pattern: removing learned attention hurts brain alignment, suggesting that the information integrated by learned attention in deep layers is genuinely important for capturing the kind of contextual structure that the brain's language network processes.

The fact that this finding is layer-specific and difficulty-dependent (varying with context length) means it cannot be explained by a simple claim like "learned attention is overrated" or "uniform attention is better." It requires a more nuanced account: at different depths, the transformer is doing qualitatively different things with attention, and the pretraining objective (masked language modeling + next sentence prediction) drives shallow layers to learn attention patterns that optimize the training loss but happen to diverge from the information-processing strategies that the brain uses. The deeper layers, processing more abstract features, converge on attention patterns that align better with brain-like processing.

The significance of this finding is amplified by the fact that it transfers to NLP tasks (Section 5, Table 1). The brain-aligned BERT variant (with uniform attention in layers 1–6) outperforms the original pretrained BERT on 8 of 13 syntactic probe tasks from Marvin and Linzen (2018), with statistically significant improvements on tasks involving subject-verb agreement across prepositional phrases, relative clauses, and object relative clauses. This closes the loop from interpretation to improvement: a modification suggested by brain data (not by NLP benchmarks) produces measurable gains on downstream linguistic competence tests. The converse also holds: uniform attention in layer 11 (which hurt brain alignment) performs on par or worse than the base model on 12 of 13 tasks.

This result is conceptually distinct from the typical "better NLP benchmark numbers" finding. The improvement is not on end-task accuracy (sentiment analysis, question answering) but on syntactic probe tasks that specifically test whether the model has acquired grammatical competence—exactly the kind of linguistic knowledge that cognitive scientists care about. The paper is demonstrating not just that brain alignment correlates with better performance, but that the brain provides a signal about linguistic structure that is complementary to standard pretraining objectives, and that incorporating that signal (even through a crude intervention like replacing attention with uniform averaging) can improve the model's implicit grammatical knowledge.

Innovation 3: Demonstrating That Long-Context Integration in Transformers Degrades in Deep Layers, Establishing a "Context Window" That Peaks in Middle Layers

The paper uncovers a consistent layer-depth × context-length interaction across all three multi-layer models studied (ELMo, BERT, and Transformer-XL): the middle layers are optimal for representing long-range context, while the deepest layers show a characteristic pattern of strong performance at short context lengths (<10 words) followed by degradation as context length increases (Figure 4). This finding is significant because it reveals a non-obvious organizational principle in deep NLP models that challenges the simple intuition that "deeper layers = more abstract, more integrated representations."

Prior to this work, the dominant mental model for layer depth in NLP architectures was roughly hierarchical: lower layers capture local, surface-level features (morphology, local collocations), and higher layers capture progressively more abstract, context-integrated features (syntax, semantics, discourse). This is the pattern observed in computer vision, where early layers detect edges and textures and later layers detect objects and scenes. The brain-alignment results suggest this picture is too simple for language transformers. The deepest layers of BERT and T-XL do not show monotonically improving long-context brain prediction—instead, they show a "sharp increase in performance at short-range context (fewer than 10 words), followed by a decrease in performance" (Section 4, Figure 4). This means the deepest layers are actually losing information about long-range context that was present in the middle layers, or re-organizing it in a way that aligns less well with human brain processing.

The finding that Transformer-XL is the only model that continues to increase performance as context length is increased (Figure 4, rightmost panel) is particularly instructive because it validates the architectural motivation behind T-XL's segment-level recurrence mechanism. Dai et al. (2019) designed T-XL specifically to capture longer context than standard transformers, and the brain-alignment results provide independent evidence that it succeeds—not on a perplexity benchmark, but in terms of encoding the kind of context-integrated information that the human brain represents during natural reading. This is a form of architectural validation that goes beyond standard metrics: T-XL wasn't designed to predict brain activity, yet its representations show exactly the long-context integration properties (sustained improvement out to 40 words) that its designers intended, and that the brain's language network demands.

The significance of this finding is that it provides a biological plausibility constraint on model architectures. If the human brain—our only working example of a system that understands language—integrates context in a way that continues to build over long spans (as the Group 2 brain regions require, being activated only by connected multi-word text), then architectures that degrade in their context representations at deep layers (like BERT) may be fundamentally limited in their ability to capture discourse-level phenomena. Conversely, architectures that sustain context integration (like T-XL) may be better suited for tasks requiring long-range linguistic dependencies. This is not a claim about benchmark performance (T-XL and BERT perform comparably on many tasks) but about the representational organization that underlies that performance—a distinction that matters for building models that generalize robustly rather than exploiting dataset-specific shortcuts.

Innovation 4: The Brain as a "Found Translation" Between NLP Architectures, Enabling Comparison Without Shared Benchmarks

A subtle but important conceptual contribution of the paper is its use of brain activity as a common reference frame that allows different NLP architectures (ELMo, BERT, USE, T-XL) to be compared in terms of what linguistic information they encode, despite having entirely different internal organizations, dimensionalities, and training objectives. The authors are explicit about the danger of direct cross-model comparison ("information is likely organized differently across networks and such comparisons are misleading"), and their solution—comparing each model to the same brain data, then comparing the patterns of brain alignment across models—is a methodological innovation that addresses a real problem in the interpretability literature.

The problem is this: if Model A's layer 7 predicts brain region R better than Model B's layer 7, does that mean Model A is "better" at encoding the information R processes? Not necessarily—Model A might be a 12-layer transformer and Model B a 2-layer LSTM, and "layer 7" means entirely different things in these architectures. The brain provides a fixed external reference: we can ask, for each model, which layers best predict activity in the Group 2 regions (long-range context), and compare the intra-model patterns. The finding that ELMo, BERT, and T-XL all show optimal long-context prediction in their middle layers (despite having 2, 12, and 19 layers respectively) is meaningful precisely because it emerges from comparing across architectures using a common reference frame—it suggests that the "middle layer peak" is a general property of deep NLP models, not an artifact of a specific architecture.

This is analogous to the role that brain data plays in cognitive neuroscience as a "common currency" for comparing different theories: if Theory A posits that syntactic processing happens in Broca's area and Theory B posits that it happens in Wernicke's area, fMRI data can adjudicate between them without requiring the theories to share a common formalism. Similarly, the brain-alignment framework can compare ELMo and BERT without requiring their representations to have the same dimensionality or internal structure. The brain's functional organization provides the translation layer.

The results that emerge from this common-reference-frame approach—such as the qualitative comparison in Figures 2 and 3 showing that USE's long-context representations predict a much smaller subset of Group 2 regions than the other models—gain their force from this methodology. We're not claiming USE is "worse" than BERT (it's designed for a different purpose), but we can observe that its sentence-level deep averaging network produces representations that lack the kind of word-level contextual integration that the brain's Group 2 regions respond to. This is a functional characterization that standard NLP benchmarks cannot provide: USE performs well on semantic textual similarity tasks, but the brain data reveals how its representations differ from token-level models in terms of context integration—a finding with implications for when USE-like architectures are appropriate.

This contribution is methodological and foundational rather than a specific empirical result. It establishes a pattern for how future work can use brain data to compare models across architectures, training regimes, and scales—provided they can be run on the same naturalistic text to extract comparable representations. As larger and more diverse models are developed, this common-reference-frame approach becomes increasingly valuable for understanding what architectural choices actually change about the linguistic information a model captures.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses fMRI data from 8 subjects and MEG data from 3 subjects reading Chapter 9 of Harry Potter and the Sorcerer's Stone (Rowling, 2012), presented one word at a time for 0.5 seconds each. The fMRI data was originally published by Wehbe et al. (2014b), and the MEG data by Wehbe et al. (2014a). The fMRI recording spans approximately 45 minutes with a TR of 2 seconds; the MEG recording covers the same chapter at 1kHz sampling, downsampled to 25ms non-overlapping time bins, yielding 20 timepoints per word across 306 sensors organized into 102 locations. The "dataset" for model evaluation is therefore not a traditional NLP benchmark but rather this multi-modal neuroimaging data, which provides the ground-truth brain activity patterns against which model representations are compared.
Base model(s). Four pre-trained NLP models are evaluated: ELMo (Peters et al., 2018), a 2-layer bidirectional LSTM language model; BERT-base (Devlin et al., 2018), a 12-layer transformer with 12 attention heads and 768 hidden units; USE (Cer et al., 2018), a sentence encoder based on a deep averaging network with 512-dimensional output; and Transformer-XL (T-XL; Dai et al., 2019), a 19-layer transformer with segment-level recurrence and 1024 hidden units. All models are used in their publicly-available pre-trained form without further fine-tuning. The selection is motivated by spanning diverse architectural paradigms—recurrent (ELMo), pure transformer (BERT), hybrid recurrent-transformer (T-XL), and sentence-embedding (USE)—allowing investigation of whether representational properties are architecture-specific or general.
Metrics. The primary evaluation metric is classification accuracy in a two-alternative forced-choice task: given a chunk of real brain data (20 consecutive TRs for fMRI, 20 randomly sampled words for MEG) and two candidate prediction chunks (the correct one vs. a randomly selected incorrect one), the classifier must identify which candidate matches the real data based on Euclidean distance. Accuracy is averaged over "a large number of times" (random draws of chunks and incorrect candidates) per measurement channel, producing a spatially-resolved accuracy map. For fMRI, results are reported on the MNI template brain; for MEG, results are reported per sensor location (102 locations) × timepoint (20 bins, 0–500ms after word onset). Statistical significance is determined via an empirical FDR-controlling procedure that estimates the False Discovery Proportion from the symmetry of chance accuracies around 0.5, with $q = 0.05$ .
Baselines. The paper does not frame its analyses around explicit "baselines" in the traditional NLP sense, but rather builds comparisons into its controlled manipulations. However, several implicit baselines exist: (1) The non-contextualized word embedding (layer 0 in ELMo/BERT, or the single-word representation at context length $k=1$ ) serves as the baseline for evaluating whether longer-context representations capture additional brain-relevant information beyond isolated word properties. (2) The pretrained BERT model with learned attention serves as the baseline for the uniform-attention manipulation (Section 4 and Section 5). (3) Majority-voting/chance-level accuracy of 0.5 serves as the baseline for the classification task. (4) For the proof-of-concept (Supplementary Section 11), explicit word-length and part-of-speech feature vectors serve as known-information baselines against which the ELMo embedding's shared variance is measured. The paper does not compare against probe-task-based interpretability methods or other interpretability frameworks, which would be a natural baseline for establishing the unique value of the brain-alignment approach.
Generation budget / compute accounting. Since the paper evaluates frozen pre-trained models rather than performing training or generation, there is no "generation budget" in the traditional sense. Instead, the relevant "compute" axis is the context length $k$ —the number of preceding words fed through the network to produce a representation for the current word. This is swept from $k=1$ (current word only) to $k=40$ (the 40 most recent words) for ELMo, BERT, and T-XL. For the uniform-attention manipulation (BERT only), the intervention is applied to a single layer at a time while keeping all other layers fixed, meaning the "cost" is identical to a standard forward pass with the modified attention weights—there is no additional computational overhead beyond the architectural modification itself. The encoding models themselves involve training separate ridge regression models (one per voxel or sensor/timepoint per layer per context length), which is computationally significant but not reported as a budget constraint. The FLOPs-matched comparison that appears in similar papers (e.g., Toneva and Wehbe do not compare pretraining vs. inference compute) is absent; this paper operates entirely in the evaluation regime on pre-trained models.
Cross-validation / statistical protocol. All encoding models are trained and evaluated using four-fold cross-validation: the ~1,350 TRs (~45 min / 2s TR) of fMRI data or the ~5,176 words of MEG data are split into four folds, with three folds used for training and one for held-out evaluation, rotating so every fold serves as validation once. The ridge regularization parameter $\lambda_i$ for each voxel (or each sensor/timepoint tuple for MEG) is selected via nested 10-fold cross-validation within each training fold—the training data is further split, a range of $\lambda$ values is evaluated, and the value minimizing validation error is chosen before fitting the full training fold. This ensures that $\lambda$ is optimized without touching the held-out test fold. For the statistical significance of fMRI voxels, the paper's empirical FDR procedure (described in Section 3.4) controls the False Discovery Proportion at $q = 0.05$ . For the syntactic NLP tasks in Section 5, statistical significance of performance differences relative to the base BERT model is assessed using a paired t-test at significance level 0.01 with FDR correction for multiple comparisons (Benjamini and Hochberg, 1995) across the 13 tasks.

Main Quantitative Results

Proof of Concept: ELMo Embedding Encodes Word Length and Part-of-Speech (MEG)

The paper first validates that its methodology can recover known properties of word embeddings by measuring the shared predictive accuracy between ELMo's non-contextualized embedding and explicit representations of word length and part-of-speech on the MEG data (Supplementary Section 11, Figure 8). This is not a main quantitative result in the sense of establishing new knowledge, but it is the foundational demonstration that the framework produces interpretable, non-spurious outputs.

The headline finding is that the ELMo word embedding predicts brain activity that overlaps with word-length information in posterior/visual sensors starting approximately 100ms after word onset, and with part-of-speech information in left frontal sensors approximately 200–250ms after word onset—exactly the temporal and spatial patterns expected from prior electrophysiology studies (Sudre et al., 2012 for visual word form processing; Frank et al., 2015 for syntactic category processing). The authors compute shared accuracy as $\text{Acc}_{\text{shared}} = \text{Acc}_{\text{embedding}} + \text{Acc}_{\text{feature}} - \text{Acc}_{\text{embedding} \cup \text{feature}}$ , where $Acc_{\text{feature}}$ is the classification accuracy using only the explicit feature (word length or POS), and $\text{Acc}_{\text{embedding} \cup \text{feature}}$ is the accuracy using both concatenated.

Figure 8 (bottom rows) shows the spatial-temporal maps of shared accuracy. "Word embedding predicts a part of the activity at the top of the helmet [frontal sensors], and this is shared mostly with the POS tags and not with word length" (Supplementary Section 11, commentary on the bottom-right comparison panel). The authors conclude: "From these results we can hypothesize that the word-embedding contains both word length and POS information, as was expected."

Quantitative specifics from Figure 8: The shared accuracy maps show sensor-level classification performance at each of the 102 sensor locations and 20 timepoints. The word-length overlap peaks in posterior sensors (locations at the back of the helmet, over visual cortex) in the 100–150ms window; the POS overlap peaks in more anterior/left-lateral sensors in the 200–250ms window. The paper does not report exact numeric accuracies for these peaks (the figure uses a color scale), but the qualitative pattern—posterior-early for word length, anterior-later for POS—is unambiguous and matches the expected neuroscience results. This serves as a validation that the encoding-model-to-classification pipeline produces anatomically and temporally plausible decompositions, giving license to apply the same methodology to the unknown properties of contextual representations.

Long-Range Context Representations: Qualitative Comparison Across Models (fMRI)

The paper's first major fMRI analysis compares, for each of the four models, the brain regions predicted by a word embedding (the representation of the most recent word only) versus a 10-word representation (derived from a middle layer: layer 1 for ELMo, layer 7 for BERT, layer 11 for T-XL) (Section 4, Figures 2 and 3). The goal is to assess whether each model's longer-context representations capture additional brain-relevant information beyond what is present in single-word embeddings—specifically, whether they predict activity in Group 2 regions (which are activated only by connected multi-word text and therefore index long-range context processing).

Headline finding from Figure 2 (qualitative brain maps): For ELMo, BERT, and T-XL, the 10-word (red) representations predict activity in subsets of both Group 1 regions and Group 2 regions, and the word-embedding-predicted regions (blue) are largely a subset of those predicted by the 10-word representation (very few blue-only voxels are visible in these models). The authors conclude: "These results may be due to the fact that all these models are at least partially trained to predict a word at a given position. They must encode long range information and also local information that can predict the appropriate word." In contrast, USE's 10-word representation predicts a "much smaller subset of Group 2 regions" and shows substantial blue-only regions (areas predicted by the word embedding but not by the 10-word context representation in Group 1 regions). The authors interpret this as: "The low performance of the USE vectors might be due to the deep averaging which might be composing words in a crude manner. The low performance in predicting group 1 regions is most probably because USE computes representations at a sentence level and does not have the option of retaining recent information like the other models. USE long context representations therefore only have long range information."

Quantitative summary from Figure 3 (bar plots): Figure 3 quantifies the percentage of voxels within Group 1b regions and Group 2 regions that are significantly predicted by each representation type (word embedding, 10-word context, or both). The results show:

ELMo: In Group 1b regions, approximately 75–80% of voxels are predicted by both representations, about 10–15% by the word embedding alone, and about 5–10% by context alone. In Group 2 regions, approximately 45–50% are predicted by context alone, about 5–10% by both, and nearly none by the word embedding alone—confirming that the ELMo 10-word representation captures Group-2-specific contextual information.
BERT: In Group 1b regions, approximately 70% of voxels are predicted by both, about 15% by word embedding alone, and about 5–10% by context alone. In Group 2 regions, approximately 55–60% are predicted by context alone, about 5% by both, and nearly none by word embedding alone—a very similar pattern to ELMo.
T-XL: In Group 1b regions, approximately 60% are predicted by both, about 20% by word embedding alone, and about 10% by context alone. In Group 2 regions, approximately 65–70% are predicted by context alone, about 5% by both, and approximately 2% by word embedding alone. T-XL shows the strongest context-only prediction in Group 2 regions among the four models.
USE: In Group 1b regions, approximately 50% are predicted by the word embedding alone, only about 10–15% by both, and nearly none by context alone—strikingly different from the other three models. In Group 2 regions, approximately 10–15% are predicted by context alone, about 2% by both, and nearly none by word embedding alone. USE's context representation predicts a dramatically smaller fraction of both Group 1b and Group 2 regions compared to the other models, which the authors attribute to the deep averaging network architecture that composes words without retaining token-level information.

The error bars (standard error over the 8 subjects) in Figure 3 indicate that the qualitative patterns are consistent across subjects—particularly the dominance of context-only prediction in Group 2 for ELMo, BERT, and T-XL, and the near-absence of context-only prediction in Group 1b for USE.

Layer Depth × Context Length Interaction (fMRI)

The paper's most systematic quantitative analysis examines how brain prediction performance varies as a function of layer depth and context length for ELMo, BERT, and T-XL (Section 4, Figure 4). For each layer of each model, encoding models are trained and evaluated at context lengths ranging from 1 to 40 words, producing a set of curves (one per layer) showing how classification accuracy changes with increasing context.

Headline finding 1: Middle layers are optimal for long-range context. Across all three models, the middle layers achieve the highest brain prediction accuracy when context length exceeds approximately 15 words. In ELMo (2 layers), layer 1 (the intermediate layer) outperforms layer 2 (the output layer) for context lengths above ~15 words. In BERT (12 layers), layers 6–8 (the middle third) show the highest performance for context lengths from 20–40 words, with layer 7 appearing to be the peak. In T-XL (19 layers), layers 9–12 (again, roughly the middle of the 19-layer stack) achieve the best long-context performance. This finding is non-obvious: if representations simply became richer with depth, the deepest layers would dominate—but they do not.

Headline finding 2: Deepest layers show a "peak-then-decline" pattern with context length. In all three models, the deepest layers (layer 2 for ELMo, layers 10–12 for BERT, layers 16–19 for T-XL) exhibit a characteristic shape: a sharp increase in brain prediction accuracy as context grows from 1 to approximately 5–10 words, followed by a decrease in performance as context extends to 40 words. The paper states: "The deepest layers across all networks show a sharp increase in performance at short-range context (fewer than 10 words), followed by a decrease in performance" (Section 4). This means that the deepest layers are not simply "more context-integrated"—they actually lose brain-relevant information about longer contexts that was present at middle layers, or reorganize it in a way that aligns less with human brain processing.

Headline finding 3: Transformer-XL is the only model with sustained performance gains from longer context. Figure 4 (rightmost panel) shows that T-XL's encoding model performance "continues to increase performance as the context length is increased" out to 40 words, while ELMo's performance plateaus or declines at longer contexts, and BERT's middle-layer performance peaks and then levels off. The authors interpret: "T-XL was designed to represent long range information better than a usual transformer and our results suggest that it does."

Quantitative specifics from Figure 4: The y-axis shows "% change in acc from L1" (for BERT, the change in classification accuracy relative to layer 1's performance), while for ELMo and T-XL the y-axis presumably shows raw or normalized classification accuracy (the figure labels are not fully specified in the text but the relative scaling is clear from the layer ordering). For BERT:

Layer 1 shows modest improvement from ~0% to ~2–3% increase in accuracy as context grows from 1 to 40 words.
Layers 6–8 peak at approximately 4–6% increase over layer 1 for context lengths of 20–40 words.
Layers 10–12 show a distinctive shape: accuracy rises sharply to ~4% at context length ~5 words, then declines to ~0–2% by 40 words.
Layer 12 (the deepest) shows the most dramatic decline—rising to approximately 3% at ~5 words, then dropping below 0% (i.e., performing worse than layer 1) at context lengths beyond 20 words.

For T-XL:

The middle layers (~9–11) achieve the highest performance, showing sustained improvement from context length 1 to 40, reaching approximately 6–8% above baseline.
The deepest layers (~16–19) show the peak-then-decline pattern, with performance dropping from a peak at ~5–10 words to near-zero improvement at 40 words.
Unlike BERT, T-XL's best-performing layers continue to benefit from additional context at all tested lengths.

For ELMo (only 2 layers):

Layer 1 outperforms layer 2 for contexts beyond ~10 words.
Layer 2 (output) shows a modest peak at ~10 words then declines, while layer 1 shows sustained or slightly improving performance.

BERT layer 1 anomaly (Figure 5). The authors note that "layer 1 in BERT behaves differently from the first layers in the other two networks." To investigate this, they re-plot BERT's layer performance as the change from layer 1's performance (i.e., subtracting layer 1's accuracy from each layer's accuracy at each context length). The resulting curves (Figure 5) "resemble the ones in T-XL" more closely, with layers 6–8 showing the strongest long-context benefits. The authors hypothesize that "BERT layer 1 combines the information from the token-level embeddings in a way that limits the retention of longer context information in the layer 1 representations"—meaning the unusual behavior of BERT's early layers is attributable to a specific transformation at layer 1 that is suboptimal for context retention, and subsequent layers partially recover from this.

Effect of Attention on Layer Representation (BERT Only, fMRI)

The paper's most causally-informative experiment replaces the learned multi-head attention in a single BERT layer with uniform attention (element-wise averaging over the previous layer's outputs) and measures the change in brain prediction performance at that layer, for context lengths from 1 to 40 words (Section 4, Figure 6). All other layers retain their pretrained attention weights.

Headline finding: The effect of replacing learned attention with uniform attention is layer-dependent in sign—shallow layers (1–6) benefit, while deep layers (7–11) are harmed, and the output layer (12) is unaffected or slightly improved.

Quantitative specifics from Figure 6: The y-axis shows "% change in acc from learned attention"—positive values mean uniform attention improves over learned attention; negative values mean uniform attention degrades performance.

Layers 1–6 (shallow): For context lengths up to approximately 25 words, uniform attention produces positive changes in accuracy relative to learned attention. Layer 1 shows the strongest benefit, with approximately +1.5% to +2% improvement at context lengths of 5–20 words, declining to near zero at 40 words. Layers 2–6 show more modest but consistently positive improvements (~+0.5% to +1%) in the 5–25 word range. The authors state: "shallow layers benefit from the uniform attention for context lengths up to 25 words."
Layers 7–11 (deep): Uniform attention produces negative changes across nearly all context lengths. Layer 8 and layer 11 show the most severe degradation, with approximately –1% to –2.5% change in accuracy. The authors state: "The performance of deep layers, other than the output layer, is harmed by the change in attention."
Layer 12 (output): Uniform attention produces a small positive change (~+0.5%) across most context lengths, making it an exception among the deep layers.

Interpretation. This result is surprising because learned, content-dependent attention is widely assumed to be uniformly beneficial. The finding that shallow layers perform better when forced to average uniformly over the context window suggests that the learned attention patterns in early BERT layers actually discard or distort information that is useful for brain-like language processing—perhaps by over-focusing on specific positions or attending to spurious statistical patterns. The deep layers' dependence on learned attention suggests that at higher levels of abstraction, the attention patterns genuinely capture structure that aligns with human language processing. The positive effect at layer 12 may reflect that the final layer benefits from the cumulative effect of uniform attention in earlier layers, or that the output layer's task (predicting masked tokens) is less sensitive to the specific attention pattern.

Controls. Only a single layer is altered at a time, and brain prediction is measured at that same layer's output. This isolates the direct effect of the attention change rather than cascading effects through the network. The manipulation is performed identically for all layers: all attention heads in the layer have their $W^Q$ and $W^K$ matrices zeroed and $W^V$ set to the identity matrix, yielding uniform attention across the previous layer's outputs.

From Brain Alignment to NLP Task Improvement (Syntactic Probe Tasks)

Section 5 tests whether the brain-alignment finding—that uniform attention in shallow BERT layers improves brain prediction—transfers to improved performance on linguistic competence tasks that do not require fine-tuning beyond pretraining. The evaluation uses the syntactic probe tasks from Marvin and Linzen (2018), which measure subject-verb agreement across various sentence types, following the protocol of Goldberg (2019): BERT is fed a complete sentence with the focus verb masked (e.g., [CLS] the game that the guard hates [MASK] bad .), the prediction for the masked position is obtained from the pretrained language-modeling head, and accuracy is computed as the fraction of test sentences where the correct verb form (e.g., is) receives a higher probability than the incorrect form (e.g., are).

Models evaluated:

Base BERT (pretrained, no attention modifications).
uni L1 through uni L6: BERT with uniform attention at a single shallow layer (1–6), one model per layer, keeping all other layers at pretrained weights.
uni L11: BERT with uniform attention at layer 11 (a deep layer that showed the strongest degradation in brain prediction under uniform attention; Figure 6).

Headline finding from Table 1 (and Supplementary Table 3 for layers 3–5): The uniform-attention models at shallow layers (L1, L2, L6) significantly outperform base BERT on 8 of the 13 syntactic tasks and achieve parity (no statistically significant difference) on 4 of the remaining 5 tasks. The only task where any brain-aligned model performs worse than base BERT is "reflexive anaphora: in a sentential complement," where uni L1 scores 0.88 vs. base's 0.89—a negligible difference. In contrast, uni L11 (the deep-layer model with attention removed where brain prediction was most harmed) performs on par or worse than base BERT on 12 of 13 tasks, exactly mirroring the brain-alignment result.

Detailed task-by-task results from Table 1:

Task (with count of test examples)	base	uni L1	uni L2	uni L6	uni L11	Best performer (stat sig vs. base)
simple (120)	1.00	1.00	1.00	1.00	0.98	All except L11 tie at ceiling
in a sentential complement (1440)	0.83	0.83	0.83	0.83	0.83	All tie
short VP coordination (720)	0.89	0.88	0.90	0.91	0.88	uni L6 (+0.02 over base, not marked as significant in table)
long VP coordination (400)	0.98	0.96	0.97	1.00**	0.96	uni L6 (+0.02, significant)
across a prepositional phrase (19440)	0.85	0.86	0.93**	0.88	0.82	uni L2 (+0.08, significant)
across a subject relative clause (9600)	0.84	0.83	0.83	0.85**	0.83	uni L6 (+0.01, significant)
across an object relative clause (19680)	0.89	0.87	0.91	0.92**	0.86	uni L6 (+0.03, significant)
across an object relative clause (no that) (19680)	0.86	0.87	0.80	0.87	0.84	uni L1/L6 (+0.01 over base, not marked significant)
in an object relative clause (15960)	0.95	0.97**	0.95	0.91	0.93	uni L1 (+0.02, significant)
in an object relative clause (no that) (15960)	0.79	0.83**	0.72	0.74	0.72	uni L1 (+0.04, significant)
reflexive anaphora: simple (280)	0.94	0.91	0.94	0.99**	0.95	uni L6 (+0.05, significant)
reflexive anaphora: in a sent. complem. (3360)	0.89	0.88	0.85	0.86	0.85	base (0.89, uni L1 not sig. worse)
reflexive anaphora: across a rel. clause (22400)	0.80	0.79	0.84**	0.79	0.76	uni L2 (+0.04, significant)

Double asterisks (**) mark differences from base that are statistically significant after paired t-test with FDR correction for 13 comparisons.

Key observations from the per-task breakdown:

The largest absolute gains are on high-count tasks: "across a prepositional phrase" (19,440 examples), uni L2 improves from 0.85 to 0.93 (+0.08); "in an object relative clause (no that)" (15,960 examples), uni L1 improves from 0.79 to 0.83 (+0.04); "reflexive anaphora: across a relative clause" (22,400 examples), uni L2 improves from 0.80 to 0.84 (+0.04). These are the tasks where statistical power is highest, and the improvements are not merely statistically significant but substantively meaningful—a 4–8 percentage point gain on already-strong baselines.
Ceiling effects limit observable gains on small-count tasks: "simple" (120 examples) is at ceiling (1.00) for all models; "in a sentential complement" (1440 examples) shows identical performance (0.83) for all models. The lack of improvement on these tasks may reflect insufficient test-set size to detect differences at ceiling or near-ceiling performance, rather than a failure of brain alignment to transfer.
uni L11 (deep-layer uniform attention) degrades performance across the board: on 12 of 13 tasks, uni L11 performs at or below base BERT, with notable drops on "across an object relative clause" (0.86 vs. 0.89) and "reflexive anaphora: across a relative clause" (0.76 vs. 0.80). This directly mirrors the brain-alignment result from Figure 6, where deep-layer uniform attention harmed brain prediction, and suggests a consistent relationship: modifications that improve brain alignment also improve syntactic competence, and modifications that degrade brain alignment also degrade syntactic competence.

Supplementary Table 3 (complete results for layers 1–6): The supplementary results for layers 3–5 show similar patterns—uniform attention in these intermediate shallow layers generally produces performance on par with or slightly above base BERT, though with fewer statistically significant improvements than layers 1, 2, and 6. Layer 4 notably underperforms on "reflexive anaphora: across a rel. clause" (0.68 vs. base's 0.80), suggesting that the optimal layers for uniform attention are not uniformly distributed across the shallow half—some layers (1, 2, 6) benefit more than others (4, 5).

Ablation Studies and Robustness Checks

The paper's experimental design embeds several ablation-like comparisons into the main analyses, though it does not label them as formal ablation studies. The following are the key robustness-relevant analyses:

Cross-model comparison of context representation quality (Figure 3): The finding that USE's 10-word representation predicts a much smaller subset of Group 2 regions than ELMo, BERT, and T-XL is not an ablation in the traditional sense, but it serves as a model-family robustness check—the brain-alignment methodology produces differential results across architectures that align with architectural differences (deep averaging vs. token-level processing), suggesting the method is sensitive to genuine representational differences rather than producing uniform or random outputs. The standard errors in Figure 3 also serve as a subject-level robustness check: the qualitative pattern of context-only prediction in Group 2 regions is consistent across the 8 subjects.

Controlled single-layer attention manipulation (Figure 6): The key robustness feature of the uniform-attention experiment is that only one layer is modified at a time, with all other parameters held at their pretrained values. This prevents the results from being confounded by cascading effects through multiple modified layers. However, the paper does not perform the converse ablation: it does not test whether increasing the sophistication of shallow-layer attention (e.g., adding more attention heads, or training BERT with a different initialization scheme for early layers) would further improve brain alignment. This leaves open whether the benefit of uniform attention is about removing harmful learned attention patterns specifically in the pretrained model, or about a general principle that shallow layers should perform simple averaging.

BERT layer 1 anomaly investigation (Figure 5): The re-analysis of BERT's layer-depth × context-length results by subtracting out layer 1's performance is a form of diagnostic ablation that isolates whether the anomalous behavior of BERT's early layers is concentrated in layer 1. The finding that the adjusted curves resemble T-XL more closely suggests that the layer 1 transformation is the locus of the anomaly—a claim that could be strengthened by testing whether initializing or training BERT with a different layer-1 structure changes the pattern, but the paper does not pursue this.

MEG proof-of-concept as methodological validation (Figure 8): The shared-variance analysis between ELMo embedding and explicit word features (length, POS) serves as a methodological ablation: it tests whether the encoding-model-to-classification pipeline can recover known relationships before applying it to unknown ones. The confirmatory results at the expected times (100ms for word length, 200ms for POS) and locations (posterior for visual, frontal for syntactic) validate that the method produces non-spurious, anatomically-plausible results. However, this validation is limited to word-level properties—the paper does not provide an analogous validation for contextual properties (e.g., showing that a representation known to encode multi-word syntax predicts Group 2 regions), which would be the next step in establishing the methodology for the paper's main claims about context.

Layer depth effect replicated across three architectures (Figure 4): The finding that middle layers outperform deep layers for long-range context prediction is replicated in ELMo (2 layers), BERT (12 layers), and T-XL (19 layers). This cross-architecture replication strengthens the claim that this is a general property of deep NLP models rather than an artifact of a specific architecture or training procedure. The fact that the "middle" is at different absolute depths (layer 1 of 2, layer 7 of 12, layer 10 of 19) but consistently in the middle third of the network suggests a principled organizational gradient rather than a fixed-depth effect.

No formal test-retest reliability analysis: The paper does not report split-half reliability, test-retest correlations, or other standard reliability metrics for the encoding models. Given that the fMRI experiment has no repeated stimuli (subjects read the chapter only once), the standard neuroscience approach of averaging across repeated presentations to improve SNR is not available. The classification task's use of 20-TR chunks and the cortical searchlight pooling are designed to compensate, but the paper does not quantify how reliable the single-subject encoding models are, which limits assessment of whether layer-difference results are robust within individual subjects or only emerge in group averages.

Critical Assessment

Do the Experiments Support the Paper's Central Claims?

The paper makes four central contributions (Section 1.2): (1) a new method to interpret network representations using brain data; (2) analysis and hypotheses about ELMo, BERT, USE, and T-XL; (3) the finding that middle layers of transformers are best at predicting brain activity, that T-XL's performance doesn't degrade as context increases, and that uniform attention in early BERT layers improves brain prediction; (4) the demonstration that altering BERT to better align with brain recordings improves its syntactic NLP task performance. Each claim requires separate scrutiny.

Claim 1: "We present a new method to interpret network representations and a proof of concept for it."

The method is genuinely novel in its direction of inference (brain-to-model rather than model-to-brain) and in its use of the brain's functional organization as a decomposition tool rather than designing explicit probe tasks. The proof of concept (Figure 8) demonstrates that the method recovers known word-level properties (length, POS) at the expected neural times and locations. This is a reasonable validation for the word-embedding level of the method.

However, what the proof of concept does not demonstrate is that the method accurately decomposes contextual information. The word-length and POS validations operate on non-contextualized embeddings and test against features (word length, POS tags) that are themselves non-contextual. The paper's main contributions concern contextual representations—how layers integrate multi-word information, how attention affects context processing, etc.—and there is no analogous validation for the contextual claims. We don't have an independent "ground truth" for what contextual information a representation should contain in the way we have for word length (we know word embeddings approximately encode word length, and we can verify the method detects this). To validate the contextual decomposition, one would want to show, for instance, that a representation known to encode syntactic parse depth (from probe tasks) predicts Group 2 regions, or that manipulating context length in a controlled way (e.g., scrambling word order) produces predictable changes in Group 2 prediction. Without such a validation, the paper's contextual claims rest on the assumption that the method transfers from word-level to context-level without loss of validity—an assumption that is plausible but untested.

The method's claimed advantage over probe tasks—that it doesn't require specifying what to look for—is demonstrated in principle, but the actual analyses in the paper are all interpreted against the Group 1/Group 2 prior, which is a relatively coarse decomposition (isolated-word vs. multi-word processing). This is an improvement over a single "overall brain prediction accuracy" number, but it doesn't yet achieve the fine-grained decomposition that the introduction's demultiplexer analogy promises (e.g., separating syntactic from semantic from discourse-level information). The prior used is data-driven but also low-dimensional (two functional groups), and the brain's language network is known to be more functionally differentiated than this binary split (e.g., Fedorenko and Thompson-Schill, 2014). The method would need a richer brain prior—perhaps derived from additional naturalistic experiments that functionally characterize sub-regions within Groups 1 and 2—to fulfill its promise of decomposing representations without specifying linguistic categories in advance.

Claim 2: "We use our method to analyze and provide hypotheses about ELMo, BERT, USE and T-XL."

The analyses are genuine and produce interpretable results, but several of the "hypotheses" are post-hoc interpretations of patterns that could have alternative explanations. For example:

USE's poor Group 2 prediction is attributed to "the deep averaging which might be composing words in a crude manner." This is plausible—deep averaging networks compute the element-wise mean of word embeddings and then pass this through feedforward layers, which could lose token-level ordering information important for Group 2 regions. But an alternative explanation is that USE's training objective (a Skip-thought-like sentence coherence task) simply doesn't encourage representations that align with the brain's language processing pathways, irrespective of the architecture. Distinguishing these would require testing a model with the same architecture but a different training objective, or a model with the same training objective but a token-level architecture—neither of which is done.
BERT layer 1's anomalous behavior (Figure 5) is attributed to the layer "combining information from the token-level embeddings in a way that limits the retention of longer context information." This is a reasonable inference from the data (subtracting layer 1's performance makes subsequent layers resemble T-XL), but it's an interpretive claim, not a demonstrated causal mechanism. The paper doesn't test whether re-initializing or re-training layer 1 changes the pattern, which would be needed to support this specific hypothesis.

These are not fatal weaknesses—the paper is clear that it is providing "hypotheses," and the data genuinely constrain the space of possible explanations. But the reader should understand these as informed interpretations rather than experimentally verified causal mechanisms.

Claim 3: Middle layers of transformers are better at predicting brain activity; T-XL performance doesn't degrade as context is increased; uniform attention in early BERT layers improves brain prediction.

These three sub-claims have different levels of support:

Middle layers are optimal for long context (strongly supported across architectures): Figure 4 shows this pattern consistently in ELMo, BERT, and T-XL. The cross-architecture replication is the strongest evidence—it's unlikely to be an artifact of a single model's training or architecture. However, the paper measures only "brain prediction accuracy" (averaged over the whole brain or within specific regions), which is a single scalar per layer per context length. This doesn't tell us what kind of information the middle layers capture that deep layers lose—is it syntactic structure, discourse coherence, semantic composition? The finding is robust as a descriptive claim about prediction performance, but its interpretation (e.g., "deep layers lose long-range context information") requires assuming that brain prediction accuracy is a monotonic proxy for "amount of long-range context information encoded." This is plausible—Group 2 regions specifically require multi-word context—but the paper doesn't break down prediction accuracy by brain region for the layer-depth analysis, which would strengthen the claim. If deep layers show decreased Group 2 prediction specifically while maintaining strong Group 1 prediction, that would directly support the interpretation. Without this spatial decomposition, the decreased accuracy could reflect that deep layers shift toward representing information that the fMRI signal is less sensitive to (e.g., very abstract semantic features) rather than "losing" context.

T-XL continues to increase performance as context grows (supported with qualifications): Figure 4 shows this clearly—T-XL's middle-layer curves rise monotonically with context length, while BERT and ELMo plateau or decline. This validates T-XL's architectural design intent. The qualification is that the context range tested is limited to 40 words; whether T-XL continues to benefit from even longer contexts (100+ words) is unknown, and the fMRI experimental design (words presented at 0.5s each, with hemodynamic lag smearing responses over multiple seconds) may not have the temporal resolution to distinguish 40-word from 80-word context integration. The finding is solid for the tested range but shouldn't be extrapolated to arbitrarily long contexts.

Uniform attention in shallow BERT layers improves brain prediction (supported with evidence of an interaction): Figure 6 demonstrates that uniform attention helps shallow layers and hurts deep layers, and the effect interacts with context length (benefit holds for up to ~25 words). This is the paper's most causally-informative result because it involves an experimental manipulation rather than passive observation. The finding is genuine.

However, there is a significant missing control: the paper does not test random attention as an intermediate condition. Does any deviation from pretrained attention help shallow layers (suggesting that the pretrained attention is actively harmful)? Or does uniform attention specifically help (suggesting a benefit to equal weighting per se)? If replacing shallow-layer attention with random fixed weights also improved brain alignment, the interpretation would be "learned attention in shallow layers is worse than any fixed pattern"—a different claim than "shallow layers should average uniformly." This ablation would be straightforward to implement and would strengthen the paper's mechanistic interpretation.

Additionally, the paper does not report the baseline brain prediction accuracy in absolute terms, only as "% change from learned attention" in Figure 6. If the absolute brain prediction accuracy of layer 1 is already very low (say, 0.52 classification accuracy, barely above the 0.5 chance level), then a +2% change represents a very small absolute improvement—from 0.52 to 0.53. Without knowing the absolute baseline, the practical significance of the attention effect cannot be assessed.

Claim 4: "Altering BERT to better align with brain recordings" leads to "better language understanding" as measured by syntactic probe tasks (strongly supported for the specific tasks and alteration tested).

Table 1 provides clear evidence: brain-aligned BERT (uniform attention in shallow layers) outperforms base BERT on 8/13 syntactic tasks with statistical significance, and the control model (uniform attention in layer 11, which hurt brain alignment) performs worse. This is the paper's most impactful claim, and it is well-supported by the data presented.

However, the scope of this claim requires careful qualification:

The alteration is brain-alignment-motivated, not brain-alignment-optimized. The paper takes the qualitative finding from Figure 6 ("shallow layers benefit from uniform attention") and implements a binary intervention (learned attention → uniform attention). This is a proof-of-concept that brain-alignment insights can transfer to NLP, not a demonstration that maximizing brain alignment maximizes NLP performance. A stronger test would be to train BERT with an auxiliary objective that directly maximizes brain-alignment during pretraining, then evaluate syntactic performance—which is closer to what the paper's conclusion ("altering a model to better align with brain recordings may lead to better language understanding") envisions, but is not attempted here.
The NLP tasks are syntactic probes, not downstream benchmarks. The Marvin and Linzen (2018) tasks test whether BERT has acquired grammatical competence—specifically, subject-verb agreement across various syntactic constructions. These are important measures of linguistic knowledge, but they are not the same as end-task performance (e.g., GLUE benchmark, question answering, summarization). The paper's claim is about "language understanding," but the improvement is demonstrated only on a narrow (though linguistically meaningful) slice of syntactic competence. It's possible that the brain-aligned BERT performs worse on other NLP tasks (e.g., named entity recognition, where attention to specific local context might be important). The paper does not test this—the uniform-attention models are evaluated only on the Marvin and Linzen tasks. This is a reasonable scope for an initial demonstration, but the claim of "better language understanding" is broader than the evidence supports.
The improvement is layer-specific and content-dependent. The brain-alignment finding only holds for shallow layers and for context lengths up to ~25 words (Figure 6). The NLP tasks use complete sentences of varying lengths; the paper doesn't analyze whether the improvement is concentrated in shorter sentences (where the uniform-attention benefit applies) vs. longer sentences. If the improvement is driven entirely by short-sentence examples, the practical significance is more limited.
The baseline BERT is strong, leaving limited headroom. On many tasks, base BERT already achieves 0.83–1.00 accuracy (Table 1). The improvements, while statistically significant, are often small in absolute terms (+0.01 to +0.08 on already-high accuracies). The largest absolute gains are on "across a prepositional phrase" (+0.08, from 0.85 to 0.93) and "in an object relative clause (no that)" (+0.04, from 0.79 to 0.83)—meaningful but not transformative. This doesn't invalidate the claim but contextualizes it: brain alignment provides a modest but real boost to syntactic competence in this setting.
The direction of causality is correlational, not causal, at the level of the NLP claim. The paper's logic is: (Step 1) we observe that uniform attention in shallow layers improves brain alignment; (Step 2) we hypothesize that this brain alignment reflects better linguistic representations; (Step 3) we test whether the same modification improves NLP task performance and find that it does. This is a correlational bridge: brain alignment and NLP performance co-vary under the attention manipulation. But this doesn't prove that brain alignment causes better NLP performance—it could be that uniform attention independently improves both brain prediction and syntactic processing for unrelated reasons (e.g., it acts as a regularizer that reduces overfitting to surface statistics in the pretraining data). To establish causality, one would need to selectively optimize for brain alignment and show that NLP performance improves as a result—which the paper does not do.

What Would Strengthen the Claims?

Missing baselines: The paper does not compare its brain-alignment-based interpretation to interpretations derived from probe tasks. For example, for the layer-depth × context-length analysis, one could train syntactic and semantic probe tasks on each BERT layer at different context lengths, and compare the resulting "probe-task-difficulty curves" to the brain-prediction curves. If the two methods agreed (e.g., both showing peak syntactic encoding in middle layers), that would validate the brain method against an established approach. If they diverged, that would reveal what unique information the brain data provides. This comparison is feasible with existing tools and would substantially strengthen the claim that brain alignment offers complementary interpretability.

Missing scale: All experiments use a BERT-base model (12 layers, 768 hidden units). The findings would be more compelling if replicated on BERT-large (24 layers) to test whether the "middle layer peak" scales with depth and whether the uniform-attention benefit for shallow layers generalizes to models with more layers. Transformer-XL (19 layers) provides some cross-architecture evidence, but within the BERT family, only the base model is tested.

Missing spatial decomposition for the layer-depth analysis: The Figure 4 analysis reports whole-brain or average prediction accuracy. Breaking this down by Group 1 vs. Group 2 regions would directly test whether deep layers lose contextual information specifically (Group 2 prediction drops) while retaining word-level information (Group 1 prediction stays high), vs. losing information uniformly. This analysis is within reach given the data already collected—it would require re-running the classification evaluation separately for voxels in Group 1 and Group 2 regions.

Missing attention ablation (random attention): As noted above, testing whether any fixed attention pattern helps shallow layers (random weights) vs. only uniform averaging would strengthen the mechanistic interpretation and potentially reveal a general principle about early-layer attention.

Missing validation of contextual decomposition: To validate that the Group 2 prior correctly identifies contextual information, the paper could test a model where context integration is experimentally degraded—e.g., BERT with scrambled word order, or BERT trained only on shuffled sentences—and verify that Group 2 prediction drops while Group 1 prediction is maintained. This would confirm that the brain prior is actually sensitive to the kind of contextual information the paper claims it tracks.

Missing generalization of NLP improvement: Evaluating the brain-aligned BERT on a broader set of NLP tasks—including semantic tasks (e.g., semantic textual similarity), discourse tasks, and standard GLUE benchmarks—would establish whether the benefit of uniform shallow-layer attention is specific to syntactic agreement or generalizes to broader measures of language understanding. The paper's conclusion that brain alignment leads to "better language understanding" would be much stronger if supported by generalization across task types.

Genuine Weaknesses

Single stimulus text: All experiments use one chapter of one book (Harry Potter and the Sorcerer's Stone, Chapter 9). This is a single text with specific linguistic properties (narrative fiction, certain vocabulary, specific syntactic constructions). The brain-alignment results may be specific to narrative text processing and not generalize to other genres (news, dialogue, scientific text). The fMRI experiment uses only 45 minutes of data (~5,000 words), which is small by neuroimaging standards—modern naturalistic fMRI studies often use hours of data per subject (e.g., Huth et al., 2016 use 2+ hours). The small dataset limits statistical power and may make the encoding models unstable.

Small subject sample: 8 subjects for fMRI and 3 for MEG is typical for neuroimaging studies of this era but limits generalizability. Individual differences in brain anatomy and language processing strategy are substantial; patterns that hold in 8 subjects may not replicate in a different sample. The paper does not report between-subject variance in the encoding model performance, which would indicate how robust the layer-depth and attention findings are across individuals.

No cross-text or cross-subject validation: The encoding models are trained and tested on the same text (cross-validated across time within the chapter). There is no demonstration that the layer-to-brain-region mappings learned on Harry Potter generalize to different text (a held-out story, or a different genre). This limits the claim that the method reveals "what representations the model captures about language in general" as opposed to "what representations the model captures about this specific text."

The brain prior is coarse. The two-group decomposition (isolated words vs. multi-word sequences) is a starting point, but language processing involves many more functionally distinct processes—phonological encoding, morphological decomposition, syntactic parsing, semantic composition, discourse integration, pragmatic inference, etc. The current prior cannot separate these, which limits the method's interpretability resolution. The paper acknowledges this implicitly by focusing on "context" as the main dimension of variation, but the demultiplexer analogy promised more fine-grained decomposition.

The FLOPs comparison paper structure is absent. Unlike the example paper (which includes a FLOPs-matched comparison between pretraining and test-time compute), Toneva and Wehbe do not include any computational cost analysis. The encoding model training (ridge regression per voxel per layer per context length) is computationally intensive, and the paper doesn't report how this cost scales with model size, layer count, or dataset duration. For the method to be practically adopted by NLP researchers, the computational cost of the brain-alignment analysis itself would need to be characterized and justified.

The NLP improvement is demonstrated on syntactic probes only. While the improvement on Marvin and Linzen tasks is genuine and statistically significant, these tasks measure a specific aspect of linguistic competence (sensitivity to syntactic agreement constraints). Whether brain-aligned modifications improve performance on applied NLP benchmarks (sentiment analysis, reading comprehension, summarization) is an open question. The paper's claim that the altered BERT has "better language understanding" would benefit from a broader evaluation, or from qualifying the claim as "better syntactic understanding as measured by subject-verb agreement probes."

No integration with other interpretability methods. The paper positions brain alignment as an alternative to probe tasks and theoretical analysis, but a more compelling demonstration would show how these approaches complement each other—e.g., showing that brain-alignment reveals contextual integration patterns that probe tasks miss, or that probe tasks reveal specific syntactic knowledge that brain alignment cannot distinguish from general compositional processing. The current framing as "a departure from existing studies that go the other way" (Section 1.1) sets up a false competition rather than a synthesis.

Limited model diversity for 2019: The four models represent major architectural families of the era, but only BERT receives the detailed attention-manipulation experiment. ELMo, USE, and T-XL are characterized observationally (context length, layer depth, region prediction) but not experimentally manipulated. The uniform-attention finding is therefore only demonstrated for BERT—it's unknown whether ELMo or T-XL would show similar layer-dependent effects if their attention/recurrence mechanisms were analogously altered. The generalizability of the paper's most actionable insight (brain alignment → NLP improvement) is supported only for the BERT architecture.

These weaknesses do not invalidate the paper's contributions—the method is genuinely novel, the findings are interpretable and replicable in principle, and the NLP transfer result is a compelling existence proof. But they bound the strength of the conclusions: the paper demonstrates feasibility and promise of brain-based NLP interpretability and improvement, not a mature, validated framework ready for broad deployment. The authors are appropriately cautious in their claims, using language like "we hypothesize," "suggests," and "may lead to," and the paper is best read as an invitation to a research program rather than a definitive solution.

6. Limitations and Trade-offs

Single Stimulus Text and Small Subject Sample Constrain the Inference That Findings Reflect General Language Processing

The assumption or constraint. All experiments in this paper use a single stimulus text—Chapter 9 of Harry Potter and the Sorcerer's Stone—presented to 8 subjects for fMRI and 3 subjects for MEG. The total fMRI recording duration is approximately 45 minutes, yielding roughly 1,350 TRs of data per subject. The paper's interpretations about what linguistic information different NLP model layers encode implicitly assume that the brain-alignment patterns observed on this specific narrative text generalize to language processing more broadly. The authors do not explicitly claim universality, but the framing—"we can use brain activity recordings as a proxy for these representations" and the demultiplexer analogy—positions the brain as a general-purpose interpretability tool, which requires that one chapter of fantasy fiction be representative of linguistic processing.

The consequence. Three distinct generalizability problems cascade from this constraint:

First, genre specificity: Harry Potter is narrative fiction with particular linguistic properties—dialogue interspersed with descriptive prose, fantasy-specific vocabulary, a consistent authorial style, and a coherent narrative arc. A model's representations aligned with brain activity during narrative comprehension may not align with brain activity during other genres (news articles, scientific abstracts, conversational transcripts, legal documents). For example, discourse-level features like character tracking and plot coherence, which are prominent in the Harry Potter text and likely engage Group 2 regions, may not be the same kind of "long-range context" as tracking an argument across paragraphs of a scientific paper. If the brain-alignment patterns are genre-specific, then conclusions like "T-XL is the only model that continues to increase performance as context length is increased" (Section 4, Figure 4) might reflect T-XL's superior handling of narrative context specifically, not context integration in general.

Second, subject sample size: 8 subjects for fMRI is standard for neuroimaging studies of this era but is small by the standards of replicable individual-differences research. Language processing exhibits substantial individual variability—in anatomy (the precise location of language-responsive cortex varies across subjects by centimeters), in reading strategy (some readers engage visual imagery more heavily, others focus on propositional content), and in baseline brain signal-to-noise ratio. The paper reports group-average results on the MNI template, but does not report between-subject variance in encoding model performance. If, for example, the "middle layers are optimal for long context" finding holds in 5 of 8 subjects but reverses in the other 3, the group-average result is meaningful but fragile—a different sample of 8 subjects drawn from the same population could produce a different optimal layer depth. The standard error bars in Figure 3 provide some indication of subject-level consistency for the word-embedding vs. context comparison, but the layer-depth × context-length analysis (Figure 4) and the attention-manipulation results (Figure 6) are presented without between-subject variance estimates.

Third, no cross-text validation: All encoding models are trained and evaluated on the same text chapter via cross-validation across time. There is no demonstration that a model trained to map layer-7 BERT representations to a subject's brain activity on Harry Potter Chapter 9 can generalize to predict that subject's brain activity on Chapter 10. If the brain-region-to-layer mapping is text-dependent—for instance, if layer 7 best predicts Group 2 regions for Harry Potter but layer 9 best predicts them for a news article—then the framework reveals text-specific representational alignment, not a stable property of the NLP model's architecture. Without cross-text validation, the paper's conclusions about model architecture (e.g., "shallow layers benefit from uniform attention," Section 4) are confounded with the specific text.

What evidence exists in the paper. The paper provides no cross-text validation and does not report between-subject variance for the layer-depth or attention analyses. The authors do not explicitly discuss genre generalizability as a limitation, though they do note in Section 1.1 that the brain prior is "data-driven" and derived from Lerner et al. (2011), whose experiments used a narrated story—also narrative text. This means even the prior itself (the Group 1/Group 2 decomposition) may be most valid for narrative comprehension and less applicable to other genres.

Mitigation status. Not addressed. The paper does not suggest that genre or text specificity is a limitation, nor does it propose cross-text validation as future work. The Future Work section (Section 6) focuses on expanding the MEG analysis and developing "more informative fMRI brain priors," not on testing generalizability. A practitioner considering deploying this framework would need to independently validate that the brain-alignment patterns replicate on their target text domain.

Difficulty Estimation Cost Is Dominant and Unaccounted For

The assumption or constraint. The paper's framework requires, for each NLP model layer and context length under investigation, training a separate encoding model (ridge regression with per-voxel nested cross-validation) and running the classification-based evaluation across the entire brain. The authors do not report computational cost, but the procedure is intensive: for BERT-base alone, 12 layers × multiple context lengths (1, 5, 10, 15, 20, 25, 30, 35, 40) × 4 cross-validation folds × ~30,000 cortical voxels per subject × nested 10-fold CV for $\lambda$ selection × 8 subjects. Each voxel-level encoding model requires solving a ridge regression problem multiple times (once for each $\lambda$ candidate, then a final fit). The attention-manipulation experiments add another factor: 12 single-layer alterations × context lengths × the same CV procedure. The total computation—while not pretraining-scale—is substantial and must be repeated for each new model, each new text, and each new subject cohort. This cost is entirely unaccounted for in the paper's analyses; there is no discussion of wall-clock time, FLOPs, or memory requirements.

The consequence. The practical consequence is a cost-to-insight ratio problem for adoption. A researcher wanting to use this method to interpret their own model would need to: (1) acquire brain imaging data on subjects reading text representative of their domain (the Harry Potter data is pre-existing but may not generalize); (2) run their model's forward passes on the same text for every context length and layer; (3) train and evaluate tens of thousands of encoding models. The computational cost of steps (2) and (3) is likely orders of magnitude larger than standard probe-task evaluation (where one trains a small linear classifier on frozen layer representations for a few dozen NLP tasks). The paper's framing as an interpretability tool competes with probe tasks and theoretical analysis, but the cost comparison is never made. For many NLP practitioners, the computational overhead—including the need for neuroimaging data and expertise—would be prohibitive relative to the interpretability gain over simpler methods.

A second consequence is that the cost limits comparative analysis scale. The paper evaluates four models (ELMo, BERT, USE, T-XL) with limited layer-depth and context-length sweeps. Scaling this to compare 10+ models (the modern LLM landscape), to sweep more context lengths, to test more architectural modifications (e.g., ablating specific attention heads, varying layer width, testing different pretraining objectives), or to evaluate on multiple texts would multiply the already-unaccounted cost. This means the method, as presented, is suited for detailed case studies of a few models rather than large-scale comparative interpretability, which limits its utility for guiding architecture search or pretraining objective design.

A third, more subtle consequence is that the encoding model quality itself is unquantified as a function of this cost. The ridge regression with nested CV is a standard pipeline, but the paper does not report the $R^2$ or prediction error of the encoding models in absolute terms—only classification accuracy from the downstream two-alternative task. If the encoding models are poor fits (e.g., $R^2$ near zero, with classification accuracy barely above chance), the entire interpretability pipeline rests on a weak signal. The classification task aggregates 20 TRs to boost SNR, but the underlying regression quality is never assessed. A practitioner cannot determine whether the computational investment in nested CV and per-voxel $\lambda$ tuning is actually producing well-fit encoding models or whether simpler methods (fixed $\lambda$ , linear regression without temporal lags) would suffice.

What evidence exists in the paper. None. The paper does not report encoding model fit quality, computational cost, wall-clock time, or any efficiency metric. The supplementary materials (Section 10) describe the encoding model procedure in detail but provide no cost estimates.

Mitigation status. Not addressed. The paper does not acknowledge computational cost as a limitation. A partial mitigation—which the paper does not explore—would be to demonstrate that a reduced-cost version of the pipeline (e.g., using a fixed $\lambda$ for all voxels, skipping nested CV, using fewer context lengths) produces qualitatively identical layer-depth and attention-effect results. Without such a cost-sensitivity analysis, the default assumption is that the full pipeline as described is necessary for the reported findings.

The Improvement in NLP Task Performance Is Demonstrated Only for Syntactic Agreement Probes, Not for General Language Understanding

The assumption or constraint. Section 5 claims that "altering BERT to better align with brain recordings would enable it to also better understand language" (abstract) and that the brain-aligned models "significantly outperform the pretrained model" (Section 5, Table 1). The evaluation of this claim is restricted to the 13 syntactic probe tasks from Marvin and Linzen (2018), all of which measure subject-verb agreement across various sentence constructions. These tasks test whether the model can distinguish the correct verb form (e.g., is) from an incorrect verb form (e.g., are) given a masked position in a complete sentence. This is a syntactic competence test—it measures sensitivity to grammatical number agreement—not a test of general language understanding. The paper's language implicitly generalizes from syntactic agreement to broader language comprehension: "this result shows a transfer of knowledge from the brain to NLP tasks and validates our approach" and "the model with increased brain-alignment outperforms the original model" (abstract).

The consequence. The gap between what was measured and what was claimed has two important dimensions:

First, the tasks are narrow in linguistic scope. The Marvin and Linzen tasks test one specific syntactic phenomenon: number agreement between a subject and a verb, in contexts where the subject and verb are separated by intervening linguistic material (prepositional phrases, relative clauses, sentential complements). Performing well on these tasks requires that the model track the syntactic subject across intervening words and select the appropriate verb inflection—a genuine test of syntactic processing. However, language understanding encompasses much more: semantic composition, discourse coherence, pragmatic inference, coreference resolution, named entity recognition, sentiment analysis, question answering, and on and on. The brain-aligned BERT's improvement on subject-verb agreement does not imply that brain alignment improves any of these other capabilities. In fact, because the uniform-attention modification removes learned attention—forcing the model to average uniformly over context—it could plausibly harm tasks that require selective attention to specific context words (e.g., coreference resolution, where the model must link a pronoun to a specific antecedent, not average over all preceding nouns). The paper does not test this, and the possibility is not discussed.

Second, the tasks are evaluated in a zero-shot probing setup, not as downstream benchmarks with task-specific fine-tuning. The evaluation protocol from Goldberg (2019) uses BERT's pretrained masked-language-modeling head to predict the masked verb—this tests what syntactic knowledge is implicitly encoded in the representations before any task-specific adaptation. Improving zero-shot syntactic agreement is a meaningful result, but it does not demonstrate that brain alignment improves performance after fine-tuning on downstream tasks (the standard deployment paradigm for BERT). If the brain-aligned BERT showed worse fine-tuning performance (e.g., because uniform shallow attention removes attention patterns that are useful initialization for task-specific fine-tuning), the practical value of the brain-alignment insight would be limited.

What evidence exists in the paper. Table 1 provides per-task accuracies for the 13 Marvin and Linzen tasks, comparing base BERT against uniform-attention models at layers 1, 2, 6, and 11. The results show statistically significant improvements on 8 of 13 tasks, no significant difference on 4 of 5 remaining tasks, and a significant difference favoring brain alignment on "across a prepositional phrase" (+0.08) and "in an object relative clause (no that)" (+0.04) as the largest-gain tasks. The paper provides no evaluation on any other NLP benchmark: no semantic tasks, no discourse tasks, no GLUE tasks, no reading comprehension, no generation tasks.

Mitigation status. Not addressed in the paper. The authors do not qualify the "better language understanding" claim to "improvement on syntactic agreement probes," and the Future Work section (Section 6) does not mention expanding the NLP evaluation. The abstract's framing—"Probing the altered BERT using syntactic NLP tasks reveals that the model with increased brain-alignment outperforms the original model"—is technically accurate (they did probe with syntactic tasks and did find improvement), but the surrounding language about "better understand language" is broader. A practitioner reading the paper should understand that the demonstrated NLP benefit is restricted to syntactic agreement and that generalization to other linguistic phenomena or downstream tasks is untested.

The Method Cannot Resolve Whether Brain-Aligned Representations Are Better Because They Are Brain-Like or Because Uniform Attention Acts as a Beneficial Regularizer

The assumption or constraint. The paper's central causal chain is: (1) uniform attention in shallow BERT layers improves brain alignment (Figure 6); (2) the same modification improves syntactic NLP task performance (Table 1); therefore (3) improving brain alignment leads to better language understanding. This chain implicitly assumes that the NLP improvement is mediated by increased brain alignment—that brain-aligned representations are functionally better for language processing. However, the experimental design cannot distinguish this from an alternative explanation: uniform attention acts as a architectural regularizer that coincidentally improves both brain prediction and syntactic competence through a mechanism unrelated to brain-likeness.

The consequence. If the alternative explanation is correct, then the brain data was a useful heuristic for discovering a beneficial architectural modification (uniform shallow attention), but the benefit does not depend on brain alignment per se. This would fundamentally change the paper's contribution: instead of "we can use the brain to improve NLP models" (a claim about cross-domain knowledge transfer), it would be "we accidentally discovered that early BERT layers are over-parameterized in their attention" (a claim about model architecture). The practical implication diverges sharply: under the brain-mediated account, one would pursue more brain alignment (e.g., training models to directly maximize encoding model performance, using richer brain priors, incorporating MEG temporal constraints) to further improve NLP. Under the regularizer account, one would pursue less attention in early layers (e.g., simpler parameterizations, fixed pooling operations) regardless of brain data, and the brain experiment was merely the serendipitous discovery mechanism.

Several specific alternative mechanisms are consistent with the data:

Uniform attention prevents overfitting to spurious position-specific statistics in the pretraining data. The pretrained BERT's learned attention may have specialized to statistical regularities of the BooksCorpus and English Wikipedia (the pretraining corpora) that do not generalize to the syntactic agreement tasks or to the brain's language processing strategies. Uniform attention removes this specialization, acting as a variance-reduction technique.
Uniform attention reduces representational capacity in early layers, forcing the model to defer complex processing to deeper layers where it can be more effective. This is a form of architectural bottleneck that improves generalization—analogous to how reducing the capacity of early layers in convolutional networks can improve robustness.
The effect is specific to the pretrained weights and would disappear if BERT were re-initialized and re-trained with uniform early attention from the start (i.e., the benefit is about removing harmful learned patterns, not about uniform attention per se being superior).

What evidence exists in the paper. The paper provides no experiments that test causal mediation. There is no condition where brain alignment is manipulated independently of attention (e.g., by adding a brain-alignment auxiliary loss during pretraining) to see if NLP performance tracks brain alignment specifically. There is no comparison between uniform attention and other fixed attention patterns (e.g., random fixed weights, position-biased attention, local-window attention) to see if the NLP benefit is specific to uniform attention or shared by any non-learned pattern. There is no experiment where brain alignment is measured for an unrelated architectural modification (e.g., varying layer width, removing residual connections) to establish that brain alignment uniquely tracks NLP-relevant properties rather than general model quality.

Figure 6 and Table 1 together establish a correlation between brain alignment and NLP performance under the attention manipulation, but this is consistent with both the brain-mediated and the regularizer explanations. The paper's language throughout assumes the brain-mediated interpretation without acknowledging the alternative.

Mitigation status. Partially addressed by the uni L11 control in Table 1. The fact that uniform attention in layer 11 (which harms brain alignment) also harms NLP performance provides converging evidence that brain alignment and NLP performance covary. But this only strengthens the correlation, not the causal direction. The regularizer hypothesis can accommodate the layer-11 result if uniform attention acts as a beneficial regularizer specifically in early layers (where it may prevent overfitting to low-level statistics) and a harmful capacity reduction in deep layers (where learned attention is genuinely needed for syntactic processing). The paper does not discuss this alternative or propose experiments to distinguish it.

The Brain Prior Is Too Coarse to Deliver on the Demultiplexer Promise, Limiting the Method's Interpretability Resolution

The assumption or constraint. The paper's central conceptual contribution is the demultiplexer analogy: "In contrast to the brain, a network-derived representation might encode information that is related to multiple of these processes without a specific organization. When we align that specific network representation with fMRI and MEG data, the result will be a decomposition of the representation into parts that correspond to different processes and should therefore be more interpretable" (Section 1.1). This promise requires a brain prior that differentiates multiple language sub-processes—syntactic parsing, semantic composition, discourse integration, lexical access, morphological decomposition, etc. The actual prior used in the paper divides the language network into only two functional groups: Group 1 (regions activated by both isolated words and connected text) and Group 2 (regions activated only by connected text). This is a single binary distinction—isolated-word processing vs. multi-word sequence processing—which is far coarser than the rich functional differentiation known to exist in the language network.

The consequence. The binary prior constrains what the method can discover. The paper's main results demonstrate that ELMo, BERT, and T-XL 10-word representations predict activity in Group 2 regions (Figure 2), confirming that these models encode information relevant to multi-word processing—but this only tells us that the models capture some kind of contextual information, not what kind. A layer could predict Group 2 regions because it encodes syntactic parse depth, or semantic composition, or discourse coherence, or some combination. The method cannot distinguish these because the prior does not distinguish them. The demultiplexer analogy promised decomposition into "different processes," but the actual decomposition achieved is "processes that care about word order vs. processes that don't"—a single bit of functional resolution.

This is particularly limiting for the layer-depth analysis (Figure 4). The finding that middle layers best predict "brain activity" (averaged across regions or in unspecified whole-brain measures) is suggestive but underspecified. If the authors could show that middle layers specifically predict syntactic-processing regions, while deep layers predict discourse-level regions, the interpretation would be richer and more actionable. With only the Group 1/Group 2 prior, the most the paper can say is that middle layers best predict "long-range context" processing—useful, but far from the fine-grained functional decomposition the demultiplexer analogy envisioned.

The proof-of-concept (Figure 8, Supplementary Section 11) demonstrates that MEG's temporal resolution can separate word length (early visual) from part-of-speech (later frontal) information within a single word embedding, suggesting that the method can achieve finer-grained decomposition when the brain modality supports it. But the fMRI-based main analyses—which are the paper's primary contributions—do not exploit this temporal dimension and are limited to the two-group spatial prior.

What evidence exists in the paper. The prior is described in Section 1.1 and Figure 1, with the specific brain regions listed in Supplementary Table 2. The paper explicitly acknowledges that this is a simplified prior: "Fig. 1 shows a simple approximation of these areas on the Montreal Neurological Institute (MNI) template." The authors also cite Jain and Huth (2018) as corroborating evidence, but Jain and Huth's work is itself a context-length manipulation (varying LSTM context windows), which aligns with the Group 1/Group 2 distinction but does not provide finer functional sub-parcellation.

Mitigation status. The limitation is partially acknowledged by the Future Work (Section 6), which states: "Our next steps are to... derive more informative fMRI brain priors that contain specific conceptual information that is linked to brain areas, and use them to study the high level semantic information in network representations." This indicates the authors recognize the coarseness of the current prior as a limitation and plan to address it. However, the paper does not discuss how this coarseness limits the current findings—the caveat is buried in future work rather than presented as a constraint on the present results. A practitioner using the method would need richer brain priors (e.g., functionally-defined regions of interest from meta-analyses or additional naturalistic experiments) to achieve the decomposition resolution the paper advertises.

The NLP Evaluation Does Not Test Whether Brain Alignment Outperforms Simpler, Cheaper Architectural Modifications

The assumption or constraint. Section 5 demonstrates that BERT with uniform attention in shallow layers outperforms base BERT on syntactic agreement tasks. The paper interprets this as evidence that brain-alignment insights transfer to NLP. However, the experimental design lacks a critical control: does the uniform-attention modification outperform random or theory-agnostic architectural modifications of comparable simplicity? Without this control, we cannot determine whether brain alignment is necessary for the improvement, or whether any reasonable perturbation of early-layer attention would produce similar gains (making the brain experiment a costly route to a result that could have been discovered by random search or architectural sweeps).

The consequence. If randomly zeroing attention weights in randomly selected layers also improves syntactic agreement performance, then the brain alignment was incidental—it selected one beneficial modification from a space where many modifications are beneficial. This would not invalidate the paper's claim that the brain-aligned modification improves NLP, but it would undermine the claim that brain alignment specifically provides useful guidance for model improvement. The value proposition of the method depends on brain data providing better or more efficient guidance than alternative approaches to model modification. If 1,000 randomly chosen shallow-layer attention ablations were tested and the 50 with the best validation performance also improved syntactic agreement, then the brain experiment is no more informative than random architectural search—just far more expensive.

More specifically, the paper does not compare against:

Random layer ablation: Selecting a random subset of shallow layers to apply uniform attention to (not just single layers, but combinations) and testing NLP performance. The current experiment only modifies one layer at a time.
Attention head dropout: Randomly dropping attention heads in shallow layers (a standard regularization technique) rather than replacing entire layers with uniform attention.
Weight decay or attention regularization during pretraining: Training BERT with explicit regularization on attention entropy (encouraging more uniform attention) and testing whether this improves syntactic probes.
Simpler fixed attention patterns: Replacing learned attention with position-biased attention (e.g., favoring nearby words), or with a learned but layer-invariant attention pattern, to test whether the benefit is specific to uniform attention or to non-learned attention generally.

What evidence exists in the paper. The uni L11 control demonstrates that the modification is not universally beneficial (it hurts deep-layer attention), which provides some specificity. But this only shows that the modification must be applied to the right layers—it doesn't show that the right layers are uniquely identifiable through brain alignment rather than through simpler methods (e.g., a sweep of single-layer attention ablations evaluated on a syntactic validation set, which would be computationally cheaper than acquiring and analyzing brain data).

The paper also shows that layers 3–5 (Supplementary Table 3) produce smaller or less consistent NLP improvements than layers 1, 2, and 6, suggesting that not all shallow-layer uniform attention is equally beneficial. But the paper does not test whether this pattern of layer-specific NLP improvement matches the pattern of layer-specific brain alignment improvement (i.e., does the layer that shows the largest brain alignment gain from uniform attention also show the largest NLP gain?). Such a within-paper correlation analysis would strengthen the claim that brain alignment specifically tracks NLP benefit, but it is not reported.

Mitigation status. Not addressed. The paper does not discuss alternative architectural modification strategies or compare against non-brain-informed modifications. This is partly a scope limitation—the paper's goal is to demonstrate that brain alignment can yield NLP improvements, not that it is the most efficient route to such improvements—but the claim of "knowledge transfer from brain to NLP" could be strengthened by demonstrating that the brain provides information not obtainable through simpler means. The strongest version of this would be: a modification identified through brain alignment that is counterintuitive from an NLP perspective (i.e., that an NLP researcher would not have thought to try) but that nonetheless improves NLP performance. The uniform-attention finding has this character—replacing learned attention with uniform averaging is not an obvious NLP-motivated modification—but without the control experiments, it remains possible that motivated NLP researchers would have discovered the same modification through standard architectural search.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new category of interpretability tool for NLP: using the human brain's functional organization as a biological readout that decomposes neural network representations without requiring researchers to specify in advance which linguistic properties to test for. The magnitude of the shift is best characterized as opening a new axis of evaluation, not a paradigm shift that replaces existing methods. Probe tasks, attention visualization, and theoretical capacity analysis remain valuable; this work adds brain alignment as a complementary approach that is uniquely suited for studying complex, high-level linguistic phenomena that are difficult to operationalize as classification tasks. The paper's own admission that it cannot yet decompose representations at fine functional granularity (Section 6, Future Work) makes clear that this is the beginning of a research program, not a mature replacement for existing tools.

The most landscape-changing aspect of the work is the closing of the loop between NLP and cognitive neuroscience. Prior to this paper, the dominant direction of interaction was model-to-brain: researchers used NLP representations as feature spaces to model brain activity (Wehbe et al., 2014a; Jain and Huth, 2018), treating NLP models as engineering tools for neuroscience. This paper inverts that relationship, demonstrating that brain data can inform model design and improvement—specifically, that a brain-alignment observation (uniform attention helps shallow BERT layers predict brain activity) transfers to measurable gains on syntactic competence tasks (Table 1, Section 5). This is a proof of existence: brain-guided model modification can improve NLP performance. Even if the specific finding (uniform attention in layers 1–6) turns out to be architecture- or task-specific, the demonstration that the brain-to-model direction yields actionable insights establishes a new feedback loop that did not previously exist.

The paper also reconciles a latent tension in how the field thinks about attention mechanisms. Since Vaswani et al. (2017), the dominant assumption has been that learned, content-dependent attention is uniformly beneficial—more sophisticated attention patterns should produce better representations at every layer. This paper provides counter-evidence: in pretrained BERT, the learned attention in shallow layers actually produces representations that are less brain-aligned than simple uniform averaging, and the uniform-attention variant improves syntactic agreement performance on 8 of 13 probe tasks. This does not overturn the value of attention mechanisms generally (the deep layers rely on learned attention for both brain alignment and NLP performance), but it reframes the question from "is attention good?" to "where in the network is learned attention beneficial, and what is it doing at different depths?" The finding that the benefit of learned attention is layer-dependent—and that the optimal pattern varies with context length (Figure 6)—shifts attention research toward understanding functional specialization across network depth, a question that had been under-explored relative to studying attention patterns at a single layer or averaged across layers.

For the interpretability community specifically, this work makes probe-task-only approaches less attractive as a complete solution. The limitation that probe tasks "can only detect linguistic properties that researchers explicitly design tasks for" (Section 2, prior sections) was already recognized, but this paper provides a concrete alternative rather than merely critiquing the status quo. The proof-of-concept (Supplementary Figure 8) demonstrates that brain alignment can recover known linguistic properties (word length, part-of-speech) without designing tasks for them—the brain's temporal and spatial organization provides the decomposition. This does not make probe tasks obsolete; they remain essential for testing specific hypotheses about whether a particular syntactic or semantic feature is encoded. But for exploratory analysis—"what does this layer represent?" without a prior hypothesis—brain alignment offers a path that probe tasks structurally cannot provide. The paper thus shifts the interpretability landscape from a single-method paradigm (design tasks → test for known properties) toward a multi-method one where brain alignment handles open-ended discovery and probe tasks handle hypothesis-driven confirmation.

For the model improvement community, the paper introduces a new source of architectural guidance that is orthogonal to standard validation metrics. The typical workflow for improving NLP models is to propose an architectural change, train the model, and evaluate on downstream benchmarks. This paper demonstrates a third signal: modifying the model to better predict brain activity, then testing whether that modification improves NLP performance. The fact that this worked (Table 1) even with a crude intervention (replacing learned attention with uniform averaging) suggests that brain alignment could serve as a development-phase diagnostic for model design—a way to evaluate architectural decisions before incurring the full cost of end-task training and evaluation. This is not yet practical (the encoding model pipeline is computationally expensive, as discussed in Section 6), but it establishes a principle: brain data can provide a training-free signal about representational quality that correlates with downstream linguistic competence.

Finally, the paper changes how the field should think about Transformer-XL's architectural contribution. Dai et al. (2019) designed T-XL's segment-level recurrence to capture longer context than standard transformers, and demonstrated improvements on language modeling perplexity. This paper provides independent, biologically-grounded evidence that the mechanism works as intended: T-XL is the only tested model whose brain prediction performance "continues to increase as the context length is increased" out to 40 words (Figure 4). This is a form of architectural validation that goes beyond benchmark metrics—T-XL was not trained to predict brain activity, yet its representations show sustained context integration that aligns with the brain's Group 2 regions (which require multi-word sequences). This finding makes biological plausibility a more attractive auxiliary criterion for evaluating architectural innovations: a new recurrence or attention mechanism can be assessed not just on perplexity or downstream accuracy, but on whether it produces representations that track context the way the human brain does. This is not a replacement for task-based evaluation, but an additional dimension that may reveal representational properties invisible to standard metrics.

Follow-Up Research This Work Enables

Training models with an explicit brain-alignment auxiliary objective during pretraining. The paper shows that a post-hoc modification to BERT (replacing learned attention with uniform averaging in shallow layers) improves both brain alignment and syntactic NLP performance. The obvious next step is to train a model from scratch with a joint objective: standard masked language modeling loss plus a term that penalizes deviation between the model's layer representations and the brain activity patterns they should predict. Concretely, for a subset of training sentences that have corresponding brain data (requiring a larger neuroimaging dataset than the single Harry Potter chapter used here), one would add a ridge-regression encoding loss at each layer, encouraging the representations to linearly predict the recorded brain activity. The key question is whether this joint training produces representations that (a) predict brain activity better than post-hoc alignment, (b) outperform post-hoc modified BERT on syntactic probe tasks, and (c) generalize to standard NLP benchmarks (GLUE, question answering) without degrading performance—addressing the concern from Section 6 that the current benefit is restricted to syntactic agreement. This experiment would also help resolve the causal ambiguity identified in Section 6: if directly optimizing for brain alignment improves NLP performance, that strengthens the brain-mediated account; if it does not, the uniform-attention benefit may be a regularizer effect orthogonal to brain-likeness.

Constructing and validating finer-grained brain priors for sub-process decomposition. The paper's demultiplexer analogy promises that brain alignment will decompose network representations into parts corresponding to "different processes," but the current Group 1/Group 2 prior is a binary distinction (isolated-word vs. multi-word processing). A high-priority follow-up is to develop richer functional parcellations of the language network from existing data. One approach: use the naturalistic fMRI datasets from Huth et al. (2016) or Wehbe et al. (2014b) to define regions of interest (ROIs) based on voxel-wise encoding model weights for different linguistic feature spaces—e.g., regions that are best predicted by syntactic parse features vs. semantic word vectors vs. discourse-level coherence metrics. These functionally-defined ROIs would replace the binary Group 1/Group 2 prior with a multi-way decomposition: "this layer predicts syntactic regions but not semantic regions," "this layer predicts discourse regions but not syntactic regions." The validation would be to show that these new priors produce layer-wise brain-alignment profiles that match probe-task results (e.g., a layer that predicts syntactic brain regions should also show high syntactic probe accuracy), establishing convergent validity. This extension directly addresses the "coarse prior" limitation from Section 6 and moves the method toward its advertised capability.

Testing whether brain alignment generalizes across text genres and subject populations. All experiments use Chapter 9 of Harry Potter with 8 fMRI subjects and 3 MEG subjects. A necessary stress test is to replicate the main findings—middle-layer optimality for context, T-XL's sustained context integration, and the uniform-attention benefit for shallow BERT layers—on a different text genre with a different subject sample. Concretely: use an existing naturalistic neuroimaging dataset with a different genre (e.g., the Moth radio stories from Huth et al., 2016; scientific lectures; conversational transcripts) and a larger subject sample (20+), run the full encoding-model-to-classification pipeline for BERT at minimum (layers 1–12, context lengths 1–40), and test whether (a) the layer-depth × context-length interaction curves (Figure 4) replicate, (b) the uniform-attention benefit for shallow layers (Figure 6) replicates, and (c) the layer that benefits most from uniform attention is the same across genres. A negative result on any of these would bound the generalizability of the current findings and indicate that brain-alignment properties are text-dependent—a critical qualification for the method's claimed utility as a general interpretability tool. A positive result across genres would substantially strengthen the paper's conclusions.

Systematic comparison of brain-alignment-guided vs. probe-task-guided layer-wise analysis. The paper positions brain alignment as complementary to probe tasks but never directly compares them. A rigorous follow-up would take a single model (e.g., BERT-base), apply both methods to every layer, and quantify agreement and divergence. For brain alignment: train encoding models per layer, compute classification accuracy in specific ROIs (Group 1, Group 2, and ideally finer-grained syntactic/semantic ROIs). For probe tasks: use existing suites like SentEval (Conneau et al., 2018) or the Marvin and Linzen (2018) tasks, training linear classifiers on each layer's representations. The key analysis is a layer-wise correlation matrix: does the layer that best predicts syntactic brain ROIs also show the highest syntactic probe accuracy? Does the layer that best predicts semantic brain ROIs show the highest semantic probe accuracy? Where do the methods diverge, and what do those divergences reveal—do some layers encode brain-relevant information that no probe task captures (evidence for brain alignment's unique value), or do some probes detect information that brain alignment misses (evidence for complementarity)? This experiment requires no new data collection, only running existing probe suites on the same representations already extracted for the brain-alignment analysis, and would directly address the "missing baseline" critique from Section 6.

Adversarial testing of the brain-alignment-to-NLP transfer claim using architectures where brain alignment is deliberately degraded. The paper's headline result is that a brain-aligned BERT variant outperforms base BERT on syntactic probes. To test whether this relationship is causal rather than correlational, one could systematically degrade brain alignment in a controlled way and measure the effect on NLP performance. For example: train BERT variants with attention patterns that are explicitly manipulated to reduce brain prediction accuracy in specific ROIs (e.g., by forcing shallow-layer attention to focus on a single random token rather than averaging uniformly, which should degrade Group 1 prediction), then evaluate these variants on the Marvin and Linzen tasks. If NLP performance tracks the brain-alignment degradation monotonically (i.e., worse brain alignment → worse syntactic agreement), that strengthens the causal interpretation. If some manipulations degrade brain alignment but leave NLP performance unchanged, or improve NLP performance while degrading brain alignment, that would reveal boundary conditions on the transfer claim. A particularly informative variant: what happens if you make shallow-layer attention more sophisticated (e.g., adding more heads, using relative position encodings) instead of less? Does this improve brain alignment (contrary to the uniform-attention benefit) and if so, does NLP performance also improve? This experiment would clarify whether the benefit is about simplifying shallow attention specifically or about matching the brain's context integration strategy more generally.

Scaling to larger models and more recent architectures. The paper's analyses use BERT-base (12 layers, 768 hidden), ELMo (2 layers), T-XL (19 layers), and USE—all models from the 2018–2019 era. A natural extension is to apply the same encoding-model pipeline to more recent architectures: BERT-large (24 layers), RoBERTa, GPT-2/3 (to study unidirectional vs. bidirectional context integration), T5 (to study encoder-decoder architectures), and instruction-tuned models (to study how task conditioning changes representational alignment with the brain). The specific question is whether the paper's findings scale: does the "middle layer peak" for long-context integration shift proportionally with total depth (e.g., to layers 12–16 in a 24-layer model)? Does the uniform-attention benefit for shallow layers hold in models with different pretraining objectives (e.g., GPT's causal language modeling vs. BERT's masked language modeling)? A particularly informative comparison would be between a standard pretrained model and its instruction-tuned or RLHF-tuned variant, tested on the same text—does alignment with human brain activity increase or decrease after fine-tuning to follow instructions, and what does that tell us about what fine-tuning does to linguistic representations? This scaling experiment requires no new brain data (the existing Harry Potter dataset can be used) but would substantially extend the generalizability of the framework beyond the specific models tested.

Practical Applications and Downstream Use Cases

Model development diagnostic: screening architectural variants before full training. The paper demonstrates that brain-alignment measurements on a pretrained model can identify architectural inefficiencies—specifically, that BERT's shallow-layer learned attention is counterproductive for encoding brain-like linguistic structure (Figure 6). For an industrial NLP team developing a new transformer variant, a practical workflow would be: (1) acquire a small neuroimaging dataset on representative text (even a single subject, single session of ~1 hour of fMRI reading data, following the encoding model protocol); (2) run frozen pretrained candidate architectures through the encoding-model-to-classification pipeline; (3) use the resulting brain-alignment profiles (layer depth × brain region matrices) to compare architectures before committing to full downstream evaluation. The specific metrics that showed signal in this paper—the layer at which Group 2 prediction peaks, whether context integration is sustained or degrades at deep layers, whether shallow-layer attention is beneficial or harmful—can serve as development criteria. The cost of the neuroimaging data acquisition and encoding model training must be weighed against the cost of training and evaluating multiple candidate models on downstream benchmarks. For large models where a single pretraining run costs hundreds of thousands of dollars, a $10,000–$ 50,000 neuroimaging study that identifies a beneficial architectural modification (analogous to the uniform-attention finding) would be cost-effective. The key prerequisite not yet demonstrated is that brain-alignment quality during development predicts downstream task performance—the paper shows this for one modification on syntactic probes, but generalization to broader benchmarks needs validation.

Neuroscience-informed architecture search for long-context models. The paper's finding that Transformer-XL is the only tested model with sustained context integration out to 40 words (Figure 4) provides a concrete design principle: models intended for tasks requiring long-range linguistic dependencies (document summarization, multi-turn dialogue, narrative understanding) should be evaluated not just on perplexity or task accuracy, but on whether their intermediate representations show sustained improvement in brain prediction as context length grows. A practical application for a team building a long-context model would be: during architecture development, measure the brain-alignment context-length curve (as in Figure 4) for candidate recurrence/attention mechanisms. A mechanism that shows a plateau or decline in brain prediction beyond some context length—like BERT's deep layers (Figure 4, middle panel)—may have a fundamental limitation for long-range linguistic processing that is not visible in perplexity scores (which can be driven by local statistical patterns). The T-XL curve provides a positive template: the ideal architecture's middle-layer brain prediction should rise monotonically with context length, indicating that the model is genuinely integrating information across the full available window in a way that aligns with human language processing. This use case requires no new methodology beyond what the paper already provides—the layer-depth × context-length encoding model sweep—applied to the new candidate architectures.

Quality assurance for NLP models deployed in brain-computer interface or neurolinguistic applications. In clinical or research settings where NLP models are used to process or generate text that will be presented to human subjects in neuroimaging experiments (e.g., generating stimuli for fMRI studies of language, building assistive communication devices that present text to users), the brain-alignment framework provides a direct way to evaluate whether the model's representations match human language processing. If a text-generation model produces sentences whose BERT/BERT-variant representations poorly predict activity in the language network (particularly Group 2 regions), the generated text may place unusual processing demands on human readers—even if the text is grammatically well-formed. A concrete application: before using an NLP model to generate stimuli for an fMRI experiment on syntactic processing, run the stimuli through the encoding model pipeline with an existing brain dataset to verify that the model's representations of those stimuli show the expected brain-region prediction profile (Group 2 regions predicted by middle layers, sustained context integration, etc.). This is a domain-specific use case, but it directly leverages the paper's demonstrated finding that different models' representations differentially predict Group 2 activity (Figure 3, USE vs. BERT/T-XL), providing a principled way to select models for human-facing language applications based on their alignment with biological language processing rather than just corpus-derived metrics.