Memorizing Transformers

ArXiv: 2012.00363

🎯 Pitch

You can surgically edit facts inside a Transformer without retraining from scratch—but counterintuitively, limiting weight updates to the first or last layers works better than fine-tuning the entire model. This constrained, layer-specific approach lets language models forget outdated knowledge and learn new facts while leaving the rest of their memory intact.

1. Executive Summary

This paper proposes a new task of explicitly modifying specific factual knowledge implicitly stored in Transformer language model parameters while preserving performance on unmodified facts. The authors create a benchmark from T-REx and zsRE datasets and evaluate constrained fine-tuning approaches on BERT-Base, BERT-Large, ALBERT-XXLarge, and the memory-augmented FaE model. The core mechanism is constrained fine-tuning — applying an ℓ∞ norm constraint on weight changes during optimization on modified facts (projecting weights back into a δ-ball around the original parameters after each gradient step) — combined with layer-specific modification, where updating only the first or last Transformer block proves more effective than full-model fine-tuning. The best configuration achieves an average accuracy of 60.62% for BERT-Base and 66.19% for FaE when modifying 32 facts on T-REx, establishing that explicit symbolic memory does not necessarily make knowledge modification easier than updating implicit parameters alone.

2. Context and Motivation

The Core Problem: We Can't Edit What Language Models Know

The fundamental question this paper tackles is both simple and deeply practical: if a Transformer language model has memorized a fact, how do you change that fact without breaking everything else the model knows? This question arises from a convergence of two well-established findings in the NLP literature that had not previously been connected as a single engineering challenge.

The first finding, established by Petroni et al. (2019) and Roberts et al. (2020), is that large pretrained Transformer language models (BERT, T5, GPT series) do not merely learn linguistic patterns — they implicitly store substantial factual knowledge in their parameters. When you ask BERT "[MASK] is the capital of France," it answers "Paris" not because it has seen that exact cloze sentence during pretraining, but because it has encoded the factual relationship (France, capital, Paris) somewhere in its 110M+ parameters. Petroni et al. (2019) demonstrated this systematically through the LAMA benchmark, which converts facts from knowledge bases like Wikidata into natural-language cloze statements and measures whether pretrained models can fill in the blanks. The results were striking: BERT-Base achieved non-trivial accuracy on factual queries without any fine-tuning on the target facts, suggesting that the model's pretraining on Wikipedia and BookCorpus had effectively compressed a knowledge base into its weight matrix.

The second finding, more diffuse but equally important, is that the world's facts change, and models that rely on implicit memorization have no mechanism for keeping up. An athlete switches teams. An election changes a country's leadership. A scientific consensus shifts. A user tells their AI assistant to remember that their favorite restaurant has moved. In traditional knowledge bases — SQL databases, key-value stores, graph databases — updating a fact is straightforward: you locate the relevant record and modify it. The infrastructure for querying and updating is the primary design consideration. But in a Transformer model, the "record" for any given fact is distributed across millions of parameters, with no clear mapping between a specific weight and a specific fact.

This paper identifies the gap directly in its opening paragraphs (Section 1):

"Different from conventional KBs that explicitly memorize factual knowledge, Transformers implicitly memorize knowledge in their model parameters. As a result, Transformers lack one key advantage of the conventional databases: efficiently modifying the factual knowledge stored in the model."

This is not merely an academic observation — it represents a fundamental architectural limitation that prevents language models from serving as dynamic, maintainable knowledge repositories.

Why This Matters: The Three-Use-Case Argument

The paper grounds the importance of knowledge modification in three concrete scenarios that span privacy, correctness, and fairness (Section 1). Each scenario represents a different failure mode of static implicit memorization:

Staleness and temporal updates. Facts in the world change continuously. A language model trained on a static corpus snapshot — say, Wikipedia from 2019 — will retain incorrect information indefinitely unless updated. The example the paper gives is a sports player who changes teams over time. More broadly, any deployment of a language model in a production setting faces an expiration date on the accuracy of its memorized knowledge. Without a mechanism for targeted updates, organizations face a choice between serving stale information or retraining the entire model — a process that, for models the size of BERT-Large (340M parameters) or larger, is computationally prohibitive, especially if only a handful of facts need updating.

Privacy and the right to be forgotten. Carlini et al. (2019) demonstrated that large language models can unintentionally memorize sensitive information from their training data — phone numbers, addresses, personal identifiers. Under regulations such as GDPR's "right to erasure," organizations may be legally obligated to remove such information from deployed models. Retraining from scratch on a cleansed corpus is one approach, but it is expensive and requires identifying all instances of the sensitive information in an unstructured pretraining corpus — a task that is itself technically challenging. The ability to surgically remove or overwrite specific memorized facts would provide a more practical compliance mechanism.

Bias elimination. Language models are well-documented to absorb and amplify biases present in their training data — gender stereotypes (Bolukbasi et al., 2016), racial biases (Blodgett et al., 2020), and other forms of representational harm (Bordia and Bowman, 2019). Mitigating these biases typically requires either careful corpus curation before pretraining (which cannot anticipate all biases) or post-hoc debiasing of model outputs (which treats symptoms rather than causes). The knowledge modification framework offers a third path: directly modify the specific biased associations stored in the model's parameters. For instance, if a model has learned a spurious correlation between a profession and a gender from distributional statistics in its training data, targeted modification could overwrite that association without requiring a full retraining pipeline.

These three motivations are not merely rhetorical framing — they represent genuine failure modes that arise from the mismatch between how Transformers store knowledge (distributed, implicit, static after training) and how knowledge actually behaves in the world (localized, explicit, dynamic).

Prior Approaches and Where They Fall Short

The paper identifies four categories of existing work that either partially address the problem or provide natural baselines, each with specific limitations that motivate the constrained fine-tuning approach.

Retraining from scratch or on modified corpora. The most reliable approach is to update all training data to be consistent with new facts and retrain the model. The paper acknowledges this (Section 3.2) but dismisses it on two practical grounds. First, identifying and updating modified facts in unstructured pretraining corpora is "highly non-trivial" — if the model needs to update "the capital of country X is now Y," you would need to find every sentence in Wikipedia and BookCorpus that references the old capital and modify it, which is essentially an information extraction and text generation problem at massive scale. Second, even if corpus modification were feasible, retraining the model from scratch is computationally prohibitive for modifying a small number of facts. This creates a fundamental asymmetry: the cost of updating one fact in a traditional database is O(1) (modify one record), while the cost of updating one fact in a Transformer via retraining is O(total training cost).

Unconstrained fine-tuning on modified facts. The most intuitive lightweight approach is to simply fine-tune the model on supporting evidence for the modified facts (Equation 1 in Section 3.2). For example, if the model needs to change "Eliud Kipchoge holds the marathon world record," you would train it on sentences stating the new record holder. The paper demonstrates (Table 3) that this approach achieves high accuracy on the modified facts — up to 82.50% for BERT-Base when fine-tuning the 11th Transformer block — but catastrophically forgets the unmodified facts, with accuracy on unmodified facts dropping to near zero (e.g., 0.37% when fine-tuning all of BERT-Base from a fine-tuned starting point). This is the classic catastrophic forgetting problem (Kirkpatrick et al., 2017) in a particularly acute form: because the fine-tuning data consists of only the modified facts (e.g., 32 examples), the model rapidly overfits to those specific input-output mappings, overwriting the distributed representations that encoded the original knowledge.

The key insight behind this failure is that each weight in a Transformer affects the model's prediction for many different facts simultaneously. When you aggressively update weights to maximize the probability of the new answer for one fact, you are necessarily changing the model's behavior for all other facts that depended on the same weights. There is no natural isolation between facts stored in the same parameter vector.

Fine-tuning on a mixture of modified and unmodified facts. A natural refinement is to include examples of unmodified facts in each training batch alongside the modified ones, hoping to maintain performance on the original knowledge while learning the new facts. The paper explores this (denoted as FTA in Section 4.4, with results in Table 4) and finds that it does not significantly improve over constrained fine-tuning on only modified facts. The reason is subtle: even when each minibatch contains a 50/50 mix of modified and unmodified examples, the optimizer loops over the modified facts multiple times per epoch (since the modified set is small), effectively giving them disproportionate weight. As the paper notes (Appendix B):

"the optimizer repeatedly loops over DM, which effectively makes the model 10 times as much more biased towards minimizing the expected loss on DM (as we train 10 epochs) than on DF\S"

This creates an inherent tension: you need enough exposure to modified facts to learn them well, but that same exposure inevitably biases optimization away from preserving unmodified facts. Simply mixing data does not resolve this tension — it only shifts the balance point without eliminating the underlying tradeoff.

Memory-augmented models with symbolic components. Several models introduce explicit memory structures alongside the Transformer to make knowledge more accessible and modifiable — Févry et al. (2020)'s Entities as Experts (EaE) adds entity memory, Verga et al. (2020)'s Facts as Expert (FaE) builds on this with additional fact memory, and Khandelwal et al. (2020)'s kNN-LM adds a nearest-neighbor retrieval mechanism over training examples. One of the explicit motivations for FaE is that modifying knowledge should be easy: you simply update the symbolic links in the explicit memory, and the model's predictions should change accordingly.

The paper tests this claim directly (Section 4.6, Table 5) and arrives at a surprising finding: it does not work as intended. When only the symbolic memory links are modified without touching the Transformer parameters, FaE achieves only 46.88% accuracy on the modified facts — higher than the 30% reported by Verga et al. (2020) but still far below what constrained fine-tuning achieves. The paper explains this as a consistency problem between implicit and explicit memory:

"since these models rely on both the contextual representation and the symbolic links, inconsistency between its implicit memory (realized via contextual representation) and the explicit symbolic memory can result in wrong predictions"

In other words, FaE stores knowledge redundantly: the BERT-based Transformer backbone has implicitly memorized facts during training, and the explicit symbolic memory also encodes them. When you update only the explicit memory, the implicit memory in the Transformer parameters continues to influence predictions through the contextual representations, creating contradictory signals. The model sometimes follows the updated explicit link and sometimes follows the original implicit knowledge encoded in the Transformer weights. To achieve high accuracy on modified facts, you need to modify the Transformer parameters as well — which means FaE does not actually simplify the knowledge modification problem relative to standard BERT.

This finding is significant because it undermines a key claimed advantage of memory-augmented architectures. It suggests that implicit memorization in Transformer weights is not simply a redundant backup of explicit memory but rather an integral part of how these models compute predictions, and it must be dealt with directly regardless of whether explicit memory modules are present.

The kNN-LM approach (Appendix F, Table 6) fares even worse: modifying only the value tokens in the nearest-neighbor datastore yields at most 12.50% accuracy on modified facts, and the paper identifies a fundamental limitation — all facts sharing the same object (e.g., all facts where the answer is "Paris") will be affected by modifying any one of them, because the nearest-neighbor mechanism does not distinguish between different relations. This is a form of collateral damage that arises from the non-compositional nature of the kNN retrieval: the model retrieves based on contextual embedding proximity without understanding whether the fact relation matches.

How This Paper Positions Itself

The paper frames its contribution around a novel task formulation rather than a novel architecture or training objective. This is an important distinction: the paper is not proposing a new type of Transformer or a fundamentally new optimization algorithm. Instead, it defines a benchmark and evaluation protocol for what it calls "explicitly modifying specific factual knowledge in Transformer models while ensuring that model performance does not degrade on the unaltered facts" (Section 1), and then evaluates a family of simple, natural approaches against that benchmark.

The task formulation itself constitutes a contribution because prior work had not defined the problem with this level of precision. Prior literature had studied knowledge probing (what do models know?), knowledge injection (how do we add new knowledge?), and catastrophic forgetting (how do we prevent performance degradation when learning new tasks?) — but these three threads had not been combined into the specific challenge of modifying existing knowledge while preserving everything else intact. The paper explicitly distinguishes its task from continual learning (Section 2), noting:

"Similar to continual learning, memory modification also expects the predictions to be updated efficiently (potentially without access to the unmodified facts) while preserving the accuracy for the unmodified facts. In this case, both settings suffer from catastrophic forgetting, but memory modification further requires the model to memorize new facts that conflict with previously learned facts, posing new challenges to existing continual learning approaches."

This is a critical distinction. Continual learning typically involves learning a new task where the model has not previously stored conflicting information. Knowledge modification requires the model to overwrite existing knowledge with contradictory information — the model must learn that what was previously true is now false. This creates a direct conflict: the same weights that correctly predicted the old fact must now predict the new fact, while simultaneously continuing to correctly predict all other facts that depended on those same weights. This is a harder constraint than typical continual learning settings because the modification itself targets facts that are already well-learned.

The paper's positioning is also notable for what it does not claim. It does not claim to have solved the knowledge modification problem — the best average accuracy is 66.19% (FaE on T-REx with 32 modified facts, Table 2), and performance degrades as the number of modified facts increases (Figure 3). It does not claim that the constrained fine-tuning approach is optimal — the Fisher information approximation (Appendix C) is mentioned as a potentially better constraint that was attempted but did not outperform the simple ℓ∞ norm. It does not claim that the approach works for all types of knowledge — all experiments are on factual triples (subject, relation, object) from structured knowledge bases, leaving open questions about modifying procedural knowledge, commonsense knowledge, or linguistic knowledge.

What the paper does claim is more foundational: that knowledge modification in Transformers is possible to a meaningful degree, that layer-specific modification is more effective than full-model modification, and that explicit symbolic memory does not simplify the problem as much as one might expect. These claims collectively establish the contours of a new subproblem — much as Petroni et al. (2019) established the contours of knowledge probing — and provide baselines that future work can improve upon.

The paper also positions itself within a broader narrative about the relationship between memorization and generalization. It cites Feldman (2020) and Feldman and Zhang (2020), who showed theoretically that optimal generalization on long-tailed distributions requires memorization of specific training examples from rare subpopulations. In this light, knowledge modification is not just about fixing stale facts — it is about making memorization-based learning systems maintainable. If memorization is necessary for generalization, then the ability to update what is memorized is necessary for the system to remain correct over time. This connection to generalization theory elevates the task from a practical engineering challenge to a fundamental requirement for any learning system that relies on memorization.

3. Technical Approach

3.1 Reader Orientation

This section explains the constrained fine-tuning framework — a simple optimization procedure that modifies a specific subset of a Transformer model's weights (often just one layer) to update memorized factual knowledge, while preventing the model from forgetting everything else it knows by keeping all weight changes within a tight mathematical bound around the original parameters. The core idea is that factual knowledge in Transformers, though distributed across many parameters, can be selectively overwritten by identifying which layers are most amenable to modification and then solving a constrained optimization problem that maximizes accuracy on the new facts subject to a hard limit on how far the weights can move from their original values.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

Base Transformer model (BERT-Base, BERT-Large, ALBERT-XXLarge, or FaE) — the pretrained language model that has implicitly memorized a collection of facts $\mathcal{F}$ in its parameters $\theta_0$ . It accepts cloze-style inputs (sentences with a [MASK] token) and outputs a probability distribution over vocabulary tokens for the masked position.
Modification dataset $\mathcal{D}_M$ — a small set of training examples (supporting evidence sentences, synonymously called "evidences" in the paper) for the facts we want to change. Each example is a masked sentence where the correct answer has been replaced with the new, modified object. For instance, if the original fact was "(Natalie Lowe, place of birth, Sydney)" and we want to change the birthplace to London, all training examples that previously required predicting "Sydney" now require predicting "London."
Constraint mechanism — an $\ell_\infty$ norm ball of radius $\delta$ centered at the original parameters $\theta_0$ . After every gradient step during fine-tuning, the updated parameters are projected back into this ball — meaning no individual weight can change by more than $\delta$ from its starting value. This is the core mechanism that prevents catastrophic forgetting.
Layer selector — a design choice (not a learned component) that determines which subset of the Transformer's layers are updated during constrained fine-tuning. The paper experiments with updating only the 0th (first) Transformer block, only the 5th (middle) block, only the 11th or 23rd (last) block, or all blocks simultaneously.
Evaluation protocol — after modification, the model is tested on two disjoint sets: (a) $\mathcal{M}$ , the set of modified facts (using held-out test questions that are phrased differently from the training evidences), and (b) $\mathcal{F} \setminus \mathcal{S}$ , all other facts that were not modified. The metric $\bar{A} = (A_\mathcal{M} + A_{\mathcal{F}\setminus\mathcal{S}}) / 2$ averages these two accuracies.

The information flow is: (1) select a subset of facts $\mathcal{S} \subset \mathcal{F}$ to modify; (2) construct $\mathcal{D}_M$ by replacing the answer labels for all training evidences of facts in $\mathcal{S}$ with the new desired objects; (3) choose a layer or set of layers to fine-tune; (4) perform projected gradient descent (Adam with $\ell_\infty$ projection after each step) on $\mathcal{D}_M$ only, with a sweep over constraint strengths $\delta$ ; (5) evaluate on both modified and unmodified test facts; (6) select the $\delta$ that maximizes $\bar{A}$ .

3.3 Roadmap for the Deep Dive

First, the formal constrained optimization objective (Equations 2 and 3), which defines what "constrained fine-tuning" means and why the $\ell_\infty$ norm is chosen over $\ell_2$ or Fisher information constraints.
Second, the projected gradient descent algorithm (Algorithm 1 in Appendix D), which operationalizes the constraint during training — the exact projection operations for both $\ell_2$ and $\ell_\infty$ norms.
Third, the benchmark construction process (datasets, fact selection, evidence modification, train/test splits), since the benchmark itself is a contribution and its design choices affect what the results mean.
Fourth, the layer-specific modification strategy — why different layers produce dramatically different tradeoffs, and why the optimal layer changes with the number of modified facts and the model's initial state.
Fifth, the extension to memory-augmented models (FaE) and why modifying explicit symbolic memory alone fails without also updating Transformer parameters.
Sixth, the kNN-LM modification approach (Appendix F) and the fundamental reason nearest-neighbor editing causes collateral damage to unmodified facts sharing the same object.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis and benchmark paper whose core idea is that knowledge modification in Transformers can be achieved through constrained optimization on modified training examples, and that the choice of which layer to modify is the key design variable that determines the tradeoff between learning new facts and retaining old ones.

The Constrained Optimization Framework

Why a constraint is necessary. The naive approach of unconstrained fine-tuning on the modified facts $\mathcal{D}_M$ solves the following problem:

$\text{minimize}_{\theta \in \Theta} \frac{1}{m} \sum_{x \in \mathcal{D}_M} \mathcal{L}(x; \theta)$

where $m = |\mathcal{D}_M|$ is the number of training evidences for the modified facts. This achieves high accuracy on $\mathcal{M}$ but catastrophically destroys performance on $\mathcal{F} \setminus \mathcal{S}$ because the optimizer is free to move weights arbitrarily far from $\theta_0$ , overwriting the distributed representations that encoded the unmodified facts.

The ideal constrained objective. The paper formulates the ideal version of the problem (Equation 2) as:

$\text{minimize}_{\theta \in \Theta} \frac{1}{m} \sum_{x \in \mathcal{D}_M} \mathcal{L}(x; \theta)$ $\text{subject to} \quad \frac{1}{n} \sum_{x' \in \mathcal{D}_{\mathcal{F}\setminus\mathcal{S}}} \left(\mathcal{L}(x'; \theta) - \mathcal{L}(x'; \theta_0)\right) \leq \delta$

where $n = |\mathcal{D}_{\mathcal{F}\setminus\mathcal{S}}|$ is the number of training evidences for the unmodified facts, $\mathcal{L}(x'; \theta)$ is the loss on unmodified evidence $x'$ under the new parameters $\theta$ , $\mathcal{L}(x'; \theta_0)$ is the loss under the original parameters, and $\delta > 0$ is a small constant specifying the maximum acceptable increase in average loss on unmodified facts.

What it computes: The objective minimizes the average loss on the modified facts (same as unconstrained fine-tuning), but with a hard constraint that the average loss on all unmodified training evidences must not increase by more than $\delta$ from the original model's loss. The constraint is evaluated over the full training set of unmodified facts $\mathcal{D}_{\mathcal{F}\setminus\mathcal{S}}$ , which contains $n$ examples.

Why this form: This formulation directly encodes the goal: learn the new facts while explicitly enforcing that the model's behavior on everything else stays approximately the same. The constraint is on the change in loss rather than absolute loss because the original model may have high loss on some facts (it's not perfect), and we only care about preventing degradation, not improving what was already suboptimal. The $\delta$ parameter controls the allowable tradeoff — larger $\delta$ permits more forgetting in exchange for potentially higher accuracy on modified facts.

The practical approximation. The ideal constraint is computationally prohibitive because evaluating it requires computing the loss over potentially millions of unmodified training examples at each optimization step. The paper approximates it using a local smoothness assumption (Equation 3):

$\text{minimize}_{\theta \in \Theta} \frac{1}{m} \sum_{x \in \mathcal{D}_M} \mathcal{L}(x; \theta)$ $\text{subject to} \quad \|\theta - \theta_0\| \leq \delta$

where $\|\cdot\|$ denotes a norm in the parameter space, $\theta$ is the current parameter vector, and $\theta_0$ is the original (pretrained or fine-tuned) parameter vector before any modification.

What it computes: Instead of constraining the loss change on unmodified data directly, this constrains the maximum distance that any parameter can move from its original value according to some norm. The intuition is that if the weights don't change much, the model's predictions on unmodified inputs won't change much either — this is the local continuity assumption: small perturbations in parameter space produce small perturbations in function space.

Why this form works: The constraint becomes computationally trivial to enforce — it requires only the current parameters $\theta$ and the stored original parameters $\theta_0$ , with no need to evaluate the model on training data during the constraint check. The projection step (described below) is closed-form for both $\ell_2$ and $\ell_\infty$ norms and adds negligible overhead per gradient step. The price is that the relationship between $\delta$ (a bound on weight change) and the actual loss change on unmodified facts is indirect and must be tuned — there is no guarantee that a particular $\delta$ corresponds to a particular loss increase.

Choice of norm: $\ell_\infty$ vs. $\ell_2$ . The paper experiments with both $\ell_2$ (Euclidean distance, constraining the sum of squared weight changes) and $\ell_\infty$ (maximum absolute change per weight, constraining each weight independently) and finds:

" $\ell_\infty$ consistently leads to more stable results for knowledge modification."

This choice is significant. The $\ell_2$ norm allows some weights to change a lot as long as others change very little, keeping the total squared distance within $\delta$ . This can permit large changes to a small number of "important" weights that strongly affect predictions for specific facts — which is exactly the kind of change that causes catastrophic forgetting for facts that depend on those same weights. The $\ell_\infty$ norm, by contrast, treats every weight equally: no single weight can change by more than $\delta$ , regardless of what other weights do. This prevents the optimizer from concentrating all the adaptation into a few weights that might be critical for many different facts, forcing it to spread the modification across many parameters in a more distributed way that is less likely to catastrophically interfere with any single unmodified fact.

The Fisher information alternative (Appendix C). The paper also derives a theoretically more principled approximation using a second-order Taylor expansion of the loss constraint. When the number of modified facts is small (so we are still near a minimum of the loss on unmodified facts), the linear term vanishes and the constraint becomes:

$\sum_{ij} \Delta\theta_i \Delta\theta_j \left( \frac{1}{2n} \frac{\partial}{\partial\theta_i} \frac{\partial}{\partial\theta_j} \sum_{x' \in \mathcal{D}_{\mathcal{F}\setminus\mathcal{S}}} \mathcal{L}(x'; \theta_0) \right) + O(\Delta\theta^3) \leq \delta$

where the quantity in parentheses is the Fisher information matrix — the expected outer product of gradients on the unmodified data, which measures how sensitive the loss on each unmodified example is to changes in each pair of parameters. This would provide a more precise constraint: weights that are important for unmodified facts (high Fisher information) would be permitted only very small changes, while weights that are unimportant (low Fisher information) could change more freely.

However, the paper reports:

"We experimented with an approximation of the Fisher information computed with batch size 128, and found that it did not outperform the $\ell_\infty$ norm with (3). We leave the detailed exploration of the Fisher metric for the memory modification task to future work."

The likely reason is that computing the Fisher information accurately requires averaging over the full training set of unmodified facts, which is expensive and difficult to parallelize — the batch-128 approximation may have been too noisy to provide better guidance than the simple $\ell_\infty$ norm, which has the advantage of being exact and trivial to compute.

Projected Gradient Descent (Algorithm 1)

The constrained optimization problem in Equation 3 is solved using projected gradient descent with Adam as the base optimizer. The procedure is formalized as Algorithm 1 in Appendix D.

Standard Adam update (unconstrained). At each iteration $t$ , the algorithm:

Draws a minibatch $\mathcal{S}_t$ from the modification dataset $\mathcal{D}_M$ .
Computes the average gradient $g_t = \frac{1}{|\mathcal{S}_t|} \sum_{x_k \in \mathcal{S}_t} \nabla \mathcal{L}(x_k; \theta_{t-1})$ .
Updates biased first and second moment estimates: $m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$ , $v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$ , with hyperparameters $0 < \beta_1 < 1$ , $0 < \beta_2 < 1$ .
Computes bias-corrected estimates and forms the candidate update: $\tilde{\theta}_t = \theta_{t-1} - \eta_t \frac{\sqrt{1-\beta_2^t}}{1-\beta_1^t} \frac{m_t}{\sqrt{v_t} + \epsilon}$ , where $\eta_t$ is the learning rate at step $t$ and $\epsilon > 0$ is a small constant for numerical stability.

Projection step. After computing $\tilde{\theta}_t$ (which may lie outside the constraint set), the algorithm projects it back:

$\theta_t = \Pi_{\|\theta - \theta_0\| \leq \delta}(\tilde{\theta}_t)$

This is the critical step that enforces the constraint. The projection operator $\Pi$ finds the closest point to $\tilde{\theta}_t$ that satisfies the norm constraint, effectively clipping the parameter update to stay within the allowed region.

$\ell_2$ projection (Equation 8):

$\Pi_{\|\theta - \theta_0\|_2 \leq \delta}(\theta) = \theta_0 + (\theta - \theta_0) \min\left\{\frac{\delta}{\|\theta - \theta_0\|_2}, 1\right\}$

What it computes: If the Euclidean distance from the updated parameters to the original parameters is less than or equal to $\delta$ , do nothing (the factor is 1). If the distance exceeds $\delta$ , scale the displacement vector $\theta - \theta_0$ down by the factor $\delta / \|\theta - \theta_0\|_2$ so that the resulting parameters lie exactly on the surface of the $\ell_2$ -ball of radius $\delta$ centered at $\theta_0$ . This preserves the direction of the update while limiting its magnitude.

$\ell_\infty$ projection (Equation 9):

$\Pi_{\|\theta - \theta_0\|_\infty \leq \delta}(\theta) = \theta_0 + \min\left\{\max\{\theta - \theta_0, -\delta\}, \delta\right\}$

where the $\min$ and $\max$ operations are applied element-wise (per parameter).

What it computes: For each individual parameter $\theta_i$ , if its displacement $\theta_i - \theta_{0,i}$ is already within $[-\delta, \delta]$ , leave it unchanged. If it exceeds $+\delta$ , clip it to $+\delta$ (the parameter can increase by at most $\delta$ ). If it falls below $-\delta$ , clip it to $-\delta$ (the parameter can decrease by at most $\delta$ ). The $\max$ operation first floors the displacement at $-\delta$ (preventing it from being more negative than $-\delta$ ), then the $\min$ operation caps it at $+\delta$ (preventing it from being more positive than $+\delta$ ). The result is that every parameter's change is independently clamped to the interval $[-\delta, +\delta]$ .

Why element-wise clamping matters: The $\ell_\infty$ projection treats parameters as independent — each weight has its own $\pm\delta$ budget that is not shared with other weights. This prevents the optimizer from compensating for large changes in some weights by keeping others very close to their original values (which the $\ell_2$ projection permits, since it constrains total squared change). Under $\ell_\infty$ , the model cannot, for example, double the value of one attention head's output projection while keeping all other weights frozen — every weight is individually bounded. This is particularly important for Transformers because attention heads often specialize, and allowing even one head to change dramatically could alter the model's behavior on many different facts.

The $\delta$ sweep. The constraint strength $\delta$ is not learned or derived analytically — it is treated as a hyperparameter that must be chosen per model, per layer, per number of modified facts. The paper sweeps over a range of $\delta$ values (the specific range is not stated in the main text but can be inferred from Appendix F to span at least $10^{-3}$ to $4 \times 10^{-3}$ for the $\ell_\infty$ norm on BERT-Base) and selects the one that maximizes $\bar{A} = (A_\mathcal{M} + A_{\mathcal{F}\setminus\mathcal{S}}) / 2$ , the average accuracy on modified and unmodified facts. As $\delta$ approaches zero, no modification occurs and $A_\mathcal{M}$ remains low while $A_{\mathcal{F}\setminus\mathcal{S}}$ stays high. As $\delta$ approaches infinity, the constraint becomes inactive (equivalent to unconstrained fine-tuning), $A_\mathcal{M}$ becomes high, and $A_{\mathcal{F}\setminus\mathcal{S}}$ collapses. The optimal $\delta$ lies somewhere in between, at the point where the tradeoff between the two accuracies is most favorable.

Benchmark Construction

The paper constructs two knowledge modification benchmarks from existing factual knowledge datasets: T-REx (Elsahar et al., 2018) and Zero-shot Relation Extraction (zsRE, Levy et al., 2017). The construction process involves several design decisions that affect how results should be interpreted.

Source datasets and their properties.

T-REx: Contains 34,039 facts across 41 Wikipedia relations, with a total of 1,282,567 training evidences and 34,039 test evidences (Table 1). Each fact is a (subject, relation, object) triple, e.g., "(Natalie Lowe, place of birth, Sydney)." The version used comes from the LAMA benchmark (Petroni et al., 2019). All object labels are single-token — they can be predicted by the masked language model in one forward pass. This property simplifies evaluation (exact string match is sufficient) but also restricts the benchmark to facts expressible as single tokens, potentially excluding multi-word entities and complex facts.
zsRE: Contains 147,905 training facts and 47,156 test facts, with 197,829 training evidences and 59,527 test evidences (Table 1). The version comes from the KILT benchmark (Petroni et al., 2020). Unlike T-REx, zsRE has multiple template questions per fact with varied phrasing, and the answers can be multi-token (up to 20 tokens when using the uncased BERT tokenizer). The model's prediction is considered correct only when all predicted tokens match the label exactly. The training and test splits for zsRE are constructed from the original training set of KILT since the relations in different KILT splits do not overlap — for each fact, two of its questions are assigned to the test set if it has more than three questions; one question is preserved in training if it has only one; and one question goes to each set otherwise.

Fact modification procedure. To create $\mathcal{M}$ modifications for a subset $\mathcal{S} \subset \mathcal{F}$ of size $|\mathcal{M}|$ :

Select $m = |\mathcal{M}|$ facts uniformly at random from the full set $\mathcal{F}$ .
For each selected fact, replace its object with a new object sampled from the pool of all objects that share the same relation in the training set, with sampling probability proportional to frequency. For example, if modifying "(Natalie Lowe, place of birth, Sydney)," the new object would be another birthplace (e.g., London, Paris, New York) sampled according to how often it appears as someone's birthplace in the training data.
Consistently update all supporting evidences for each modified fact — both the training evidences (the Wikipedia sentences or template questions used during fine-tuning) and the test evidences (the held-out evaluation questions). This is critical: if the training evidence is "Natalie Lowe was born in [MASK]" with the new label being "London," but the test evidence is a different phrasing like "Natalie Lowe's birthplace is [MASK]," the model must learn to predict "London" for any question about Natalie Lowe's birthplace, not just the specific phrasing it was trained on. The different phrasings between training and test evidences test whether the model has genuinely updated its factual knowledge rather than merely memorizing a surface pattern.

Why frequency-based sampling? The new object is sampled according to its frequency in the training set for that relation. This ensures that the modified facts are ecologically plausible — the model is being asked to learn that someone was born in London (a common birthplace) rather than some rare location that might be easier to memorize but less realistic. It also means the task difficulty varies: modifying a fact to a high-frequency object means the model has already seen many examples of that object in other contexts (which could help or hurt, depending on interference), while modifying to a low-frequency object tests the model's ability to learn a novel association.

The train/test gap by construction. For T-REx, the training evidences come from Wikipedia sentences (natural text with the object replaced by [MASK]), while the test evidences come from template-generated cloze questions ("Natalie Lowe was born in [MASK]."). The templates are structurally simpler and syntactically different from the Wikipedia sentences. This means the model cannot succeed on the test set by memorizing surface-level lexical patterns from the training evidences — it must generalize across phrasings, which requires updating the underlying factual association rather than learning a shallow mapping from a specific sentence to a specific answer.

For zsRE, both training and test evidences are template-generated questions, but with different templates for the same fact — e.g., training might ask "What is the continent that Della Pia Glacier is located?" while test asks "What continent is Della Pia Glacier found on?" Again, the model must generalize across phrasings.

Scale of modification. The paper experiments with $|\mathcal{M}| \in \{32, 128, 512, 2048\}$ modified facts. These represent a range from a handful of targeted updates (e.g., correcting a few stale facts in a deployed system) to a more substantial knowledge overhaul (e.g., updating all facts about a particular domain). The number of modified facts dramatically affects which layer is optimal and what accuracy tradeoff is achievable (Figure 3).

Layer-Specific Modification Strategy

The paper's most striking empirical finding is that which layer you modify matters enormously, and the optimal layer depends on the model's initial state and the number of modified facts.

What "modifying a layer" means mechanically. When the paper says "fine-tune block 11," it means: freeze all parameters in the Transformer model (embeddings, all other Transformer blocks, the task-specific prediction head if present) and only update the weights of the specified Transformer block during constrained fine-tuning. A Transformer block (or layer) in BERT consists of a multi-head self-attention sublayer followed by a feed-forward sublayer, each with residual connections and layer normalization — modifying "block 11" means updating all parameters in both sublayers of that block, plus its two layer normalization components, while keeping everything else frozen.

Why different layers behave differently. Table 3 presents unconstrained fine-tuning results for BERT-Base on T-REx with 32 modified facts, comparing three training regimes (RI+FTM, FTM, FT+FTM) and three layers (0, 5, 11). The key patterns:

Random initialization (RI+FTM): All layers perform poorly — $A_\mathcal{M}$ around 19–21% and $A_{\mathcal{F}\setminus\mathcal{S}}$ near zero. Without pretraining, the model has no linguistic knowledge to build on, and 32 examples are insufficient to learn both language and facts from scratch, even for a single layer.
Pretrained only (FTM): The model achieves high $A_\mathcal{M}$ (67–75%) but $A_{\mathcal{F}\setminus\mathcal{S}}$ collapses to near zero regardless of which layer is modified. The pretrained model has factual knowledge distributed across all layers, so even modifying a single layer disrupts the distributed representation sufficiently to destroy most unmodified facts.
Fine-tuned then modified (FT+FTM): This is where layer choice matters dramatically. Modifying block 0 gives $A_\mathcal{M} = 77.50\%$ and $A_{\mathcal{F}\setminus\mathcal{S}} = 0.37\%$ — essentially complete forgetting. Modifying block 5 gives $A_\mathcal{M} = 77.50\%$ and $A_{\mathcal{F}\setminus\mathcal{S}} = 15.09\%$ — some retention. Modifying block 11 gives $A_\mathcal{M} = 82.50\%$ and $A_{\mathcal{F}\setminus\mathcal{S}} = 1.12\%$ — near-total forgetting again.

The non-monotonic pattern (layer 5 is uniquely good at preserving unmodified facts under unconstrained fine-tuning) is surprising and hints at a functional specialization across layers. The middle layers (block 5) appear to store factual knowledge in a form that is less brittle — modifying them overwrites fewer unmodified facts — while the early layers (block 0, which processes input embeddings and low-level features) and late layers (block 11, which feeds into the prediction head) store knowledge in ways that are more entangled with other facts.

The effect of constraints: why constrained fine-tuning changes the optimal layer. When constraints are applied (Figures 1, 2, 3, and Table 2), the picture shifts:

With constraints and $|\mathcal{M}| = 32$ on BERT-Base (Table 2): Block 11 is optimal for FT+FTM ( $\bar{A} = 60.62\%$ ), while block 0 is optimal for FTM ( $\bar{A} = 47.47\%$ ). The constraint $\delta$ prevents the catastrophic forgetting that made block 11 terrible in the unconstrained setting, while still allowing it to achieve high $A_\mathcal{M}$ .
As $|\mathcal{M}|$ grows (Figure 3, zsRE benchmark, FT+FTM setting): For 32 modified facts, block 11 achieves the highest $\bar{A}$ . For 128 modified facts, blocks 0 and 11 are roughly comparable. For 512 and 2048 modified facts, block 0 becomes clearly optimal. The paper summarizes (Section 4.5.2):

"the block with highest $\bar{A}$ changed from the last one (block 11 or 23) to the first one (block 0) for both BERT-Base and BERT-Large"

Why this shift occurs. The paper does not provide a mechanistic explanation, but the pattern is consistent with the hypothesis that early layers encode more general, composable knowledge that can be updated with less interference, while late layers encode more task-specific, entangled representations that give high accuracy for small modifications but conflict more as the number of modifications grows. When only a few facts need changing, a late layer can be repurposed to memorize the new associations without disturbing the earlier layers' general knowledge — the model essentially adds a "patch" at the top. When many facts need changing, repurposing a late layer runs into capacity limits and interference, and it becomes more effective to update the early layers that process input tokens into initial representations, effectively "re-encoding" the facts from the ground up.

ALBERT's limitation. ALBERT-XXLarge shares weights across all Transformer blocks (a design choice to reduce parameter count from 340M+ to 235M while maintaining depth). As the paper notes: "the only option here is to finetune all its blocks on the modified facts." This means ALBERT cannot benefit from layer-specific modification, which may partially explain why it achieves only $\bar{A} = 50.49\%$ (Table 2) despite being a large model — it must modify representations at all layers simultaneously, increasing the risk of interference with unmodified facts.

Full-model vs. layer-specific modification (Figure 1). When all Transformer blocks are modified simultaneously under constraints, performance is consistently worse than modifying only a single well-chosen block. In Figure 1 (T-REx, $|\mathcal{M}| = 32$ ), full-model fine-tuning of BERT-Base achieves $A_\mathcal{M} \approx 45\%$ and $A_{\mathcal{F}\setminus\mathcal{S}} \approx 50\%$ ( $\bar{A} \approx 47.5\%$ ), while modifying only block 0 achieves $A_\mathcal{M} \approx 71\%$ and $A_{\mathcal{F}\setminus\mathcal{S}} \approx 18\%$ ( $\bar{A} \approx 44.5\%$ ) — comparable average but qualitatively different tradeoffs. For BERT-Large and ALBERT, full-model fine-tuning is strictly worse than single-layer modification in terms of $\bar{A}$ (Figure 1).

This is a counterintuitive result: one might expect that updating all layers with a small $\delta$ per layer would be equivalent or better than updating one layer with a larger $\delta$ , since the total capacity available for modification is larger. The fact that it is worse suggests that the $\ell_\infty$ constraint interacts with layer-wise gradient structure in a non-trivial way: when all layers are modified, gradients flow through the entire network and the optimizer may make coordinated changes across layers that, while individually small, collectively move the function more than the $\ell_\infty$ per-weight bound would suggest. Modifying a single layer limits the "path length" through which parameter changes can compose.

The Role of Initial Model State: FT vs. FTM vs. FT+FTM

The paper distinguishes three starting points for modification, denoted by different prefixes (Section 4.4):

FTM: Start from an off-the-shelf pretrained model (e.g., BERT-Base as released) and fine-tune directly on the modified facts $\mathcal{D}_M$ with constraints. The model has general language understanding and some factual knowledge from pretraining, but has not been specifically trained on the T-REx or zsRE formats.
FT+FTM: First fine-tune the pretrained model on the original unmodified T-REx or zsRE training set to teach it the task format and the original facts, achieving 50.50% (BERT-Base), 51.39% (BERT-Large), 47.96% (ALBERT-XXLarge), or 60.38% (FaE) accuracy on all facts. Then modify the specific facts $\mathcal{S}$ using $\mathcal{D}_M$ with constraints.
FTA: During the modification step (FTM), include both modified and unmodified training examples in each minibatch (50/50 split), with the constraint still applied to weight changes.

Why the FT prefix matters (Table 2 and Figure 2). The accuracy on unmodified facts after modification ( $A_{\mathcal{F}\setminus\mathcal{S}}$ ) is dramatically higher for FT+FTM than for FTM. For BERT-Base modifying 32 facts on T-REx with constraints, FTM on block 0 achieves $A_{\mathcal{F}\setminus\mathcal{S}} = 17.69\%$ while FT+FTM on block 11 achieves $A_{\mathcal{F}\setminus\mathcal{S}} = 43.40\%$ — a 25.7 percentage point improvement. The fine-tuning step (FT) has compartmentalized the factual knowledge: by training the model specifically on the T-REx format with the original facts, the knowledge becomes more localized to task-relevant parameters, making it less susceptible to disruption when other specific facts are modified. The pretrained model's factual knowledge is more diffusely distributed (it learned facts incidentally from diverse pretraining contexts), so modifying even a single layer affects a broader swath of facts.

The FTA variant (Table 4). Including unmodified examples in each minibatch during constrained modification provides a small improvement for some layers (block 11: $\bar{A}$ increases from 42.29% to 42.97%) but decreases performance for others (block 0: $\bar{A}$ drops from 46.96% to 45.91%). The effect is small and inconsistent, suggesting that the constraint $\|\theta - \theta_0\| \leq \delta$ already does most of the work of preserving unmodified facts, and adding explicit unmodified training examples provides limited additional benefit. The paper attributes this to the imbalanced exposure: the optimizer loops over the small set of modified facts multiple times per epoch, giving them disproportionate weight even when each batch is 50/50.

Modifying Memory-Augmented Models: The FaE Case Study

Facts as Expert (FaE) (Verga et al., 2020) is a memory-augmented Transformer designed to combine implicit knowledge in BERT parameters with explicit symbolic knowledge stored in structured memory modules. The architecture has three components (Section 4.6 and Table 5):

BERT-base Transformer — a standard BERT model that produces contextual representations.
Symbolic memory modules — entity memory (inherited from EaE, Févry et al., 2020) and fact memory, which store structured (subject, relation, object) triples and can be queried during inference. The entity memory contains learned embeddings for entities; the fact memory explicitly encodes relational triples.
Additional weights (AWT) — parameters that connect the Transformer's contextual representations to the symbolic memory modules, mapping between the continuous representation space and the discrete symbolic space. These are the "additional weights outside the Transformer part."

The naive approach: symbolic-only modification. The intuitive appeal of FaE is that modifying knowledge should be simple — change the symbolic links in the fact memory and the model should output the new facts. Table 5 shows that modifying only the symbolic links (row "NONE") yields $A_\mathcal{M} = 46.88\%$ , which is significantly better than the 30% reported by Verga et al. (2020) but far below the 75%+ achievable by modifying Transformer parameters. More importantly, $A_{\mathcal{F}\setminus\mathcal{S}}$ stays at 60.38% — perfect preservation — because the Transformer weights are untouched.

Why symbolic-only modification fails for high $A_\mathcal{M}$ . The paper explains that FaE computes predictions by combining signals from both the Transformer's contextual representation (implicit memory) and the explicit symbolic memory. Even after updating the symbolic link, the Transformer's contextual representation — which was trained on data where the original fact was true — still encodes the old fact. The model therefore receives contradictory signals: the explicit memory says "London," the implicit memory says "Sydney." The prediction is a weighted combination, and the implicit signal often dominates (or the conflict reduces confidence in both answers). This is the "inconsistency between its implicit memory (realized via contextual representation) and the explicit symbolic memory" that the paper identifies.

Achieving high $A_\mathcal{M}$ requires modifying Transformer parameters. Table 5 shows results for modifying different components under constraints (FT+FTM setting), with $\delta$ chosen so that $A_\mathcal{M}$ matches the 75% achieved by BERT-Base:

AWT only: $A_\mathcal{M} = 75.00\%$ , $\Delta A_{\mathcal{F}\setminus\mathcal{S}} = -3.00\%$ — the best tradeoff. Modifying only the weights that connect the Transformer to the symbolic memory is sufficient to resolve the inconsistency while minimizing disruption to the Transformer's encoding of unmodified facts.
Block 3 + AWT: $A_\mathcal{M} = 78.12\%$ , $\Delta A_{\mathcal{F}\setminus\mathcal{S}} = -15.16\%$ — higher modified-fact accuracy but much larger forgetting.
Block 7 + AWT: $A_\mathcal{M} = 81.25\%$ , $\Delta A_{\mathcal{F}\setminus\mathcal{S}} = -19.32\%$ — even more forgetting.
All weights: $A_\mathcal{M} = 75.00\%$ , $\Delta A_{\mathcal{F}\setminus\mathcal{S}} = -7.01\%$ — worse tradeoff than AWT only (larger forgetting for the same $A_\mathcal{M}$ ), reinforcing the pattern that modifying too many weights is counterproductive.

The key comparative finding (Table 2). FaE with AWT-only modification achieves $\bar{A} = 66.19\%$ (with $A_{\mathcal{F}\setminus\mathcal{S}} = 57.38\%$ at $A_\mathcal{M} = 75.00\%$ ), compared to BERT-Large at $\bar{A} = 58.75\%$ ( $A_{\mathcal{F}\setminus\mathcal{S}} = 44.70\%$ at $A_\mathcal{M} = 72.80\%$ ). FaE's higher $\bar{A}$ is primarily due to starting from a much higher baseline accuracy (60.38% vs. 51.39%) — it has more room to lose accuracy on unmodified facts while maintaining a high average. The paper's crucial claim (Section 4.6) is that FaE "does not have a significant advantage in terms of tradeoff between $A_\mathcal{M}$ and $A_{\mathcal{F}\setminus\mathcal{S}}$ when we require $A_\mathcal{M}$ to be high":

"BERT-Large can achieve an $A_\mathcal{M}$ of 77.50% with a drop of less than 4.00% in $A_{\mathcal{F}\setminus\mathcal{S}}$ . In contrast, FaE reaches an $A_\mathcal{M}$ of 75.00% with a drop of 3.00% in $A_{\mathcal{F}\setminus\mathcal{S}}$ ."

The tradeoff — how much unmodified-fact accuracy you lose per unit of modified-fact accuracy gained — is comparable between BERT-Large and FaE. Explicit symbolic memory does not fundamentally decouple knowledge modification from implicit parameter updates; it only changes which parameters (AWT vs. Transformer blocks) are most efficient to modify.

The kNN-LM Modification Approach (Appendix F)

The paper also tests kNN-LM (Khandelwal et al., 2020), a retrieval-augmented approach where the model's prediction for a [MASK] token is interpolated with a nearest-neighbor lookup over a datastore of (contextual embedding, answer token) pairs constructed from training data.

Adapting kNN-LM to masked language models. The paper constructs a datastore from the modified facts $\mathcal{D}_M$ only: for each training evidence of a modified fact, the key is the contextual embedding $c(x; \theta_0)$ of the [MASK] token (produced by the frozen original model), and the value is the new, modified answer token $y'$ . At test time, for an input $x$ , the model computes: