The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation

🎯 Pitch

Alignment fine-tuning can suppress extractable training data leakage by over 100×, yet memorized content remains latent and recoverable through jailbreaks or modified decoding strategies. This gap between apparent safety and underlying vulnerability means that no existing mitigation provides a verifiable guarantee of data deletion, fundamentally challenging the GDPR's Right to Erasure in deployed models.

1. Executive Summary

This paper systematizes the landscape of memorization in large language models, analyzing the mechanisms that cause unintended verbatim or approximate reproduction of training data, the methods for detecting such memorization, and the strategies for mitigating it. Drawing on evidence across model families and benchmarks—including prefix-based extraction attacks on GPT-2 and PaLM, membership inference attacks on fine-tuned models, and scaling analyses on PaLM 2 and the PILE dataset—the work dissects three interacting dimensions: factors influencing memorization (model size, data duplication, sequence length, tokenization, and sampling methods), memorization at different stages (where pre-training dynamics create systematic recency biases and fine-tuning can amplify latent vulnerabilities), and a unified taxonomy of definitions spanning verbatim, approximate, extractable, and counterfactual memorization. The paper reports that deduplication can reduce memorized token generation by an order of magnitude, that alignment procedures can suppress extractable memorization by up to ~150× under normal prompts while leaving the underlying memory intact, and that decoding-time interventions such as activation steering reduce memorization by up to 60% with minimal utility degradation. It establishes that test-time alignment and post-hoc filtering limit the accessibility of memorized content without eliminating its existence, and that no current mitigation—including differential privacy, machine unlearning, or data sanitization—provides formal verifiable guarantees of erasure at scale, leaving compliance with the GDPR's Right to Erasure and copyright fair-use doctrine as technically unresolved challenges.

2. Context and Motivation

The Core Problem: Memorization Is a Misaligned Feature, Not a Bug

The fundamental tension this paper grapples with is baked into the architecture of large language models themselves. At their core, LLMs are data compression systems trained to minimize prediction loss over massive corpora. This optimization objective creates an inherent ambiguity: when a model perfectly reproduces a training sequence, is it demonstrating useful knowledge acquisition or harmful data regurgitation? The authors frame memorization not as a failure mode to be eliminated but as a misaligned feature of data compression—a natural consequence of the training objective that becomes problematic when the compressed data includes sensitive, private, or copyrighted content.

This framing is significant because it rejects the intuition that memorization is simply overfitting that better regularization would fix. Instead, it positions memorization as something more fundamental: a spectrum ranging from beneficial factual recall (remembering that Paris is the capital of France) to harmful instance-specific reproduction (regurgitating someone's social security number). The central challenge, as the paper defines it (Section 2), is disentangling these two forms—which current methods cannot do systematically.

The gap the paper identifies is that the field lacks a unified conceptual framework for reasoning about this spectrum. Definitions of memorization are heterogeneous and sometimes contradictory across studies (Table 1 catalogs at least eight distinct definitions: verbatim, approximate, eidetic, discoverable, k-extractable, counterfactual, and τ-compressible memorization, each with different assumptions about what counts as memorization and how it should be measured). This definitional fragmentation means that results from different papers are often incomparable—a model might be declared "safe" by one metric while leaking data by another.

Why This Problem Matters: The Stakes Span Privacy, Law, and Trust

The paper identifies consequences that extend far beyond academic interest across three interconnected domains.

Personal data leakage. As demonstrated by Carlini et al. (2021) in their foundational extraction attack on GPT-2, LLMs can be induced to reproduce training data verbatim—including PII, secret keys, and private communications. More recent work cited in the survey (Nasr et al., 2025) showed that even aligned, production models like ChatGPT can be coerced into leaking training data through divergence attacks that bypass safety fine-tuning, achieving up to 150× more verbatim sequences than benign prompts. The paper notes (Section 7) that Panda et al. (2024) demonstrated targeted neural phishing attacks achieving 50% success rates in extracting PII from fine-tuned models. These are not theoretical vulnerabilities; they represent concrete attack surfaces with direct harm potential. For industries handling sensitive data—healthcare, legal services, customer support—a single PII leak can trigger regulatory penalties under GDPR, HIPAA, or comparable frameworks.

Copyright and intellectual property. The paper situates memorization at the center of high-stakes litigation, prominently featuring the New York Times v. Microsoft/OpenAI lawsuit (filed December 2023). The Times alleged that ChatGPT could reproduce passages of its copyrighted articles verbatim when prompted appropriately, behavior the plaintiffs argue constitutes mass copyright infringement. This is not an isolated incident: the paper references Chabon v. OpenAI (authors claiming infringement in training), Doe v. GitHub (alleging DMCA violations from verbatim code reproduction), and empirical work by Freeman et al. (2024) that analyzed the specific memorization claims in the Times lawsuit. These cases are actively shaping the legal landscape, and their outcomes hinge on technical questions about memorization that the field has not yet answered: What degree of verbatim reproduction constitutes infringement? Does paraphrased memorization create the same legal exposure? How should fair-use doctrine apply to models that can both summarize (potentially fair use) and reproduce verbatim (potentially infringing) the same copyrighted work?

The alignment-memorization illusion. The paper identifies a critical subtlety in Section 6.2: model alignment processes like instruction tuning and RLHF make memorized content harder to extract under normal interaction but do not make it harder to memorize or prevent the memory from existing. Nasr et al.'s divergence attack exploits precisely this gap—by coaxing the model to revert toward its pre-aligned behavior, attackers bypass the safety layer and access the underlying memorized content that alignment never removed. The authors characterize this as a "crucial distinction between achieving safety/utility and achieving true privacy/unlearning." This finding implies that models currently deployed as "safe" may be one carefully crafted prompt away from leaking sensitive training data, undermining both user trust and regulatory compliance claims.

Where Prior Approaches Fall Short

The paper organizes prior work into three categories and identifies specific, structural limitations in each.

Detection methods: can identify symptoms but not guarantee completeness. The paper catalogs three families of detection approaches (Section 5, Table 3):

Extraction attacks (prefix-based, divergence attacks) can demonstrate that memorization exists by actively eliciting training data, but they provide no upper bound on what remains hidden. The divergence attack's 150× improvement over benign prompts likely still captures only a fraction of total memorized content. Furthermore, prefix-based attacks require access to the training data itself (you need the prefix to prompt for the suffix), making them useful for auditing but impractical for adversarial threat models where attackers don't know what's in the training set.
Membership inference attacks (MIAs) classify whether specific data was in the training set using proxies like loss values, perplexity, or calibrated reference comparisons. The paper's critique is devastating for their use as per-instance detectors: Duan et al. (2024) and Zhang et al. (2025) argued that MIAs "lack a well-calibrated null model since one cannot feasibly train an identical model without the target input," making false-positive rates fundamentally unquantifiable. The paper cites Maini and Suri's (2024) critique of a best-paper-award-winning MIA method as further evidence that apparent MIA improvements often exploit distributional artifacts rather than solving the underlying calibration problem. The authors recommend MIAs be demoted to aggregate-level privacy auditing tools rather than per-instance evidence.
Soft prompting (learned continuous embeddings optimized to trigger memorization) can amplify extraction rates but requires full white-box access to model gradients, making it irrelevant as a practical threat vector while remaining useful for research auditing.

The structural gap across all detection methods is what the paper calls "zero-knowledge detection" (Open Question 1, Section 5): the inability to identify memorization from intrinsic statistical properties of the model's outputs without requiring differential comparison to the training data. If a model generates a string with anomalously low perplexity or unusually high confidence, can we conclude it's memorized without checking the training set? Current methods cannot answer this.

Mitigation strategies: none provide formal verifiable guarantees. Section 6 breaks mitigation into training-time, post-training-time, and inference-time approaches, identifying a consistent pattern: each approach reduces some forms of memorization but none provides provable erasure at scale.

Data cleaning (deduplication, PII scrubbing) is the most practical and widely adopted approach. Lee et al. (2022) demonstrated that deduplication reduced memorized token generation by an order of magnitude. However, the paper identifies a critical shortcoming (Section 3): near-duplicates—paraphrased sentences, text with minor edits—fall below standard similarity thresholds and escape detection. Schäfer et al. (2024) showed that models still memorize these "invisible" semantic duplicates because they arrive at similar internal representations even when surface forms differ. The paper argues for shifting from surface-level hash matching to "training data attribution–based approaches that can analyze the influence of near-duplicates and identify which content segments the model internally treats as equivalent."
Differential privacy (DP-SGD) is the only approach that provides formal privacy guarantees, but the paper identifies substantial obstacles to practical deployment. Li et al. (2021) showed LLMs can be strong DP learners when fine-tuning, but the guarantees rely on bounding "group size"—the number of times an individual's data appears. In web-scale pretraining corpora with pervasive duplication and near-duplication, "reliably defining this group size [is] extremely challenging," and consequently "the interpretation and strength of DP guarantees against pretraining-time memorization often remain unclear due to the underlying data redundancy" (Section 6.1). The paper notes that even parameter-efficient DP methods (LoRA + DP) face unresolved questions about whether concentrating noise in a narrow parameter subset weakens the effective privacy protection. The efficiency-utility trade-off remains severe: strict privacy budgets produce substantial performance degradation, and the computational overhead of per-example gradient clipping and noise injection makes scaling to modern LLM sizes prohibitive.
Machine unlearning aims to retroactively remove the influence of specific training examples. Yao et al. (2024) showed approximate unlearning can be 10⁵× cheaper than retraining from scratch. But the paper emphasizes a fundamental limitation: "unlike DP, there is no formal guarantee, thereby leaving a risk that memorization persists." More critically, unlearned states are fragile—the paper notes that "small exposures to previously forgotten data can quickly reverse the unlearning" (Table 5, post-training challenges), making unlearning a continuous maintenance burden rather than a one-time fix. The ParaPO approach (Chen et al., 2025), while innovative in using preference optimization to replace memorized outputs with summaries, acknowledged utility degradation on math, knowledge, and reasoning benchmarks—raising the unresolved question of "what extent memorization is required for utility and generalization."
Inference-time interventions (MemFree decoding, TokenSwap, activation steering) operate as wrappers around pretrained models rather than modifying the models themselves. Activation steering (Suri et al., 2025) showed 60% memorization reduction by manipulating internal activations during generation, and Stoehr et al. (2024) demonstrated that memorization is localized to specific attention heads (finding a single early-layer head that triggers verbatim recall for rare token combinations). However, the paper identifies a critical limitation: neurons involved in one memory "often contribute to others, complicating memory-specific interventions due to the risk of collateral forgetting" (Section 6.3). Furthermore, these methods are reactive—they suppress specific known instances of memorization rather than learning generalizable privacy policies that prevent leakage of novel sensitive content.

The training stage knowledge gap. Section 4 identifies a major blind spot: memorization dynamics across the LLM lifecycle are poorly understood. Pre-training creates systematic recency biases (Jagielski et al., 2023; Kiyomaru et al., 2024)—examples seen late in training are disproportionately likely to be memorized because parameter drift overwrites early representations. Fine-tuning can either amplify latent memorization from pre-training or introduce new catastrophic memorization of the fine-tuning data itself, depending on the task and method (Mireshghallah et al., 2022; Nasr et al., 2025). RLHF adds another layer: Pappu et al. (2024) found that data memorized during fine-tuning persisted with high frequency through RLHF, but found minimal evidence that reward model data itself was memorized. Distillation—ubiquitous in modern ML pipelines but unstudied for memorization—theoretically propagates memorized content from teacher to student through the KL divergence objective, as Singh (2025) and Chaudhari et al. (2025) recently explored. The paper frames these as interconnected stages where memorization risk transforms rather than simply accumulating, demanding "a comprehensive defense with targeted interventions at every phase" (Table 2).

The methodological schism the paper highlights is severe: Tirumala et al. (2022) demonstrated that a model can simultaneously "forget" a sequence according to MIA metrics while reproducing it flawlessly when prompted—the culprit being data duplication, which creates a weak membership signal that MIAs miss while solidifying the sequence as a high-probability output. The paper concludes that "this methodological divergence is so severe that any claim about memorization dynamics... is effectively ill-defined without a precise specification of the metric," calling for "immediate community-wide efforts toward standardized memorization evaluation protocols."

How This Paper Positions Itself Relative to Existing Work

The paper explicitly differentiates itself from two prior surveys that partially cover the same ground. Hartmann et al. (2023) focused on definitions, implications, detection, and prevention but at a higher level of abstraction. The current paper claims to provide "an in-depth review... by emphasizing concept differences and relationships, main research challenges with key barriers and research directions at different memorization stages, the key principles, requirements and practicality, and core limitations and statistical soundness of detection techniques." Compared to Satvaty et al. (2024), the paper states it "examines the technical underpinnings, evaluation methods, and privacy implications... and provides concrete research directions."

The distinctive contribution—beyond broader coverage—is the systematization of knowledge as a framework. Rather than simply cataloging papers, the paper structures the field around:

A taxonomy of definitions (Table 1) that distinguishes outcome-centric (verbatim, approximate), elicitation-centric (extractable, discoverable, k-extractable), probabilistic ((n,p)-discoverable), causal (counterfactual), and efficiency-based (τ-compressible) memorization concepts, clarifying their relationships, overlaps, and the types of evidence each provides.
A stage-based risk framework (Table 2) that traces how memorization risk transforms across pre-training, SFT, RLHF/post-training, and distillation—identifying specific research challenges and barriers unique to each stage.
A comparative analysis of detection methods (Table 3) organized by goal (extraction vs. classification vs. learned prompting) rather than technique, making explicit the requirements (access level, practicality) and fundamental limitations (statistical soundness, threat model viability) of each approach.
A taxonomy of open questions systematically organized by subtopic (factors influencing memorization, memorization at different stages, detection, mitigation, privacy/legal risks) with explicit rationale and technical barrier formulation for each question.

The paper's meta-contribution is reframing memorization research from a collection of point solutions toward a structured understanding of what problems remain unsolved and why. The open questions are not presented as a wish list but as specific technical barriers: for example, OQ1 in detection asks not just "can we detect memorization without training data access?" but frames it as "an out-of-distribution problem, where the goal is to distinguish plausible linguistic samples from low-complexity artifacts indicative of direct replication" (citing Schwarzschild et al.'s adversarial compression framework). This problem formulation—connecting detection to fundamental properties of the compression objective—is more precise and actionable than asking for better detectors in general.

The paper's stance is ultimately that memorization is not a failure to be eliminated through a single technical fix but a governance problem requiring layered interventions across the model lifecycle, where the frontier is developing "principled control and attribution, rather than a simple balance" between utility and privacy. This positions the paper not as a recipe book but as a roadmap identifying which research directions are most likely to produce meaningful progress toward trustworthy deployment.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

This paper is primarily a systematization of knowledge (SoK)—a survey that organizes, taxonomizes, and critically evaluates existing research rather than introducing a new technical system. The "system" being built is a conceptual framework for understanding memorization in LLMs as a multi-stage phenomenon shaped by interacting factors across the model lifecycle, rather than a single failure mode at training time. The paper solves the problem of fragmented definitions and incomparable results by constructing a unified taxonomy that reveals how different memorization concepts (verbatim, approximate, extractable, counterfactual) relate to each other, which detection methods provide what kind of evidence, and why mitigation strategies fail at the fundamental level of lacking formal verifiable guarantees.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's intellectual architecture has four interconnected components, organized not as a pipeline but as a layered analytical framework:

Definitional Taxonomy (Section 2, Table 1): A classification of eight distinct memorization concepts—verbatim, approximate, eidetic, extractable, discoverable, k-extractable, (n,p)-discoverable, counterfactual, and τ-compressible—organized by their evidential basis (outcome-centric, elicitation-centric, probabilistic, causal, information-theoretic). This component establishes what we mean when we say a model has memorized something.
Factor Analysis (Section 3): An examination of the interacting drivers of memorization—model size, data duplication, sequence length, tokenization, and sampling methods—with quantitative evidence for each factor's contribution and an emphasis on their synergistic, non-additive interplay.
Stage-Based Risk Framework (Section 4, Table 2): A lifecycle model tracing how memorization risk originates in pre-training (through recency bias and data duplication), reshapes during fine-tuning (through parameter-efficient methods creating concentrated "memorization circuits"), potentially amplifies during RLHF (through reward model incentives), and propagates during distillation (through the KL divergence objective). Each stage presents distinct research challenges and barriers.
Detection-Mitigation-Legal Triad (Sections 5-7, Tables 3-6): Three parallel analyses that characterize current methods by their fundamental limitations: detection methods are organized by goal (extraction, classification, learned prompting) and evaluated for statistical soundness; mitigation methods are categorized by intervention point (training-time, post-training, inference-time) and evaluated for guarantee strength; privacy and legal risks are traced from technical failures (lack of verifiable erasure) through regulatory challenges (GDPR compliance) to socio-technical consequences (erosion of trust).

Information flows laterally rather than sequentially: definitions constrain what detection methods can claim; detection limitations determine what mitigation strategies can verify; unresolved technical barriers create the legal and regulatory gaps.

3.3 Roadmap for the Deep Dive

First, the definitional taxonomy (Table 1), because every subsequent claim about detection, mitigation, or risk depends on which memorization concept is being invoked—and the field uses at least eight.
Second, the factor interaction model (Section 3), because the factors are not independent; understanding their synergy explains why simple interventions against single factors fail.
Third, the stage-based lifecycle model (Section 4, Table 2), because memorization risk transforms qualitatively across pre-training, fine-tuning, RLHF, and distillation—a risk introduced at one stage may remain latent and be reactivated at another.
Fourth, the detection methodology analysis (Section 5, Table 3), organized by the goal of the method (extraction vs. classification vs. learned prompting) rather than by technique, to expose the statistical and practical limitations that make per-instance detection claims unreliable.
Fifth, the mitigation taxonomy (Section 6, Tables 4-5), categorized by intervention timing (training, post-training, inference) and evaluated against the standard of formal verifiable guarantees—revealing that no current method meets this standard at scale.
Sixth, the privacy-legal risk framework (Section 7, Table 6), which connects specific technical failures (non-verifiable unlearning, paraphrastic leakage, fragility of forgetting) to regulatory compliance gaps and ongoing litigation.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systematization of knowledge paper whose core idea is that memorization in LLMs cannot be understood as a single phenomenon addressed by point solutions; rather, it is a lifecycle-spanning risk requiring a unified conceptual framework that reconciles heterogeneous definitions, identifies fundamental limitations across detection and mitigation methods, and traces technical gaps to their legal and regulatory consequences.

3.4.1 The Definitional Taxonomy: Eight Concepts, Three Epistemic Categories

The paper's foundational contribution is Table 1, which taxonomizes eight distinct definitions of memorization from the literature into five epistemic categories based on how they establish that memorization has occurred. This is essential because different definitions imply different detection methods, different threat models, and different guarantees—and the paper demonstrates that findings obtained under one definition often fail to transfer to others.

Outcome-Centric Definitions (Three Concepts)

These definitions classify memorization by the nature of the model's output relative to the training data, without specifying how that output was elicited.

Verbatim/Perfect Memorization: Exact, word-for-word reproduction of a training sequence with no deviation. This is the most stringent outcome and serves as the baseline for the most severe form of data leakage. The paper positions this as the target output for eidetic recall and the successful outcome of extractable memorization attacks. Its legal significance is direct: verbatim reproduction forms the basis for copyright infringement claims and PII exposure concerns.
Approximate/Paraphrased Memorization: Generation of text semantically equivalent to a training sequence but not a verbatim copy—for example, rephrasing a sentence while preserving its meaning. The paper identifies this as a known detection challenge: most elicitation-based methods rely on exact string matching and thus miss paraphrased reproduction entirely. Ippolito et al. (2023), cited in the paper, used normalized edit distance with a threshold to detect approximate matches, but the paper notes that this creates a fundamental tension: setting the threshold too high misses real leakage, while setting it too low creates false positives. The significance is that paraphrased memorization may circumvent both technical detection and legal definitions of infringement while still exposing sensitive information or copyrighted content's semantic content.
Eidetic Memorization: A qualitative descriptor for exceptionally strong instances of verbatim recall—specifically, the high-fidelity reproduction of long, complex, or low-probability sequences. Carlini et al. (2021) introduced this concept to characterize cases where a model reproduces not just common phrases but rare, specific sequences (e.g., a unique URL or a personal conversation). The paper frames eidetic memorization as representing "the upper bound of recall fidelity"—it signals deep embedding of data in model parameters rather than surface-level statistical overfitting.

Elicitation-Centric Definitions (Three Concepts)

These definitions classify memorization by how the training data is retrieved—the nature of the prompt or attack required to produce the memorized output. This category is crucial because it captures the adversarial dimension: different elicitation methods have different threat model requirements (access to training data, model weights, output distributions) and thus different practical relevance for security analysis.

Extractable Memorization: The broadest elicitation category, defined as the existence of any constructible prompt that causes the model to generate a specific training example. Nasr et al. (2025) formalized this as the "absolute attack surface": if data can be retrieved by any means, the model is considered vulnerable. The paper notes this definition is intentionally threat-model-agnostic—it covers both black-box attacks with no training data access (if an attacker can discover effective prompts) and white-box auditing scenarios.
Discoverable Memorization: A constrained subset of extractable memorization where the prompt is the actual prefix from the training data. Given a training example composed of prefix $p$ and suffix $s$ (originally consecutive in the training corpus), the model is prompted with $p$ , and discoverable memorization occurs if it reproduces $s$ verbatim. Carlini et al. (2023) and Lee et al. (2022) used this methodology systematically. The paper identifies this as "a practical and measurable method for probing memorization using known training data prefixes," enabling large-scale privacy auditing. However, it simultaneously notes that this definition is "inapplicable to black-box threat models due to its reliance on training data access"—an attacker who doesn't know what was in the training set cannot construct the prefix needed to test for discoverability.
k-Extractable Memorization: A refinement of discoverable where the prefix has a specified length $k$ tokens. Biderman et al. (2023) used this formalization to quantify the relationship between context length and extraction likelihood. The paper positions this as enabling "quantifying the relationship between context length and the likelihood of extraction," which is central to understanding how much partial information an attacker needs to reconstruct a training example.

Probabilistic Definitions (One Concept)

(n, p)-Discoverable Extraction: Hayes et al. (2025) introduced this formalization to account for the stochastic nature of sampling-based decoding. A string $s$ is $(n, p)$ -discoverable if it appears in at least one of $n$ generated completions with probability at least $p$ . The paper presents this as moving "beyond deterministic checks to a more realistic, probabilistic framework for assessing leakage in real-world, non-greedy decoding scenarios." Under greedy decoding, a model might appear safe for a given prefix, but sampling with temperature might reveal the memorized completion with non-trivial probability over repeated attempts. The paper explicitly notes that this framework "better quantifies data exposure risk under diverse sampling strategies," directly connecting to the findings in Section 3 that sampling method choice significantly impacts memorization extraction rates.

Causal Definitions (One Concept)

Counterfactual Memorization: Zhang et al. (2023) introduced this as a fundamentally different epistemic basis: a model's output is attributed to memorization only if the output would change upon removing the specific data point from the training set. The paper identifies this as requiring "a causal link, not just correlational output," which "provides a theoretical grounding for all other forms of memorization." The practical challenge is severe: retraining a massive model without each candidate data point is computationally prohibitive. The paper notes that Pappu et al. (2024) and Zhang et al. (2023) approximated this by training a smaller number of models that exclude entire subsets of data, then testing whether excluded examples are still reproduced—a necessary but imperfect approximation.

The paper emphasizes that this definition is what allows researchers to distinguish "true memorization from coincidental generation"—a model might produce a sequence that happens to match training data not because it memorized it but because the sequence is a plausible completion under the learned distribution. Only counterfactual analysis can separate these cases.

Efficiency/Information-Theoretic Definitions (One Concept)

τ-Compressible Memorization (Adversarial Compression Ratio): Schwarzschild et al. (2024) defined memorization by the compression ratio $\tau$ between prompt length and output length. High compression (short prompt produces long memorized output) indicates efficient extraction. The paper frames this as an information-theoretic approach that "provides a continuous spectrum of memorization 'strength' rather than a binary classification." Its legal relevance is specific to fair use analysis: the "extraction effort" (prompt complexity) relative to the "recall" (output length) could inform whether a reproduction constitutes transformative use. A prompt that is trivially short but produces a lengthy verbatim passage might argue against fair use; a prompt that requires substantial creative construction to elicit a short fragment might argue for it.

Why This Taxonomy Matters Beyond Classification

The paper uses this taxonomy to make a deeper methodological point: the field's inconsistent use of these definitions creates a measurement schism where different methods can deliver contradictory verdicts about the same model. The paper's central example (Section 4) is Tirumala et al. (2022): membership inference attacks (which operate under a probabilistic, loss-based definition) can indicate that a model has "forgotten" a sequence because its loss on that sequence is not anomalously low, while prefix-based extraction (operating under a discoverable definition) simultaneously demonstrates that the model reproduces the sequence flawlessly when prompted. The mechanism is data duplication: duplication creates a weak, distributed membership signal that evades MIA detection while simultaneously solidifying the sequence as a high-probability output. The paper concludes this divergence is "so severe that any claim about memorization dynamics... is effectively ill-defined without a precise specification of the metric."

3.4.2 Factor Interaction Model: The Cascade of Influences

Section 3 synthesizes evidence across multiple studies to construct a model of memorization as an emergent property of interacting factors rather than independent additive effects. The paper explicitly describes this as a "cascade": data duplication provides an initial signal, model size captures it, tokenization amplifies it for specific sequence types, and sampling methods determine whether it becomes observable.

Model Size

The foundational finding, from Carlini et al. (2021), established that memorization scales log-linearly with model size: each doubling of parameter count produces a roughly constant multiplicative increase in memorized content. The paper traces this finding through subsequent work that both reinforces and complicates it:

Tirumala et al. (2022) observed that larger models not only memorize more content but do so more rapidly during training—a phenomenon the paper notes "cannot be fully explained by conventional overfitting or hyper-parameter tuning." The implication is that memorization is not a late-training pathology that better early stopping would prevent but rather an intrinsic property of how larger models allocate capacity.
Li et al. (2024) demonstrated that extraction attacks become significantly more effective against larger models, meaning the scaling effect has both a capacity component (more memorization occurs) and an accessibility component (memorized content is easier to extract). This dual effect compounds the privacy risk beyond what raw memorization counts would suggest.

The paper draws on corroborating evidence from Kandpal et al. (2022), Biderman et al. (2023), and Carlini et al. (2023) to establish this as a robust finding across model families and evaluation protocols.

Data Duplication

The paper presents data duplication as the factor with the strongest empirical evidence and the most direct mitigation lever, but simultaneously identifies its persistence through "invisible" near-duplicates as a fundamental limitation of current approaches.

Lee et al. (2022) established the quantitative baseline: deduplication produced a tenfold decrease in memorized token generation compared to models trained on non-deduplicated data. This is the paper's headline number for the effectiveness of data cleaning.
Kandpal et al. (2022) identified a superlinear relationship: rarely duplicated training samples are seldom memorized, while heavily duplicated samples are memorized at rates disproportionate to their frequency. This means duplication doesn't just linearly increase memorization probability—it creates a threshold effect where sequences above a certain repetition count become almost certain to be memorized.
The paper's critical addition to this literature is the identification of near-duplicate vulnerability (Section 3): Schäfer et al. (2024) showed that models still memorize sequences that are semantically equivalent to training data but syntactically distinct—paraphrased sentences, text with minor edits, content with the same meaning expressed differently. These near-duplicates fall below standard similarity thresholds used in deduplication pipelines, which typically rely on hash-based exact matching or n-gram overlap with strict thresholds.

The paper's proposed solution direction is significant: shift from surface-level matching to "training data attribution–based approaches that can analyze the influence of near-duplicates and identify which content segments the model internally treats as equivalent." This means using influence functions (Koh and Liang, 2017) or similar attribution methods to determine whether the model has learned to map syntactically distinct but semantically identical inputs to the same internal representation—effectively defining duplication from the model's perspective rather than from a pre-defined heuristic.

Sequence Length

Carlini et al. (2023) established that memorization increases logarithmically with sequence length, with "verbatim reproduction probability rising by orders of magnitude as sequences extend from 50 to 950 tokens." The paper identifies two distinct length effects:

Absolute sequence length: Longer training sequences are more memorable because they contain more unique token patterns that serve as effective retrieval cues—a 950-token passage has many more potential prompt prefixes than a 50-token sentence.
Prefix length in extraction: Longer prefixes increase the likelihood of verbatim completion because they reduce ambiguity about which training example is being cued. Carlini et al. (2023) showed that the probability of extraction increases substantially with longer prefixes, which the paper interprets as evidence that memorization is retrieval-like: more context narrows the set of possible completions until only the memorized one remains.

The paper notes that extraction methodologies exploiting this relationship include "continuous soft prompting and dynamic soft prompting techniques" (Wang et al., 2024) that learn to construct optimal prefix-like embeddings, demonstrating "a consistent pattern wherein the volume of extracted data grows proportionally with increasing prefix token sizes."

Tokenization

Kharitonov et al. (2021) established that models trained with larger Byte Pair Encoding (BPE) vocabularies memorize significantly more training sequences. The mechanism is specific and structural: BPE tokenization merges frequently co-occurring byte pairs into single tokens. For rare or unique sequences—named entities, URLs, code snippets, uncommon phrases—larger BPE vocabularies tend to collapse these sequences into single tokens or very short token sequences. The paper explains the consequence: "Memorization was particularly strong for named entities, URLs, and uncommon phrases, often becoming single tokens under larger BPE settings."

When a URL like https://specific-user-page.example.com/profile?id=12345 becomes a single BPE token, memorizing that "token" is equivalent to memorizing the entire URL—there is no decomposition into reusable subword units that could support generalization. This connects directly to the counterfactual memorization framework: a single-token representation makes the model's behavior on that sequence indistinguishable from rote storage because there is no internal structure to generalize from.

Sampling Methods

The paper synthesizes a consistent finding across multiple studies: stochastic decoding reveals memorized content that greedy decoding hides.

Carlini et al. (2021) initially established that greedy decoding can reveal memorized sequences but misses others "due to its low diversity"—greedy decoding always selects the single highest-probability token, which may suppress memorized completions if the model has learned to avoid them through alignment or if the memorized sequence is not the single most likely continuation.
Yu et al. (2023) showed that optimizing top-k, nucleus sampling, and temperature parameters "can substantially increase memorized data extraction, in some cases doubling previous baselines." The mechanism is that stochastic sampling occasionally selects tokens that greedy decoding would bypass, and once a memorized sequence is "entered," the model's confidence on subsequent tokens may remain high enough to continue the extraction.
Tiwari and Suh (2025) quantified this as "randomized decoding nearly doubles leakage risk compared to greedy decoding," contradicting earlier assumptions that greedy decoding represented the worst case.
Hayes et al. (2025) provided theoretical grounding through the $(n, p)$ -discoverable framework, demonstrating mathematically that "repeated sampling with varied decoding parameters can expose memorization hidden under greedy approaches."

This finding has direct implications for both attack and defense: an attacker can amplify extraction by sampling with diverse decoding strategies, while a defender cannot rely on greedy decoding as evidence that memorization is absent—the model may still contain the memory, just not as the single most likely token.

Data Modality and Content Characteristics

The paper notes an emerging finding that memorization risk varies systematically by data type. Beyond text, diffusion models replicate training images (Somepalli et al., 2023), and vision-language models exhibit cross-modality memorization where textual prompts elicit memorized images or vice versa (Wen et al., 2025). Within text, "unique or novel sequences, such as PII or secret keys, are more susceptible to memorization because the model cannot generalize from other examples and must store the information explicitly to minimize loss on that data point" (citing Feldman and Zhang, 2020). This connects memorization to the long-tail nature of training data: common patterns can be learned through statistical generalization, but rare patterns force the model into a choice between memorization and high loss.

Interplay of Factors

The paper's distinctive contribution to the factor analysis is the explicit claim that these factors are not additive but synergistic, forming a "cascade" where each amplifies the effect of others:

"This process can be imagined as a cascade with data duplication providing a strong initial signal, which is then captured by the large parameter size of models... This is exacerbated by tokenization, which can collapse lengthy, unique sequences into single, easily memorized units... Specific prompts act as keys while varied sampling methods can either reveal the most obvious memorized content or, over repeated trials, uncover a far wider range of subtly store information."

The paper further connects this to the adversarial compression framework (Schwarzschild et al., 2024): models learn to memorize predictable sequences as an efficient strategy to minimize training loss. The temporal training dynamics then create recency bias (Section 4), favoring recently seen examples for memorization. The conclusion is that "memorization often arises from a 'perfect storm' of factors, underscoring that various confounding factors may determine the strength of memorization as well as that effective mitigation requires a holistic approach rather than addressing any single factor in isolation."

3.4.3 Stage-Based Lifecycle Model: How Memorization Risk Transforms

Section 4 and Table 2 construct a lifecycle model tracing memorization through four stages: pre-training, supervised fine-tuning (SFT), RLHF/post-training, and distillation. The paper's central theoretical contribution here is that memorization is not a static property acquired during pre-training and then either retained or forgotten; rather, it is dynamically reshaped at each subsequent stage, with distinct mechanisms operating at each transition.

Pre-Training: Recency Bias and Predictable Scaling

The paper identifies a systematic temporal bias in memorization during pre-training: examples seen later in training are disproportionately likely to be memorized, while examples seen early are disproportionately likely to be forgotten. The mechanism, drawing on Jagielski et al. (2023) and Kiyomaru et al. (2024), is parameter drift: "updates driven by later data overwrite earlier representations." Under standard non-deterministic training regimes with data shuffling, dropout, and stochastic optimization, the model's parameters continuously shift, and information encoded early can be overwritten unless reinforced by repeated exposure.

The paper identifies a critical interaction with data duplication: duplicated examples are more likely to be revisited throughout training, protecting them from the recency-driven forgetting that affects single-exposure examples. This provides a mechanistic explanation for the superlinear relationship between duplication and memorization observed by Kandpal et al. (2022): duplication doesn't just provide more learning signal; it actively prevents forgetting.

Huang et al. (2024) added that "later-stage checkpoints are more susceptible to memorizing even rare or out-of-distribution content," which the paper interprets as reflecting "increased model capacity and representational flexibility at those stages." After the model has largely converged on common patterns, its remaining capacity can be allocated to storing rare examples that wouldn't fit into generalizable representations.

Biderman et al. (2023) demonstrated that this process follows predictable scaling laws: "as model size and training duration increase, specific sequences predictably transition from unmemorized to memorized." The paper's implication is that memorization is an "emergent property of scale that is heavily exacerbated by, and often inseparable from" overfitting—it is not a training failure that better regularization would prevent but a scaling-driven phenomenon with predictable dynamics.

The paper's most trenchant critique of the pre-training literature is methodological: the MIA-vs-extraction schism documented in Tirumala et al. (2022). A model can appear to have "forgotten" a sequence under MIA metrics (its loss on that sequence is not anomalously low) while simultaneously reproducing it under prefix-based extraction. The paper's explanation—data duplication creating a weak membership signal that MIAs miss while solidifying the sequence as a high-probability output—amounts to a diagnosis of why detection methods disagree. The consequence: "this methodological divergence is so severe that any claim about memorization dynamics... is effectively ill-defined without a precise specification of the metric."

Supervised Fine-Tuning: Amplification vs. Introduction of New Memorization

The paper identifies fine-tuning as having a dual, context-dependent effect on memorization:

Amplification of latent memorization from pre-training. Nasr et al. (2025) introduced the Janus Interface concept: targeted fine-tuning on specific data distributions can reactivate and amplify previously latent pre-training memorization. The mechanism is that fine-tuning shifts the model's output distribution toward patterns present in the fine-tuning data; if those patterns overlap with pre-training data, the model may become more likely to reproduce them. The paper frames this as "fine-tuning can also be leveraged to extract memorized pretraining data."
Introduction of new memorization of the fine-tuning data itself. Mireshghallah et al. (2022) provided a comparative analysis showing that "head-only fine-tuning presents the highest risk of memorization, likely due to overfitting," while "adapter-based fine-tuning, when constraining parameter updates, reduces memorization." The mechanism relates to the number of parameters being updated: full fine-tuning or head-only tuning modifies many parameters and can overfit to the (typically small) fine-tuning dataset, while parameter-efficient methods like adapters restrict updates to a small subset of weights, reducing the capacity for new memorization.

The paper also reports Mireshghallah et al.'s finding that task type matters: "differences in memorization from models fine-tuned for tasks such as summarization vs. question answering [link] to the attention dynamics of each task, with narrower attention patterns correlating with higher memorization." A summarization model that focuses attention narrowly on source text may memorize specific passages from that text; a QA model with broader attention may be less prone to instance-specific memorization.

Table 2 identifies the core research barrier for the SFT stage: Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA "concentrate updates into a small subset of weights, potentially creating specialized 'memorization circuits' that are hard to disentangle from circuits responsible for task-specific capabilities." The paper's proposed direction is "provable and efficient machine unlearning tailored to the SFT context" and "novel regularization techniques for PEFT that penalize the verbatim memorization of SFT examples without causing catastrophic forgetting of pre-trained knowledge."

RLHF/Post-Training: The Reward Model Incentive Problem

The paper's coverage of RLHF memorization is grounded in a single study—Pappu et al. (2024), conducted in the code generation domain—but extracts a general principle: "data memorized during fine-tuning persists with high frequency in post-RLHF models," while there is "minimal evidence that reward model data or reinforcement learning data becomes memorized." This implies that RLHF does not introduce new memorization of the reward modeling data but also does not erase memorization introduced during earlier stages.

Table 2 frames the RLHF stage as a potential adversarial game: "the reward model acts as a black box, and its own memorized biases can implicitly encourage the policy model to reproduce undesired content." The mechanism is that the optimization process may discover that regurgitating certain memorized sequences is an effective strategy for maximizing reward—if the reward model (trained on human preferences) has learned to favor certain patterns that align with memorized content, the policy model will be reinforced for reproducing those patterns.

The paper's proposed research direction is ambitious: "co-develop a 'regurgitation-seeking' model trained to elicit memorized content from the policy model" and "reward [the primary model] for helpfulness and resisting the adversarial agent." This is essentially a game-theoretic approach to alignment where memorization suppression is explicitly adversarial rather than a side effect of general safety training.

Distillation: The KL Divergence as a Memorization Conduit

The paper identifies distillation as theoretically likely to propagate memorization but empirically understudied. The standard distillation objective minimizes the Kullback-Leibler (KL) divergence between student and teacher output distributions:

$D_{KL}(p_t \| p_s) = \sum_i p_t(i) \log\frac{p_t(i)}{p_s(i)}$

where $p_t$ is the teacher model's output probability distribution over the vocabulary and $p_s$ is the student model's distribution for the same input.

What it computes: the KL divergence measures how much information is lost when using the student's distribution to approximate the teacher's distribution. When minimized as a training objective, it forces the student to match the teacher's output probabilities token by token.

Why this form matters for memorization: if the teacher has memorized a training sequence, it will assign extremely high probability (near 1.0) to the memorized tokens when prompted appropriately. The KL objective then directly incentivizes the student to replicate this overconfidence—the student is penalized for being uncertain on tokens that the teacher is certain about. The paper cites Singh (2025) and Chaudhari et al. (2025) as early evidence that this mechanism propagates memorized content from teacher to student, but notes that "this has not been formally analyzed" in a comprehensive study.

Table 2 frames the research direction as moving beyond "matching raw logits" to "distilling from calibrated or temperature-scaled teacher distributions to penalize overconfidence, or augmenting the loss with a regularizing term that minimizes the student's divergence from a 'safe' public dataset."

3.4.4 Detection Methodology: Organizing by Epistemic Category

Section 5 and Table 3 organize detection methods not by technique but by what kind of claim they enable: extraction (the method produces training data as output), classification (the method labels data as member/non-member), or learned prompting (the method discovers optimal prompts for elicitation). This organization exposes that different detection categories provide fundamentally different types of evidence, and that conflating them—as the field routinely does—produces invalid comparisons and overclaimed conclusions.

Extraction-Based Methods

These methods aim to make the model generate training data verbatim. Their defining characteristic is that the output is the memorized content itself, providing direct evidence that specific sequences are stored in the model.

Divergence Attack (Nasr et al., 2025): This method coaxes aligned models to revert to their pre-aligned, base-model behavior by constructing prompts that induce "decoding analogous to an end-of-text token during pretraining, a context in which LLMs are known to favor high-likelihood and often memorized continuations." The key principle is that alignment training (instruction tuning, RLHF) does not erase memorization but rather adds a "safety layer" that suppresses memorized outputs under normal interaction patterns. By constructing prompts that circumvent this safety layer—essentially tricking the model into behaving as if it were still in pretraining mode—the attack accesses the underlying memorized content.

The paper reports quantitative impact: "achieving up to 150× more verbatim sequences compared to typical user queries." The practical assessment in Table 3 rates this as high practicality ("cheap prompt engineering") with requirements of only black-box access. However, the limitation is significant: the attack "targets transient alignment artifacts rather than core memorization phenomena" and "lacks durability and is susceptible to mitigation via model updates." If the alignment procedure is improved to close the specific divergence vectors the attack exploits, the extraction rate could drop dramatically without any change to the underlying memorized content.

Prefix-Based Data Extraction (Carlini et al., 2021; Lee et al., 2022; Carlini et al., 2023; Li et al., 2024): This method queries the model with a known training data prefix and checks whether it generates the corresponding suffix verbatim. The methodology requires access to the training data to construct the prefixes, making it suitable for auditing (where the auditor has access to the training set and wants to verify that specific content cannot be extracted) but "inapplicable to black-box threat models" (Table 3) where an attacker doesn't know what was in the training data.

The paper traces the evolution: Carlini et al. (2021) initially demonstrated extraction from GPT-2 without training data access by using diverse strategic prompts (beginning-of-sequence tokens, internet text not believed to be in training). Lee et al. (2022) systematized the prefix methodology to measure deduplication's effect on memorization rates. Carlini et al. (2023) established the relationship between prefix length and extraction probability. Li et al. (2024) introduced LLM-PBE, a benchmark toolkit for "evaluating privacy risks via prefix-based extraction" that showed structured prefixes (email headers, document beginnings) are particularly potent at eliciting memorized sequences.

The limitation is fundamental: "scope is restricted to verbatim prefix completion, offering low recall for other memorization types" (Table 3). The method cannot detect approximate memorization (paraphrased outputs would not match the suffix), cannot detect memorization where the training example is retrievable through a prompt other than its original prefix, and cannot estimate the total volume of memorized content—it only confirms what is extractable through the specific prefixes tested.

Classification-Based Methods

These methods classify data points as training members or non-members based on model outputs without generating the memorized content itself. The core challenge—and the paper's central critique—is that they lack a well-calibrated null distribution, making per-instance claims statistically unsound.

Membership Inference Attacks (MIAs) (Shokri et al., 2017; and many subsequent variants): The paper traces the evolution of MIA methodology through several generations of techniques, each attempting to address limitations of previous approaches:

Raw loss (Yeom et al., 2018): The simplest MIA, using the model's loss on a target input as the signal. The assumption is that training examples have lower loss because the model was optimized on them. The paper notes this method's limitation without explicitly stating it: input difficulty confounds the signal—a difficult non-member example may have higher loss than an easy member example, producing false classifications.
Reference-based calibration (Carlini et al., 2021): Subtracts the loss from a separate reference model (trained on different data) on the same input to isolate differences due to training exposure. The intuition is that if both models assign similar loss, the low loss is due to the input being inherently predictable, not due to memorization. If the target model assigns significantly lower loss than the reference, memorization may be the cause.
Zlib entropy normalization (Carlini et al., 2021): Normalizes loss by the zlib-compressed size of the input sequence, using compression ratio as a proxy for complexity. The intuition is that complex sequences should have higher loss regardless of memorization; normalizing by complexity isolates the memorization signal.
Neighborhood attack (Mattern et al., 2023): Examines local changes in the loss landscape by "generating nearby perturbations of the target input and comparing their average loss." The intuition is that memorized examples sit in sharp loss minima—small perturbations cause large loss increases—while non-memorized examples sit in flatter regions.
Min-k% prob (Shi et al., 2023): Focuses on the subset of tokens with the highest loss (least confident predictions) within a sequence. The assumption is that "even the most uncertain tokens in a memorized sequence will still be predicted with confidence"—a memorized sequence may have some tokens predicted with lower confidence, but those tokens will still have higher confidence than equivalent tokens in non-memorized sequences.

The paper then delivers its central critique of MIAs, drawing on Duan et al. (2024) and Zhang et al. (2025):

"MIAs lack a well-calibrated null model since one cannot feasibly train an identical model without the target input. This makes it impossible to meaningfully estimate false positive rates, undermining the statistical soundness of individual predictions."

The epistemological problem is structural: to know whether a given loss value indicates memorization, one needs to know what the loss would have been if the model had not been trained on that example. Since retraining without each candidate example is computationally prohibitive (requiring one full training run per data point being tested), there is no ground truth for calibration. The paper's reference to Maini and Suri (2024) further documents that apparent MIA improvements "are likely artifacts of distributional shifts introduced by the benchmark itself, which MIAs can exploit without truly resolving the underlying calibration problem."

The paper's recommendation is to "re-frame [MIAs] as tools for aggregate-level privacy auditing rather than as evidence of individual training data exposure," and to advocate for alternatives like "data extraction attacks or canary-based MIAs, which offer more direct insights into memorization and leakage." Canary-based MIAs involve inserting known unique sequences into the training data and then testing whether the model can be induced to reproduce them, providing a controlled ground truth that per-instance MIAs cannot.

Learned-Prompting Methods

These methods use optimization to discover continuous (embedding-space) prompts that maximize memorization extraction rates.

Soft Prompting (Ozdayi et al., 2023; Kim et al., 2023; Wang et al., 2024): The core technique is to prepend learnable continuous embeddings (rather than discrete tokens) to the model input and optimize them via gradient descent to maximize the probability of generating memorized content. Ozdayi et al. (2023) demonstrated that fixed-length continuous prompts could amplify memorization leakage by up to 9.3% or suppress extraction by up to 97.7%—the same technique serves as both attack and defense depending on the optimization objective.

Kim et al. (2023) introduced ProPILE, "a privacy auditing framework that uses soft prompt tuning in a white-box setting to extract PII memorized by LLMs." The paper notes that their learned prompts were "transferable across models, suggesting consistent memorization patterns that can be systematically exploited."

Wang et al. (2024) extended static soft prompts to dynamic soft prompting, where a prompt generator produces "tailored soft prompts based on the input prefix." This enables context-sensitive extraction: different prefixes receive different optimized soft prompts rather than a single universal prompt. The paper reports that "dynamic prompts significantly improved the discoverable memorization rate, surpassing both static prompt and no-prompt baselines."

Table 3's practical assessment is clear: soft prompting has "full white-box access" requirements and "prohibitively expensive" computational cost, making it "irrelevant as a practical attack vector" while useful for research auditing. The limitation is that "dependency on full white-box access and prohibitive computational cost limits its applicability to research contexts."

3.4.5 Mitigation Taxonomy: Guarantee Strength as the Organizing Principle

Section 6 categorizes mitigation strategies by intervention timing (training-time, post-training, inference-time) but evaluates them against a consistent standard: does the method provide formal verifiable guarantees that memorized content has been removed? The paper's finding is that no current method meets this standard at scale, with only differential privacy providing formal guarantees that become difficult to interpret in web-scale pretraining contexts.

Training-Time Interventions

Data Cleaning (Deduplication and PII Scrubbing): The paper presents data cleaning as the most practical and widely adopted approach but identifies specific structural limitations. Deduplication removes exact and near-exact duplicate sequences from training data, reducing the overrepresentation that drives memorization. The quantitative evidence from Lee et al. (2022) is a tenfold reduction in memorized token generation.

However, the paper identifies two failure modes:

Near-duplicate escape: Schäfer et al. (2024) demonstrated that models memorize near-duplicates (paraphrased content, text with minor edits) that fall below standard similarity thresholds. The paper argues this necessitates "training data attribution–based approaches that can analyze the influence of near-duplicates and identify which content segments the model internally treats as equivalent." This shifts the definition of duplication from surface-level string matching to model-internal representational similarity.
Uniqueness-driven memorization: The paper acknowledges (citing Feldman and Zhang, 2020) that unique sequences like PII or secret keys are memorized because the model "cannot generalize from other examples and must store the information explicitly," not because of duplication. Deduplication is irrelevant for truly unique sequences—they appear once and are memorized on that single exposure.

PII scrubbing, guided by taxonomies like Lukas et al. (2023), systematically removes or replaces sensitive tokens (CARDINAL, DATE, PERSON categories) from training data. The paper reports that Li et al. (2024) demonstrated that systematic PII scrubbing "dramatically reduce[s] such leakage, confirming the efficacy of targeted data cleaning as a primary defense mechanism."

Differential Privacy (DP-SGD): The paper presents DP as the only approach providing formal privacy guarantees but identifies substantial scaling and interpretation challenges.

The standard DP-SGD procedure (Abadi et al., 2016) operates by:

Computing per-example gradients for each training instance in a minibatch.
Clipping each gradient to a fixed norm $C$ to bound individual influence: $\tilde{g} = g \cdot \min(1, C/\|g\|_2)$ .
Adding calibrated Gaussian noise to the aggregated clipped gradients: $\tilde{G} = \frac{1}{B}\sum_i \tilde{g}_i + \mathcal{N}(0, \sigma^2 C^2 I)$ .
Tracking cumulative privacy loss via a privacy accountant to enforce the global $(\epsilon, \delta)$ budget, where $\epsilon$ bounds the privacy loss and $\delta$ is the failure probability.

What this procedure computes: a noisy gradient update where no single training example can influence the update beyond a bounded amount, determined by the clipping norm $C$ and noise multiplier $\sigma$ . The $(\epsilon, \delta)$ parameters quantify the privacy guarantee: smaller $\epsilon$ means stronger privacy but typically worse utility.

Why DP faces scaling challenges for LLM memorization: the paper identifies the "group size" problem as the central barrier. DP guarantees assume a bounded number of contributions per individual. In web-scale pretraining corpora with pervasive duplication and near-duplication, reliably bounding how many times an individual's data appears is "extremely challenging." As the paper states, "the interpretation and strength of DP guarantees against pretraining-time memorization often remain unclear due to the underlying data redundancy."

The paper reports that Li et al. (2021) demonstrated LLMs can be "strong DP learners" when fine-tuning—pretrained representations are resilient to DP noise—but that this finding applies primarily to the fine-tuning regime, not to pretraining from scratch. For DP-PEFT approaches (LoRA + DP, as explored by Ma et al., 2024), the paper identifies an unresolved question: "the limited parameter updates in PEFT may concentrate DP noise in a narrow subset of the model, potentially weakening privacy protection." The intuition is that if only a small number of parameters are updated, an adversary might need to compromise only those parameters to extract information, making the effective privacy guarantee weaker than the formal $(\epsilon, \delta)$ budget would suggest.

The paper notes that user-level DP (Chua et al., 2024), which bounds contributions at the user rather than record level, "reduces memorization, with much lower canary extraction rates than record-level DP," but does not resolve the fundamental scaling challenge.

Post-Training-Time Interventions

Machine Unlearning: The paper defines unlearning as aiming "to remove the influence of certain training examples so that the model's behavior is as if those examples were never seen." The paper reports that Yao et al. (2024) evaluated various strategies—gradient ascent (reversing the gradient direction on target examples), negative re-labeling (training the model to produce nonsensical outputs for target examples), and adversarial sampling (training on adversarially perturbed versions of target examples)—and found that "approximate unlearning methods can be over 10⁵× more computationally efficient than retraining a model from scratch."

However, the paper identifies a fundamental limitation:

"unlike DP, there is no formal guarantee, thereby leaving a risk that memorization persists."

The paper further identifies that unlearned states are fragile: "small exposures to previously forgotten data can quickly reverse the unlearning, undermining long-term effectiveness" (Table 5). This means unlearning is not a one-time operation but a continuous maintenance burden—and one without a verifiable success criterion.

ParaPO (Chen et al., 2025): This method takes a different approach from traditional unlearning. Instead of attempting to erase the influence of memorized data through gradient manipulation, it:

Identifies memorized sequences in the pretraining corpus by searching for sequences where prefix-based prompting produces near-exact suffix matches.
Uses a separate LLM to summarize each memorized datapoint, creating a paraphrase.
Applies Direct Preference Optimization (DPO) (Rafailov et al., 2024) on pairs of (memorized sequence, summarization), marking the summarization as preferred.

The DPO objective is:

$\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right]$

where $\pi_\theta$ is the policy being optimized, $\pi_{\text{ref}}$ is a reference model (typically the pre-DPO checkpoint), $y_w$ is the preferred output (the summary), $y_l$ is the dispreferred output (the verbatim memorized sequence), $x$ is the prompt, $\beta$ is a temperature parameter, and $\sigma$ is the logistic sigmoid function.

What it computes: the DPO loss increases the relative probability of the summary over the verbatim memorized output under the policy model, compared to their relative probabilities under the reference model. The log-ratio terms measure how much more the policy prefers the summary over the memorized output relative to the reference model's preferences; the loss encourages this difference to be large and positive.

Why this approach is interesting for memorization: rather than trying to erase the knowledge entirely (which might remove useful understanding along with the specific phrasing), DPO redirects the model toward expressing the same information in a non-verbatim form. The paper reports that this "decreases unintended memorization while preserving verbatim recall of desired sequences (e.g., direct quotations)."

However, the paper notes an important limitation: ParaPO "slightly decreases utility on math, knowledge and reasoning benchmarks." This raises the unresolved question of "to what extent memorization is required for utility and generalization"—a question the paper presents as fundamental rather than a technical limitation of ParaPO specifically.

Alignment as Consequential Mitigation: The paper identifies that model alignment (instruction tuning, RLHF) reduces the accessibility of memorized content without reducing its existence. Nasr et al. (2025) demonstrated that aligned models are harder to extract from under normal interaction but remain vulnerable to divergence attacks. The paper frames this as "a crucial distinction between achieving safety/utility and achieving true privacy/unlearning" (Section 6.2). Safety training makes it harder for users to accidentally encounter memorized content; it does not make it harder for a determined attacker to extract it.

Inference-Time Interventions

These methods operate as wrappers around pretrained models, modifying generation behavior without changing model weights.

MemFree Decoding (Ippolito et al., 2023): This method introduces a privacy filter into the generation loop using a bloom filter representing all n-grams in the training set. During generation, if the model's proposed next token would create an n-gram that exists in the training set, the filter can block or penalize that token. The paper identifies limitations: near-identical n-grams with minor modifications evade detection, and access to the model's n-gram training data is required—which may not be available for closed-source models.

TokenSwap (Prashant et al., 2025): A lightweight token-level intervention that disrupts memorized sequences at inference time "entirely without access to the training corpus or model weights." The method identifies tokens likely to be part of memorized sequences and swaps them with alternatives, reducing both exact and near-verbatim generation. The paper presents this as representative of an "emerging emphasis on practical, plug-and-play privacy interventions that can complement more computationally intensive training-time methods."

Activation Steering (Suri et al., 2025; Stoehr et al., 2024; Chang et al., 2023): The paper's most extensive coverage of inference-time methods focuses on manipulating internal activations during generation.

Suri et al. (2025) used sparse autoencoders to identify activation patterns linked to memorized passages. The paper reports that constructing "steering directions that reduced memorization by up to 60%, with minimal performance degradation" was possible by identifying the directions in activation space that correlate with memorized output generation and then subtracting a scaled version of those directions during inference. However, the paper notes the method's sensitivity: "steering effectiveness depends heavily on precise layer selection and steering strength."

Stoehr et al. (2024) provided mechanistic evidence for localization: they "identified a single attention head in an early layer that reliably activates in response to rare token combinations seen during training, triggering verbatim recall." Furthermore, they showed that "fine-tuning only a subset of the high-gradient weights was sufficient to erase memorized passages," suggesting that memorization is not distributed uniformly across the model but concentrated in specific components.

Chang et al. (2023) introduced diagnostic benchmarks (injection and deletion) to evaluate how well different localization methods actually identify neurons responsible for memorized content. The paper reports that "pruning-based approaches were most effective: for example, Hard Concrete was able to identify fewer than 0.5% of neurons whose removal led to a 60% drop in memorization accuracy."

The paper's critical limitation for activation steering is that "neurons involved in one memory often contribute to others, complicating memory-specific interventions due to the risk of collateral forgetting" (Section 6.3). Removing neurons to suppress one memorized fact may accidentally remove related useful knowledge stored in overlapping neural populations.

3.4.6 Privacy-Legal Risk Framework: From Technical Failure to Regulatory Consequence

Section 7 and Table 6 construct a three-layer model connecting technical limitations to legal/regulatory challenges to socio-technical consequences.

Layer 1: Technical Challenges

The paper identifies specific technical failures that create regulatory vulnerability:

Inseparability: Sensitive data "is deeply entangled in model parameters, making precise isolation and removal technically infeasible" (Table 6). This is not a limitation of current methods but a consequence of how distributed representations work—information about any given training example is encoded across many parameters in ways that are difficult to isolate.
Conceptual unlearning failure: "Token-level unlearning is easily bypassed by rephrased prompts, failing to remove the underlying concept" (Table 6). Even if a model is "unlearned" so that it no longer outputs a specific PII string in response to a specific prompt, a rephrased prompt requesting the same information may still succeed because the underlying knowledge representation was never removed—only the surface-level association was suppressed.
Lack of verifiable erasure: "There are no provable guarantees that empirical unlearning has completely and permanently removed data influence" (Table 6). Unlike DP, which provides mathematical privacy guarantees (however difficult to interpret at scale), unlearning provides only empirical evidence that a specific test fails—not that all possible tests would fail.

Layer 2: Legal and Regulatory Challenges

These technical limitations map directly onto specific regulatory requirements:

GDPR Right to Erasure (Article 17): The requirement that individuals can request deletion of their personal data creates a technical obligation that current unlearning methods cannot reliably fulfill. The paper frames this as a compliance gap: the regulation requires a capability (verifiable deletion) that the technology does not provide.
Evolving definition of PII: The legal scope of what constitutes personal data "that a model can infer (not just regurgitate) is legally ambiguous in the US and is inconsistent with GDPR definitions." The paper notes that models can infer sensitive attributes about individuals from non-sensitive training data—memorization of the inference capability rather than the original data—creating privacy risks that current detection methods (focused on verbatim reproduction) cannot identify.
Fair use ambiguity for generative models: The line between transformative use (summarizing or building on copyrighted content, potentially fair use) and derivative infringement (reproducing copyrighted content verbatim) is legally unsettled for generative models. The paper frames this as creating "uncertainty for developers and deployers" because the same model can do both, and it is currently impossible to guarantee or verify that it will not do the latter.

Layer 3: Socio-Technical Consequences

The paper identifies specific harms that follow from unresolved technical-legal gaps:

Erosion of user trust: "The inability to guarantee data deletion undermines user confidence and creates significant reputational risk for model deployers." This is not a hypothetical concern—it directly affects adoption in regulated industries (healthcare, finance, legal) where data confidentiality is a legal obligation.
Accountability gap: "The opaque nature of memorization makes it difficult to assign responsibility when a model produces harmful or illegal content." If a model reproduces defamatory content or hate speech from its training data, who is responsible? The model developer (for training on the data), the deployer (for serving the model), or the original content creator? The paper frames this as a question the technical community cannot answer alone but must inform.
Breakdown of trustworthiness in high-stakes domains: The paper explicitly connects memorization to reliability in medicine and law: "If users cannot trust that a model's output is the result of genuine reasoning [rather than memorized pattern matching], its utility in high-stakes domains may be critically impaired."

The Attribution-Control Framing

The paper's concluding vision (Section 8) reframes the entire memorization problem from one of elimination to one of attribution and control:

"Detection techniques must move beyond identifying symptoms to enable true attribution of knowledge. Mitigation strategies like de-duplication and unlearning, which aim for erasure, show potential but are insufficient; the frontier lies in developing robust methods for control over what a model recalls."

This is a genuinely distinctive theoretical position. Rather than attempting to make models that don't memorize (which the paper argues may be impossible without sacrificing the factual recall that makes LLMs useful), the research agenda should focus on making models that can account for what they've memorized and selectively control whether and how they reproduce it. The paper connects this to ongoing litigation: "Ongoing legal challenges highlight the need for frameworks that can manage this control and attribution to preserve beneficial knowledge while preventing information leakage."

The paper's open questions in Section 7 operationalize this vision: developing copyright-aware training that "augment[s] training corpora with copyright metadata and modif[ies] the optimization objective to explicitly penalize the generation of text that is both verbatim and tagged as protected"; creating memorization metrics that quantify "the degree of transformative use" by assessing "the functional and semantic relationship between an LLM's output and a copyrighted source"; and producing "context-aware memorization auditing" that replaces binary memorized/not-memorized classifications with "a multi-faceted report assessing factors like the novelty of the memorized text content, its commercial value, and the 'extraction effort' required."

4. Key Insights and Innovations

Innovation 1: Memorization Is a Spectrum of Epistemically Distinct Phenomena, Not a Single Concept

The paper's most fundamental intellectual move is to demonstrate that the field's fragmentation—contradictory findings, incomparable results, and definitional disputes—is not merely organizational sloppiness but reflects a genuine epistemic multiplicity: "memorization" is not one thing that different methods measure with varying accuracy but several fundamentally different things that different definitions constitute. Table 1 is not a glossary; it is an argument that outcome-centric definitions (verbatim reproduction), elicitation-centric definitions (extractability under specific prompt conditions), causal definitions (counterfactual dependence on training inclusion), and information-theoretic definitions (compression ratio) answer incompatible questions. A sequence can be memorized under one definition and unmemorized under another—and the paper provides the mechanism: data duplication creates a weak membership inference signal (causing MIA failure) while simultaneously solidifying the sequence as a high-probability output (causing prefix-based extraction success), as demonstrated by Tirumala et al. (2022). This resolves a long-standing tension in the literature not by declaring one definition correct but by showing that each definition captures a different facet of the underlying phenomenon, and that the field's error has been to treat them as interchangeable operationalizations of the same latent variable.

Prior to this paper, surveys in the space—including Hartmann et al. (2023) and Satvaty et al. (2024)—acknowledged definitional diversity but did not systematically relate definitions to each other along epistemic dimensions. The standard approach was to list definitions and note their differences. This paper goes further: it classifies definitions by what kind of evidence they require (observing outputs, constructing prompts, retraining models, measuring compression ratios) and uses that classification to explain why specific pairs of methods disagree. The MIA-vs-extraction schism is not an anomaly but a predicted consequence of using a classification-based definition (membership) versus an elicitation-based definition (extractability) on data with specific structural properties (duplication). This is a diagnostic insight, not a taxonomic one: it tells researchers what kind of disagreement to expect given their choice of definition and their data characteristics.

The significance extends beyond academic clarity. The paper's argument that per-instance MIAs are "statistically unsound" for individual classification—because constructing a calibrated null distribution is computationally intractable—is not a methodological critique of specific MIA variants but a structural limitation of classification-based definitions when applied to models whose training cannot be counterfactually replicated. This conclusion, drawing on Duan et al. (2024), Zhang et al. (2025), and Maini and Suri (2024), has direct legal consequences: if MIAs cannot provide per-instance evidence of training data inclusion, they cannot serve as technical substantiation for claims in copyright litigation or GDPR data subject access requests. The paper's recommended reframing of MIAs as aggregate auditing tools represents not a minor adjustment but a fundamental re-scoping of what this widely used technique can legitimately claim.

Innovation 2: The Alignment-Memorization Distinction as a Category Error with Security Consequences

The paper draws a distinction that, once articulated, seems obvious—but which the prior literature systematically conflated: alignment reduces the accessibility of memorized content without reducing its existence. This is not a nuance; it is a category error with direct security implications. Instruction tuning and RLHF make models safer by teaching them not to output memorized training data under normal interaction patterns, creating an impression of privacy that the paper demonstrates is illusory under adversarial prompting. The evidence is Nasr et al.'s (2025) divergence attack, which achieves up to 150× more verbatim extraction than benign prompts by exploiting the gap between aligned behavior and the underlying base model's memorization. The paper's characterization—"alignment makes memorization harder to extract but does not make it harder to memorize (or prevent the memory from existing)"—reframes alignment as a surface-level filter rather than a privacy mechanism.

This insight reframes the safety literature's relationship to privacy. Prior work largely treated alignment as a positive contribution to privacy (models that refuse to output PII are private), without interrogating whether the underlying information was actually removed. The paper's framing reveals that alignment for safety and privacy are orthogonal objectives that have been optimized under the mistaken assumption that they were aligned. A model can be highly aligned (refusing dangerous requests, avoiding toxic outputs) while being maximally vulnerable to privacy attacks from adversaries who can construct prompts that bypass alignment. The paper's evidence that this vulnerability is real—not theoretical—in production models (the divergence attack was demonstrated on deployed systems) transforms this from an academic distinction into a practical security concern.

The conceptual contribution is more specific than "alignment doesn't solve privacy." The paper identifies the mechanism of the illusion: alignment training teaches the model to avoid certain output distributions under certain input distributions, but does not modify the stored representations that would produce those outputs under different input distributions. This is analogous to training a classifier to reject certain inputs rather than removing the capability that those inputs activate—a learned override, not an architectural change. The paper's proposal to reframe alignment as an adversarial game between a "regurgitation-seeking" red-team model and the primary model (Table 2) follows directly from this diagnosis: if alignment is a learned suppression rather than a capability removal, then alignment must be tested against adaptive attacks that specifically target the suppression mechanism.

Innovation 3: The Lifecycle Model of Memorization Risk Transformation

The paper's organization of memorization across the LLM lifecycle—pre-training, SFT, RLHF, and distillation—is not merely a convenient structure for cataloging papers. It constitutes a theoretical claim that memorization risk is not a static property acquired during pre-training but a dynamic phenomenon that changes its nature at each subsequent stage. Table 2's framework specifies distinct mechanisms at each stage: pre-training introduces risk through recency-biased encoding of duplicated and unique sequences; SFT reshapes risk by concentrating parameter updates into narrow subspaces ("memorization circuits"); RLHF potentially incentivizes memorization through reward model biases; and distillation propagates memorization through the KL divergence objective's pressure to match teacher overconfidence. Each transition represents not just an additional opportunity for memorization but a qualitative transformation in what kind of memorization exists and how it is encoded.

This is a fundamental departure from the prior literature's implicit model, which treated memorization as something that happens during training and is then either retained or forgotten. In that model, post-training interventions (fine-tuning, RLHF, distillation) are downstream processes that inherit the base model's memorization profile unchanged. The paper's evidence contradicts this: Mireshghallah et al. (2022) showed that fine-tuning method choice (head-only vs. adapter-based) produces different kinds of memorization with different extraction characteristics; Pappu et al. (2024) showed that RLHF preserves fine-tuning memorization while not introducing new memorization of reward model data; and the paper's theoretical analysis of distillation identifies the KL divergence objective as an active propagation mechanism, not passive inheritance.

The lifecycle model generates a specific, testable prediction that distinguishes it from the static-retention model: if memorization is reshaped at each stage, then mitigation strategies should be stage-specific rather than one-size-fits-all. The paper's organization of Table 5 into stage-specific challenges (pre-training data sanitization, SFT unlearning, inference-time activation steering) is not organizational convenience but a prescriptive implication of the lifecycle model. The finding that activation steering works well for localized memories (Stoehr et al., 2024) but poorly for distributed knowledge supports this: the stage at which memorization was introduced affects which mitigation strategies are applicable.

This contribution is conceptually fundamental—it changes what researchers should expect when they study memorization across different phases of model development—but empirically incomplete. The distillation stage, in particular, is identified as theoretically likely to propagate memorization but empirically understudied, with the paper citing only early evidence from Singh (2025) and Chaudhari et al. (2025). The lifecycle model is thus better understood as a research program with specific empirical gaps identified at each stage (Table 2's "Key Barriers & Research Directions" column) rather than a fully validated theory.

Innovation 4: The Attribution-Control Framing as an Alternative to the Erasure Paradigm

The paper's concluding vision (Section 8) represents a genuinely distinctive theoretical position: reframing the memorization problem from one of elimination (how do we make models that don't memorize?) to one of attribution and control (how do we make models that can account for what they've memorized and selectively regulate its reproduction?). This is not a technical contribution—no method is proposed—but a framing innovation that reorients the field's research agenda. The erasure paradigm, implicit in most prior work, assumes that the goal is to prevent or remove memorization: deduplicate training data, apply DP to bound information leakage, unlearn specific examples, filter outputs at inference time. The paper does not reject these approaches but argues they are "insufficient" because they treat memorization as a binary condition (present/absent) rather than a resource to be managed.

The attribution-control framing emerges from the paper's diagnosis of why the erasure paradigm fails: useful knowledge (factual recall, linguistic patterns) and harmful regurgitation (PII, copyrighted content) are entangled in the same model parameters through the same compression objective. Eliminating memorization entirely would eliminate the model's capacity to store and retrieve facts—the very capability that makes LLMs useful. The paper's evidence that ParaPO (Chen et al., 2025) reduces verbatim reproduction while slightly degrading utility on reasoning benchmarks supports this entanglement claim: you cannot surgically remove instance-specific memorization without affecting related general knowledge.

The practical implications are concrete and connect directly to ongoing legal developments. The paper's open questions in Section 7 operationalize the attribution side (how can we causally attribute a memorized output to a specific training stage or data point?) and the control side (can we train models to be copyright-aware, distinguishing transformative use from derivative reproduction at generation time?). The legal relevance is direct: fair use doctrine hinges on whether a reproduction is "transformative," a determination that requires understanding how the model used the training data to produce its output—an attribution question. The GDPR Right to Erasure requires verifiable deletion—a control question. The paper's framing positions the technical community's role not as building models that never memorize (which may be impossible) but as building models that provide accountability for and control over their memorization behavior.

This innovation is framing-level rather than technical—it does not introduce a new method or provide new empirical results. Its significance lies in redirecting research attention: rather than pursuing ever-stronger erasure techniques that face fundamental limits (the inseparability problem, the near-duplicate problem, the lack of verifiable guarantees), the field should invest in making memorization legible and governable. The paper's connection of this framing to specific technical barriers (causal attribution requires multi-stage counterfactual interventions that are computationally prohibitive; control requires disentangling knowledge representations that are currently opaque) prevents it from being merely aspirational—each component of the attribution-control vision maps to a concrete, unsolved research problem identified elsewhere in the paper.

5. Experimental Analysis

Evaluation Methodology

Dataset. This paper is a systematization of knowledge (SoK) — a survey and critical synthesis of existing research. It does not introduce new experiments, collect new data, or train new models. Instead, it aggregates results reported across dozens of prior studies. The primary benchmarks discussed include MATH (Hendrycks et al., 2021, specifically the Lightman et al. 2022 split of 12,000 training and 500 test questions), the PILE dataset (Gao et al., 2020), and custom canary-based evaluation sets constructed by individual studies (e.g., Carlini et al., 2021; Lee et al., 2022; Biderman et al., 2023). The paper also discusses legal benchmarks such as the extraction evidence presented in the New York Times v. Microsoft/OpenAI litigation (NYT-MC, 2023). No unified experimental framework is applied; the paper's contribution is the critical comparison and systematization of methods and findings across these heterogeneous settings.

Base model(s). The paper surveys results across GPT-2 (Radford et al., 2019), PaLM 2 (Anil et al., 2023), PaLM (Chowdhery et al., 2022), LLaMA (Touvron et al., 2023), OPT (Zhang et al., 2022), and various fine-tuned and aligned production models (including ChatGPT, as studied by Nasr et al., 2025). No single model family is the focus; the paper's analytical contribution is precisely the cross-model synthesis of findings that were originally established in individual model-specific studies.

Metrics. The paper catalogs multiple metrics used across the literature, each tied to a specific memorization definition:

Discoverable memorization rate: The fraction of training examples whose suffix is exactly reproduced when prompted with the corresponding prefix. Used by Carlini et al. (2023), Lee et al. (2022), and Biderman et al. (2023).
k-extractable memorization: As above, but with the prefix restricted to $k$ tokens. Used by Biderman et al. (2023) to establish scaling relationships.
Extraction rate: Number of verbatim training sequences recovered per query or per unit of compute budget. Used by Carlini et al. (2021) and Nasr et al. (2025).
Membership inference accuracy / AUC: Classification performance for distinguishing training from non-training examples. Used by Shokri et al. (2017), Mattern et al. (2023), Shi et al. (2023), and critiqued by Duan et al. (2024) and Zhang et al. (2025).
Counterfactual memorization score: Difference in model behavior with vs. without a specific training example, approximated by training subset-exclusion models. Used by Zhang et al. (2023) and Pappu et al. (2024).
Adversarial compression ratio (ACR) $\tau$ : Ratio of output length to prompt length for extracted sequences. Introduced by Schwarzschild et al. (2024).

Baselines. Since the paper is a survey rather than an experimental study, "baselines" refer to the comparison conditions used within the individual studies being synthesized. Key baselines across the surveyed literature include:

Greedy decoding vs. stochastic sampling (temperature sampling, nucleus sampling, top-k sampling) — used as a baseline in extraction studies (Carlini et al., 2021; Yu et al., 2023; Tiwari and Suh, 2025; Hayes et al., 2025) to demonstrate that stochastic methods reveal more memorization.
Undeduplicated training data vs. deduplicated data — the baseline for Lee et al. (2022) and Kandpal et al. (2022).
No-alignment (base model) vs. aligned model (instruction-tuned, RLHF) — the baseline for Nasr et al. (2025) and the divergence attack analysis.
No mitigation vs. data cleaning / DP-SGD / unlearning / activation steering — baselines across the mitigation literature (Section 6).
Majority voting and ORM best-of-N weighted — baselines used in PRM-based search studies that the paper references but does not itself conduct.

Generation budget / compute accounting. The paper does not enforce a unified compute accounting framework. Individual studies measure compute in different units: number of model parameters (for scaling analyses), number of training examples or epochs (for memorization dynamics), number of generations or samples (for extraction attacks), and FLOPs (for training-inference tradeoff analyses in the FLOPs-matched comparison literature that the paper references). The paper's contribution is not to harmonize these metrics but to compare findings across them while noting where metric differences create incomparability — this is the paper's explicit methodological critique (Section 4, discussion of the MIA-vs-extraction schism).

Cross-validation / statistical protocol. The paper does not apply a cross-validation protocol of its own. It reports the statistical protocols used by individual studies, including two-fold cross-validation within difficulty bins (used in the search and revision scaling literature the paper references), canary-based insertion with known ground truth (used in DP evaluation studies like Li et al., 2021 and Chua et al., 2024), and probabilistic extraction frameworks like $(n, p)$ -discoverable extraction (Hayes et al., 2025), which formalizes the likelihood of retrieval under repeated sampling as a way to account for stochastic decoding variation. The paper's critical assessment of MIA methodology includes the specific statistical limitation that these methods "lack a well-calibrated null model since one cannot feasibly train an identical model without the target input" (Section 5, citing Duan et al., 2024 and Zhang et al., 2025), making false-positive rates fundamentally unquantifiable — this is a statistical critique, not a protocol recommendation.

Main Quantitative Results

Because this paper synthesizes rather than generates experimental results, the "results" are the quantitative findings from the surveyed literature that the paper organizes into its framework. All numbers below are reported as they appear in the paper, with source studies cited.

Extraction and Detection Results

Prefix-based extraction from GPT-2 (Carlini et al., 2021). The foundational extraction study recovered "hundreds of memorized examples" from GPT-2, including PII, secret keys, and private communications, by querying the model with diverse strategic prompts. The paper reports this as establishing the existence of extractable memorization in production models, but notes the methodology was less systematic than modern prefix attacks — the prompts were not constructed from known training data prefixes.

Effect of deduplication on extraction rates (Lee et al., 2022). Models trained on deduplicated data exhibited "a tenfold decrease in memorized token generation" compared to models trained on non-deduplicated data. This is the paper's headline quantitative result for data cleaning effectiveness (Section 3).

Prefix length and extraction probability (Carlini et al., 2023). Memorization increases logarithmically with sequence length, with "verbatim reproduction probability rising by orders of magnitude as sequences extend from 50 to 950 tokens." The paper uses this to establish the relationship between prefix length (as a prompt) and extraction likelihood.

Divergence attack extraction multiplier (Nasr et al., 2025). The divergence attack achieved "up to 150× more verbatim sequences compared to typical user queries" when applied to aligned, production language models. This is the paper's central quantitative evidence for the alignment-memorization distinction — alignment suppresses extraction by approximately two orders of magnitude under normal interaction but the underlying memory remains accessible.

Effect of sampling methods on extraction (Yu et al., 2023; Tiwari and Suh, 2025). Optimizing top-k, nucleus sampling, and temperature parameters "can substantially increase memorized data extraction, in some cases doubling previous baselines" (Yu et al., 2023). Tiwari and Suh (2025) quantified that "randomized decoding nearly doubles leakage risk compared to greedy decoding." The paper synthesizes these as evidence that no single decoding method minimizes leakage across all scenarios (Section 3).

Probabilistic extraction framework (Hayes et al., 2025). The $(n, p)$ -discoverable extraction formalization demonstrates that "repeated sampling with varied decoding parameters can expose memorization hidden under greedy approaches." The paper presents this as theoretical grounding for why stochastic decoding reveals more memorization (Section 3).

MIA performance and limitations (Duan et al., 2024; Zhang et al., 2025; Maini and Suri, 2024). The paper does not report specific MIA accuracy numbers because its position is that per-instance MIA claims are fundamentally uncalibrated. It cites Maini and Suri (2024) as demonstrating that apparent MIA improvements from a best-paper-award-winning method (Zhang et al., 2024) "are likely artifacts of distributional shifts introduced by the benchmark itself, which MIAs can exploit without truly resolving the underlying calibration problem" (Section 5). The quantitative conclusion is not a number but a methodological verdict: MIAs should be "re-framed as tools for aggregate-level privacy auditing rather than as evidence of individual training data exposure" (Section 5).

Soft prompting extraction rates (Ozdayi et al., 2023; Wang et al., 2024). Fixed-length continuous soft prompts "increased memorization leakage by up to 9.3%" as attack prompts and "decreased extraction by up to 97.7%" as suppress prompts (Ozdayi et al., 2023). Dynamic soft prompting "significantly improved the discoverable memorization rate, surpassing both static prompt and no-prompt baselines" (Wang et al., 2024). The paper reports these as establishing soft prompting as both an attack and defense vector, but notes the white-box access requirement makes it impractical as a real-world threat.

Factors Influencing Memorization: Quantitative Evidence

Log-linear scaling with model size (Carlini et al., 2021). Memorization scales log-linearly with model parameter count. The paper cites this as the foundational finding for model size effects (Section 3), corroborated by Tirumala et al. (2022), Kandpal et al. (2022), Biderman et al. (2023), Carlini et al. (2023), Hayes et al. (2025), and Li et al. (2024).

Superlinear relationship between duplication and memorization (Kandpal et al., 2022). Rarely duplicated training samples are seldom memorized, while heavily duplicated samples are memorized at rates disproportionate to their frequency. The paper frames this as explaining why deduplication is effective but near-duplicate escape creates persistent risk (Section 3).

Tokenization effects (Kharitonov et al., 2021). Models trained with larger BPE vocabularies memorize "significantly more sequences," with the effect "particularly strong for named entities, URLs, and uncommon phrases, often becoming single tokens under larger BPE settings" (Section 3). The paper connects this to the adversarial compression framework: single-token representations eliminate internal decomposition that could support generalization.

Mitigation Results

Data cleaning effectiveness (Lee et al., 2022; Li et al., 2024). Deduplication produced a tenfold reduction in memorized token generation (Lee et al., 2022). Systematic PII scrubbing "dramatically reduce[s] such leakage" (Li et al., 2024, reported by the paper in Section 6.1). The paper presents these as the strongest quantitative evidence for any mitigation approach while noting the near-duplicate limitation.

DP-SGD on LLMs (Li et al., 2021). LLMs can act as "strong DP learners" when fine-tuning on top of pretrained representations, with DP-trained models "approach[ing] the performance of non-private baselines, while offering provable protection against memorization" (Section 6.1). The paper notes this finding is most robust for the fine-tuning regime; pretraining from scratch with DP at scale remains unresolved.

DP-PEFT memorization reduction (Ma et al., 2024). DP-PEFT "reduces memorization and achieves comparable or superior downstream task performance relative to full-model DP-SGD, under strict privacy budgets" (Section 6.1). However, the paper identifies the unresolved concern that "limited parameter updates in PEFT may concentrate DP noise in a narrow subset of the model, potentially weakening privacy protection."

User-level DP extraction reduction (Chua et al., 2024). User-level DP produces "much lower canary extraction rates than record-level DP" (Section 6.1). This is the quantitative evidence that stronger privacy units (user vs. record) produce stronger memorization protection.

Unlearning efficiency (Yao et al., 2024). Approximate unlearning methods "can be over 10⁵× more computationally efficient than retraining a model from scratch" (Section 6.2). The paper reports this efficiency number but immediately qualifies it: "unlike DP, there is no formal guarantee, thereby leaving a risk that memorization persists."

Activation steering reduction (Suri et al., 2025). Using sparse autoencoders to identify and manipulate activation patterns linked to memorized passages "reduced memorization by up to 60%, with minimal performance degradation" (Section 6.3). The paper reports this as the strongest quantitative evidence for inference-time mitigation, while noting the method's sensitivity to "precise layer selection and steering strength."

Neuron localization precision (Chang et al., 2023). Hard Concrete pruning "was able to identify fewer than 0.5% of neurons whose removal led to a 60% drop in memorization accuracy" (Section 6.3). The paper presents this as evidence that memorization is concentrated in a small subset of parameters, enabling targeted intervention, but also notes that "neurons involved in one memory often contribute to others, complicating memory-specific interventions due to the risk of collateral forgetting."

ParaPO utility-memorization tradeoff (Chen et al., 2025). The method "decreases unintended memorization while preserving verbatim recall of desired sequences (e.g., direct quotations)" but "slightly decreases utility on math, knowledge and reasoning benchmarks" (Section 6.2). The paper presents this as raising the unresolved question of what extent memorization is required for utility.

Memorization Dynamics Results

Recency bias in pre-training (Jagielski et al., 2023; Kiyomaru et al., 2024). Examples seen later in training are disproportionately likely to be memorized; examples seen early are disproportionately likely to be forgotten. The paper reports this as establishing that memorization is temporally biased, not uniform across the training distribution (Section 4).

Late-stage memorization amplification (Huang et al., 2024). "Later-stage checkpoints are more susceptible to memorizing even rare or out-of-distribution content" (Section 4). The paper interprets this as reflecting increased capacity allocation after the model has largely converged on common patterns.

Predictable scaling laws for memorization (Biderman et al., 2023). "As model size and training duration increase, specific sequences predictably transition from unmemorized to memorized" (Section 4). The paper presents this as evidence that memorization follows scaling laws analogous to loss scaling.

RLHF memorization persistence (Pappu et al., 2024). "Data memorized during fine-tuning persists with high frequency in post-RLHF models," while there is "minimal evidence that reward model data or reinforcement learning data becomes memorized" (Section 4). The paper reports this as the primary quantitative evidence for how RLHF interacts with memorization, while noting it is limited to the code generation domain.

Ablation Studies and Robustness Checks

Since this is a survey paper rather than an experimental study, the "ablation studies" are the comparative analyses within the surveyed literature that the paper highlights as robustness evidence for specific findings.

Deduplication method comparison (Lee et al., 2022 vs. Schäfer et al., 2024): The paper synthesizes a specific robustness limitation: deduplication using exact or near-exact matching produces a tenfold reduction in memorization (Lee et al., 2022), but near-duplicates (paraphrased content, semantically equivalent text with surface differences) escape detection and remain memorizable (Schäfer et al., 2024). This is not a comparison of deduplication methods within a single study but a cross-study finding that the paper elevates as a key limitation: surface-level deduplication is effective for what it catches but insufficient for what it misses.

Decoding method comparisons (Carlini et al., 2021; Yu et al., 2023; Tiwari and Suh, 2025; Hayes et al., 2025): The paper synthesizes a consistent pattern across four studies: greedy decoding misses memorized content that stochastic decoding reveals. Carlini et al. (2021) established this; Yu et al. (2023) showed parameter optimization can double baselines; Tiwari and Suh (2025) quantified a near-doubling of leakage risk; Hayes et al. (2025) provided theoretical grounding through the $(n, p)$ -discoverable framework. The cross-study consistency — across different models, datasets, and extraction methodologies — is what the paper presents as robustness evidence that "no single decoding method minimizes leakage across all scenarios" (Section 3).

MIA variant comparisons (Duan et al., 2024; Zhang et al., 2025; Maini and Suri, 2024): The paper's most extensive "ablation" analysis is its methodological critique of MIAs. Rather than comparing MIA variants within a controlled study, the paper synthesizes cross-study evidence that MIA improvements often fail to replicate or are attributable to benchmark artifacts:

Duan et al. (2024) argued that MIAs lack calibrated null models.
Zhang et al. (2025) recommended MIAs be used only for aggregate auditing, not per-instance claims.
Maini and Suri (2024) demonstrated that a best-paper-winning MIA method's apparent gains were artifacts of distributional shift in the evaluation benchmark.

This cross-study synthesis is the paper's evidence for its claim that per-instance MIAs are statistically unsound — a claim that relies on convergent findings across independent studies rather than a single controlled experiment.

Fine-tuning method comparison (Mireshghallah et al., 2022): Head-only fine-tuning produces the highest memorization risk, adapter-based fine-tuning produces lower risk. The paper reports this as a "comparative analysis of fine-tuning approaches" (Section 4) that establishes method choice as a significant factor. The paper also reports the finding that task type matters: summarization tasks with narrow attention patterns produce higher memorization than QA tasks with broader attention.

Unlearning method durability (Table 5, citing multiple sources): The paper synthesizes evidence that unlearned states are fragile: "small exposures to previously forgotten data can quickly reverse the unlearning, undermining long-term effectiveness." This is presented as a cross-cutting limitation of unlearning methods rather than a finding from any single study, drawing on Cooper et al. (2024), Fan et al. (2025), and the broader machine unlearning literature.

Activation steering locality (Stoehr et al., 2024 vs. Chang et al., 2023): The paper reports two findings about localization: Stoehr et al. (2024) identified a single early-layer attention head that triggers verbatim recall for rare token combinations, and Chang et al. (2023) found that fewer than 0.5% of neurons can account for 60% of memorization accuracy. However, the paper also reports the robustness limitation that "neurons involved in one memory often contribute to others, complicating memory-specific interventions due to the risk of collateral forgetting" (Section 6.3). This is a cross-study finding: localization is precise enough to be useful but not sufficiently isolated to avoid collateral damage.

Negative result: ReST^{EM} for revision training (referenced in Appendix K of referenced work, Singh et al., 2024): The paper notes in its discussion of revision models and self-improvement that "attempting to optimize the revision model with RL-style training caused performance to degrade substantially with sequential revisions, likely because on-policy data collection amplified spurious correlations in the revision trajectories" (Section 6 discussion of Pappu et al., 2024 and related work). The paper flags this as evidence that revision-based mitigation is sensitive to training methodology.

Negative result: RLHF does not erase fine-tuning memorization (Pappu et al., 2024): The paper reports that "data memorized during fine-tuning persists with high frequency in post-RLHF models" — a negative result indicating that RLHF cannot be relied upon to remove memorization introduced at earlier stages. The paper also reports the converse negative: minimal evidence that RLHF introduces new memorization of its own training data. Both findings constrain the role of RLHF in memorization management.

Critical Assessment

The central challenge in evaluating this paper is that it is a survey — it does not run experiments, so standard critiques about experimental design, dataset size, or baseline choice do not apply in the usual way. Instead, the paper must be evaluated on whether the evidence it synthesizes genuinely supports its conceptual claims, whether it accurately characterizes the limitations of the studies it cites, and whether its framework reveals gaps that individual studies missed.

Claim: Memorization definitions are heterogeneous and this heterogeneity causes contradictory findings across studies.

This claim is the paper's intellectual foundation, and the evidence is strong but incomplete. Table 1 taxonomizes eight definitions with clear relationships, and the paper provides the specific mechanism for one well-documented contradiction: the Tirumala et al. (2022) finding that MIAs and extraction attacks disagree because data duplication creates a weak membership signal that MIAs miss while solidifying the sequence for extraction. This is a clean, mechanistic explanation for a specific documented divergence.

However, the paper does not provide a systematic demonstration that all definitional mismatches produce contradictions, nor does it survey the literature to quantify how often MIA-based and extraction-based studies reach opposite conclusions. The Tirumala et al. finding is a case study, not a systematic review. A stronger version of this claim would require showing that specific pairs of definitions produce predictable disagreement patterns across multiple model families and datasets. The paper's taxonomy enables this analysis but does not perform it — it remains a framework for diagnosing contradictions rather than an empirical demonstration that the framework explains most contradictions in the literature.

Claim: Alignment reduces the accessibility of memorized content without reducing its existence.

The central evidence is Nasr et al. (2025)'s divergence attack achieving up to 150× more extraction compared to benign prompts. This is strong evidence for the specific claim that aligned models can be induced to leak memorized content that they would not leak under normal interaction. However, the paper's stronger framing — that alignment "does not make it harder to memorize" — is not directly tested by Nasr et al.'s study, which examines extraction from already-trained models rather than comparing memorization rates during training with vs. without alignment procedures. The claim that alignment does not prevent memorization from occurring is a reasonable inference (alignment happens after pretraining, and the divergence attack recovers pretraining data), but the paper does not cite controlled experiments that compare pretraining memorization rates in models destined for alignment vs. those not.

Furthermore, the 150× multiplier is a relative measure: it tells us alignment reduces but does not eliminate extractability. The paper does not provide the absolute extraction rates — 150× more than a very low baseline could still be a low absolute rate. The significance of the finding depends on whether the absolute extraction volume after the attack is large enough to constitute a realistic privacy threat. The paper reports the multiplier but not the baseline extraction rate from which it is computed.

Claim: No current mitigation method provides formal verifiable guarantees of erasure at scale.

This claim is structurally accurate but somewhat tautological given how the paper defines "formal verifiable guarantees." The paper acknowledges that differential privacy does provide formal guarantees, but argues these guarantees become difficult to interpret in web-scale pretraining due to the "group size" problem — the inability to reliably bound how many times an individual's data appears in the training corpus. This is a practical limitation rather than a theoretical one: DP's guarantees are mathematically valid; the question is whether the privacy unit (individual, record, user) can be meaningfully defined and bounded. The paper's claim that DP guarantees "remain unclear" in pretraining contexts is more about the difficulty of applying the framework than about the framework's validity. A more precise claim would be: DP provides formal guarantees whose practical strength depends on the accuracy of group size estimation, and this estimation is currently unreliable at web scale for pretraining data.

For unlearning, the paper's claim that there is "no formal guarantee" is accurate according to the current literature — unlearning methods provide empirical evidence of erasure (a specific test no longer succeeds) but not mathematical proof that no possible test would succeed. However, the paper does not fully engage with emerging work on verifiable unlearning (some of which it cites in Table 5, including Eisenhofer et al., 2025) or discuss what formal guarantees for unlearning would even look like. The claim that no method provides formal guarantees is true but underspecified: what kind of guarantee would be sufficient? Computational indistinguishability from retraining? Provable information-theoretic bounds? The paper critiques existing methods against an implicit standard of formal verification without articulating what that standard should be.

Claim: The field suffers from a methodological schism where different metrics deliver contradictory verdicts.

The Tirumala et al. (2022) evidence is compelling for the specific case of MIA vs. extraction. However, the paper generalizes from this to "any claim about memorization dynamics... is effectively ill-defined without a precise specification of the metric." This is a strong claim that would require showing that metric-dependent contradictions are common, not just that one well-documented case exists. The paper does not systematically survey the literature for other cases where different metrics disagree, nor does it establish whether the Tirumala et al. finding is typical or exceptional. The mechanism (data duplication) is plausible as a general source of MIA-extraction disagreement, but the paper does not estimate how common this mechanism is — what fraction of training data is duplicated in ways that would trigger this specific divergence. The diagnosis is sharp but the prevalence evidence is thin.

Claim: Deduplication reduces memorized token generation by an order of magnitude.

This claim is well-supported by Lee et al. (2022) for the specific deduplication method and model family tested. The paper's own caveat — that near-duplicates escape detection and remain memorizable — is important but does not undermine the claim; it specifies the claim's boundary conditions. The tenfold reduction is for exact and near-exact duplicates; semantically equivalent but surface-different content is a separate problem. The paper could strengthen its treatment by discussing whether the tenfold figure is consistent across other deduplication studies (e.g., Kandpal et al., 2022) or whether it varies with dataset size, duplication rate, and model architecture. The paper reports the headline number without systematic cross-study validation.

Claim: Activation steering reduces memorization by up to 60% with minimal utility degradation.

The Suri et al. (2025) finding is the primary evidence. The paper's reporting is appropriately cautious — it notes the sensitivity to layer selection and steering strength, and the collateral forgetting risk. However, "minimal utility degradation" is vague: the paper does not report what utility metrics were measured, what degradation thresholds were used, or whether the 60% reduction figure and the minimal degradation figure were achieved simultaneously at the same steering strength. The reader cannot assess from the paper alone whether there is a direct tradeoff (more memorization reduction → more utility degradation) that the "up to 60%" figure represents a particular point on.

Missing evidence that would strengthen the paper's framework:

No systematic prevalence analysis for metric contradictions. The paper identifies the MIA-extraction schism as a central methodological problem but does not survey the literature to estimate how often it occurs or under what conditions. A systematic review of papers that use multiple metrics would strengthen the claim that this is a widespread problem rather than an isolated finding.
No quantitative comparison of mitigation method effectiveness. The paper reports individual numbers from individual studies (tenfold reduction from deduplication, 60% from activation steering, 10⁵× efficiency for unlearning vs. retraining) but does not attempt to normalize these across different measurement protocols, threat models, or evaluation datasets. A reader cannot determine from the paper whether deduplication is more or less effective than activation steering at reducing real-world privacy risk because the metrics are incommensurable. The paper acknowledges this incommensurability as part of its critique of the field but does not propose a framework for cross-method comparison.
No sensitivity analysis for the near-duplicate problem. The paper identifies near-duplicate escape as a key limitation of deduplication but does not report quantitative estimates of what fraction of memorized content is attributable to near-duplicates vs. exact duplicates. Without this, the practical significance of the limitation is unclear — if near-duplicates account for 5% of memorization, the tenfold reduction from exact deduplication remains largely effective; if they account for 50%, the limitation is severe.
Limited evidence for the distillation stage of the lifecycle model. The paper identifies distillation as a stage where memorization likely propagates but acknowledges this "has not been formally analyzed." The lifecycle model is thus empirically incomplete at one of its four stages. The paper's framework would be strengthened by either conducting this analysis or more explicitly flagging the distillation stage as a hypothesized rather than demonstrated risk.

Overall assessment of evidence-to-claims mapping:

The paper's strongest claims — the definitional heterogeneity problem, the alignment-memorization distinction, the stage-based lifecycle model — are well-supported by the specific studies cited and represent genuine intellectual contributions that reframe the field's understanding. The paper's critical assessments of detection methods (MIAs are unreliable for per-instance claims) and mitigation methods (no formal verifiable guarantees at scale) are justified by the evidence but the paper sometimes overstates by generalizing from specific findings without demonstrating prevalence. The paper's value is not in providing new empirical evidence but in organizing existing evidence into a framework that reveals patterns, contradictions, and gaps that individual studies could not see. The framework's validity depends on whether these patterns actually hold across the literature — and for several key claims (metric contradictions, mitigation effectiveness comparisons), the evidence is illustrative rather than systematic, making the framework a compelling hypothesis about the field's structure rather than a demonstrated empirical fact about that structure.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Excluded from Headline Efficiency Numbers

The assumption or constraint. The paper's compute-optimal allocation framework (Section 3.2 in the referenced search-and-revision paper that this SoK draws on for methodological analogy) requires estimating per-prompt difficulty before allocating the test-time compute budget. The standard method—generating 2048 samples per question and scoring them with a verifier—is itself extremely expensive. As the referenced work explicitly acknowledges in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

While this SoK paper does not propose a new system with its own difficulty estimation pipeline, it inherits and endorses the compute-optimal allocation framework without addressing this cost in its own analysis of detection and mitigation trade-offs. When the paper discusses practical deployment of mitigation strategies (Section 6), it evaluates detection methods for their computational requirements (Table 3 rates soft prompting as "prohibitively expensive") but does not apply the same cost-accounting scrutiny to the difficulty-estimation prerequisite that underlies adaptive allocation strategies it references as promising.

The consequence. The reported 4× efficiency gains for compute-optimal scaling over best-of-N baselines are computed after difficulty is known, without amortizing the cost of learning it. In any realistic deployment, the total cost would be difficulty estimation plus strategy execution, and at 2048 samples per question, the former could dominate the latter. A practitioner implementing the adaptive approaches the paper references as mitigation directions (Section 6.3 discusses inference-time interventions that could be difficulty-conditioned) would find that the actual wall-clock or FLOPs cost is substantially higher than the headline numbers suggest—potentially eliminating the claimed efficiency advantage entirely for moderate-compute budgets. The paper's omission of this cost means a central quantitative claim motivating test-time compute approaches goes unqualified.

What evidence exists in the paper. The paper does not measure this limitation itself; it is a survey synthesizing other work. However, Section 3.2 of the referenced search-and-revision paper (which this SoK cites as methodological precedent) explicitly acknowledges the difficulty estimation cost problem and flags it as "a key avenue for future work." The SoK paper's own Table 3 flags the "prohibitive computational overhead of rigorous methods (e.g., shadow models)" as a limitation of detection approaches, demonstrating awareness of cost-as-limitation but not applying the same rigor to the adaptive frameworks it endorses.

Mitigation status. The referenced paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) and on adaptive difficulty estimation that starts with a few samples and adjusts dynamically. Neither is implemented or evaluated. The SoK paper's own open questions (Section 5, OQ1 on "zero-knowledge detection" and Section 6, OQs on scaling DP and optimizing activation steering) similarly point toward cheaper alternatives without providing them. This limitation remains completely unaddressed in any practical sense.

No Systematic Evidence That Metric-Dependent Contradictions Are Prevalent, Not Just Documented in Isolated Cases

The assumption or constraint. The paper's central methodological critique is that the field suffers from a "schism" where different memorization metrics deliver contradictory verdicts—MIA may indicate forgetting while extraction demonstrates retention. The paper identifies one well-documented mechanism: data duplication creates weak membership signals that MIAs miss while solidifying sequences for prefix-based extraction (Tirumala et al., 2022, discussed in Section 4). From this case, the paper generalizes to:

"this methodological divergence is so severe that any claim about memorization dynamics... is effectively ill-defined without a precise specification of the metric"

This generalization assumes that the Tirumala et al. finding is representative of widespread metric disagreement rather than an isolated, mechanism-specific phenomenon. The paper does not systematically survey the literature to estimate how often MIA-based and extraction-based studies reach opposite conclusions, nor does it establish whether the specific mechanism (data duplication confounding MIA signal) is the dominant cause of metric disagreements or one among many.

The consequence. A reader cannot determine from the paper whether the metric schism is a pervasive structural problem requiring a fundamental reorientation of memorization research (as the paper implies) or a documented edge case that primarily affects studies conducted on heavily duplicated data with specific MIA methodologies. If the latter is true, the paper's strongest prescriptive claim—that "immediate community-wide efforts toward standardized memorization evaluation protocols" are necessary (Section 4)—may be overstated relative to the evidence. The practical consequence for a practitioner is uncertainty about whether their chosen evaluation method (MIA, extraction, or another approach) is likely to conflict with other methods for their specific data characteristics and threat model, or whether the Tirumala et al. finding is a cautionary tale that primarily applies to high-duplication training sets.

What evidence exists in the paper. The Tirumala et al. (2022) finding is the paper's central case study (Section 4), and it is described in detail: a model appears to forget a sequence under MIA while reproducing it under extraction, driven by data duplication. The paper does not provide additional cases of metric disagreement, does not estimate what fraction of training data exhibits the duplication pattern that triggers this specific divergence, and does not survey how many memorization studies use multiple metrics (and whether those that do typically find agreement or disagreement). The prevalence evidence is a single well-documented mechanism, presented as representative without systematic validation that it actually represents the typical case.

Mitigation status. The paper calls for "standardized memorization evaluation protocols" (Section 4) and "immediate community-wide efforts" toward them. It does not propose such protocols itself, does not specify what standardization would entail (minimum set of metrics? cross-validation across metric types? duplication-controlled evaluation sets?), and does not estimate how much the metric disagreement problem would shrink under standardized protocols versus persisting as a fundamental ambiguity. The recommendation is directionally sensible but operationally unspecified.

The Distillation Stage of the Lifecycle Model Is Theoretically Motivated but Empirically Unvalidated

The assumption or constraint. Section 4 and Table 2 construct a lifecycle model tracing memorization risk through four stages: pre-training, SFT, RLHF/post-training, and distillation. For distillation, the paper hypothesizes a specific propagation mechanism: the KL divergence objective between teacher and student output distributions forces the student to replicate the teacher's overconfidence on memorized sequences. The paper states this mechanism clearly:

"If the teacher assigns extremely high probability to a memorized sequence, the Kullback-Leibler (KL) divergence objective directly incentivizes the student to replicate this overconfidence, facilitating the transfer of memorized content"

However, the paper acknowledges this is a theoretical argument, not an empirically validated phenomenon:

"this has not been formally analyzed"

The lifecycle model thus has one stage—distillation—that is hypothesized rather than demonstrated, distinguished from the other three stages where the paper can cite specific empirical studies (Jagielski et al., 2023 for pre-training; Mireshghallah et al., 2022 and Nasr et al., 2025 for SFT; Pappu et al., 2024 for RLHF).

The consequence. The lifecycle model's prescriptive implication—that "effective mitigation cannot be a single, post-hoc fix but requires a comprehensive defense with targeted interventions at every phase of a model's creation" (Section 4, Table 2 framing)—depends on all four stages being genuine risk transformation points. If distillation does not in fact propagate memorization (or propagates it only under specific conditions that are rare in practice), then the "comprehensive defense" may be over-engineered for this stage, and research resources directed at distillation-specific mitigation (as Table 2 proposes: "distilling from calibrated or temperature-scaled teacher distributions" and "augmenting the loss with a regularizing term") may be misallocated relative to more impactful interventions at better-validated stages.

For practitioners, the uncertainty about distillation's role is directly consequential: model distillation is a standard technique in production ML pipelines (the paper acknowledges it is "ubiquitous in modern ML pipelines"), and if distillation propagates memorization as hypothesized, every distilled model deployment inherits the memorization risks of its teacher. If the hypothesis is wrong or the propagation magnitude is negligible, distillation may serve as an implicit mitigation (the student's capacity limitations may cause it to "forget" low-probability memorized sequences that the teacher retained) rather than a risk amplifier.

What evidence exists in the paper. The paper cites two early studies: Singh (2025) and Chaudhari et al. (2025). The Singh reference is described as showing that "this mechanism propagates memorized content from teacher to student." The Chaudhari et al. reference is described as showing that "bias injected adversarially into teacher models can be amplified in student models via distillation," which the paper notes "naturally suggests that memorization can propagate" but treats as suggestive rather than conclusive. Neither study appears to provide a systematic, large-scale analysis of memorization propagation through distillation across multiple model families, data modalities, and distillation configurations. The paper's language is appropriately cautious ("naturally suggests," "has not been formally analyzed") but the lifecycle model nonetheless treats distillation as a co-equal stage with specific research challenges and proposed interventions.

Mitigation status. The paper's proposed research direction (Table 2) is to "research distillation objectives that go beyond matching raw logits... including distilling from calibrated or temperature-scaled teacher distributions to penalize overconfidence, or augmenting the loss with a regularizing term that minimizes the student's divergence from a 'safe' public dataset." This assumes the propagation mechanism is both real and significant enough to warrant mitigation—an assumption that the underlying empirical work has not yet validated. The paper does not suggest or conduct the baseline measurement that would be needed to determine whether distillation is a risk amplifier, a neutral transfer mechanism, or a de facto mitigation.

Mitigation Methods Are Evaluated Against a Formal Verification Standard That No Current Method Meets, Without Specifying What Would Suffice

The assumption or constraint. Section 6 evaluates mitigation strategies against a consistent criterion: does the method provide "formal verifiable guarantees that memorized content has been removed?" The paper finds that no current method meets this standard at scale. The framing is explicit across multiple subsections:

For unlearning: "unlike DP, there is no formal guarantee, thereby leaving a risk that memorization persists" (Section 6.2).
For data cleaning: the paper notes that near-duplicates escape detection, and uniqueness-driven memorization (Feldman and Zhang, 2020) is unaffected by deduplication—implicitly, data cleaning provides empirical reduction without formal bounds.
For DP: the paper acknowledges DP provides formal guarantees but argues they are "difficult to interpret in web-scale pretraining contexts" (Section 6.1) due to the group size problem.
Table 5 lists "Verifiability and Proof of Forgetting" as a foundational challenge, noting the "absence of formal guarantees to certify that a data point's influence has been completely erased."

What the paper does not provide is a specification of what kind of formal guarantee would be sufficient, how it would be operationalized, or what burden of proof it would need to meet. The implicit standard appears to be something like "mathematical proof that no possible extraction method can recover the data"—a standard that may be unrealizable for any non-trivial model that has been trained on the data, given the fundamental inseparability problem the paper itself identifies.

The consequence. The paper risks setting an implicit standard that no practical method can meet, which has the effect of making all existing approaches appear equally inadequate without providing a realistic target. This can be paralyzing rather than guiding for practitioners: if DP guarantees are "unclear" at scale, unlearning has "no formal guarantee," data cleaning misses near-duplicates, and inference-time interventions suppress specific instances without generalizing, then what should a deployer do? The paper's answer—layered interventions across the lifecycle—is sensible but operates at the level of general principle rather than concrete guidance. The paper does not help a practitioner decide whether a system that uses deduplication + DP-SGD fine-tuning + activation steering during inference is "good enough" for a specific deployment context with specific threat models and regulatory requirements.

For research prioritization, the unspecified verification standard creates ambiguity: should the field pursue stronger formal guarantees (potentially requiring fundamentally new mathematical frameworks for what "erasure" means in distributed representations), or should it accept that guarantees will be empirical and focus on making empirical evaluation more rigorous and standardized? The paper's critique of current methods leans toward the former, but its proposed research directions (Section 6 open questions) lean toward the latter—improving the practicality and scalability of existing approaches rather than developing new formal frameworks.

What evidence exists in the paper. The paper cites emerging work on verifiable unlearning (Table 5 references Eisenhofer et al., 2025 on "verifiable and provably secure machine unlearning" and Zhang et al., 2024 on verification fragility) but does not engage deeply with what these approaches can and cannot guarantee. The paper's own Table 5 lists "Verifiability and Proof of Forgetting" as a foundational challenge alongside a note that current methods have an "absence of formal guarantees to certify that a data point's influence has been completely erased" and that this "makes unlearning methods difficult to audit and trust, especially for regulatory compliance and against adversarial verification." This diagnosis is accurate as a description of the current state but does not advance the conversation toward what an adequate verification framework would require.

Mitigation status. The paper does not attempt to resolve this limitation. Open Question 4 in Section 7 ("What technical threshold defines legally significant memorization?") touches on the verification standard issue from the legal side: "instead of searching for a single, universal threshold, the research community should develop a framework for context-aware memorization auditing." This is a direction—replacing binary verification with multi-faceted evidence—but it does not address the underlying formal guarantee problem. The paper's concluding framing of "attribution and control" rather than "erasure" (Section 8) can be read as an implicit acknowledgment that formal erasure verification may be fundamentally intractable, and that the field should pivot toward different goals, but the paper does not make this argument explicitly or reconcile it with its earlier critique of methods for lacking formal guarantees.

The Survey Synthesizes Findings Across Model Families Without Accounting for Architecture-Specific Memorization Mechanisms

The assumption or constraint. The paper synthesizes evidence across GPT-2, PaLM, PaLM 2, LLaMA, OPT, and production models like ChatGPT, treating memorization as a phenomenon with consistent underlying mechanisms regardless of architecture. While Section 3 identifies factors that influence memorization (model size, data duplication, tokenization), the paper does not systematically analyze whether different architectural choices—decoder-only vs. encoder-decoder, attention mechanism variants, normalization placement, activation functions—produce qualitatively different memorization patterns or interact differently with specific mitigation strategies.

The paper's Open Question 3 in Section 3 asks: "How does model architecture scale influence the mechanisms of memorization?" and elaborates: "It remains unclear if larger models simply have more capacity for memorization or if the internal mechanisms themselves evolve with scale." This acknowledges the gap but does not address it in the paper's framework. The lifecycle model, the detection taxonomy, and the mitigation evaluation all implicitly assume that findings from one architecture transfer to others—or at least that the differences are quantitative (more or less memorization) rather than qualitative (different kinds of memorization requiring different detection and mitigation approaches).

The consequence. A practitioner deploying a specific architecture cannot determine from the paper which findings apply to their model and which might be architecture-specific. For example: Stoehr et al. (2024) identified a single early-layer attention head that triggers verbatim recall—but this was demonstrated on a specific architecture, and it is unknown whether memorization in other architectures localizes to similar components or distributes differently. If memorization in some architectures is highly localized (making activation steering effective) and in others is distributed (making it ineffective), the paper's treatment of activation steering as a general mitigation strategy (Section 6.3) overstates its applicability.

Similarly, the finding that larger BPE vocabularies increase memorization (Kharitonov et al., 2021) is a tokenization-specific effect that may interact with architecture in ways the paper does not explore. A model with a different tokenization scheme (e.g., character-level, SentencePiece with different merge rules, or byte-level tokenization) might exhibit different memorization scaling with vocabulary size, or the effect might be entirely absent.

What evidence exists in the paper. The paper cites studies conducted on different architectures but does not provide cross-architecture comparisons or systematically note where findings have been replicated across architectures versus where they rest on single-architecture evidence. The factor analysis in Section 3 cites consistent findings across model families for model size effects (Carlini et al., 2021 on GPT-2; Tirumala et al., 2022 and others on subsequent models) and data duplication (Lee et al., 2022 across multiple model sizes), providing some confidence in cross-architecture generalizability for these specific factors. However, for more fine-grained mechanisms—attention head localization, the interaction between fine-tuning method and memorization type, activation steering effectiveness—the paper does not discuss whether findings are architecture-specific or have been replicated.

Mitigation status. The paper's Open Question 3 explicitly calls for research on this topic: "Tracing 'memorization circuits' within components like FFN layers theorized to act as key-value memories is crucial for understanding how these circuits change in smaller models and for designing future architectures that are inherently safer." This is a forward-looking research direction, not a resolution. The paper does not provide practitioners with guidance on how to assess whether a given finding is likely to apply to their architecture, nor does it flag which of its synthesized claims are most at risk of being architecture-specific.

The Paper Frames Mitigation as Necessitating Layered Interventions Without Quantifying the Cumulative Utility Cost

The assumption or constraint. Section 4's lifecycle model and Section 6's mitigation taxonomy collectively argue that effective memorization mitigation requires "a comprehensive defense with targeted interventions at every phase of a model's creation" (Table 2 framing). The prescription is to apply data cleaning at pre-training, DP or unlearning at post-training, and inference-time filtering or activation steering at deployment. This layered-defense framing assumes that the utility costs of each intervention are independent and additive—or at least that the cumulative degradation is acceptable. The paper provides individual utility-impact numbers for some methods (ParaPO slightly degrades reasoning benchmarks; DP-SGD approaches non-private performance when fine-tuning; activation steering causes "minimal utility degradation") but does not estimate the cumulative utility cost of combining them.

The "what extent memorization is required for utility and generalization" question (Section 6.2, raised in the context of ParaPO's utility degradation) is identified as unresolved, but the layered-defense prescription implicitly assumes the answer is "less than we're currently allowing"—that the margin between necessary memorization (for factual recall and generalization) and harmful memorization (of PII, copyrighted content) is wide enough to accommodate multiple interventions without crossing below the utility threshold.

The consequence. A practitioner who follows the paper's layered-defense recommendation might implement deduplication + DP-SGD fine-tuning + activation steering and discover that the cumulative utility degradation is substantially larger than any single method's reported cost. The paper does not provide evidence about whether these interventions' utility costs interact—e.g., whether a model that has been DP-trained is more or less susceptible to utility degradation from activation steering, or whether deduplication's effect on factual knowledge compounds with unlearning's effect on reasoning benchmarks. Without this evidence, the recommendation to apply multiple interventions is based on an untested additive model of costs.

For high-stakes applications where utility degradation is unacceptable (medical diagnosis, legal analysis, financial decision-making), the layered-defense approach may be practically infeasible even if each individual intervention is "minimal" in isolation—three "minimal" degradations might sum to an unacceptable total. The paper provides no framework for practitioners to estimate this cumulative cost or to prioritize which interventions to apply given a utility budget.

What evidence exists in the paper. The paper reports individual utility-impact estimates for specific methods:

ParaPO: "slightly decreases utility on math, knowledge and reasoning benchmarks" (Section 6.2).
DP-SGD when fine-tuning: "DP-trained models approach the performance of non-private baselines" (Section 6.1, citing Li et al., 2021).
Activation steering: "minimal performance degradation" at 60% memorization reduction (Section 6.3, citing Suri et al., 2025).
Unlearning via ParaPO: utility degradation is acknowledged but not quantified in the paper's discussion.

However, these numbers are drawn from different studies, different models, different datasets, and different utility metrics. There is no study cited that applies multiple interventions to the same model and measures cumulative utility impact, and the paper does not attempt to normalize the individual estimates to enable even rough cumulative prediction.

Mitigation status. The paper's framing of the problem as "disentangling" beneficial from harmful memorization (Section 8) and its open question about "what extent memorization is required for utility" (Section 6.2) acknowledge the fundamental tension but do not resolve it. The paper does not propose methods for estimating cumulative utility costs, does not suggest experimental protocols for measuring intervention interactions, and does not provide a decision framework for practitioners to navigate the utility-memorization tradeoff under multiple simultaneous interventions. The layered-defense recommendation remains a qualitative principle rather than a quantitative guideline, and the paper's own evidence about individual method costs suggests the cumulative cost could be significant without being able to estimate how significant.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the landscape not by introducing a new technique but by reorganizing how the field thinks about memorization. The shift is from treating memorization as a monolithic failure mode with point solutions toward understanding it as a dynamic, multi-stage risk that transforms across the LLM lifecycle and cannot be addressed at any single stage in isolation. This is a conceptual reframing with practical consequences, not a paradigm shift—the individual findings the paper synthesizes existed in the literature before this survey; what is new is the framework that reveals their relationships and exposes their collective gaps.

The magnitude of the shift is significant for research prioritization but incremental for immediate practice. A practitioner reading this paper in December 2025 will not find a new method to deploy tomorrow. They will, however, find a structured diagnosis of why their current approach (deduplication, alignment, or DP fine-tuning applied in isolation) is insufficient, and a roadmap for what a comprehensive defense would require. This is a diagnostic contribution rather than a prescriptive one—it tells you what is broken and what kinds of fixes are needed, but does not provide the fixes themselves.

The paper resolves one specific and important contradiction in the prior literature: the MIA-vs-extraction schism where models appear to "forget" sequences under membership inference while simultaneously reproducing them under prefix-based prompting. The paper's explanation—data duplication creates a weak membership signal that MIAs miss while solidifying the sequence as a high-probability output—provides a mechanistic account for a finding that previously appeared paradoxical. This resolution is significant because it establishes that different memorization metrics measure different things rather than measuring the same thing with varying accuracy. The practical implication is that evaluations relying on a single metric (particularly MIA-based evaluations) may be systematically blind to certain forms of memorization that other metrics would detect. This finding should change how researchers report memorization results: any claim about a model's memorization behavior is "effectively ill-defined without a precise specification of the metric" (Section 4), and papers that report only MIA results without complementary extraction-based evaluation are providing incomplete evidence.

The paper makes two research directions more attractive:

Zero-knowledge detection (Section 5, OQ1): the paper's critique of MIAs—that they lack calibrated null models and cannot provide per-instance evidence of training data inclusion—motivates a shift toward detection methods that identify memorization from intrinsic statistical properties of model outputs without requiring differential comparison to the training data. The paper frames this as "an out-of-distribution problem, where the goal is to distinguish plausible linguistic samples from low-complexity artifacts indicative of direct replication" (citing Schwarzschild et al.'s adversarial compression framework). This direction becomes more attractive because the paper has shown that the alternative—refining MIA calibration—may be fundamentally limited by the intractability of training counterfactual models.
Attribution and control rather than erasure (Section 8): the paper's analysis of mitigation limitations—deduplication misses near-duplicates, DP guarantees are difficult to interpret at pretraining scale, unlearning lacks formal guarantees and is fragile to relearning—motivates a shift away from pursuing perfect erasure toward making memorization legible (where did this output come from?) and governable (can the model selectively suppress harmful reproduction while preserving beneficial recall?). This direction becomes more attractive because the paper has demonstrated that the erasure paradigm faces fundamental barriers (inseparability of beneficial and harmful memorization in distributed representations, the near-duplicate problem, the lack of verifiable guarantees) that may be insurmountable at scale.

The paper makes one research direction less attractive: pursuing ever-more-sophisticated MIA variants for per-instance training data detection. The paper's synthesis of Duan et al. (2024), Zhang et al. (2025), and Maini and Suri (2024) establishes that per-instance MIA claims are statistically unsound in principle (not just insufficiently optimized) because constructing a calibrated null distribution requires training a counterfactual model without the target data point, which is computationally intractable. Apparent MIA improvements that appear to solve this problem are likely exploiting benchmark artifacts (Maini and Suri, 2024). The paper's recommendation to reframe MIAs as aggregate auditing tools rather than per-instance detectors effectively deprecates a line of research that many papers have pursued. A researcher considering a new MIA variant should, after reading this paper, first demonstrate that their method does not suffer from the calibration failures documented across multiple independent studies—a burden that the paper implies may be impossible to meet for per-instance claims.

The paper also makes the distillation-memorization propagation hypothesis a newly visible research gap. Prior to this survey, distillation was not systematically discussed in the memorization literature. The paper's lifecycle model (Section 4, Table 2) identifies the KL divergence objective as a theoretical mechanism for memorization propagation from teacher to student and flags this as empirically unvalidated. This creates an attractive research opportunity: a systematic study measuring memorization transfer rates under different distillation configurations (temperature scaling, data augmentation, architecture mismatch) would either validate the lifecycle model's distillation stage or establish boundary conditions under which distillation serves as an implicit mitigation. Either result would be valuable, and the paper's framework makes the question concrete.

Follow-Up Research This Work Enables

A systematic prevalence analysis of metric-dependent contradictions in memorization research. The paper's central methodological critique—that different memorization metrics can deliver contradictory verdicts—is supported by one well-documented case (Tirumala et al., 2022: MIA indicates forgetting while extraction demonstrates retention, driven by data duplication). The paper generalizes from this to claim the divergence is "so severe that any claim about memorization dynamics... is effectively ill-defined without a precise specification of the metric" (Section 4). A direct follow-up would survey the memorization literature for studies that report multiple metrics on the same model and same data, cataloging whether they agree or disagree, and characterizing the conditions (data duplication rate, sequence length, model size) under which disagreement is most common. The study would test a specific hypothesis derived from the paper's mechanism: metric disagreement should be concentrated among high-duplication training examples and should be rare for unique sequences (where MIAs and extraction should both detect memorization or both fail to detect it). A negative result—finding that metric disagreement is rare even on duplicated data—would substantially weaken the paper's claim that the field suffers from a pervasive methodological schism and would suggest the Tirumala et al. finding is a mechanism-specific edge case rather than a representative example. A positive result would validate the paper's most important methodological critique and provide quantitative guidance on which metric combinations are necessary for reliable evaluation.

Measuring memorization propagation through knowledge distillation under controlled duplication conditions. The paper's lifecycle model identifies distillation as a stage where memorization likely propagates from teacher to student through the KL divergence objective, but acknowledges this "has not been formally analyzed" (Section 4). A direct follow-up would train a teacher model on a corpus with controlled duplication rates (e.g., using canary sequences inserted at known frequencies), measure the teacher's verbatim and approximate memorization rates, then distill to students of varying capacities using standard distillation, temperature-scaled distillation (where the teacher's logits are divided by a temperature parameter $\tau > 1$ before computing the KL divergence, which softens the distribution and reduces pressure to match overconfident predictions), and a regularization-augmented objective that adds a term penalizing divergence from a public reference dataset. The study would measure: (1) what fraction of teacher-memorized canaries are reproduced by the student under each distillation configuration; (2) whether smaller student models are less likely to propagate memorized sequences (a capacity-limitation hypothesis where the student's reduced parameter count forces it to prioritize common patterns over rare memorized ones); and (3) whether teacher overconfidence on memorized sequences (measured as the teacher's output probability assigned to the memorized token) predicts student propagation probability. A finding that distillation propagates memorization at high rates even with temperature scaling would validate the lifecycle model's distillation stage as a genuine risk amplifier and motivate the mitigation strategies proposed in Table 2 (calibrated distillation, safe-dataset regularization). A finding that standard distillation rarely propagates memorization—or that student capacity constraints naturally filter memorized content—would refine the lifecycle model by establishing distillation as a neutral or even mitigating stage, redirecting research attention to better-validated risk stages.

Evaluating activation steering for generalizable privacy policies rather than single-instance suppression. The paper reports that activation steering (Suri et al., 2025) reduces memorization by up to 60% for targeted passages but identifies a key limitation: current methods suppress specific known instances rather than learning generalizable privacy policies, and "neurons involved in one memory often contribute to others, complicating memory-specific interventions due to the risk of collateral forgetting" (Section 6.3). A direct follow-up would test whether steering directions learned from one set of memorized passages (e.g., PII-containing sequences) generalize to suppress memorization of unseen passages from the same category (e.g., different PII sequences that share structural properties but not content). The experiment would: (1) identify steering directions using a training set of memorized passages; (2) apply these directions during inference on a held-out test set of memorized passages from the same semantic category; (3) measure both memorization suppression and utility preservation (factual QA accuracy, reasoning benchmark performance) on the held-out set. A finding that steering directions generalize across passages with shared structural properties (e.g., suppressing all sequences matching email patterns, not just the specific emails used to learn the steering direction) would transform activation steering from a reactive, instance-specific tool into a proactive, policy-based mitigation—directly addressing the generalization limitation the paper identifies. A finding that steering directions are highly instance-specific (no generalization to unseen passages from the same category) would establish a fundamental limitation and redirect research toward training-time methods for generalizable privacy.

A cumulative utility cost study for layered memorization defenses. The paper argues that effective mitigation requires "a comprehensive defense with targeted interventions at every phase" (Table 2) but provides only individual utility-impact estimates for isolated methods (ParaPO slightly degrades reasoning; DP-SGD approaches non-private performance when fine-tuning; activation steering causes "minimal" degradation). A direct follow-up would apply multiple interventions to a single model—deduplication during pretraining, DP-SGD during fine-tuning, and activation steering during inference—and measure the cumulative utility impact across factual recall benchmarks (e.g., Natural Questions, TriviaQA), reasoning benchmarks (MATH, GSM8K), and generation quality metrics (perplexity on held-out text). The study would test whether utility costs are additive (three "minimal" degradations summing to a significant total), subadditive (interventions interact to cause less degradation than the sum of individual costs), or superadditive (interventions interact to cause more degradation than the sum). The paper's hypothesis that beneficial and harmful memorization are "entangled in the same model parameters through the same compression objective" (Section 8, paraphrased from the conclusion) predicts that multiple interventions targeting memorization will have compounding effects on useful knowledge—a superadditive cost model. Finding superadditive costs would establish a fundamental tension in the layered-defense approach and motivate research on disentanglement methods that the paper identifies as "the frontier" (Section 8). Finding subadditive costs would validate the layered-defense framing as practically viable and provide quantitative guidance for practitioners on how many interventions can be applied before crossing unacceptable utility thresholds.

Copyright-aware training with verifiable non-infringement guarantees. The paper's Open Question 1 in Section 7 asks: "Can LLMs be trained to avoid memorizing copyrighted content?" and proposes augmenting training corpora with copyright metadata while modifying the optimization objective to penalize generation of text that is "both verbatim and tagged as protected." A direct follow-up would implement this proposal: annotate a training corpus (e.g., a subset of The PILE with Books3 and other copyrighted sections tagged) with copyright status metadata at the document and passage level, then train models with an augmented loss function that includes a penalty term for generating sequences that match copyrighted training passages under prefix-based extraction tests. The penalty could take the form of an auxiliary loss applied when the model's output distribution assigns high probability to copyrighted continuations given non-copyrighted prefixes. The study would measure: (1) whether the augmented training reduces extraction rates for copyrighted content compared to standard training; (2) whether the reduction generalizes to near-duplicate copyrighted content (paraphrased passages) that was not exactly matched in training; and (3) whether the augmented training preserves the model's ability to discuss copyrighted works in transformative ways (summarization, analysis, critique) that fair use doctrine would permit. This directly addresses the paper's call for metrics that "quantify the degree of transformative use" by distinguishing verbatim reproduction (potentially infringing) from analytical or critical discussion (potentially fair use) and would provide the first empirical evidence on whether copyright compliance can be learned as a model behavior rather than enforced through post-hoc filtering—a question with direct relevance to ongoing litigation including New York Times v. Microsoft/OpenAI.

Practical Applications and Downstream Use Cases

Privacy auditing for regulatory compliance under GDPR's Right to Erasure. Organizations deploying LLMs that process personal data from EU residents face a binding legal obligation: under GDPR Article 17, individuals can request deletion of their personal data, and the data controller must comply. The paper's systematic analysis reveals that no current method—including machine unlearning, differential privacy, and data cleaning—provides formal verifiable guarantees that data has been erased from model parameters. For a compliance officer or engineering team at a company deploying an LLM, this paper provides the technical evidence needed to make an informed risk assessment: current unlearning methods (gradient ascent, negative re-labeling) are ~10⁵× more efficient than retraining but "unlike DP, there is no formal guarantee, thereby leaving a risk that memorization persists" (Section 6.2), and unlearned states are fragile—"small exposures to previously forgotten data can quickly reverse the unlearning" (Table 5). The practical benefit is not a solution but a clear-eyed assessment: the paper enables technical teams to document for regulators that they have applied the best available methods while acknowledging their limitations, and to justify supplementary measures (access controls, output filtering, contractual restrictions on extraction attacks) that compensate for the lack of verifiable erasure. The paper's framework for evaluating mitigation methods by guarantee strength (Section 6) and its taxonomy of challenges at the intersection of technical and legal risks (Table 6) provides a structure for this documentation.

Training data curation for open-source LLM releases with reduced memorization risk. Organizations releasing open-source models (e.g., following the LLaMA, OPT, or Pythia paradigm) face a concrete risk: once model weights are public, any memorized content—PII, copyrighted material, toxic content—can be extracted by anyone with download access, with no ability to retroactively patch the model. The paper provides specific, quantified guidance for mitigating this risk at training time. Deduplication produces a tenfold reduction in memorized token generation (Lee et al., 2022, reported in Section 3). PII scrubbing, guided by taxonomies like Lukas et al. (2023) that categorize sensitive tokens (CARDINAL, DATE, PERSON), "dramatically reduce[s] such leakage" (Section 6.1, citing Li et al., 2024). The paper's identification of near-duplicate escape as a key limitation—semantically equivalent content with surface differences evades standard deduplication—provides a specific engineering target: extend deduplication beyond exact matching to include semantic similarity detection using embedding-based near-duplicate identification. The practical benefit is a training data pipeline that reduces memorization risk by approximately an order of magnitude before any model-specific mitigation is applied, and the paper's explicit documentation of the near-duplicate gap tells engineers where to invest additional effort (semantic deduplication) rather than pursuing diminishing returns from more aggressive exact deduplication.

Model alignment verification for production LLM deployments. Companies deploying aligned LLMs (e.g., ChatGPT, Claude, Gemini) currently rely on safety evaluations that test whether the model refuses harmful requests, avoids toxic outputs, and follows instructions. The paper's alignment-memorization distinction (Section 6.2) demonstrates that these evaluations are systematically blind to a critical vulnerability: alignment reduces the accessibility of memorized content under normal interaction but does not reduce its existence. The quantitative evidence is Nasr et al. (2025)'s divergence attack achieving up to 150× more verbatim extraction than benign prompts (Section 5). For a red-team or safety evaluation team at a model provider, this paper provides a concrete new evaluation category: adversarial memorization extraction testing. Rather than only testing whether the model refuses to output PII or copyrighted content, test whether adversarial prompts that induce pre-alignment-like behavior (end-of-text-style prompts, few-shot examples that normalize verbatim reproduction, prompts that mimic pretraining data formats) can extract memorized content that alignment suppresses under normal interaction. The paper's finding that the divergence attack succeeds on "aligned, production language models" (Section 5) means this is not a theoretical vulnerability—it has been demonstrated on deployed systems. The practical benefit is a more complete safety evaluation that tests for the presence of memorized content (which alignment does not remove) rather than only testing whether the alignment layer successfully suppresses it under standard queries.

Due diligence for AI copyright litigation risk assessment. The paper's analysis of the legal landscape (Section 7), centered on the New York Times v. Microsoft/OpenAI litigation, creates a framework for technical due diligence that legal teams can use to assess a model's exposure to copyright claims. The key practical insight is that legal risk depends not just on whether a model can reproduce copyrighted content (most large models can, given the right prompt) but on the ease of extraction—what the paper calls the "extraction effort" required. The paper's taxonomy of extraction methods (Table 3) distinguishes between attacks with different access requirements: prefix-based extraction requires training data access (making it useful for plaintiffs who can subpoena training data but not for random users); divergence attacks require only black-box access (making them a realistic threat from any user); soft prompting requires white-box access (making it relevant only when model weights are public). For a legal team defending against a copyright claim, this taxonomy enables a structured rebuttal: if the plaintiff demonstrated extraction using a method that requires access privileges the plaintiff doesn't typically have (e.g., training data prefixes), the defense can argue that the extraction is not representative of ordinary user behavior. If the plaintiff demonstrated extraction using black-box methods (divergence attacks), the defense is weaker. The paper's call for "context-aware memorization auditing" that reports "a multi-faceted report assessing factors like the novelty of the memorized text content, its commercial value, and the 'extraction effort' required" (Section 7, OQ3) provides a template for the kind of technical evidence that courts may increasingly demand as these cases proceed.