Do Membership Inference Attacks Work on Large Language Models?

🎯 Pitch

Membership inference attacks (MIAs) are essentially useless against the pre-training data of large language models, barely outperforming random guessing due to massive training sets and near-single-epoch training—when they do appear to work, it's an illusion caused by temporally shifted non-members with artificially low n-gram overlap, not genuine membership leakage.

1. Executive Summary

This paper studies the difficulty of membership inference attacks (MIAs) against the pre-training data of large language models, evaluating five state-of-the-art MIAs against the Pythia model suite (160M to 12B parameters) trained on the Pile. The work finds that MIAs barely outperform random guessing across most domains and model sizes, identifying two root causes: the combination of massive training data with near-one-epoch training and an inherently fuzzy boundary between members and non-members driven by high n-gram overlap—for example, Wikipedia and ArXiv non-members exhibit average 7-gram overlaps of 32.5% and 39.3% with training data. When MIAs do appear successful, the paper demonstrates that this can be attributed to distribution shift in candidate set selection—such as using temporally shifted non-members that concentrate at substantially lower n-gram overlap (13.9% for temporal Wikipedia vs. 39.3% for natural non-members)—rather than genuine membership leakage, establishing that existing MIAs are highly sensitive to even small lexical or semantic modifications and classifying semantically close paraphrases as non-members with high confidence.

2. Context and Motivation

The Core Problem: Do Membership Inference Attacks Actually Work on Large Language Models?

The fundamental question this paper tackles is deceptively simple: can an adversary reliably determine whether a specific text was used to train a large language model? This matters because membership inference attacks serve as the primary tool for measuring how much private information a model leaks about its training data. If MIAs don't work, our ability to audit LLM privacy is severely compromised. If they work only under specific conditions, we need to understand those conditions precisely.

The paper identifies a critical gap: while MIAs have been extensively studied on traditional machine learning models—classifiers trained on tabular data, computer vision models, and even fine-tuned language models—there has been limited systematic work studying MIA effectiveness against the pre-training phase of large language models. This gap is significant because pre-training is where LLMs ingest the vast majority of their data, including potentially sensitive information from sources like web crawls, academic papers, code repositories, and dialogue forums. If we cannot reliably detect when this data is "memorized" by the model, we cannot audit for privacy violations, cannot verify compliance with data usage agreements, and cannot assess the risk that an adversary might extract training data.

Why This Problem Matters: Privacy Auditing, Copyright, and Contamination Detection

The paper situates MIAs within a broader ecosystem of practical concerns (Section 1):

Privacy auditing and regulatory compliance. MIAs are used to verify whether models satisfy privacy guarantees, particularly in the context of differential privacy (Steinke et al., 2023). If an MIA cannot reliably distinguish training members from non-members, it provides evidence that the model does not leak individual-level information. But if MIAs are simply broken against LLMs—failing not because the model is private but because the attacks are too weak—then the auditing tool itself becomes unreliable, creating a dangerous false sense of security.

Copyright and intellectual property concerns. Recent work has used MIAs to investigate whether models have "memorized" copyrighted text, with implications for ongoing litigation around training data usage. If MIAs produce near-random results, they cannot serve as evidence either for or against memorization, leaving legal questions unresolved.

Test-set contamination detection. If a model's evaluation benchmark was accidentally included in its training data, performance metrics become inflated. MIAs offer one approach to detecting such contamination (Shi et al., 2023; Oren et al., 2023). If attacks don't work, contamination can go undetected, undermining the validity of benchmark comparisons.

Data extraction risk assessment. MIAs are frequently used as a proxy for measuring memorization and, by extension, the risk that training data can be extracted verbatim (Carlini et al., 2021; Mireshghallah et al., 2022b). While the paper distinguishes MIAs from extraction attacks—different assumptions, different threat models—effective MIAs provide a lower bound on leakage risk. If MIAs fail, extraction risk may still exist undetected, or conversely, the failure may genuinely indicate low memorization. Disambiguating these possibilities requires understanding why MIAs fail.

The paper identifies several specific limitations in the existing MIA literature that motivated their large-scale evaluation:

Most MIA research has focused on classifiers or fine-tuned models, not pre-trained LLMs. Zarifzadeh et al. (2023), Bertran et al. (2023), and Lukas et al. (2023) achieved high attack performance, but their analyses were limited to traditional classifier architectures or supervised fine-tuning settings where models are trained for many epochs on relatively small datasets. These conditions—multiple epochs, small data, concentrated training signal—are fundamentally different from LLM pre-training, where models see massive datasets for approximately one epoch. The paper notes that "the performance of existing MIAs on LLMs and their pre-training data is largely unexplored" (Section 1), identifying a genuine gap rather than an incremental extension.

Conflicting evidence about memorization in LLMs. On one hand, extraction attacks have demonstrated that LLMs do memorize and can regurgitate training data verbatim (Carlini et al., 2021; Biderman et al., 2023a). On the other hand, these extraction successes are often concentrated in specific high-duplication subsets of the data (Kandpal et al., 2022) and may not generalize to the broader training distribution. This creates an apparent paradox: extraction attacks show some memorization exists, but existing MIAs—which should detect that same memorization—show near-random performance. Resolving this tension requires understanding whether the attacks are broken, the memorization signal is too weak, or the standard membership definition is ill-suited to natural language.

Recent work reports high MIA performance, but the paper questions the evaluation methodology. Specifically, the paper notes that Shi et al. (2023) and Meeus et al. (2023) report state-of-the-art MIAs achieving >0.7 AUC ROC on pre-trained LLMs—substantially higher than the near-random results in this paper's Table 1. The paper does not dismiss these results but investigates why they differ, hypothesizing that candidate set selection—how non-members are chosen—may introduce unintended distribution shift that inflates apparent attack performance. This is a methodological critique: if non-members are drawn from a different temporal range or otherwise differ from the true member distribution, distinguishing them may reflect domain shift detection rather than genuine membership inference.

A Conceptual Tension: What Does "Membership" Even Mean for Language Data?

The paper introduces a deeper conceptual problem that goes beyond attack methodology. In traditional machine learning, membership is unambiguous: a record is either in the training set or it isn't, with no overlap between members and non-members. But natural language doesn't work this way. Documents contain repeated phrases, common quotations, standard boilerplate, and domain-specific terminology that naturally overlap even between "different" samples. For example, two Wikipedia articles about different topics will still share phrases like "born in," "according to," or "references." Two academic papers in the same field will share technical terminology, citation formats, and even identical mathematical definitions.

This creates what the paper terms a "fuzzy boundary" between members and non-members. A non-member sample may contain many substrings that appear verbatim in training data—substrings the model genuinely "saw" during pre-training—even though the complete document was never in the training set. The paper quantifies this overlap through n-gram analysis, finding that "the Wikipedia, ArXiv, and PubMed Central domains have average 7-gram overlaps of 32.5%, 39.3%, and 41.0%, respectively" with the training data (§3.2.2). At the extreme, domains like GitHub have 76.9% average 7-gram overlap, largely due to the repetitive nature of code (copyright notices, function definitions, syntax patterns).

This high overlap raises a fundamental question: if a non-member shares 40% of its 7-grams with training data, in what sense is it truly a "non-member" from the model's perspective? The model has seen 40% of the document's substrings during training, meaning it has partial knowledge of the document even though the complete document was never presented. The standard binary membership definition—seen exactly vs. not seen at all—may not be the right framework for measuring information leakage in generative language models. As the paper puts it in Section 5:

"The definition of membership in the standard MI game treats only records seen exactly during training as members... However, this may be at odds with what adversaries and privacy auditors care about when concerning information leakage."

How This Paper Positions Itself

The paper positions itself not as proposing a new attack or claiming that LLMs are inherently private, but rather as providing a systematic diagnostic of why existing MIAs fail on pre-trained LLMs, what conditions make them succeed, and what the implications are for how we think about membership in natural language. This is explicitly a measurement and analysis contribution, not a methods contribution.

The paper makes several distinct positioning moves:

It challenges the transferability of MIA methodology from classical ML to LLMs. The characteristics of LLM pre-training—billions of tokens, near-one-epoch training, inherent n-gram overlap in natural language—create conditions that existing MIAs were never designed to handle. The paper's finding that MIAs perform near-random (Table 1) isn't presented as "LLMs are private" but as "our existing privacy measurement tools may be inadequate for this regime."

It reinterprets prior positive MIA results through the lens of distribution shift. Rather than contradicting Shi et al. (2023) or Meeus et al. (2023), the paper offers an alternative explanation: carefully chosen non-member candidates may inadvertently introduce distribution shift, making the task easier in ways that don't reflect genuine membership inference. The temporal shift experiments (§4) provide concrete evidence for this interpretation, showing that using Wikipedia articles from after the training cutoff as non-members inflates AUC from ~0.55 to ~0.80.

It opens a conversation about rethinking membership for generative models. The final section (§5) on "Revisiting Membership" is forward-looking, arguing that the standard membership inference game may need to be extended to account for semantically or lexically close "neighbors" of true members. The paper demonstrates that existing MIAs classify semantically similar paraphrases as non-members with high confidence (Table 4), suggesting that the current definition of membership misses forms of information leakage that auditors care about.

It provides infrastructure for the community. The release of MIMIR, a unified benchmark package with implementations of all existing MIAs, positions this work as enabling future research rather than being a final word on the topic. This practical contribution acknowledges that the primary gap is not lack of attack ideas but lack of standardized evaluation and understanding of when and why attacks succeed or fail.

The paper's motivation is thus not to claim that LLMs are secure, but to argue that our tools for measuring their privacy leakage are insufficiently understood and potentially misleading—and that fixing this requires both better attacks and a more nuanced understanding of what membership means in natural language.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

This paper is a large-scale empirical evaluation and diagnostic study rather than a new method proposal. The system being evaluated is not a single model but an ecosystem of five existing membership inference attacks applied to pre-trained language models from the Pythia suite, all wrapped in a unified benchmarking framework called MIMIR. The core problem is: given a target language model and a candidate text sequence, can any of these attacks reliably determine whether that text was in the model's pre-training data? The "shape" of the solution is a systematic measurement campaign—test every attack against every model size and domain, then debug why performance is near-random by examining characteristics of LLM training (data scale, epoch count, recency effects) and inherent properties of natural language (n-gram overlap between members and non-members).

3.2 Big-picture architecture (diagram in words)

The evaluation framework has six major components:

Target Models (Pythia suite): Five auto-regressive transformer language models ranging from 160M to 12B parameters, trained on the Pile dataset for approximately one epoch. Both deduplicated (PYTHIA-DEDUP) and non-deduplicated (PYTHIA) variants are used. These models output next-token probability distributions—the raw material all attacks consume.
Candidate Data (Members and Non-Members): Pairs of text sequences drawn from the same domain (e.g., Wikipedia, ArXiv, GitHub) where one is known to be in the training data (member) and one is held-out test data (non-member). Each sequence is 100–200 words, sampled from documents longer than 100 words. The paper constructs multiple non-member sets to probe different hypotheses: natural Pile test splits, n-gram overlap thresholded sets, and temporally shifted sets (Wikipedia articles from after the training cutoff, ArXiv preprints from successive months after July 2020).
Five Membership Inference Attacks: Black-box attacks that take a target model and a candidate sequence and output a scalar membership score. These are LOSS, Reference-based, Zlib Entropy, Neighborhood, and Min-k% Prob. Each uses different signals (raw loss, loss relative to a reference model, loss calibrated by compression size, loss curvature via perturbation, or minimum token probabilities) to quantify how "surprised" the model is by the sequence—with lower surprise implying membership.
Reference Model (for calibrated attacks): A separate language model (STABLELM-BASE-ALPHA-3B-V2) used by the Reference-based attack to calibrate the target model's loss. This calibration accounts for intrinsic sequence complexity: a hard sequence will have high loss under any model, so the difference between target and reference loss isolates the memorization signal. The choice of reference model is critical and empirically determined through ablation (Section 3.1, Table 6 in Appendix).
n-gram Overlap Analyzer: A bloom filter-based infrastructure that precomputes n-gram membership across the entire Pile training set, enabling efficient quantification of how much substring overlap exists between candidate non-members and the training data. This is the diagnostic tool that explains why MIAs struggle.
Evaluation Framework (MIMIR): The unified Python package that orchestrates experiments—loading models, sampling candidates, running attacks, computing AUC ROC and TPR@low%FPR, and producing bootstrap confidence intervals. This is released as open-source infrastructure to enable reproducibility and future work.

Information flows as follows: (1) a target model and domain are selected; (2) members and non-members are sampled from the Pile train/test splits (possibly with additional n-gram filtering or temporal constraints); (3) each attack computes a membership score for every candidate sequence against the target model (and optionally the reference model); (4) scores are thresholded and evaluated against ground-truth membership labels using AUC ROC and TPR@low%FPR, with 95% confidence intervals via 1,000 bootstrap samples.

3.3 Roadmap for the deep dive

First, the formal membership inference game definition—this establishes the precise mathematical framework, what the adversary knows, and how scores become binary predictions—since all five attacks share this common structure.
Second, the five attacks individually—their scoring functions, what signal each exploits, and their computational requirements—because understanding why attacks fail requires understanding exactly what each attack measures.
Third, the target model and data pipeline—model selection, candidate sampling, deduplication procedures, and the specific parameters used—since these operational details determine the difficulty of the inference task.
Fourth, the n-gram overlap measurement infrastructure—how bloom filters enable efficient substring matching at Pile scale, and how overlap distributions are computed—because this is the key diagnostic tool that explains attack failure.
Fifth, the evaluation protocol—AUC ROC, TPR@low%FPR, bootstrapping, and the cross-validation logic for reference model selection—since the paper's claims rest on rigorous measurement.
Sixth, the experimental variants that probe why attacks struggle—training data size scaling (using intermediate checkpoints), epoch count experiments (Datablations, SILO), recency analysis, and temporal shift benchmarks—because these controlled experiments isolate specific causal factors.

3.4 Detailed, sentence-based technical breakdown

This is primarily an empirical analysis paper whose core idea is that existing membership inference attacks fail against pre-trained LLMs not because LLMs are inherently private, but because the combination of massive single-epoch training and natural n-gram overlap in language makes the member/non-member boundary inherently fuzzy—and when attacks appear to succeed, this is often attributable to unintended distribution shift in candidate set construction rather than genuine membership detection.

The Membership Inference Game: Formal Definition

All five attacks operate within the same abstract framework. Given a target auto-regressive language model $M$ that was trained on dataset $D$ , a candidate sequence $x = x_1, x_2, ..., x_n$ consisting of $n$ tokens, and a membership scoring function $f(x; M)$ that outputs a real-valued score, the adversary's task is to predict whether $x \in D$ (member) or $x \notin D$ (non-member).

The scoring function $f(x; M)$ does not directly output a binary prediction. Instead, it produces a score where lower values typically indicate higher confidence that $x$ was in the training data—though this convention varies slightly by attack. The adversary then chooses a threshold $\tau$ and classifies $x$ as a member if $f(x; M) \leq \tau$ , producing a binary decision.

What the adversary knows: The paper focuses on the black-box setting where the adversary has query access to the model's output probabilities $P(x_t | x_1...x_{t-1}; M)$ for any prefix, but does not have access to model parameters, gradients, or training data samples beyond the candidate sequences being tested. This is the most realistic threat model for deployed LLMs accessed via API. The paper specifically excludes attacks requiring meta-classifiers trained on ground-truth member/non-member subsets, noting that "access to such a subset can be difficult to guarantee in practice, especially as the inclusion of samples in pre-training corpora becomes more ambiguous as these corpora continue to expand." More powerful attack settings (white-box parameter access, shadow model training, multiple reference model ensembles) are discussed as alternatives in the literature but not evaluated here due to computational infeasibility at LLM scale and the paper's focus on practically deployable attacks.

The evaluation metric is not the raw score but its discriminative power. Rather than reporting absolute score values, the paper evaluates attacks by how well their scores separate the member and non-member distributions. This is measured via AUC ROC (area under the receiver operating characteristic curve, which integrates the tradeoff between true positive rate and false positive rate across all possible thresholds) and TPR@low%FPR (true positive rate when the false positive rate is constrained to 1%, 5%, or 10%—measuring attack performance in high-confidence regimes where false accusations are rare). All metrics are computed with 95% confidence intervals via 1,000 bootstrap resamples of the benchmark, providing statistical rigor beyond point estimates.

Why this evaluation framework matters beyond raw accuracy: AUC ROC is threshold-invariant—it measures the inherent separability of the two distributions without requiring the adversary to choose a specific operating point. This is important because the optimal threshold depends on the adversary's cost ratio (how much worse is a false positive than a false negative?), which varies by application. AUC ROC of 0.5 means the attack is no better than random guessing; AUC ROC of 0.6 means the attack has some discriminative power but is still weak; AUC ROC above 0.8 would indicate strong separability. The TPR@low%FPR metric captures performance in the regime most relevant to privacy auditing: if an auditor wants to identify training data with high confidence (few false accusations), can the attack still find members? A TPR@1%FPR of 2.5% means that when the attack is tuned to falsely accuse only 1% of non-members, it correctly identifies only 2.5% of members—which is useful for auditing but requires testing many candidates to find a few confident predictions.

Attack 1: LOSS Attack

The simplest and most foundational attack, proposed by Yeom et al. (2018) and adapted for language models by Carlini et al. (2019):

$f_{\text{LOSS}}(x; M) = L(x; M)$

where $L(x; M)$ is the model's cross-entropy loss on the target sequence $x = x_1, ..., x_n$ , defined as:

$L(x; M) = -\frac{1}{n}\sum_{t=1}^{n} \log P(x_t | x_1, ..., x_{t-1}; M)$

where $P(x_t | x_1, ..., x_{t-1}; M)$ is the probability the model assigns to token $x_t$ given the preceding context, and $n$ is the number of tokens in the sequence.

What it computes: for each position in the sequence, the attack queries the model for the probability it assigns to the token that actually appears at that position, given all preceding tokens. It takes the negative log of these probabilities (converting them to "surprise" values—a token the model assigns high probability gets low surprise), averages across all $n$ positions, and outputs this average as a single scalar. Intuitively, if the model was trained on $x$ , it should be less "surprised" by the tokens in $x$ (lower loss) than if $x$ is novel.

Why this form: The LOSS attack is the simplest possible membership signal—it assumes that training causes the model to assign higher likelihood to its training data than to unseen data. This assumption is theoretically well-founded: models minimize loss on training data during optimization, so training points should have lower loss than held-out points, especially when the model overfits. However, the attack has no calibration mechanism—it cannot distinguish between a sequence having low loss because it was memorized versus having low loss because it is intrinsically predictable (e.g., containing common phrases, simple syntax, or repetitive patterns). This lack of calibration is the primary weakness that all other attacks attempt to address.

Attack 2: Reference-Based Attack

A calibrated attack following Sablayrolles et al. (2019) and Watson et al. (2022), which uses a separate "reference" language model $M_{\text{ref}}$ trained on disjoint data to normalize away intrinsic sequence difficulty:

$f_{\text{Ref}}(x; M) = L(x; M) - L(x; M_{\text{ref}})$

where $L(x; M)$ is the target model's loss on $x$ (as defined above) and $L(x; M_{\text{ref}})$ is the reference model's loss on the same sequence.

What it computes: the difference between how surprised the target model is by $x$ and how surprised the reference model is by $x$ . If both models assign similar probability to $x$ , this difference is near zero, suggesting $x$ is not memorized—its loss is explained by intrinsic complexity that any reasonable language model would exhibit. If the target model assigns substantially lower loss (less surprise) than the reference model, the difference is negative and large in magnitude, suggesting the target model has memorized $x$ beyond what the reference model's general language understanding would predict.

Why this form: This calibration addresses the fundamental weakness of the raw LOSS attack. Common sequences (e.g., "the cat sat on the mat") will have low loss under any language model because they are intrinsically predictable. Without calibration, these sequences would be incorrectly flagged as members. By subtracting the reference model's loss, the attack isolates the excess memorization—the degree to which the target model is less surprised than a model that has comparable language understanding but no specific memory of this sequence. The key assumption is that the reference model is trained on data from a similar distribution but disjoint from the target model's training data. However, the paper notes that this assumption is "hard to impose at the scale of pre-training corpora; common practice is to collect all the data available on the web, leading independently collected datasets to naturally overlap with each other." If the reference model's training data overlaps with the target model's data, it may also have lower loss on members, reducing the calibration's effectiveness.

Reference model selection: The paper empirically ablates across eight candidate reference models (Table 6 in Appendix), including GPT-2 (124M), DistilGPT2 (82M), OPT-1.3B, GPT-Neo-1.3B, SILO-PDSWBY-1.4B, LLaMA-7B, StableLM-Base-Alpha-3B-V2, and Pythia-Dedup-1.4B (the non-deduplicated variant of a smaller Pythia model). STABLELM-BASE-ALPHA-3B-V2 is selected because it yields the highest AUC ROC across most domains. The paper speculates that this model's success is due to two factors: (1) it is trained on a much larger corpus (trillions of tokens including RedPajama and the Pile, with an emphasis on refined web text), which gives it strong general language modeling capabilities that help it achieve losses similar to the target model on non-member data, and (2) despite this overlap, the target model's larger size (especially for Pythia-12B) means it likely "considerably overfits certain member samples," creating a detectable loss differential on members even when the reference model generalizes well. The paper also experiments with an aggregate of all reference models (averaging calibrated scores) but finds this "naive ensembling method" performs poorly, likely because different reference models have different tokenizers, making their loss values incomparable in magnitude.

Attack 3: Zlib Entropy Attack

A difficulty-calibrated attack proposed by Carlini et al. (2021) that uses compression size rather than a reference model to normalize for sequence complexity:

$f_{\text{Zlib}}(x; M) = \frac{L(x; M)}{\text{zlib}(x)}$

where $L(x; M)$ is the target model's loss on $x$ and $\text{zlib}(x)$ is the length in bytes of the sequence after zlib compression.

What it computes: the ratio of the model's surprise to the sequence's information-theoretic complexity as measured by a general-purpose compression algorithm. A sequence that compresses well (short zlib output) has low entropy and should be predictable; a sequence that compresses poorly (long zlib output) has high entropy and should be harder to predict. By dividing the model's loss by the compressed size, the attack normalizes for this intrinsic difficulty. If the ratio is low, the model is less surprised than the sequence's complexity would suggest, indicating memorization. If the ratio is high, the model's surprise is consistent with or exceeds what the sequence's complexity would predict.

Why this form: This provides an alternative calibration mechanism that requires no reference model—only the target model and a standard compression library. The key insight is that zlib compression captures a general notion of sequence complexity: repetitive text compresses well, while novel or high-entropy text compresses poorly. By normalizing loss by compressed size, the attack accounts for the fact that intrinsically simple sequences will have low loss under any reasonable model. The advantage over the reference-based attack is that zlib calibration does not depend on the availability of a suitable reference model and avoids the tokenizer incompatibility and data overlap issues that complicate reference model selection. The disadvantage is that zlib compression is a cruder measure of "language model difficulty" than an actual language model—it captures character-level redundancy (repeated strings) but not semantic predictability (a semantically predictable phrase might still have unique character sequences). The paper uses the standard Python zlib library with default compression settings.

Attack 4: Neighborhood Attack

An attack proposed by Mattern et al. (2023) that measures the curvature of the loss landscape around the candidate sequence, based on the intuition that memorized training points sit in sharp local minima of the loss function while non-member points sit in flatter regions:

$f_{\text{Neigh}}(x; M) = L(x; M) - \frac{1}{n}\sum_{i=1}^{n} L(\tilde{x}_i; M)$

where $L(x; M)$ is the target model's loss on the original sequence $x$ , $\tilde{x}_i$ for $i = 1, ..., n$ are $n$ "neighbor" sequences generated by perturbing $x$ , and $n$ is the number of neighbors.

What it computes: the difference between the model's loss on the original sequence and the average loss on perturbed versions of that sequence. The intuition is that if $x$ was memorized during training, the model has specifically optimized its parameters to assign low loss to this exact sequence—but nearby sequences (with token replacements, deletions, or insertions) receive much higher loss because the optimization was narrow and specific. If $x$ was not memorized, the model's loss should be relatively flat around $x$ —small perturbations don't dramatically change the loss. Therefore, a large positive difference (original loss much lower than neighbor average loss) indicates membership; a small difference indicates non-membership.

How neighbors are generated: The attack uses BERT (Devlin et al., 2019) as a masking model. For a given sequence, BERT randomly selects a percentage of tokens and replaces them with [MASK] tokens, then predicts the most likely tokens to fill those masks based on bidirectional context. The resulting filled-in sequence preserves local coherence (since BERT uses context from both directions to make predictions) while introducing lexical variation. The paper uses a masking percentage of 5%, meaning 5% of tokens are masked and replaced. This creates "neighbors" that are semantically and syntactically similar to the original but lexically distinct—close enough to probe local loss curvature without being so close that the model assigns them identical loss.

Why this form: The neighborhood attack captures a fundamentally different signal than the previous attacks. LOSS, reference-based, and zlib attacks all measure the model's absolute behavior on the candidate sequence. The neighborhood attack measures the relative behavior: how does the model's loss change when the sequence is slightly modified? This is inspired by the observation that overfitting creates sharp minima in the loss landscape—the model's loss increases rapidly as you move away from the training point in input space. For non-members, the loss landscape should be smoother because the model has not specifically optimized to minimize loss at that exact point. The attack requires more computation than simpler attacks (it needs to query the model $n+1$ times per candidate—once for the original and $n$ times for neighbors), making it more expensive but potentially more informative.

Attack 5: Min-k% Prob Attack

An attack proposed by Shi et al. (2023) based on the observation that non-member sequences tend to contain more individual tokens with anomalously low probability than member sequences do:

$f_{\text{Min-k\%}}(x; M) = \frac{1}{|\text{min-k}(x)|} \sum_{x_i \in \text{min-k}(x)} -\log P(x_i | x_1, ..., x_{i-1}; M)$

where $\text{min-k}(x)$ is the set of tokens in $x$ that have the $k\%$ lowest probabilities under the model, $|\text{min-k}(x)|$ is the number of tokens in that set, and $P(x_i | x_1, ..., x_{i-1}; M)$ is the model's probability for token $x_i$ given preceding context.

What it computes: rather than averaging the negative log probability over all tokens (as in the LOSS attack), this attack identifies the $k\%$ of tokens that the model finds most surprising (lowest likelihood) and averages only those. The key insight is that for a non-member sequence, there will typically be several tokens the model assigns very low probability—these are the "canary" tokens that reveal the sequence is unfamiliar. For a member sequence, even the least-likely tokens should have reasonable probability because the model was trained on them. The attack discards the tokens the model is confident about (which could be confident for either members or non-members if they contain common phrases) and focuses on the tail of the distribution.

Hyperparameter $k$ : The paper experiments with $k \in \{10, 20, 30, 40, 50\}$ , as recommended by Shi et al. (2023), and "settles on $k = 20$ for our experiments." This means the attack averages the 20% of tokens with the lowest probabilities. The choice of $k$ represents a tradeoff: too small a $k$ and the attack is noisy (averaging over very few tokens); too large a $k$ and the attack converges to the standard LOSS attack, losing the benefit of focusing on the tail. The paper's choice of 20% suggests that focusing on the bottom quintile of token probabilities provides a robust signal.

Why this form: The LOSS attack averages over all tokens, which can dilute the signal—many tokens in any sequence have high probability under a well-trained language model (common words, predictable syntax), and these high-probability tokens dominate the average, obscuring the rare low-probability tokens that genuinely indicate unfamiliarity. The Min-k% Prob attack specifically amplifies these rare low-probability signals. Consider a non-member sequence that is mostly natural and predictable (high-probability tokens) but contains a novel named entity or unusual phrasing (very low-probability token). The LOSS attack's average might be dominated by the many high-probability tokens, producing only a modestly elevated loss. The Min-k% attack isolates that single problematic token, producing a much stronger membership signal. This design is motivated by the empirical observation in Shi et al. (2023) that "non-member examples tend to have more tokens assigned lower likelihoods than member examples do."

Target Models and Their Properties

The Pythia model suite (Biderman et al., 2023b) provides the primary attack targets. The paper uses five models from the PYTHIA-DEDUP suite (trained on the deduplicated Pile): 160M, 1.4B, 2.8B, 6.9B, and 12B parameters. For comparison, it also uses the non-deduplicated PYTHIA suite with the same parameter counts (excluding 160M for most experiments). Both suites share the GPT-NeoX architecture (Andonian et al., 2023) and were trained on approximately 300B tokens from the Pile (Gao et al., 2020). The critical property that enables the paper's diagnostic experiments is that EleutherAI released intermediate checkpoints for all models, with known training data order, allowing the paper to analyze how MIA performance evolves as training progresses and to control for the recency of member samples. Specifically, checkpoints were saved every 1,000 steps initially and every 5,000 steps thereafter, with each step processing 1,024 sequences of 2,048 tokens (approximately 2.1M tokens per step).

Why Pythia specifically: The paper needs models where (1) the exact training data, data order, and intermediate checkpoints are fully known and publicly available, (2) the training setup is representative of modern LLM pre-training (large-scale, near-one-epoch, decoder-only transformer), and (3) there is a range of model sizes to study scaling effects. The Pythia suite satisfies all three requirements uniquely well. Alternative model families (GPT-Neo, OPT, LLaMA) either don't release intermediate checkpoints, don't release exact training data, or were not trained in a controlled scaling setup. The paper does replicate key findings on GPT-Neo models (Appendix A.6, Table 7) to validate that results are not specific to the GPT-NeoX architecture, finding similar near-random performance patterns.

For the deduplication comparison: The PYTHIA models were trained on the original Pile (approximately 825GB, ~300B tokens) for about 0.9 epochs. The PYTHIA-DEDUP models were trained on the deduplicated Pile, which removes exact duplicate documents and near-duplicate documents, reducing the total training data. The PYTHIA-DEDUP models are trained for slightly more than one epoch (approximately 1.0 epochs), with the paper selecting "the checkpoint that most closely matches the one epoch mark over the deduplicated Pile"—specifically checkpoint 'step99000,' where step 99,000 corresponds to 99,000 × 1,024 × 2,048 ≈ 207.6B tokens seen.

Additional model families are used for specific diagnostic experiments:

Datablations (Muennighoff et al., 2023): 2.8B-parameter models trained on 55B tokens from C4, varying the number of epochs from 1 to 14 (and one model at 44 epochs, excluded from evaluation). Since the total training data is fixed, increasing epochs directly increases data repetition—making these ideal for isolating the effect of multi-epoch training on MIA performance.
SILO (Min et al., 2023): 1.3B-parameter models trained on the Open License Corpus with domain-specific upsampling. The SILO-PDSW variant (trained on public domain and permissively licensed software data) upsamples certain underrepresented domains by 3×, creating a more realistic setting where effective epoch count varies by domain. The paper uses intermediate checkpoints to study how MIA performance evolves as a specific domain (HackerNews, DM Mathematics) is seen multiple times.
GPT-Neo (Black et al., 2021): 125M, 1.3B, and 2.7B parameter models trained on the full Pile for ~300B tokens, similar to Pythia but with a different architecture (GPT-3 replication) and tokenizer (fewer whitespace tokens). Used for cross-architecture validation.
OLMo (Groeneveld et al., 2024): 1B and 7B parameter models trained on the DOLMA dataset (Soldaini et al., 2023) with 3T and 2.5T tokens respectively. Used for preliminary experiments validating that near-random MIA performance extends to models trained on even larger, more recent corpora.

Candidate Data Pipeline: Members and Non-Members

The paper constructs membership inference benchmarks for seven individual domains from the Pile—Wikipedia, GitHub, Pile-CC (Common Crawl web text), PubMed Central (biomedical abstracts), ArXiv (computer science preprints), DM Mathematics, and HackerNews—plus an aggregate benchmark over the entire Pile. For each domain, the pipeline proceeds as follows:

Step 1: Source identification. Members are drawn from the Pile training set; non-members are drawn from the Pile test set. The Pile's creators (Gao et al., 2020) performed document-level decontamination of the test set against the training set, ensuring that no test document appears verbatim in the training data. However, the paper notes that this document-level deduplication does not prevent substring-level overlap—different documents can contain the same phrases, quotes, or boilerplate text—which is precisely the phenomenon the n-gram overlap analysis later reveals.

Step 2: Additional decontamination. Following Groeneveld et al. (2023), the paper applies a further 13-gram bloom filter decontamination step to the non-member set. For each non-member document, the system checks whether any 13-gram substring has more than 80% overlap with the training set. Documents exceeding this threshold are excluded. This provides stronger guarantees than document-level deduplication against verbatim substring contamination, but as the paper's analysis reveals, it still leaves substantial n-gram overlap (e.g., 32.5% average 7-gram overlap for Wikipedia non-members) because natural language inherently contains repeated substrings across distinct documents.

Step 3: Length filtering. Only documents longer than 100 words are retained. This threshold "reduces the impact of sample length on MIA performance" (Shi et al., 2023 observed that longer samples provide more signal for membership detection). The 100-word minimum ensures that candidate sequences contain sufficient content to be meaningful for membership inference while avoiding the trivial case of very short sequences that might be uninformative.

Step 4: Truncation. Sampled documents are truncated to 200 words from the beginning. This creates uniform-length candidate sequences and bounds computational cost while ensuring that the selected portion (the document's opening) typically contains the most contentful text.

Step 5: Sampling. For individual domains, 1,000 members and 1,000 non-members are randomly sampled from the filtered data. For the aggregate Pile benchmark, 10,000 members and 10,000 non-members are sampled from the complete Pile train and test sets respectively. These sample sizes are chosen to provide stable AUC estimates while keeping experimental costs manageable—each candidate requires a forward pass through the target model (and potentially the reference model and multiple neighbor perturbations).

For the temporal shift experiments (Section 4): The member set remains the same (Pile training samples), but non-members are replaced with more recent data from the same domain. For Wikipedia, non-members are drawn from the RealTimeData WikiText dataset (Li et al., 2023c), consisting of Wikipedia articles created between August 12, 2023 and January 8, 2024—well after the Pile's March 2020 Wikipedia dump cutoff. For ArXiv, non-members are sampled from preprints posted in successive months after July 2020 (August 2020, January 2021, June 2021, January 2022, June 2022, January 2023, June 2023), collected via the ArXiv API. The paper processes these following the same steps used by the Pile (converting LaTeX sources to Markdown, filtering conversion errors, appending titles to article bodies for Wikipedia). This creates a controlled experiment: the only difference between the natural and temporal non-member sets is the temporal range, with all other processing identical.

For the n-gram overlap thresholded experiments (Section 3.2.2): The non-member set is further filtered to include only samples with ≤20% 7-gram overlap with the training data. This is explicitly an experimental probe, not a recommendation for benchmark construction—the paper clarifies that "this step is not a suggestion for researchers to alter their benchmarks; such a processing step drifts away from the standard membership inference game." The purpose is to test the hypothesis that high n-gram overlap between members and non-members is a key factor reducing MIA performance. By artificially eliminating high-overlap non-members, the paper demonstrates that MIAs become substantially more effective, confirming the diagnostic value of the n-gram overlap measurement.

For the training data size and recency experiments (Section 3.2.1, Appendix C.1): The pipeline is modified to use intermediate checkpoints. Members are sampled from the documents seen within the most recent 100 training steps before each checkpoint, using EleutherAI's provided random seeding to deterministically map checkpoints to their training data windows. Non-members are fixed across all checkpoints (1,000 samples from the full Pile test set). This design isolates the effect of training progress: as the model sees more data, how does MIA performance change? The recency experiment flips this—the target model is fixed (checkpoint at step 99,000), but member sets are sampled from different training windows (steps 0–100, 100–200, ..., 98,900–99,000) to measure how recently-seen data is more vulnerable.

n-gram Overlap Measurement Infrastructure

This diagnostic infrastructure quantifies the degree of substring overlap between non-member candidates and the training data, providing a quantitative explanation for why MIAs struggle.

Bloom filter construction. The paper creates a bloom filter over the entire Pile training set for a given n-gram size $n$ . A bloom filter is a probabilistic data structure that supports efficient set membership queries (is this n-gram in the training data?) with a configurable false positive rate and zero false negatives—it may occasionally say "yes" when the n-gram is not actually present, but it will never say "no" when the n-gram is present. Due to the Pile's scale (approximately 800GB), the bloom filter is sharded: the training data is split in half, and two independent bloom filters are constructed. Each filter is configured to have a false positive rate below 1% (actual: 0.6%), meaning that an n-gram not in the training data has a 0.6% chance of being incorrectly flagged as present. To check n-gram inclusion in the full Pile, the system queries both shards and returns "present" if either shard returns "present."

Tokenization and hashing. For each document in the training set, the text is tokenized at the word level (not subword level—this is important because it captures lexical overlap, not tokenizer-specific overlap). A sliding window of size $n$ with stride 1 extracts all consecutive word n-grams from the document. Each n-gram is hashed and inserted into the bloom filter. The same sliding window procedure is applied to non-member candidates when computing overlap.

The n-gram overlap score for a non-member sample $x$ consisting of $m$ words is defined as:

$\text{overlap}_n(x) = \frac{1}{m - n + 1} \sum_{i=1}^{m-n+1} \mathbf{1}\{\exists y \in D : x_i...x_{i+n-1} \in y\}$

where $x_i...x_{i+n-1}$ is the n-gram starting at position $i$ in $x$ , $D$ is the training dataset, and $\mathbf{1}\{\cdot\}$ is the indicator function (1 if the n-gram appears in at least one training document, 0 otherwise).

What it computes: the fraction of n-grams in the non-member that can be found anywhere in the training data—not necessarily in a single training document, but in any document. A score of 0.40 means that 40% of the n-grams in the non-member appear in at least one training document. This is a strict lower bound on how much of the non-member the model has "seen" (as n-grams) during training.

Why n-gram overlap matters for MIA: If a non-member shares 40% of its 7-grams with training data, the model has literally seen those substrings during training and will likely assign them high probability—exactly the signal that MIAs use to infer membership. The model has no way to distinguish "this 7-gram appeared in a training document" from "this 7-gram appeared in a test document"—it only sees the tokens during training. Therefore, the presence of shared substrings creates an irreducible ambiguity: the model produces a membership-like signal (low loss) on non-member sequences because it genuinely has seen substantial portions of them during training, just not in the exact same order or complete document context.

The paper computes overlap distributions for $n \in \{4, 7, 13\}$ across all domains (Figure 3 shows 7-gram distributions; Figure 14 in Appendix shows all three n values). The choice of $n=7$ for the main text is somewhat arbitrary but represents a meaningful phrase length—shorter than a full sentence but longer than a common bigram, and roughly corresponding to multi-word expressions, proper names, and formulaic constructions. For the thresholding experiments in Table 2, the paper uses $n=7$ with a threshold of ≤20% overlap.

Evaluation Protocol

Primary metric: AUC ROC. For each (attack, model, domain) combination, the attack computes membership scores for all 1,000 members and 1,000 non-members. The AUC ROC is computed by sweeping all possible thresholds, computing the true positive rate (TPR) and false positive rate (FPR) at each, and integrating the resulting curve. AUC ROC ranges from 0.0 to 1.0, with 0.5 representing random guessing and 1.0 perfect separation. Statistical confidence is assessed via 1,000 bootstrap resamples: the 1,000+1,000 samples are resampled with replacement 1,000 times, AUC is recomputed on each bootstrap, and 95% confidence intervals are derived from the 2.5th and 97.5th percentiles of the bootstrap distribution.

Secondary metric: TPR@low%FPR. At specific FPR thresholds (1%, 5%, 10%), the paper reports the TPR achieved. This captures attack performance in the high-confidence regime where false accusations are rare—the regime most relevant for privacy auditing, where you want to identify training data with minimal chance of falsely accusing non-members. TPR@1%FPR can be very low (e.g., 0.5%) even when AUC is moderate, indicating that while the attack has some discriminative power, it cannot confidently identify many members without also falsely accusing many non-members.

Reference model selection protocol. The reference-based attack's performance depends critically on the reference model choice, but this choice cannot be optimized on the test data without biasing results. The paper addresses this by (1) ablating across eight reference models in a systematic sweep (Table 6), (2) selecting the reference model (STABLELM-BASE-ALPHA-3B-V2) that consistently performs best across domains and model sizes, and (3) using this reference model for all subsequent experiments. This is a pragmatic approach—the reference model is chosen based on held-out validation data, not test data—but the paper acknowledges that reference model selection remains "challenging and largely empirical." Future work might train dedicated reference models on carefully curated non-overlapping data, but this is computationally expensive at LLM scale.

Statistical rigor across experimental variants. For each experimental condition (different non-member set, different checkpoint, different model), the paper recomputes all metrics with the same bootstrap procedure. Shaded regions in figures represent 95% confidence intervals, enabling visual assessment of whether differences between conditions are statistically reliable. This is particularly important for the training data size experiments (Figure 2, left), where the performance curves for different model sizes cross and diverge.

Experimental Variants for Diagnostic Analysis

The paper conducts several controlled experiments to isolate specific factors affecting MIA performance. Each experiment modifies one variable while holding others constant:

Training data size experiment (Section 3.2.1, Figure 2 left): Using PYTHIA-DEDUP intermediate checkpoints at steps 1,000, 5,000, 10,000, 15,000, ..., 95,000, and 99,000, the paper measures how MIA performance changes as the model sees more training data. Members for each checkpoint are sampled from documents seen within the most recent 100 steps (to control for recency effects—see below). Non-members are fixed across all checkpoints. The key finding is that performance "generally starts as near-random, then rapidly increases within the next few thousand steps, before decreasing across successive checkpoints." This non-monotonic pattern is attributed to an initial overfitting phase (when the data-to-parameter ratio is still small) followed by progressive generalization as more data is seen.

Recency experiment (Appendix C.1, Figure 9): Fixing the target model at checkpoint step 99,000 (approximately one epoch), the paper varies which training window the members are sampled from. Member sets are drawn from the most recent 100 steps before step 1,000, 5,000, 10,000, ..., 95,000, and 99,000. The key finding is that "member data seen more recently by the given checkpoint contributes to slightly higher MIA performance," consistent with known forgetting effects in neural networks (Jagielski et al., 2023). The recency effect is more pronounced for smaller models, while larger models show more persistent memorization of earlier data.

Epoch count experiment (Section 3.2.1, Figure 2 right): Using Datablations models trained on a fixed 55B-token subset of C4 for varying numbers of epochs (1 to 14), the paper measures how MIA performance changes with data repetition. All models have the same architecture (2.8B parameters), same training data, same total tokens seen—the only difference is how many times each data point was repeated. The key finding: "performance increases linearly with the number of effective epochs." This directly supports the hypothesis that near-one-epoch training is a major factor in low MIA performance.

SILO epoch experiment (Appendix C.2, Figure 10): Extending the epoch analysis to a more realistic setting where domain-specific upsampling creates 3× repetition for underrepresented domains (HackerNews, DM Mathematics), the paper tracks MIA performance across intermediate SILO checkpoints as effective epoch count increases. Results are more nuanced: HackerNews shows an initial increase then plateau, while DM Mathematics decreases with more epochs—counter-intuitive and attributed to the symbolic nature of mathematical text making specific sample memorization unlikely even with repetition.

Temporal shift experiment (Section 4): Holding members fixed (Pile Wikipedia or ArXiv training samples), the paper replaces non-members with temporally shifted data. For Wikipedia, non-members are 2023–2024 articles. For ArXiv, non-members are preprints from successive months after July 2020 (August 2020 through June 2023). The key finding is that temporally shifted settings yield substantially higher AUC ROC (e.g., Wikipedia AUC increases from ~0.55 to ~0.80), and this increase correlates with a shift in n-gram overlap distribution (temporal non-members have much lower overlap with training data). This demonstrates that apparent MIA success in such settings may reflect distribution shift detection rather than genuine membership inference.

n-gram overlap thresholding experiment (Section 3.2.2, Table 2): Taking the natural non-member sets and filtering to retain only samples with ≤20% 7-gram overlap with training data, the paper measures how MIA performance changes. The key finding is dramatic improvement: e.g., Wikipedia LOSS attack increases from AUC 0.516 to 0.666; GitHub zlib attack increases from 0.690 to 0.908. This confirms that the overlap between members and non-members at the substring level is a primary confound reducing MIA performance in the natural setting.

Modified member experiments (Section 5): Creating "fuzzy members" by perturbing actual members, the paper measures how existing MIAs classify these samples. Two perturbation methods are used: (1) lexical modification by replacing $n$ random tokens (for $n \in \{1, 10, 25\}$ ) with random vocabulary tokens, and (2) semantic modification using GPT-4 to paraphrase member samples while preserving meaning. For each modified sample, the paper computes the attack score and determines whether it would be classified as a member or non-member using thresholds derived from the natural member/non-member distributions. The key finding is that MIAs classify these modified members as non-members with high confidence (Table 4), even when the modification is minimal (edit distance 1) or preserves semantics (GPT-4 paraphrases). This suggests that current MIAs are hypersensitive to exact token sequences and miss semantically meaningful leakage.

Design choices underlying these experiments: The paper uses a between-subjects design across domains—each domain is analyzed independently rather than pooled—because n-gram overlap distributions, model performance, and attack effectiveness vary substantially by domain. The choice of 1,000 samples per (domain, attack, model) combination balances statistical power (1,000 bootstrap resamples provide stable estimates) with computational cost (each sample requires a forward pass through a multi-billion parameter model). The paper's use of multiple non-member sets (natural, temporal, n-gram thresholded) rather than a single fixed test set enables causal analysis of why MIAs perform as they do, moving beyond descriptive measurement to mechanistic explanation.

MIMIR: The Unified Benchmark Package

The paper releases MIMIR as a Python package (http://github.com/iamgroot42/mimir) with documentation, tests, and pre-processed datasets available via HuggingFace (https://huggingface.co/datasets/iamgroot42/mimir). The package provides:

A base attack class with helper functions for computing loss, extracting token probabilities, and caching model outputs—reducing boilerplate when implementing new attacks.
Data processing utilities for filtering, sampling, and caching candidate sequences from Pile domains and other sources, with configurable length constraints and deduplication options.
Support for a wide range of target and reference models through the HuggingFace transformers interface, enabling experiments with any model that exposes next-token probability distributions.
Modular configuration files allowing multiple experiments to run in parallel without code changes, using Python 3.9.7, PyTorch 2.0.1, and GPU hardware ranging from RTX 6000 to A100.

The release of MIMIR positions this work as enabling future research—providing a standardized, reproducible platform for evaluating new attacks, testing new models, and extending the analysis to new domains or threat models. This infrastructure contribution is particularly important given the paper's central claim: that evaluating MIAs correctly requires careful attention to candidate set construction, n-gram overlap distributions, and the potential for unintended distribution shift. By providing a unified package that implements all existing attacks with consistent evaluation protocols, MIMIR lowers the barrier for researchers to diagnose whether their new attacks genuinely improve membership inference or merely exploit distribution shift.

4. Key Insights and Innovations

Innovation 1: The Membership Inference Failure as a Diagnostic Signal, Not a Privacy Guarantee

The paper's most fundamental intellectual move is reframing the near-random performance of MIAs from a potential conclusion ("LLMs are private") into a diagnostic puzzle that reveals something about the nature of language data and LLM training. Prior work on MIA had largely operated in a regime where attack success was the norm—high AUC on classifiers (Zarifzadeh et al., 2023; Bertran et al., 2023), fine-tuned LMs (Mireshghallah et al., 2022b), and even some pre-trained LM settings (Shi et al., 2023). The field's implicit assumption was that if an MIA fails, either the attack is too weak or the model genuinely doesn't memorize. This paper introduces a third possibility: the failure may reflect a fundamental mismatch between the standard membership definition and the nature of natural language generation, rather than a lack of memorization per se.

What makes this distinctive is that the paper treats the MIA's failure as the primary object of study rather than a null result to be dismissed. The question shifts from "which attack works best?" to "what does it mean that none of them work well, and under what conditions would they work?" This is a conceptual reframing that converts a series of negative results (Table 1, showing AUC near 0.5 for most domains) into a productive line of investigation.

The n-gram overlap measurement infrastructure (Figure 3, Table 2) operationalizes this reframing. Instead of stopping at "MIAs perform near random," the paper quantifies why: non-members share 32.5% (Wikipedia), 39.3% (ArXiv), and 41.0% (PubMed Central) of their 7-grams with training data on average. This isn't a failure of the attacks—it's evidence that the model has genuinely seen substantial portions of these "non-member" documents during training, just not in the exact same document context. The distinction between "this exact document was in the training set" and "40% of this document's substrings were in the training set" is precisely what the standard membership inference game cannot capture, and the paper makes this ambiguity the centerpiece rather than a footnote.

Prior work on MIAs had acknowledged that reference-based calibration (Watson et al., 2022) or difficulty normalization (Carlini et al., 2021) could improve precision, but no prior work had systematically demonstrated that the definition of membership itself—not just the attack methodology—breaks down at the scale and nature of LLM pre-training data. This is a fundamental conceptual contribution, not an incremental methodological refinement.

Evidence anchoring: The n-gram overlap thresholding experiment (Table 2) shows that when non-members are artificially restricted to ≤20% 7-gram overlap, AUC jumps from 0.516→0.666 (Wikipedia LOSS) and 0.690→0.908 (GitHub zlib). This demonstrates that the poor natural performance is largely attributable to this overlap, not to weak attacks or memorization-free models. The temporal shift experiments (Section 4, Table 3) provide converging evidence: when non-members are drawn from post-cutoff Wikipedia, their 7-gram overlap drops from 39.3% to 13.9% (Figure 4), and AUC correspondingly rises from ~0.55 to ~0.80.

Innovation 2: Distribution Shift as the Hidden Explanation for Prior "Successful" MIAs

The paper's second major contribution is a methodological critique that reframes prior positive MIA results on LLMs as potentially arising from unintended distribution shift in candidate set construction, rather than genuine membership detection. This is not merely a "replication failed" claim—it is a diagnostic framework for evaluating whether an MIA benchmark genuinely tests membership or accidentally tests something else (distribution shift, temporal change, domain mismatch).

What makes this innovation distinctive is the systematic unpacking of exactly how distribution shift inflates MIA performance, combined with a concrete diagnostic tool (n-gram overlap distribution comparison) that future researchers can use to audit their own benchmarks. Prior work on MIA evaluation (Carlini et al., 2022; Murakonda and Shokri, 2020) had noted that poor benchmark construction can inflate results, but this paper provides the first quantitative demonstration of the mechanism in the LLM pre-training setting: temporal shift causes n-gram overlap to drop, which makes members and non-members intrinsically more distinguishable, which inflates AUC independent of actual memorization.

The temporal shift experiments (Section 4) are the key evidence. When non-members are Wikipedia articles from 2023–2024 (post-Pile cutoff), AUC reaches 0.796 (Reference-based, 12B model) compared to ~0.58 for natural non-members from the same temporal range (Table 3 vs. Table 1). This isn't because the model memorized 2020 articles more than the attack revealed—it's because 2023 articles contain different terminology, reference different events, and use different linguistic patterns that make them inherently less similar to the training distribution. The paper shows this concretely: the average 7-gram overlap drops from 39.3% (natural Wikipedia non-members) to 13.9% (temporal Wikipedia non-members), and this shift in overlap distribution (Figure 4) directly parallels the shift in AUC.

The significance extends beyond a methodological caution. If prior work (Shi et al., 2023; Meeus et al., 2023) reported high MIA performance that was partially attributable to distribution shift, then the field's understanding of LLM memorization risk may be systematically miscalibrated—overestimating how easily training data can be detected while underestimating the more subtle forms of leakage that don't respect the exact-match membership boundary. The paper is careful not to claim that prior work is wrong, but rather that the interpretation of those results needs to account for candidate set construction effects that were not previously analyzed.

Table 5 (Appendix) provides the smoking gun: thresholds derived from temporal-shifted benchmarks produce much higher false positive rates when applied to natural in-distribution non-members. For example, a threshold achieving 1% FPR on the temporal Wikipedia benchmark produces a 9.8% FPR on natural Wikipedia non-members (LOSS attack). This means the decision boundary learned on the temporal benchmark is largely separating temporal era rather than membership—a clean demonstration that apparent MIA success can reflect a fundamentally different inference task.

The paper also provides a practical diagnostic: compare the n-gram overlap distribution of candidate non-members against a held-out member sample from the same domain. If the non-member distribution is noticeably shifted toward lower overlap, the benchmark may be measuring distribution shift rather than membership. This gives future researchers a concrete, quantitative check to apply before interpreting MIA results.

Innovation 3: Rethinking Membership for Generative Models—Beyond Exact Match

The paper's third contribution is to explicitly question whether the standard definition of membership—"was this exact sequence in the training data?"—is the right framework for measuring information leakage in generative language models, and to provide initial empirical evidence that the answer may be "no." This is a conceptual challenge to the foundations of membership inference, not an incremental attack improvement.

The standard membership inference game, formalized by Yeom et al. (2018) and used across hundreds of subsequent papers, treats membership as a binary property: a record either is in the training set (member) or is not (non-member), with no intermediate states. This definition works well for structured data (tabular records, images) where records are discrete and overlap is minimal. The paper argues that this definition breaks down for natural language, where "different" documents can share substantial content, and where semantically equivalent paraphrases can convey identical information through entirely different token sequences.

Section 5 provides the empirical evidence. When member samples are modified by replacing a single random token (edit distance 1), the Reference-based attack classifies these modified samples as non-members with near-zero false positive rate at low-FPR thresholds (Table 4, left: 0.1% FPR for edit distance 1 on ArXiv at 1% threshold). When members are paraphrased by GPT-4 to preserve semantics while changing the surface form, LOSS and Reference-based attacks still classify them as non-members in high-confidence regimes (Table 4, right: 0.4% vs. 3.0% FPR for Wikipedia LOSS vs. Ref at 5% threshold). Figure 5 visualizes this: the score distributions of modified members are clearly separable from actual members under the LOSS attack, meaning the attack confidently rejects sequences it "should" recognize as equivalent if the goal is to detect information leakage rather than exact token sequence memorization.

This is not simply "attacks are imperfect." It reveals a fundamental tension: the exact-match membership definition may be too strict for privacy auditing. An adversary who obtains a paraphrase of a training document has still learned the sensitive information contained in that document, even though the paraphrase would be classified as a non-member by all existing MIAs. Conversely, a non-member that shares 40% of its substrings with training data has had most of its content "seen" by the model during training, even though the complete document was held out. The paper argues that "guessing the membership of some sample via other sufficiently close samples can be useful" (Section 5) and that privacy auditors should care about this kind of leakage.

This reframing has practical implications beyond the paper's experiments. If the membership definition is too narrow, then MIAs that correctly implement the standard game may systematically underestimate privacy risk—they correctly identify that a paraphrase wasn't in the training data (true negative under exact-match definition) while missing that the model has effectively memorized the sensitive content through similar training examples (false negative under a semantic-leakage definition). The paper doesn't propose a solution to this problem, but by making the tension explicit and providing quantitative evidence of its magnitude, it opens a new research direction: extending the membership inference game to account for fuzzy, semantic, or neighbor-based membership.

The key conceptual move is recognizing that the "high n-gram overlap" finding from Section 3.2.2 and the "sensitivity to perturbations" finding from Section 5 are two sides of the same coin. High n-gram overlap means non-members are "partially members" from the model's perspective—the model has seen their substrings, so loss-based attacks cannot distinguish them. Extreme perturbation sensitivity means that even tiny changes make members appear non-member-like—the attacks are calibrated to detect exact token sequences, not the information content those sequences carry. Together, these findings suggest that the standard MIA framework is simultaneously too permissive (flagging non-members that share substrings as potentially indistinguishable from members) and too strict (failing to flag near-duplicates of members that an adversary would find equally valuable). This is a fundamental diagnostic insight, not a methodological tweak.

Innovation 4: The Pre-training Regime as a Structural Barrier to Membership Inference

The paper's fourth contribution is identifying that near-one-epoch training on massive datasets—the standard recipe for modern LLM pre-training—constitutes a structural barrier to membership inference that is qualitatively different from the overfitting-based vulnerabilities that classic MIAs exploit. This is not simply "more data makes attacks harder"—it is a claim about the shape of the privacy-vs-overfitting relationship that has not been previously characterized.

Classic MIA research (Yeom et al., 2018; Shokri et al., 2017) established that overfitting is the primary enabler of membership inference: models that overfit their training data (memorizing specific examples rather than learning general patterns) leave detectable traces that membership inference attacks can exploit. The standard prescription for defending against MIAs is to reduce overfitting—through regularization, early stopping, or differential privacy. But LLM pre-training already operates in a regime that should, by this logic, be highly private: massive datasets, single-digit epochs of training (often less than one), and explicit deduplication to remove repeated examples.

The paper's epoch-count experiments (Section 3.2.1) test this logic directly and reveal a more nuanced picture. Using Datablations models (2.8B parameters, fixed 55B token budget, 1–14 epochs), the paper shows that MIA performance increases roughly linearly with epoch count (Figure 2, right). At 1 epoch, attacks are near-random; at 14 epochs, attacks achieve AUC above 0.8. This demonstrates that the near-one-epoch pre-training regime is not just a quantitative reduction in memorization—it is a regime change. The model transitions from a state where membership is essentially undetectable (despite seeing every training example exactly once) to a state where it is highly detectable (after seeing the same examples multiple times), and this transition is gradual and monotonic.

The training data size experiment (Figure 2, left) reveals a second structural dynamic: MIA performance initially spikes early in training (when the data-to-parameter ratio is still low and the model can "overfit" its limited exposure) and then gradually declines as more data is seen and the model generalizes better. This is consistent with the double-descent and grokking phenomena observed in other contexts (Nakkiran et al., 2021) but applied specifically to the membership inference setting. The paper interprets this as evidence that "the data-to-parameter-count ratio is smaller early in training and the model may tend to overfit... but generalizes better as training progresses."

What makes this innovation distinctive is that it identifies a structural property of LLM pre-training—the combination of massive data, near-one-epoch exposure, and progressive generalization—as inherently hostile to membership inference, independent of attack sophistication. Improving attacks (e.g., better calibration, more reference models, meta-classifiers) might shift the absolute numbers but cannot overcome the fundamental fact that a single exposure to each training example, in a model trained primarily for generalization, leaves minimal per-example signal. This is a conceptual contribution that reframes the challenge: the primary barrier to MIA on pre-trained LLMs is not attack weakness but a training regime that systematically minimizes the overfitting that MIAs depend on.

The recency analysis (Appendix C.1, Figure 9) adds a practical dimension: even in the one-epoch regime, recently seen data is more vulnerable, with performance elevated for member sets from the most recent training steps. This has direct implications for fine-tuning, where models are trained for multiple epochs on small domain-specific datasets—exactly the conditions that maximize MIA vulnerability, explaining why prior work on fine-tuned models (Mireshghallah et al., 2022a; Fu et al., 2023) found much higher attack success than the pre-training setting studied here. The paper thus draws a clear distinction: pre-training is relatively safe from MIAs, but fine-tuning may not be, and the boundary between these regimes is governed by epoch count and dataset size rather than model architecture or attack methodology.

Evidence anchoring: Figure 2 (left) shows the non-monotonic trajectory—AUC rises from ~0.5 to ~0.75 within the first few thousand steps, then declines to ~0.55–0.60 by step 99,000. Figure 2 (right) shows the linear epoch effect—AUC increases from ~0.55 at 1 epoch to ~0.85 at 14 epochs for the LOSS attack. Table 1 shows the aggregate result: across all domains and model sizes, no attack exceeds AUC 0.6 except for GitHub (where n-gram overlap is extreme, as discussed in Innovation 2).

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary dataset is the Pile (Gao et al., 2020), specifically its train and test splits. The paper constructs benchmarks for seven individual domains—Wikipedia, GitHub, Pile-CC, PubMed Central, ArXiv, DM Mathematics, HackerNews—plus an aggregate benchmark over the entire Pile. For each domain, 1,000 members are sampled from the Pile training set and 1,000 non-members from the Pile test set (10,000 each for the aggregate Pile benchmark). Documents are filtered to those longer than 100 words, truncated to 200 words from the beginning, and undergo additional 13-gram deduplication against the training set following Groeneveld et al. (2023) using a bloom filter with a ≤80% overlap threshold. For temporal shift experiments, non-members are replaced with temporally-shifted data: Wikipedia articles from 2023–2024 (RealTimeData WikiText dataset; Li et al., 2023c) and ArXiv preprints from successive months after July 2020 (August 2020, January 2021, June 2021, January 2022, June 2022, January 2023, June 2023). For n-gram overlap thresholded experiments, non-members are filtered to retain only those with ≤20% 7-gram overlap with training data. For Datablations experiments, members and non-members are drawn from C4 train and validation sets, respectively. All sampling uses the authors' released MIMIR package with the described preprocessing pipeline.
Base Model(s). The primary target models are the Pythia suite (Biderman et al., 2023b): five PYTHIA-DEDUP models (trained on the deduplicated Pile) with 160M, 1.4B, 2.8B, 6.9B, and 12B parameters, plus comparison against the non-deduplicated PYTHIA models (same sizes, excluding 160M for most experiments). Both suites use the GPT-NeoX architecture and were trained on approximately 300B tokens from the Pile. The PYTHIA-DEDUP models are evaluated at checkpoint 'step99000,' the closest checkpoint to one full epoch over the deduplicated Pile. For cross-architecture validation, the paper also targets GPT-Neo models (125M, 1.3B, 2.7B parameters; Black et al., 2021) and OLMo models (1B, 7B; Groeneveld et al., 2024). For the epoch count experiments, the paper targets Datablations 2.8B-parameter models trained on 55B C4 tokens for 1–14 epochs (Muennighoff et al., 2023) and SILO-PDSW 1.3B-parameter models trained on the Open License Corpus (Min et al., 2023). For the training data size and recency experiments, intermediate checkpoints of PYTHIA-DEDUP models (every 5,000 steps from step 1,000 to step 99,000, with each step processing 1,024 sequences of 2,048 tokens) are used to track MIA performance as training progresses. The choice of Pythia is driven by the availability of exact training data, training order, and intermediate checkpoints—properties not available for most other model families.
Metrics. The primary metric is AUC ROC (area under the receiver operating characteristic curve), measuring the attack's ability to separate member and non-member score distributions across all possible thresholds. AUC ROC of 0.5 indicates random guessing; 1.0 indicates perfect separation. The secondary metric is TPR@low%FPR (true positive rate at low false positive rate), reporting the fraction of members correctly identified when the false positive rate is constrained to 1%, 5%, or 10%. This captures performance in high-confidence regimes relevant to privacy auditing. Both metrics are computed with 95% confidence intervals via 1,000 bootstrap resamples of the benchmark (resampling members and non-members with replacement, recomputing the metric on each bootstrap, and taking the 2.5th and 97.5th percentiles). Shaded regions in figures represent these confidence intervals.
Baselines. The paper does not propose new attacks but evaluates five existing MIAs against each other. The attacks are: (1) LOSS (Yeom et al., 2018), (2) Reference-based (Sablayrolles et al., 2019; Watson et al., 2022), (3) Zlib Entropy (Carlini et al., 2021), (4) Neighborhood (Mattern et al., 2023), and (5) Min-k% Prob (Shi et al., 2023). There is no single "baseline attack" per se; rather, all five attacks are compared to the random-guessing baseline of AUC 0.5, and the Reference-based attack (using STABLELM-BASE-ALPHA-3B-V2 as the reference model, chosen via ablation in Table 6) generally serves as the strongest or near-strongest method in most settings.
Generation Budget / Compute Accounting. This paper studies membership inference attacks against language models, not inference-time compute strategies for language models. There is no generation budget in the sense of beam search or sampling. Instead, the relevant compute metric is the number of model queries: the LOSS attack requires one forward pass per candidate sequence; the Reference-based attack requires two (target + reference model); the Neighborhood attack requires $n+1$ (one for the original, $n$ for neighbors, where $n$ is the number of neighbor sequences generated per candidate); Min-k% Prob and Zlib require one forward pass each but with additional post-processing (sorting token probabilities, computing compression size). All attacks are evaluated on the same 1,000-member/1,000-non-member benchmarks, making query counts comparable within each (attack, model) pair. For the Neighborhood attack, $n$ is implicitly determined by the 5% masking rate applied to the sequence, varying with sequence length. The paper does not explicitly report total compute in FLOPs or GPU-hours but releases all code and data for reproducibility.
Cross-validation / Statistical Protocol. No k-fold cross-validation is used for attack evaluation, since the attacks do not involve training or hyperparameter selection on the test data. The primary statistical protocol is bootstrap resampling: 1,000 bootstrap samples of the benchmark are drawn, metrics are recomputed on each sample, and confidence intervals are derived from the bootstrap distribution. For reference model selection, the paper performs an exhaustive ablation across eight candidate reference models (Table 6) and selects STABLELM-BASE-ALPHA-3B-V2 based on its consistently highest performance across domains and model sizes. This selection is based on the same data used for evaluation, but since the reference model is a fixed external model (not trained on the Pile test set), this does not constitute test-set leakage. The paper acknowledges that reference model selection remains "challenging and largely empirical." For the temporal shift experiments, the paper uses fixed thresholds derived from the temporally-shifted benchmarks and evaluates them on natural non-members (Table 5), which provides an independent validation of the distribution shift hypothesis without requiring cross-validation.

Main Quantitative Results

Aggregate MIA Performance Across All Domains and Model Sizes

The headline finding across Table 1 and Table 11 is that all five MIAs perform near-random against pre-trained Pythia models in most domains. For the PYTHIA-DEDUP models:

Table 1 reports AUC ROC across seven domains, five attacks, and six model sizes (70M through 12B). In Wikipedia, the best attack (Reference-based on 12B) achieves only 0.579 AUC, with all other attacks and model sizes at or below 0.524. In Pile-CC, the best performance is 0.582 (Reference-based, 12B) with most entries between 0.49 and 0.52. In PubMed Central, the best is 0.559 (Reference-based, 12B) with most entries near 0.50. In ArXiv, the best is 0.555 (Reference-based, 12B). In DM Mathematics, performance is particularly poor—the best is 0.512 (Reference-based, 12B) and some entries fall below 0.5 (e.g., Zlib on 6.9B at 0.481, Neighborhood on 2.8B at 0.467), meaning the attack is systematically wrong more often than random guessing. In HackerNews, Reference-based on 12B reaches 0.565, with earlier model sizes below 0.55. The aggregate Pile benchmark shows 0.558 (Reference-based, 12B). The GitHub domain stands as a notable exception, with Zlib achieving 0.690 on 12B and 0.717 on 2.8B—substantially above the 0.6 threshold that the paper uses to distinguish meaningful from near-random performance. The paper attributes this to GitHub's uniquely high n-gram overlap, which makes the natural non-member set contain samples that are substantially out-of-distribution relative to the training data (Appendix B.3, Figure 7).

Table 11 reports TPR@1%FPR for the same experiments. In most domains, TPR@1%FPR is below 2%, often below 1%, indicating that even when attacks have slight discriminative power (AUC slightly above 0.5), they cannot confidently identify members without also falsely accusing many non-members. For Wikipedia with 12B, TPR@1%FPR ranges from 0.6% (min-k, Ref) to 1.0% (Ne). For ArXiv, it ranges from 0.5% (zlib) to 2.5% (Ref). GitHub again stands out, with TPR@1%FPR reaching 17.4% (zlib, 12B) and 23.4% (zlib, 2.8B). The low TPR@low%FPR values suggest that even in domains where attacks achieve moderate AUC, the separation between member and non-member score distributions is concentrated at higher score ranges, making confident (low-FPR) predictions rare.

Effect of Model Size and Deduplication

MIA performance increases with model size, but the effect is modest. For Wikipedia (Reference-based attack, PYTHIA-DEDUP), AUC rises from 0.504 (160M) to 0.515 (1.4B) to 0.565 (2.8B) to 0.571 (6.9B) to 0.579 (12B)—a gradual, monotonic increase of only 0.075 AUC across a 75× increase in parameters (Table 1). Similar gradual increases appear in Pile-CC (0.487→0.564), PubMed Central (0.516→0.559), ArXiv (0.486→0.555), and HackerNews (0.490→0.565). Figure 1 visualizes this trend for Reference-based attack on Wikipedia, ArXiv, and HackerNews, with near-overlapping curves for most model families. The paper attributes this to the well-known observation that "larger models are more prone to overfitting the training data" (Nakkiran et al., 2021), but the magnitude of the effect is small enough that even the 12B model cannot reliably distinguish members from non-members.

Deduplication reduces MIA performance. Figure 1 shows that the non-deduplicated Pythia models consistently yield higher AUC than their deduplicated PYTHIA-DEDUP counterparts at the same parameter count (e.g., Wikipedia Reference-based: 0.639 vs. 0.579 at 12B; Pile-CC: 0.567 vs. 0.582 at 12B—note that this particular comparison shows a reversal, but the overall trend across most domains and model sizes favors non-deduplicated models). Table 12 provides the full non-deduplicated Pythia results: Wikipedia Reference-based AUC reaches 0.639 (12B) vs. 0.579 for deduplicated; Pile-CC Reference-based reaches 0.567 vs. 0.582 (a small reversal); ArXiv Reference-based reaches 0.575 vs. 0.555; HackerNews reaches 0.606 vs. 0.565. The consistency of this pattern across most settings confirms Kandpal et al. (2022)'s finding that deduplication mitigates memorization. Appendix Figure 12 shows that this deduplication effect holds across multiple attack types (LOSS, Min-k%, Zlib), not just the Reference-based attack.

Cross-Architecture Validation: GPT-Neo and OLMo

GPT-Neo models reproduce the near-random performance pattern. Table 7 reports AUC ROC for GPT-Neo models across Pile domains using four attacks (Neighborhood excluded). For Wikipedia with the 2.7B model, Reference-based reaches 0.545; for Pile-CC, 0.531; for ArXiv, 0.520; for PubMed Central, 0.507. These are similar in magnitude to the Pythia results (slightly lower in some domains). GitHub again shows elevated performance (2.7B: 0.700 Min-k%, 0.712 zlib). DM Mathematics and HackerNews show near-random performance (0.481–0.516 range). The aggregate Pile benchmark reaches 0.519 (Reference-based, 2.7B). These results demonstrate that the near-random MIA performance is not an artifact of the GPT-NeoX architecture or Pythia-specific training choices.

OLMo models also show near-random performance, with some results below random. Table 8 reports AUC ROC for OLMo-1B and OLMo-7B across DOLMA domains. For Wikipedia, the best attack (min-k on 7B) reaches 0.493—below 0.5. For C4, min-k on 1B reaches 0.520. For Reddit, no attack exceeds 0.501. For Common Crawl, min-k on 1B reaches 0.517. Notably, the Reference-based attack (using STABLELM-BASE-ALPHA-3B-V2) performs substantially worse on OLMo than on Pythia—e.g., 0.412 on Common Crawl for 1B, 0.376 on s2ag for 1B—which the paper interprets as reinforcing "the difficulty in finding suitable reference models for different LLMs." Several domain-attack combinations yield AUC considerably below 0.5 (e.g., s2ag LOSS at 0.449 for 1B, zlib at 0.392), indicating the attacks are systematically inverted—predicting members as non-members and vice versa. The paper notes this counter-intuitive behavior but does not fully explain it, speculating that further investigation is needed.

The Effect of Training Data Size: Intermediate Checkpoint Analysis

MIA performance follows a non-monotonic trajectory as training data size increases. Figure 2 (left) shows the Reference-based attack AUC as a function of training steps (equal to cumulative data seen) for PYTHIA-DEDUP models at 1.4B, 2.8B, 6.9B, and 12B. The pattern is consistent across model sizes (with the 2.8B model as a notable exception—see below): performance starts near 0.5 at step 1,000, rises sharply to a peak between steps 5,000–15,000 (reaching approximately 0.75 for 12B, 0.80 for 6.9B, 0.72 for 1.4B), and then gradually declines to approximately 0.55–0.60 by step 99,000. The paper speculates that "the initial low performance is due to the model warming up in training, with high losses across both member and non-member samples," with the subsequent peak reflecting a phase where the "data-to-parameter-count ratio is smaller early in training and the model may tend to overfit," and the final decline reflecting progressive generalization as more data is seen. Appendix Figure 13 shows that LOSS, Min-k%, and Zlib attacks follow similar trajectories. The 2.8B model is anomalous—its AUC remains near 0.5 throughout training, showing essentially no membership signal at any checkpoint—a finding the paper cannot explain but notes is consistent with "unexplainable behavior for this model" reported by Biderman et al. (2023a) for extractability.

The recency of member data impacts MIA performance, but the effect is modest and saturates. Appendix Figure 9 shows MIA performance when the target model is fixed at step 99,000 but member sets are drawn from different training windows. For the Reference-based attack on 12B, AUC drops from approximately 0.72 (members from the most recent 100 steps) to approximately 0.60 (members from steps 0–100), with most of the decline occurring in the first 20,000 steps and a plateau thereafter. The paper interprets this as evidence that "recently seen data is more vulnerable" and links it to known forgetting effects (Jagielski et al., 2023). Smaller models show steeper declines, suggesting that "larger models having more parameters, allowing them to capture more seen data before having to drop older knowledge." This recency effect has practical implications for fine-tuning: "data seen during fine-tuning or continued pre-training may also be increasingly vulnerable" (Appendix C.1).

The Effect of Training Epochs: Datablations and SILO

Multi-epoch training substantially increases MIA performance. Figure 2 (right) shows MIA performance against Datablations 2.8B-parameter models trained on a fixed 55B-token C4 subset for 1 to 14 epochs. For the LOSS attack, AUC increases linearly from approximately 0.55 (1 epoch) to approximately 0.85 (14 epochs). The Reference-based attack follows a similar linear trend, reaching AUC above 0.9 at 14 epochs. Min-k%, Zlib, and Neighborhood attacks also improve monotonically with epoch count, though with different slopes. The linear relationship is striking: each additional epoch compounds the memorization signal, directly supporting the paper's hypothesis that near-one-epoch pre-training is a primary structural barrier to membership inference. If standard LLM pre-training used multiple epochs (as is common in fine-tuning), the paper implies that MIAs would be substantially more effective.

In a more realistic multi-epoch setting (SILO), results are domain-dependent. Appendix Figure 10 shows MIA performance against SILO-PDSW intermediate checkpoints for HackerNews and DM Mathematics, which are upsampled 3× in this model variant. For HackerNews (Reference-based attack), AUC increases from approximately 0.62 at 2 effective epochs to 0.72 at 5 epochs, then plateaus or slightly declines to ~0.68 at 9 epochs. The paper speculates this plateau occurs because "the target model begins to memorize less of the HackerNews samples" as training progresses, possibly because the 1.3B-parameter model has limited capacity and "may tend to overfit data more so from domains with greater representation." For DM Mathematics, the trend is counter-intuitively negative: AUC decreases from approximately 0.53 at 2 epochs to 0.48 at 9 epochs for the Reference-based attack, with other attacks showing similar declines. The paper speculates this is because "the abundance of tokens from a concentrated token space (i.e., digits, variables) that are largely symbolic rather than semantic makes memorization of specific samples unlikely," and that the model "simply fails to perform well on such data even after multiple epochs."

The Role of n-gram Overlap Between Members and Non-Members

Non-members share substantial substring content with training data. Figure 3 shows the distribution of 7-gram overlap of non-member samples against the Pile training set across domains. The distributions are right-skewed and concentrated at moderate-to-high overlap values: Wikipedia non-members have a mean 7-gram overlap of 32.5%, with a mode around 20–30%; ArXiv has a mean of 39.3%, with a peak around 30–50%; HackerNews has a mean of 39.3% (estimated from the figure, since exact means are listed only for some domains in the text), with a broad distribution from 10–60%. GitHub shows substantially higher overlap, with a mean of 76.9% and a distribution concentrated above 60%. The paper also reports means for PubMed Central (41.0%), DM Mathematics (72.8%), and FreeLaw (62.3%), though distributions for the latter two are not plotted. Appendix Figure 14 shows that these patterns hold across n-gram sizes n=4, 7, and 13, with distributions shifting leftward (lower overlap) as n increases—longer n-grams are less likely to appear in both members and non-members by chance. Even at n=13, substantial fractions of non-members have 10–30% overlap with training data.

Artificially reducing n-gram overlap dramatically improves MIA performance. Table 2 compares MIA performance on the natural non-member sets versus non-member sets filtered to ≤20% 7-gram overlap, targeting PYTHIA-DEDUP-12B. For Wikipedia, LOSS AUC increases from 0.516 to 0.666, Reference-based from 0.579 to 0.677, and zlib from 0.524 to 0.631. For GitHub, the increases are even larger: LOSS from 0.678 to 0.878, Reference-based from 0.559 to 0.615, min-k from 0.683 to 0.890, and zlib from 0.690 to 0.908. For PubMed Central, LOSS increases from 0.506 to 0.780, min-k from 0.512 to 0.792, and zlib from 0.506 to 0.772. For ArXiv, LOSS increases from 0.527 to 0.787, Reference-based from 0.555 to 0.715, and min-k from 0.530 to 0.734. These are large effect sizes—AUC improvements of 0.15 to 0.30—confirming that n-gram overlap is a first-order factor suppressing MIA performance. The paper notes that this thresholding "introduces an explicit drift between member and non-member distributions" and is "not a suggestion for researchers to alter their benchmarks."

Reference-based attacks show smaller improvements under n-gram thresholding for some domains. Table 2 shows that for GitHub, Reference-based AUC increases from 0.559 to only 0.615, compared to LOSS (0.678→0.878) and zlib (0.690→0.908). Similarly, for PubMed Central, Reference-based increases from 0.559 to 0.595, versus LOSS (0.506→0.780). The paper speculates that "since numerous low n-gram overlap non-members are outliers to the relevant domain, these non-members will also be outliers to the similar/overlapping data seen by the reference model," causing the reference model to also assign high loss to these samples and canceling out the calibration benefit (Appendix B.2). This is an important diagnostic finding: calibration-based attacks can be less sensitive to distribution shift than calibration-free attacks, because both target and reference models respond similarly to out-of-distribution samples, reducing the differential signal.

Temporal Shift as a Confound in MIA Evaluation

Temporally-shifted non-members yield substantially higher MIA performance than natural non-members. Table 3 reports AUC ROC against PYTHIA-DEDUP models on the temporal Wikipedia benchmark (members: pre-March 2020; non-members: August 2023–January 2024). For the Reference-based attack on 12B, AUC reaches 0.796—compared to 0.579 for the natural Wikipedia non-members from the same period (Table 1). Min-k% achieves 0.719 (vs. 0.517 natural), and LOSS achieves 0.680 (vs. 0.516 natural). Performance increases monotonically with model size: Reference-based AUC rises from 0.602 (160M) to 0.705 (1.4B) to 0.754 (2.8B) to 0.788 (6.9B) to 0.796 (12B).

Temporal shift corresponds to a shift in n-gram overlap distribution. Figure 4 compares the 7-gram overlap distributions of natural Wikipedia non-members (mean: 39.3%) and temporal Wikipedia non-members (mean: 13.9%). The temporal non-member distribution is concentrated at much lower overlap values, with a sharp peak near 5–10% compared to the natural distribution's peak near 25–35%. This shift in overlap distribution provides a mechanistic explanation for the elevated MIA performance: temporal non-members are intrinsically less similar to the training data at the substring level, making them easier to distinguish from members regardless of memorization.

The degree of temporal shift correlates with MIA performance. Figure 6 shows MIA performance against PYTHIA-DEDUP-12B for ArXiv benchmarks where non-members are drawn from successive months after the Pile cutoff (July 2020). The Reference-based attack AUC increases from 0.723 (non-members from August 2020) to 0.746 (January 2021) to 0.763 (June 2021) to 0.779 (January 2022) to 0.789 (June 2022) to 0.794 (January 2023) to 0.795 (June 2023)—a consistent upward trend. LOSS, Min-k%, and Zlib show similar monotonic increases. However, the n-gram overlap distribution shift is relatively small across these temporal windows: the mean 7-gram overlap drops from 22.7% (August 2020) to 20.5% (June 2023), a difference of only 2.2 percentage points (Appendix B.4, Figure 8). The paper notes that this small overlap change "doesn't clearly justify the considerable difference in MIA performance" and speculates that other factors—"changes in the distribution of topics (i.e., increasing popularity of research into LLMs) and the presence of specific identifying tokens (i.e., dates, references, new terminology)"—may contribute, further reinforcing the complexity of interpreting temporal shift effects.

Thresholds learned on temporal benchmarks do not transfer to natural non-members. Table 5 provides the critical evidence that temporal benchmarks measure something other than genuine membership. For the Reference-based attack on PYTHIA-DEDUP-12B, a threshold achieving 1% FPR on the temporal Wikipedia benchmark produces a 7.5% FPR on natural Wikipedia non-members. For the temporal ArXiv benchmarks, thresholds achieving 1% FPR produce FPRs of 4.2% to 9.4% on natural non-members, increasing with temporal distance. At 5% FPR thresholds, natural FPRs reach 13.4% to 23.5% (Reference-based), and at 10% FPR thresholds, they reach 23.3% to 37.3%. The paper concludes that "decision thresholds derived using temporally-shifted non-members end up testing for temporal shift rather than membership."

Modified Members: Sensitivity to Lexical and Semantic Perturbations

MIAs are extremely sensitive to small lexical perturbations of member samples. Table 4 (left) reports FPR on modified members (treated as non-members) when using thresholds derived from the natural member/non-member distributions, targeting PYTHIA-DEDUP-12B. For the Reference-based attack on ArXiv, replacing a single random token (edit distance 1) produces an FPR of 0.1% at the 1% threshold, 0.3% at 5%, and 0.7% at 10%. For edit distance 10: 0.0% at 1%, 0.1% at 5%, and 0.3% at 10%. For edit distance 25: 0.0% at 1%, 0.1% at 5%, and 0.2% at 10%. The LOSS attack produces 0% FPR across all edit distances and thresholds on both ArXiv and Wikipedia. This means that even a single-token substitution causes the attack to confidently classify the modified member as a non-member—the attack is detecting exact token sequence memorization, not loose content familiarity. Figure 5 (top) visualizes this: the score distribution of modified members (edit distance 25) is clearly shifted relative to both members and non-members for the LOSS attack, while the Reference-based attack partially recalibrates (the distribution of modified members overlaps more with non-members, but still sits between the two groups).

Semantically close members generated via BERT masking are also classified as non-members, though less definitively. Table 4 (right) reports FPR on semantically close modified members (5% of tokens masked and replaced by BERT, the same procedure used in the Neighborhood attack). For the Reference-based attack on Wikipedia, FPR at a 1% threshold is 0.4%, rising to 3.0% at 5% and 8.2% at 10%. For LOSS on Wikipedia: 0.0% at 1%, 0.5% at 5%, and 2.3% at 10%. For ArXiv: LOSS yields 0.0% at 1%, 0.8% at 5%, and 2.5% at 10%; Reference-based yields 0.7% at 1%, 1.9% at 5%, and 4.0% at 10%. These FPRs are higher than for random-token replacement, indicating that semantically close neighbors are more "member-like" than lexically perturbed ones, but the FPRs are still substantially below what the natural member/non-member distributions would predict. Figure 5 (bottom) shows that the score distributions of semantic neighbors overlap with members more than lexically modified samples do, especially for the Reference-based attack, but the separation is still clear. The paper interprets this as evidence that "semantically close members would be classified as non-members even though they may be as useful as actual members depending on the inference goal."

GPT-4 paraphrased members behave similarly to BERT-masked semantic neighbors. Appendix D (Figure 11, Table 10) extends the analysis to members paraphrased by GPT-4 with instructions to preserve specific details while changing style. For the LOSS attack on Wikipedia, FPR at 1% threshold is 0.0%, rising to 0.0% at 5% and 0.2% at 10%. For Reference-based on Wikipedia: 0.0% at 1%, 0.2% at 5%, and 0.7% at 10%. For HackerNews (LOSS): 0.1% at 1%, 1.1% at 5%, and 1.7% at 10%. Figure 11 visualizes the score distributions: GPT-4 paraphrased members occupy an intermediate position, overlapping with both member and non-member distributions but clearly distinguishable from actual members under both LOSS and Reference-based attacks. The paper notes that these paraphrases have "significant lexical difference" from the originals, reinforcing the finding that "current MIAs are hypersensitive to exact token sequences and miss semantically meaningful leakage."

Ablation Studies and Robustness Checks

Reference model choice matters substantially, and naive ensembling of reference models does not help. Table 6 reports AUC ROC for the Reference-based attack on PYTHIA-DEDUP models across seven domains using eight different reference models. For Wikipedia on 12B, AUC ranges from 0.498 (DistilGPT2) to 0.579 (STABLELM-BASE-ALPHA-3B-V2)—a spread of 0.081. GPT-2 (0.514), OPT-1.3B (0.522), GPT-Neo-1.3B (0.528), SILO-PDSWBY (0.529), LLaMA-7B (0.546), and Pythia-1.4B non-deduped (0.517) all underperform the chosen reference model. Similar patterns hold across domains: STABLELM-BASE-ALPHA-3B-V2 is the best or near-best in most settings, but not universally (e.g., for DM Mathematics on 12B, SILO-PDSWBY at 0.485 and OPT at 0.487 are competitive with STABLELM at 0.485). The paper experiments with aggregating all reference models by averaging their calibrated scores, but notes that this "naive ensembling method" performs poorly, likely "due to the reference models having different tokenizers," making loss magnitudes incomparable. The ablation demonstrates that reference model selection is a critical, non-trivial hyperparameter with no clear automated solution.
Min-k% Prob's k parameter is chosen as 20% after sweeping {10, 20, 30, 40, 50}. The paper states that it "experiment[s] with multiple different k... but settle[s] on k = 20 for our experiments" (Appendix A.4). No sweep results are shown, so the sensitivity of the attack to this hyperparameter is not quantified. The choice of k=20% represents a tradeoff: "too small a k and the attack is noisy (averaging over very few tokens); too large a k and the attack converges to the standard LOSS attack."
GitHub is an outlier domain due to inherently high n-gram overlap and decontamination limitations. The paper notes that GitHub consistently shows higher MIA performance than other domains (Table 1: AUC up to 0.717), but attributes this to benchmark construction rather than genuine membership detection. Appendix Figure 7 shows that the natural GitHub non-member set has a heavily right-skewed 7-gram overlap distribution (mean: 76.9%), and that the ≤80% 13-gram overlap decontamination threshold "only captures a small percentile of non-members as GitHub is naturally very high overlap." The paper speculates that "the repetitive nature of code, such as copyright notices, function definitions, and syntax like HTML tags" contributes to this high overlap. Appendix Figure 15 provides a concrete example of a non-member outlier captured by the decontamination threshold: a Burmese-language news article from a language resource repository—clearly out-of-distribution for a code-dominant domain. The n-gram overlap thresholding experiment (Table 2) shows that further restricting non-members to ≤20% 7-gram overlap inflates GitHub AUC to 0.908 (zlib), confirming that the elevated natural performance is driven by distribution shift between the natural non-member set and the member distribution, exacerbated by the inadequate decontamination threshold for high-overlap domains.
The 2.8B PYTHIA-DEDUP model is anomalous across multiple experiments. Figure 2 (left) shows that the 2.8B model's MIA performance remains near 0.5 throughout training, unlike the 1.4B, 6.9B, and 12B models that show clear non-monotonic trajectories. Appendix Figure 13 confirms this holds across LOSS, Min-k%, and Zlib attacks. The paper notes that "previous work also observes unexplainable behavior for this model" (Biderman et al., 2023a) but does not further investigate the cause. This anomaly does not undermine the paper's overall conclusions (which are supported by the other three model sizes and cross-validated on GPT-Neo), but it raises questions about the reliability of this specific model size for privacy-related analyses.
The Neighborhood attack uses a 5% masking rate with BERT as the masking model. No ablation is reported for the masking percentage or the choice of masking model. The paper selects BERT (Devlin et al., 2019) and a 5% masking percentage following Mattern et al. (2023). The computational cost of the Neighborhood attack—requiring n+1 model queries per candidate, where n neighbors are generated per sample—is not quantified, but the paper notes it is "more expensive but potentially more informative" than other attacks.
Results on OLMo models (trained on DOLMA with 3T/2.5T tokens) suggest even larger training sets may further suppress MIA performance. Table 8 shows that OLMo-7B (trained on 2.5T tokens, far more than Pythia's ~300B) produces AUC values near or below 0.5 for many domains (e.g., Wikipedia: 0.481–0.500 across attacks; Reddit: 0.463–0.501; s2ag: 0.465–0.507). The paper speculates that "due to the incredibly large amounts of training data... performance across different model sizes begins to converge to near-random performance even with such distinct model sizes." The Reference-based attack (with STABLELM-BASE-ALPHA-3B-V2) performs much worse on OLMo than on Pythia (e.g., Common Crawl: 0.410–0.412 vs. 0.564–0.582), which the paper interprets as evidence that the reference model's training data may not adequately cover the DOLMA distribution. These OLMo results are preliminary (only final checkpoints, not intermediate), but they suggest that the structural barriers to MIA identified in this paper may intensify at larger training scales.
Recency of member data has a larger effect on smaller models. Appendix Figure 9 shows that for the LOSS and Min-k% attacks, the performance difference between recently-seen and earlier-seen members is more pronounced for 1.4B and 2.8B models than for 6.9B and 12B models—the curves for smaller models drop more steeply as member recency decreases. The paper interprets this as evidence that "larger models having more parameters, allowing them to capture more seen data before having to drop older knowledge."

Critical Assessment

The paper's central claims are that (1) existing MIAs perform near-random against pre-trained LLMs in most settings, (2) this poor performance is attributable to the combination of massive single-epoch training and inherent n-gram overlap between members and non-members, (3) when MIAs appear to succeed, this can often be attributed to distribution shift in non-member candidate selection, and (4) the standard membership definition may need to be extended for generative models. The experimental design provides substantial evidence for each claim, but several important caveats limit the strength and generality of the conclusions.

Claim 1: "MIAs barely outperform random guessing for most settings across varying LLM sizes and domains." The evidence in Table 1 strongly supports this for the Pythia model suite on the Pile domains tested. No attack achieves AUC above 0.6 in any domain except GitHub (and marginally ArXiv at 0.555 Reference-based on 12B), and most values cluster tightly around 0.5. The TPR@low%FPR results (Table 11) reinforce this: even where AUC is slightly elevated, confident predictions are rare. However, the scope of this claim is limited to the specific threat model (black-box access, no meta-classifiers, no shadow models) and the specific models tested. The paper acknowledges that stronger attacks—white-box access to gradients, training multiple shadow models as in LiRA (Carlini et al., 2022), or training meta-classifiers on ground-truth member/non-member subsets—could potentially achieve higher performance, and explicitly excludes these from evaluation due to computational infeasibility or impracticality. The claim should therefore be understood as "black-box, single-model MIAs barely outperform random guessing" rather than a universal statement about all possible MIAs. The GPT-Neo replication (Table 7) provides cross-architecture validation, but only on the same Pile training data—it does not test whether the finding generalizes to models trained on substantially different corpora (e.g., multilingual data, code-heavy data, curated high-quality data). The OLMo results (Table 8) begin to address this gap but are presented as preliminary and include only two model sizes, both trained on DOLMA.

Claim 2: "Poor performance can be attributed to (a) the combination of a large dataset and few training iterations, and (b) an inherently fuzzy boundary between members and non-members." The evidence for (a) comes from two complementary experiments: the intermediate checkpoint analysis (Figure 2 left) showing that MIA performance peaks early and then declines as more data is seen, and the epoch count experiment (Figure 2 right) showing a linear increase in MIA performance with additional epochs. Together, these convincingly demonstrate that the one-epoch, large-data regime suppresses MIA signals. However, the intermediate checkpoint experiment conflates two variables: the amount of data seen and the recency of member data. The paper attempts to control for recency by sampling members from the most recent 100 steps at each checkpoint, but this introduces a different confound—the members themselves change across checkpoints, so the difficulty of the membership inference task may not be constant. The recency experiment (Appendix Figure 9) partially addresses this by fixing the target model, but it cannot disentangle whether the declining MIA performance over training is due to generalization (the paper's preferred interpretation) or simply because earlier training data is less well-fitted (which would also reduce the loss difference between members and non-members). A cleaner experiment would require injecting a fixed set of canary sequences at different points in training and measuring their detectability at the final checkpoint—but this is impossible with Pythia's fixed training order. The SILO experiment (Appendix Figure 10) provides a more ecologically valid test of epoch effects in a realistic training setting, but the counter-intuitive results (DM Mathematics decreases with more epochs) suggest that the relationship is more complex than "more epochs → more detectable."

The evidence for (b) is the strongest part of the paper. The n-gram overlap distributions (Figure 3) quantify the overlap phenomenon, and the thresholding experiment (Table 2) demonstrates that reducing overlap dramatically improves MIA performance—a clean causal intervention. The effect sizes are large (AUC improvements of 0.15–0.30), and the pattern holds across domains and attacks. The primary limitation is that the n-gram overlap measurement infrastructure uses a bloom filter with a small but non-zero false positive rate (0.6%), meaning the reported overlap percentages are slight overestimates. Additionally, the bloom filter checks for n-gram inclusion anywhere in the training data, not in a single training document—an n-gram that spans two different training documents would be counted as present even though the model never saw that specific n-gram as a contiguous substring. This overcounting likely has minimal impact on the conclusions (since the overlap is large even with this overcount), but it means the reported overlap percentages should be treated as upper bounds.

Claim 3: "When LLMs have been shown to be vulnerable to MIAs, this apparent success can be attributed to a distribution shift." The evidence is strong but limited to a specific type of distribution shift (temporal). The temporal shift experiments (Section 4) convincingly demonstrate that using post-cutoff non-members inflates AUC substantially (Tables 3 vs. 1), and that this is correlated with reduced n-gram overlap (Figure 4). The threshold transfer experiment (Table 5) provides the critical evidence that temporal benchmarks are measuring something different from genuine membership. However, the paper is careful to note that it was unable to reproduce the exact settings of Shi et al. (2023) and Meeus et al. (2023) due to unavailable data or training corpora (Appendix A.3.2), and therefore cannot conclude that those specific works' findings are solely attributable to temporal shift—only that temporal shift is a plausible confound that could inflate results. The paper's temporal experiments use different models (Pythia vs. LLaMA in Shi et al.) and different non-member construction procedures, so the comparison is illustrative rather than a direct refutation. Additionally, the paper does not systematically explore other forms of distribution shift beyond temporal—data source shift, topic shift, style shift—that could similarly inflate MIA performance. The diagnostic framework (compare n-gram overlap distributions) is proposed as a general tool, but it is validated only for temporal shift.

Claim 4: "Existing MIAs are highly sensitive to even small changes in a sample... which may be at odds with leakage that privacy auditors care about." The evidence in Section 5 and Appendix D is consistent and compelling for the specific perturbation types tested. Single-token replacement causes MIAs to confidently classify modified members as non-members (Table 4 left), and GPT-4 paraphrases—which preserve semantic content—are similarly misclassified (Table 10, Figure 11). The score distribution visualizations (Figure 5, Figure 11) show clear separation between modified members and original members. However, the paper does not test whether this sensitivity is specific to loss-based attacks or applies equally to all black-box MIAs. The experiments use thresholds derived from the natural member/non-member distributions, which means the FPR values in Tables 4 and 10 depend on the specific threshold chosen. A more comprehensive analysis would report the full ROC curve for modified members—treating them as a separate class and measuring how well the attack can distinguish modified members from actual members across all thresholds. The paper's framing of this as "rethinking membership" is forward-looking and provocative, but the experiments themselves demonstrate only sensitivity of existing MIAs, not a systematic framework for evaluating fuzzy membership. The paper does not propose a concrete alternative membership definition, nor does it measure how "useful" a paraphrase or semantically close neighbor actually is to an adversary—the claim that such samples "may be as useful as actual members" is intuitive but not empirically substantiated.

Notable gaps in the experimental design:

No systematic evaluation of sequence length effects. Shi et al. (2023) found that sample length correlates with MIA performance. The paper bounds length to 100–200 words to "reduce its impact," but this also limits the generality of the findings—longer documents (which contain more signal) might yield higher MIA performance, and the paper does not quantify how much. The claim that "inherent differences in LLM training and MI evaluation would still impact evaluation on longer texts" is stated but not tested.
Limited model families tested for the main claims. The primary results are on Pythia (GPT-NeoX architecture, Pile training data). While GPT-Neo (Table 7) and OLMo (Table 8) provide cross-validation, the paper does not evaluate open-weight models from other major families (LLaMA, Mistral, Falcon) that have different architectures, tokenizers, and training data mixtures. This is partly a data availability constraint—the paper's diagnostic experiments require known training data and intermediate checkpoints, which few model families provide—but it means the findings may not generalize to models trained on curated, high-quality, or multilingual data.
The difficulty estimation through n-gram overlap, while elegant, requires computing a bloom filter over the entire training set—a computationally expensive operation that is not part of the attack itself but an analysis tool. This does not affect the validity of the paper's conclusions (since the n-gram analysis is post-hoc, not part of the attacks), but it means future researchers cannot cheaply replicate the diagnostic for their own models and datasets without access to the full training data and significant compute.
No evaluation of user-level or document-level membership inference. All experiments test sample-level membership (was this specific 200-word chunk in the training data?). The paper acknowledges that "user-level leakage" (Kandpal et al., 2023) is an important extension but does not evaluate it. User-level MIAs (testing whether any document from a given user was in the training data) might be more successful if users contribute multiple documents, creating a stronger aggregate signal.
The accessibility of the MIMIR benchmark, while a strength, also means the paper evaluates on a fixed, known test set. If future attacks overfit to this specific benchmark (e.g., by tuning hyperparameters against the known member/non-member splits), apparent improvements may not generalize. The paper does not discuss benchmark contamination or propose holdout sets for future attackers.

Overall, the paper's experimental contributions are substantial: a large-scale, systematic, and well-controlled evaluation with diagnostic experiments that isolate causal factors, rigorous statistical methodology, and thoughtful ablations. The main limitations are scope (single primary model family, specific black-box threat model, bounded sequence lengths) rather than internal validity. The paper makes these limitations explicit and positions the work as a foundation for future research rather than a final word, which is appropriate given the rapidly evolving LLM landscape.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For, Making the Proposed Solution Impractical Without Substantial Further Work

The assumption or constraint. While this paper identifies n-gram overlap as a key diagnostic signal explaining MIA failure, it does not propose a deployable attack that uses this insight. The n-gram overlap measurement infrastructure—constructing a bloom filter over the entire Pile training set, sharding it across memory, and querying it for every candidate sequence—is a post-hoc analysis tool, not a practical attack component. The paper explicitly notes this in Section 3.2.2 when describing the n-gram thresholding experiment: "We clarify that this step is not a suggestion for researchers to alter their benchmarks; such a processing step drifts away from the standard membership inference game." Similarly, the paper's core finding—that MIAs can work if non-members are filtered to reduce n-gram overlap—does not translate into an attack because the adversary does not have access to the training data to compute n-gram overlap in the first place. The paper acknowledges this asymmetry directly: the diagnostic insight that "non-members with lower n-gram overlap are more distinguishable by existing MIAs" (Section 3.2.2) is valuable for understanding why attacks fail, but it does not produce a working attack that an adversary without training data access could execute. The temporal shift experiments (Section 4) are similarly a diagnostic probe—they demonstrate that prior work's high performance may be attributable to distribution shift, but they do not provide a recipe for an adversary to construct better non-member sets without the target model's training data cutoff knowledge.

The consequence. The paper's primary practical value is diagnostic, not prescriptive. It explains why existing black-box MIAs fail on pre-trained LLMs and identifies the structural factors (one-epoch training, n-gram overlap) responsible. However, it does not provide a method that a privacy auditor or adversary could use to reliably detect training data membership in a real deployment. The n-gram overlap insight suggests that better non-member candidate selection could improve MIA performance—but the paper does not propose how to select such non-members without access to the training data or a suitable proxy. The temporal shift finding suggests that using post-cutoff data as non-members inflates performance—but this simultaneously demonstrates that such inflated performance is not genuine membership inference, creating a Catch-22 for practitioners: the techniques that make MIAs appear to work are precisely the ones the paper shows are measuring distribution shift rather than membership. A privacy auditor seeking to use MIA as a tool in 2024 is left with the paper's negative result (black-box attacks don't work) but no clear path to a positive result (here is how to make them work reliably). The MIMIR benchmark package is released as infrastructure for future work, but as of this paper, the attacks it implements do not achieve practically useful performance in realistic settings.

What evidence exists in the paper. The gap between diagnostic insight and practical attack is visible throughout the experimental results. Table 1 shows that no attack achieves AUC above 0.6 in any domain except GitHub, and Table 11 shows TPR@1%FPR below 2–3% in most settings. The n-gram thresholding experiment (Table 2) shows that AUC can reach 0.666–0.908 if non-members are filtered, but this filtering procedure requires the training data and is explicitly flagged as not a recommendation. The temporal shift experiments (Table 3, Figure 6) show AUC reaching ~0.80, but Table 5 demonstrates these thresholds fail on in-distribution non-members. The paper does not report any experiment where an attack achieves both high AUC (> 0.7) and low FPR (< 5%) on natural, in-distribution non-members without leveraging training data access or distribution shift.

Mitigation status. The paper does not attempt to close this gap. It explicitly positions itself as a measurement and diagnostic contribution, not a methods contribution. Section 1 frames the work as "explor[ing] the challenges in evaluating membership inference attacks on LLMs" and Section 6 concludes by suggesting that "the membership inference game needs to be extended for such generative models to better align with information leakage that adversaries and auditors may care about." The MIMIR release enables future research but does not itself solve the practical attack problem. The paper's proposed direction—rethinking membership definitions to account for fuzzy, semantic, or neighbor-based membership—is forward-looking and does not include a concrete proposal or evaluation. A practitioner reading this paper in 2024 would learn that existing tools are inadequate but would not find a replacement tool ready for deployment.

All Main Results Are on a Single Model Family (Pythia) Trained on a Single Dataset (the Pile), with Preliminary Cross-Validation Only on Architecturally Similar Models

The assumption or constraint. The paper's primary experimental results (Tables 1, 11, 12; Figures 1, 2, 3; the temporal shift experiments in Section 4; the modified member experiments in Section 5) all target the Pythia and Pythia-Dedup model suites, which share the GPT-NeoX architecture and were trained on the Pile dataset for approximately one epoch. The paper states in Section 4 that "we believe this model is representative of the capabilities of many contemporary LLMs," but this claim is not systematically tested. The cross-validation experiments on GPT-Neo (Table 7, Appendix A.6) use models that, while architecturally distinct (GPT-3 replication vs. GPT-NeoX), were also trained on the Pile, use similar training recipes, and share substantial data with Pythia. The OLMo experiments (Table 8, Appendix A.7) provide the only test on models trained on entirely different data (DOLMA) and at larger scales (3T and 2.5T tokens vs. Pythia's ~300B), but these are described as "preliminary" and include only two model sizes (1B, 7B), with no intermediate checkpoint analyses, no temporal shift experiments, and no n-gram overlap distributions computed (since DOLMA training data access is needed). The paper acknowledges this limitation only indirectly—noting in Appendix A.2 that Pythia was chosen because its "exact training data, data order, and intermediate checkpoints are fully known and publicly available," which is precisely the property that most other model families lack.

The consequence. It is unclear whether the paper's central findings—near-random MIA performance, the non-monotonic training trajectory, the n-gram overlap explanation—generalize to models with different architectures (dense vs. mixture-of-experts, encoder-decoder vs. decoder-only), different tokenizers (which affect how n-gram overlap translates to token-level memorization), different training data mixtures (multilingual, code-heavy, curated high-quality), or different training paradigms (curriculum learning, data repetition strategies, continued pre-training from a base model). The OLMo results (Table 8) hint at potential differences: the Reference-based attack performs substantially worse on OLMo than on Pythia (e.g., Common Crawl AUC 0.410–0.412 vs. 0.564–0.582), and many domain-attack combinations yield AUC below 0.5—a pattern not seen in the Pythia results. This could indicate that the near-random performance observed on Pythia is the high end of MIA performance, with even weaker results on larger-scale or differently-trained models, or it could indicate that the specific reference model (STABLELM-BASE-ALPHA-3B-V2) is particularly poorly matched to OLMo's training distribution, confounding the comparison. Without systematic multi-model-family experiments, the paper's headline claim that "MIAs barely outperform random guessing for most settings across varying LLM sizes and domains" must be understood as "for Pythia-family models trained on the Pile," which is a substantially narrower claim.

Additionally, the Pile is a specific dataset with known properties—it was constructed in 2020 from specific sources with specific preprocessing, and its deduplication and decontamination procedures may not be representative of more modern or more aggressively filtered corpora. If future LLMs are trained on corpora with different n-gram overlap characteristics (e.g., more aggressively deduplicated, or conversely, containing more near-duplicate web text), the n-gram overlap distributions and consequently the MIA difficulty could shift substantially.

What evidence exists in the paper. The GPT-Neo replication (Table 7) shows qualitatively similar patterns to Pythia—near-random performance in most domains, elevated performance on GitHub, AUC generally below 0.55—but the numbers are not identical. For example, Wikipedia Reference-based AUC on GPT-Neo-2.7B is 0.545 vs. 0.565 on Pythia-Dedup-2.8B; Pile-CC Reference-based is 0.531 vs. 0.537. These small differences are within what might be expected from architectural and tokenizer variation, but without more model families, there is no way to assess whether the ~0.55–0.58 ceiling observed across Pythia models represents a fundamental limit or a Pythia-specific one. The OLMo results (Table 8) diverge more substantially: no attack on OLMo-7B exceeds AUC 0.520 on any domain except Pile-CC (0.520 Min-k%) and Common Crawl (0.505 Min-k%), and many values fall below 0.5. The paper does not compute n-gram overlap for DOLMA domains, does not run intermediate checkpoint analyses on OLMo, and does not test temporal shift on OLMo—so it is unknown whether the OLMo results reflect the same mechanisms (n-gram overlap suppressing performance) or different ones.

Mitigation status. The paper includes the GPT-Neo and OLMo experiments explicitly as cross-validation, which is more than many papers in this area provide. However, the OLMo experiments are characterized as "preliminary" and are not integrated into the main narrative of the paper (they appear only in Appendix A.7, with Table 8 in the Appendix). The paper does not claim that its findings generalize to all LLMs—the abstract states the findings apply to "a suite of language models (LMs) trained on the Pile"—but the broader framing in the introduction and conclusion (e.g., "Membership Inference on LLMs is Difficult" as a section title) implies more generality than the experiments strictly support. Future work would need to replicate the full suite of analyses (intermediate checkpoints, n-gram overlap, temporal shift, epoch count) on model families trained on different corpora—a significant undertaking given that most model families do not release intermediate checkpoints or exact training data.

The Black-Box Threat Model Excludes Potentially More Powerful Attacks, and the Paper Does Not Quantify How Much Performance Would Improve Under Stronger Assumptions

The assumption or constraint. The paper evaluates five membership inference attacks, all operating in the black-box setting where the adversary has query access to next-token probability distributions but not model parameters, gradients, or the ability to train shadow models. The paper explicitly excludes several categories of potentially stronger attacks. Section 2 notes that "more involved approaches include training shadow models (Shokri et al., 2017; Ye et al., 2022) on non-overlapping data from the target model's underlying data distribution" and that "attacks like LiRA (Carlini et al., 2022) show promise [but] require training multiple copies of shadow models, which is often intractable for LLMs." Appendix A.4 further excludes "MIAs involving meta-classifiers" because "access to such a subset [of ground-truth member/non-member data] can be difficult to guarantee in practice." The paper's central claim—"MIAs barely outperform random guessing"—is therefore restricted to a specific, relatively weak threat model: single-model, black-box, no shadow models, no meta-classifiers.

The consequence. The paper cannot distinguish between "MIAs fundamentally do not work on pre-trained LLMs because the memorization signal is too weak" and "the specific black-box attacks evaluated are too weak to extract the memorization signal that does exist." This distinction matters enormously for privacy auditing. If the memorization signal is genuinely absent, then LLMs are inherently resistant to membership inference, and auditors can use MIA failure as evidence of privacy. If, instead, the memorization signal exists but requires stronger attacks to detect—attacks that are computationally expensive but feasible for a well-resourced adversary—then the near-random performance of black-box attacks provides false reassurance. Extraction attacks (Carlini et al., 2021; Biderman et al., 2023a) have demonstrated that LLMs do memorize training data verbatim, at least in high-duplication subsets—a finding that appears to conflict with the near-random MIA performance reported here, since memorization should, in principle, produce a detectable MIA signal. The paper acknowledges this tension in Section 2: "While MIAs require knowledge of candidates and only reveal directly which of those candidates are included in the training data, extraction requires knowledge of sufficient-length prefixes... Both MIA and extraction are useful techniques for studying leakage in models, but rely on different assumptions and reveal different types of leakage risks." But the paper does not explain why extraction succeeds when MIAs fail on the same models, nor does it test whether stronger MIA assumptions (e.g., white-box access, which is available for Pythia since the models are open-source) could close this gap.

Furthermore, the paper's exclusion of meta-classifier attacks may be conservative in practice. While the authors argue that "access to such a subset can be difficult to guarantee," a privacy auditor evaluating a specific model often does have access to some known member data (e.g., from the model's documented training sources) and can construct non-member data (e.g., from held-out splits of the same corpora). In such settings, training a meta-classifier on these labeled samples—while not part of the standard black-box threat model—could substantially improve performance and is practically feasible for auditors with partial knowledge of the training data.

What evidence exists in the paper. The paper provides no experiments with white-box access, shadow models, or meta-classifiers, so the performance gap between the evaluated attacks and stronger alternatives is entirely unquantified. The paper references LiRA (Carlini et al., 2022) as achieving high performance in other settings but notes the computational intractability of training multiple shadow models at LLM scale—a valid practical concern, but one that does not establish that such attacks would fail, only that they are expensive. The extraction attack literature (Carlini et al., 2021; Biderman et al., 2023a; Carlini et al., 2023) provides indirect evidence that memorization exists in Pythia models, suggesting that stronger MIAs might detect it even when black-box attacks do not. Specifically, the fact that extraction succeeds (recovering verbatim training sequences given a prefix) implies that the model assigns anomalously high probability to those sequences—exactly the signal that MIAs attempt to measure. The failure of black-box MIAs to detect this signal (Table 1) suggests that the signal is either too localized (affecting only specific token positions but not the full-sequence average loss) or too weak relative to natural variation in sequence difficulty to be detected without calibration against a stronger reference or access to internal model states.

Mitigation status. The paper partially mitigates this concern by being explicit about its threat model scope and by including a calibrated attack (Reference-based) that uses an external model to normalize for sequence difficulty—a stronger approach than the raw LOSS attack. The paper also experiments with the Neighborhood attack (Mattern et al., 2023), which measures loss curvature rather than absolute loss—a different signal that could, in principle, capture memorization that absolute loss misses. The fact that even these stronger (within the black-box constraint) attacks perform near-randomly strengthens the paper's claim but does not fully close the gap. The paper does not discuss what performance might be achievable under stronger assumptions, does not estimate the cost of training shadow models at Pythia scale (which would help readers assess whether the "intractability" claim is absolute or context-dependent), and does not propose approximate shadow model techniques (e.g., training smaller models on subsets of the Pile) that might achieve some of LiRA's benefits at lower cost. The paper's framing—"this suggests that the success of current MIAs in previous settings does not transfer well to attacking pre-trained LLMs" (Section 1)—is appropriately narrow, but readers should understand that "current MIAs" refers specifically to the black-box, single-model attacks evaluated, not to all possible membership inference approaches.

The Study Does Not Address How MIA Performance Scales with Sequence Length, Despite Evidence That It Matters

The assumption or constraint. The paper constructs candidate sequences of 100–200 words, filtered from longer documents and truncated to 200 words from the beginning. Section 3 states that "previous work (Shi et al., 2023) observes that sample length correlates with [MIA] performance, so we bound the sample length to reduce its impact while picking a reasonable threshold so that our samples are likely to contain ample signal." The paper does not systematically vary sequence length or report how MIA performance changes as sequences get longer. The choice of 200 words is described as "a reasonable threshold" but is not empirically justified—no experiment tests whether 100-word, 200-word, 500-word, or full-document sequences yield different attack performance.

The consequence. This design choice potentially underestimates MIA performance in two ways. First, longer sequences contain more tokens, each providing an independent signal about membership. The LOSS attack averages token-level losses—with more tokens, the variance of this average decreases, potentially making the member and non-member distributions more separable even if the per-token signal is weak. The Min-k% Prob attack benefits even more directly from longer sequences, since it can select the k% most informative tokens from a larger pool, potentially capturing rarer and more diagnostic tokens. Second, the truncation to the first 200 words specifically discards the latter portions of documents, which may contain more distinctive or sensitive content (e.g., specific findings in a paper's results section, unique code in a repository's implementation, personal details in a biographical article's later paragraphs). If memorization is concentrated in distinctive content rather than in generic openings (which often contain shared boilerplate, standard introductions, or common phrasings), the paper's truncation would systematically discard the most detectable portions of member documents.

Conversely, it is also possible that longer sequences reduce MIA performance by including more tokens that are high-probability under any reasonable model (increasing the denominator in the loss average and diluting the rare low-probability tokens). The paper does not test this, so the direction of the bias is unknown. What is clear is that the paper's headline numbers (AUC ~0.5–0.6) apply to 100–200 word truncated sequences, not to full documents, and practitioners interested in document-level membership inference cannot safely extrapolate from these results.

What evidence exists in the paper. The paper provides no experiment varying sequence length. The justification for the 100–200 word bound cites Shi et al. (2023)'s observation that length correlates with performance but does not quantify this correlation for the Pythia models or the evaluated attacks. The n-gram overlap analysis (Section 3.2.2) computes overlap over the truncated sequences, not original documents—so the reported overlap percentages apply to the first 200 words, not the complete documents. It is possible that n-gram overlap is higher in the opening portions of documents (which contain shared boilerplate) and lower in later portions (which contain more distinctive content), meaning the paper's length truncation may simultaneously inflate n-gram overlap (making the membership boundary appear fuzzier than it is) and deflate MIA performance (by using shorter, less informative sequences). The paper does not discuss this potential confound.

Mitigation status. The paper acknowledges the length-performance relationship but does not treat the bounding as a limitation to be explored. The truncation is presented as a controlled experimental choice to "reduce [length's] impact"—meaning the paper's goal was to measure MIA performance independent of length effects, not to characterize how length affects performance. This is a reasonable experimental design choice for isolating other factors (n-gram overlap, model size, epoch count), but it means the absolute performance numbers reported should not be interpreted as estimates of document-level MIA performance in deployment settings where candidates may be much longer. The paper does not suggest future work on length scaling and does not release experiments with varied sequence lengths in MIMIR. A practitioner seeking to use these attacks on full documents would need to conduct their own length-scaling analysis.

The Finding That MIAs Fail Does Not Distinguish Between "Models Are Private" and "Attacks Are Too Weak," Creating Potential for False Reassurance About LLM Privacy

The assumption or constraint. The paper's framing and conclusions occasionally blur the distinction between "attacks fail" and "memorization does not exist." The abstract states that "MIAs barely outperform random guessing for most settings" and attributes this to "the combination of a large dataset and few training iterations, and an inherently fuzzy boundary between members and non-members." Section 3.2 describes these as "characteristics of LLMs" and "inherent ambiguity in MIA," and the conclusion states that "our results suggest two possibilities: (1) data does not leave much of an imprint, owing to characteristics of the pre-training process at scale... and (2) the similarity between in and out members... makes this distinction fuzzy." While the paper is careful not to claim that LLMs are provably private, the overall narrative—attacks don't work, and here's why—can be read as suggesting that the primary barrier is fundamental (one-epoch training, n-gram overlap) rather than contingent (current attacks might be too weak to detect the memorization that exists).

The consequence. A reader—particularly one from the privacy regulation or policy community—might conclude that pre-trained LLMs are inherently resistant to membership inference and that privacy concerns about training data exposure are overblown. This interpretation would be premature and potentially harmful. As discussed in the third limitation above, extraction attacks on the same model families (Biderman et al., 2023a; Carlini et al., 2023) have demonstrated that verbatim memorization does occur in Pythia models and that training data can be recovered given appropriate prefixes. The paper's own finding that recent data is more vulnerable (Appendix C.1, Figure 9) and that multi-epoch training increases detectability (Figure 2, right) demonstrates that the near-random performance in the one-epoch setting is not a fundamental privacy guarantee but a contingent property of a specific training regime—one that could change if training practices shift toward multiple epochs, data repetition, or continued pre-training on smaller domain-specific datasets. The paper's results on fine-tuned models from prior work (Mireshghallah et al., 2022a; Fu et al., 2023), which achieve high MIA performance, further underscore that the pre-training setting is the exception, not the rule.

The paper does not provide a clear bottom line for privacy practitioners: is a Pythia-12B model trained on the Pile for one epoch "private enough" that training data membership cannot be reliably inferred? The answer from the black-box attacks evaluated here is "yes"—but the paper does not establish whether this answer holds under stronger attack models, for longer sequences, for specific high-risk data types (e.g., PII, copyrighted code, rare facts), or for models trained with different recipes. The absence of a clear "here's when you should worry" guidance could lead to misapplication of the paper's findings.

What evidence exists in the paper. The paper's results show that black-box MIAs achieve near-random performance, but several findings within the paper itself complicate the privacy narrative. The GitHub domain shows AUC up to 0.717 (Table 1), demonstrating that MIAs can work when n-gram overlap is naturally controlled (because code has different overlap characteristics than natural language). The temporal shift experiments show AUC up to 0.796 (Table 3), demonstrating that distribution shift can make membership appear detectable—which simultaneously shows that the capacity to detect membership exists in the model's outputs, even if it's not exploitable without distribution-shifted non-members. The n-gram thresholding experiment (Table 2) shows AUC up to 0.908, demonstrating that when the fuzzy boundary is artificially sharpened, membership is highly detectable—which implies that for specific, distinctive documents (those with naturally low n-gram overlap), MIAs might work well even in the natural setting, a hypothesis the paper does not test by stratifying members by their n-gram overlap with the rest of the training data. The recency effect (Appendix Figure 9) shows that recently-trained data is more detectable, with Reference-based AUC reaching ~0.72 for the most recent 100 steps on 12B—well above the ~0.55 aggregate performance. This means that the paper's headline "near-random" result is an average over all training data; freshly-trained data may be substantially more vulnerable, a distinction that is not highlighted in the abstract or conclusion.

Mitigation status. The paper partially addresses this by noting the two possibilities in the conclusion and by discussing the distinction between MIA and extraction in Section 2. The section on "Revisiting Membership" (Section 5) explicitly argues that the standard membership definition may be too narrow and that "guessing the membership of some sample via other sufficiently close samples can be useful"—which implies that the failure of exact-match MIAs does not mean there is no leakage, only that the leakage takes a form current MIAs don't measure. However, this nuanced position is not reflected in the abstract's stronger claim that MIAs "barely outperform random guessing," which could be misinterpreted as a privacy guarantee. The paper would benefit from a clearer statement that "the failure of existing black-box MIAs does not imply that LLMs do not memorize training data" and from a stratification of MIA performance by factors that might increase vulnerability (document distinctiveness, recency, duplication count, presence of PII or code) to help practitioners assess risk for specific data types rather than relying on aggregate averages. The MIMIR release enables such stratification analyses, but the paper itself does not conduct them. A practitioner reading this work should understand that the near-random aggregate performance masks substantial variation across data types, training stages, and attack assumptions—and that the paper provides diagnostic tools for understanding this variation but does not provide a comprehensive risk map.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around membership inference on language models from "which attack works?" to "what does it mean for an attack to work, and what are we actually measuring?" This is fundamentally a diagnostic and methodological reframing rather than a new attack proposal or a privacy guarantee. The field previously operated under an implicit assumption: if MIAs achieve high AUC, the model memorizes training data and leaks information; if MIAs fail, the model is private. This paper demonstrates that this assumption is unreliable in the LLM pre-training setting, because MIA performance is jointly determined by at least three interacting factors—memorization, n-gram overlap between members and non-members, and distribution shift in candidate set construction—and disentangling them requires controlled diagnostic experiments that prior work largely did not conduct.

The magnitude of this shift is substantial but bounded. It is not a paradigm shift in the Kuhnian sense—the standard membership inference game (Yeom et al., 2018) remains the dominant framework, and the paper does not propose a replacement. Rather, it is a methodological corrective that changes how researchers should evaluate MIAs on LLMs: not by reporting a single AUC number on a fixed benchmark, but by analyzing n-gram overlap distributions, testing multiple non-member candidate sets (natural, temporal, n-gram thresholded), and verifying that apparent attack success is not driven by distribution shift. This is analogous to how the adversarial robustness community shifted from reporting "accuracy on perturbation X" to requiring adaptive attacks and careful threat model specification—the core task remains the same, but the evaluation standards are permanently raised.

The paper resolves a specific contradiction in the literature. Prior work on MIAs for language models produced conflicting findings: some papers reported high attack performance (Shi et al., 2023; Meeus et al., 2023), while others found that self-correction and self-critique largely fail (Huang et al., 2023, cited in the paper's motivation for why revision models need fine-tuning). The paper's temporal shift experiments (Section 4, Tables 3 and 5) directly explain this contradiction: high MIA performance in prior work can be attributable to candidate set construction that introduces distribution shift (e.g., using temporally-shifted non-members), and thresholds learned on such benchmarks fail catastrophically when applied to in-distribution non-members (Table 5 shows a 1% FPR threshold on temporal Wikipedia producing a 9.8% FPR on natural Wikipedia non-members under the LOSS attack). This does not mean those prior works are wrong—their reported numbers may be accurate for their specific benchmarks—but it means those numbers measure a different inference task (temporal shift detection) than the one they were interpreted as measuring (membership inference). The paper provides a concrete diagnostic test—comparing n-gram overlap distributions between the candidate non-member set and a held-out member sample—that allows future researchers to check whether their benchmarks are genuinely testing membership.

The paper redirects research attention in several concrete ways. First, it makes improving MIA evaluation methodology at least as important as developing new attacks. The MIMIR benchmark and the n-gram overlap diagnostic provide infrastructure that lowers the barrier for rigorous evaluation. Second, it identifies verifier over-optimization analogies that apply to MIAs: just as the companion paper on compute-optimal test-time scaling found that stronger search against a verifier paradoxically degrades performance due to over-optimization (Figure 3 in that paper), this paper finds that n-gram overlap creates an irreducible ambiguity that no amount of attack sophistication can overcome without changing the membership definition. Third, it opens a new research direction on fuzzy membership by demonstrating that current MIAs fail to detect semantically meaningful variations of training data (Section 5, Tables 4 and 10)—suggesting that the field needs to develop membership definitions and attacks that capture information leakage rather than exact token sequence memorization.

The paper also shifts the burden of proof. Before this work, a paper reporting high MIA performance on a new attack could simply state the AUC and claim success. After this work, such a paper must also show that its non-member set is representative of the member distribution (through n-gram overlap analysis or equivalent diagnostics) and that its thresholds transfer to in-distribution non-members. The paper's demonstration that GitHub's apparently high MIA performance (AUC up to 0.717, Table 1) is largely due to inadequate decontamination thresholds for high-overlap domains (Appendix B.3, Figure 7) serves as a cautionary tale for any domain with repetitive structure—code, legal text, scientific boilerplate, or template-based content.

Follow-Up Research This Work Enables

1. Training a difficulty predictor to enable practical difficulty-conditioned MIA without oracle access to the training data. The paper's central diagnostic—that n-gram overlap between members and non-members suppresses MIA performance—is measured using a bloom filter over the entire Pile training set, which requires access to the training data that an adversary or auditor would not typically have. A natural next step is to train a lightweight model that predicts the n-gram overlap of a candidate sequence with the training data using only the target model's outputs (e.g., the distribution of token-level losses, the entropy of the output distribution, or features from the Neighborhood attack's perturbation responses). If such a predictor could estimate whether a candidate has low overlap (and thus is likely to be distinguishable) versus high overlap (and thus likely to be indistinguishable from members), it would enable practical difficulty-conditioned MIA: focus attack resources on low-overlap candidates where membership detection is possible, and abstain on high-overlap candidates where the fuzzy boundary makes detection hopeless. The paper's data provides the ground truth (n-gram overlap scores for 1,000 non-members per domain) needed to train and evaluate such predictors. A strong follow-up would demonstrate that a difficulty-conditioned attack achieves substantially higher TPR@low%FPR than the unconditional attacks in Table 11 while controlling the overall FPR—essentially, selective prediction applied to membership inference.

2. Stratifying membership inference performance by document properties (duplication count, recency, distinctiveness) to build a risk map rather than a single aggregate number. The paper reports aggregate AUC numbers (Table 1) that average over all members in a domain, but the recency experiment (Appendix C.1, Figure 9) shows that recently-seen data is substantially more detectable than earlier data, and prior work (Kandpal et al., 2022) has shown that deduplication reduces extractability, implying that data duplicated in the training set is more vulnerable. The Pythia suite—with its known training data, training order, and intermediate checkpoints—provides a unique opportunity to stratify MIA performance by these properties without requiring new model training. A strong follow-up would replicate the paper's evaluation but report AUC and TPR@low%FPR separately for members binned by (a) their duplication count in the training data, (b) their recency (how many steps before the final checkpoint they appeared), (c) their n-gram overlap with the rest of the training data (a measure of distinctiveness), and (d) whether they contain PII or other sensitive content types. Such a risk map would tell practitioners which data types are genuinely at risk even in the one-epoch pre-training regime, rather than providing a single reassuring but misleading aggregate number. The paper's own GitHub results (AUC 0.717 vs. 0.5–0.58 for other domains) already hint that domain-level aggregation masks substantial variation.

3. A systematic comparison of black-box vs. white-box vs. shadow-model MIAs on the same Pythia models to quantify the performance gap between threat models. The paper evaluates only black-box attacks and explicitly excludes white-box attacks (which use gradients or internal representations) and shadow-model attacks (which train replica models on disjoint data) due to computational cost. However, the Pythia models are fully open-source with available parameters, and the training data and order are known, making it feasible—if expensive—to evaluate at least one white-box attack (e.g., using gradient norms or loss curvature with respect to parameters, which are known to carry memorization signals; Sablayrolles et al., 2019) and perhaps a small-scale shadow model attack (training 2–3 smaller Pythia-scale models on Pile subsets, as a tractable approximation of LiRA; Carlini et al., 2022). A strong follow-up would quantify exactly how much performance improves under these stronger threat models. If white-box attacks also achieve near-random performance, the paper's claim that "data does not leave much of an imprint" is strengthened. If white-box attacks achieve AUC of 0.7–0.8 on the same data where black-box attacks achieve 0.5–0.58, then the paper's negative result should be understood as a limitation of black-box access specifically, not of membership inference on LLMs in general, with important implications for the privacy threat model of deployed API-only models versus open-weight models.

4. Evaluating whether the n-gram overlap effect is an artifact of the Pile's decontamination procedures or a fundamental property of natural language, by replicating the analysis on LLMs trained on differently-decontaminated corpora. The paper's n-gram overlap analysis (Section 3.2.2, Figure 3) shows that non-members from the Pile test set share 30–40% of their 7-grams with the Pile training set, but this is in part a property of how the Pile was constructed—its document-level deduplication and the specific decontamination threshold (≤80% 13-gram overlap; Groeneveld et al., 2023) used by the paper. It is unclear whether more aggressive deduplication (e.g., near-duplicate detection, semantic deduplication, or the MinHash-based approaches used in modern corpora like DOLMA, Soldaini et al., 2023) would substantially reduce n-gram overlap and consequently increase MIA performance. A strong follow-up would replicate the paper's n-gram overlap analysis and MIA evaluation on models trained on DOLMA (for which the paper's preliminary OLMo results in Table 8 suggest even lower MIA performance, though n-gram overlap distributions are not computed) and on models trained on aggressively deduplicated corpora (e.g., using the methods from Lee et al., 2022 or Abbas et al., 2023). The key question is whether the fuzzy boundary is a fundamental property of natural language (in which case improving deduplication cannot eliminate it and MIAs will remain weak regardless of corpus engineering) or a property of specific corpus construction choices (in which case better deduplication could sharpen the boundary and make MIAs more effective).

5. Developing a semantic membership inference game with concrete distance metrics and systematic evaluation. Section 5 proposes that "the membership inference game needs to be extended for such generative models to better align with information leakage that adversaries and auditors may care about," and demonstrates that current MIAs classify semantically equivalent paraphrases as non-members with high confidence (Tables 4 and 10). But the paper does not propose a concrete alternative membership definition or distance metric. A strong follow-up would formalize a semantic membership neighborhood: for each member x, define a set of "semantically equivalent neighbors" N(x) via paraphrasing, back-translation, or embedding-space perturbation with a semantic similarity constraint, and then measure whether an MIA trained on exact members generalizes to classify members of N(x) as members. The evaluation would report a new metric—semantic TPR, the fraction of semantically equivalent neighbors correctly classified as members at a fixed FPR—alongside standard exact-match metrics. This would directly measure whether attacks capture information leakage (sensitive content being extractable through close variants) rather than exact memorization (verbatim token sequences). The paper's GPT-4 paraphrasing experiment (Appendix D, Table 10) provides a template for constructing such neighbor sets, and the paper's multiple attack implementations in MIMIR provide the evaluation infrastructure. The hardest challenge is defining what counts as "semantically equivalent" in a way that is domain-appropriate and not gameable—a research question the paper opens but does not resolve.

6. A stress-test of the paper's conclusions on models trained with multiple epochs or domain-specific repetition. The paper demonstrates that multi-epoch training linearly increases MIA performance on Datablations models (Figure 2 right) and that domain upsampling has mixed effects on SILO (Appendix C.2, Figure 10). However, these models are relatively small (2.8B and 1.3B parameters, respectively) and trained on limited data (55B tokens for Datablations, the Open License Corpus for SILO). The natural follow-up is to evaluate MIA performance on larger models that intentionally incorporate data repetition—either through multiple epochs (e.g., the LLaMA family's training on multiple epochs of curated data; Touvron et al., 2023a,b) or through domain-specific upsampling (e.g., the practice of upsampling high-quality sources like Wikipedia, academic papers, and code in many LLM training recipes). A strong follow-up would target OLMo's intermediate checkpoints (which the paper notes are available but did not evaluate for training trajectory) or LLaMA-family models if training data access becomes available, measuring whether the non-monotonic training trajectory observed for Pythia (Figure 2 left) generalizes and whether the epoch effect remains linear at larger scales. This would test whether the paper's finding that "near-one-epoch training suppresses MIA" is a contingent property of Pythia's specific training recipe or a robust scaling law.

Practical Applications and Downstream Use Cases

1. Privacy auditing of pre-trained models with appropriate expectations. Organizations deploying LLMs—whether open-weight models like Pythia or OLMo, or API-based models from commercial providers—increasingly face regulatory pressure to demonstrate that their models do not leak training data. The paper's primary practical contribution for such auditors is negative: the standard black-box MIAs in widespread use (LOSS, reference-based calibration, Min-k%) should not be trusted as evidence of privacy when applied to pre-trained LLMs, because their near-random performance may reflect the fuzzy membership boundary rather than genuine privacy protection. An auditor using these tools on a Pythia-like model and obtaining AUC near 0.5 cannot conclude "this model does not memorize training data"—they can only conclude "this tool cannot detect memorization under these conditions," which is a fundamentally different claim with different regulatory implications. The paper's diagnostic framework—specifically, the recommendation to analyze n-gram overlap distributions between candidate non-members and the training data (or a suitable proxy)—provides auditors with a concrete check before interpreting MIA results: if the non-member set has substantially lower n-gram overlap than the member distribution, the MIA may be measuring distribution shift rather than membership, and the results should be interpreted cautiously. The MIMIR package provides a standardized implementation of this diagnostic, lowering the barrier for auditors to conduct this analysis.

2. Benchmark construction for future MIA research on LLMs. The paper's demonstration that candidate set construction dramatically affects MIA performance—temporal shift inflates AUC by ~0.25 (Tables 1 vs. 3), n-gram thresholding inflates it by ~0.15–0.30 (Table 2)—has direct implications for how benchmarks should be built. Researchers developing new MIAs for LLMs should (a) report n-gram overlap distributions for their non-member sets alongside a held-out member sample to establish representativeness, (b) test their attacks on multiple non-member sets (natural, temporally shifted, and n-gram filtered) to distinguish genuine membership signal from distribution shift exploitation, and (c) report threshold transfer performance (as in Table 5) to verify that learned decision boundaries do not depend on artifacts of candidate set selection. The MIMIR benchmark package provides the infrastructure for this multi-faceted evaluation, and the paper's explicit n-gram overlap measurement methodology (Appendix B.1) gives researchers a recipe for computing these diagnostics on their own corpora.

3. Fine-tuning risk assessment. While the paper focuses on pre-training, its findings have direct implications for fine-tuning, which typically involves multiple epochs on smaller, often sensitive datasets—exactly the conditions the paper shows maximize MIA vulnerability (Figure 2 right: linear increase in AUC with epoch count). Organizations fine-tuning LLMs on proprietary data (customer conversations, medical records, financial documents) should not assume that the near-random MIA performance observed on pre-training data extends to their fine-tuned models. The paper's recency experiment (Appendix C.1, Figure 9) further demonstrates that recently-seen data is more detectable, with Reference-based AUC reaching ~0.72 for the most recent training steps on the Pythia-12B model compared to ~0.58 for the aggregate. This has direct operational implications: data fine-tuned in the most recent epoch is at substantially higher risk, and privacy auditors evaluating fine-tuned models should focus their MIA testing on the most recent training data, not a uniform sample across all fine-tuning steps. The paper's diagnostic toolkit (n-gram overlap analysis, multi-reference-model ablation as in Table 6) can be applied directly to fine-tuning data to assess the difficulty of the membership inference task before drawing conclusions about model privacy.

4. Guiding the design of privacy-preserving training pipelines. The paper's finding that deduplication reduces MIA performance (Figure 1, Table 12 vs. Table 1) confirms and extends Kandpal et al. (2022)'s result that deduplication mitigates memorization. However, the paper also shows that even aggressive deduplication (the Pythia-Dedup models use the deduplicated Pile) leaves substantial n-gram overlap (32.5% average 7-gram overlap for Wikipedia non-members, Figure 3), which in turn suppresses MIA performance by creating a fuzzy boundary. This has a counter-intuitive practical implication for privacy-preserving training: if the goal is to make membership inference difficult (as a defense), then maintaining some level of n-gram overlap between training and test data—through document-level but not substring-level deduplication—may be more effective than aggressive substring-level deduplication, because the fuzzy boundary makes members and non-members intrinsically difficult to distinguish, regardless of how much memorization occurs. Conversely, if the goal is to make membership inference informative (as an auditing tool), then aggressive substring-level deduplication and careful non-member candidate selection that minimizes n-gram overlap with training data would sharpen the boundary and make MIAs more effective. The paper's n-gram overlap infrastructure provides the measurement tool needed to tune this tradeoff quantitatively—practitioners can compute overlap distributions for different deduplication strategies and predict their impact on MIA difficulty before committing to a specific pipeline.

When to Prefer This Method

The paper does not propose a new method that can be "preferred" over alternatives in a deployment sense—it is a measurement and diagnostic study, not a methods contribution. There is no new attack, defense, or training procedure to adopt. Instead, the paper provides a diagnostic framework and evaluation methodology that researchers and practitioners should use when evaluating membership inference attacks on language models. The relevant decision is not "when should I use this attack?" but "when I am evaluating an MIA, what additional checks should I perform before trusting the results?" The paper's answer, derived from its experimental findings, is:

Always compute and report n-gram overlap distributions for the non-member set against the training data (or a suitable proxy). If the non-member distribution is notably shifted toward lower overlap compared to a held-out member sample (as in the temporal shift experiments, Figure 4), the MIA may be measuring distribution shift rather than membership, and reported performance should be treated as an upper bound on genuine membership detection.
Test decision threshold transfer between candidate sets (as in Table 5). If thresholds achieving low FPR on the evaluation's non-member set produce substantially higher FPR on an alternative in-distribution non-member set, the apparent attack success is likely driven by candidate set artifacts rather than membership signal.
Stratify MIA performance by document properties rather than reporting only aggregate metrics. Documents with low n-gram overlap with the rest of the training data, documents seen recently during training, and duplicated documents are likely to be more detectable than the aggregate average (as suggested by the recency experiment, Figure 9, and the n-gram thresholding experiment, Table 2), and an aggregate number may obscure pockets of genuine vulnerability.
Use multiple reference models and report the range of performance (as in Table 6) rather than selecting a single reference model and claiming it as optimal, since reference model choice is "challenging and largely empirical" and no single reference model dominates across all domains and target models.
For pre-trained models specifically, do not interpret near-random MIA performance as evidence of privacy. The paper's results show that black-box MIAs fail to detect membership even when memorization is known to exist (as demonstrated by extraction attacks; Carlini et al., 2021; Biderman et al., 2023a). The failure may reflect the fuzzy membership boundary (n-gram overlap) rather than absence of memorization. Stronger threat models (white-box access, shadow models, meta-classifiers) should be evaluated before concluding that a model does not leak training data.