Privasis: Synthesizing the Largest "Public" Private Dataset from Scratch

🎯 Pitch

Privacy research has been stuck without real data for decades—this paper solves that by synthesizing 1.4 million private records from nothing. But the real shocker: a tiny 4B model trained on this synthetic data matches or beats GPT-5 at removing personal information, completely upending the assumption that privacy protection requires massive frontier models.

1. Executive Summary

This paper introduces Privasis, the first million-scale fully synthetic dataset built entirely from scratch for privacy research, comprising 1.4 million records with 55.1 million annotated attributes spanning medical, legal, financial, and communication domains. Using the dataset, the authors construct a parallel corpus for text sanitization via a decomposition-based sanitization pipeline (splitting records into chunks, then applying targeted abstraction or removal based on user-specified instructions rather than fixed PII categories) and train compact sanitization models (≤ 4B parameters) that outperform frontier LLMs on the task. The trained Privasis-Cleaner-4B model achieves a 72.5% full success rate on the vanilla test set, surpassing GPT-5 (70.1%) and o3 (70.3%), while matching GPT-5's performance on a harder test set (12.4% vs. 13.1%), establishing that lightweight, locally-deployable models can match or exceed the sanitization capabilities of models orders of magnitude larger—but only when the underlying dataset provides sufficient scale and diversity in privacy-rich synthetic training data, and performance drops sharply across all models on harder records requiring contextual attribute identification.

2. Context and Motivation

The Core Problem: Privacy Research Has Been Starved of Data

The fundamental bottleneck this paper addresses is deceptively simple: privacy-sensitive data, by definition, cannot be publicly shared, which means the research community lacks the large-scale, diverse, richly-annotated datasets that drive progress in virtually every other area of AI. The authors frame this contrast starkly in Section 1:

"Progress in privacy-related research has long been fundamentally limited by a drought of data. By definition, private information cannot be publicly shared. As a result, most prior work relies on small, narrowly scoped datasets—standing in stark contrast to the data-driven scaling paradigm that underpins progress in many other areas of AI."

This gap manifests across multiple dimensions simultaneously. Existing privacy datasets are typically single-domain (e.g., clinical notes only, or Reddit posts only), small-scale (hundreds to low thousands of records rather than millions), and narrow in annotation scope (covering only a handful of predefined PII categories like names and dates, without capturing the broader spectrum of sensitive information users might want to remove). Table 3 in the paper makes this quantitative: the largest prior dataset with abstraction capabilities—PANORAMA—has 384K records but only 48 words per record on average, and it provides only deletion without rewritten alternatives. The Self-Disclosure Corpus provides abstracted rewrites but spans only 4.8K records across a single domain (Reddit posts) with an average of just 29 words per record.

The consequence is that researchers developing privacy-preserving techniques—whether data sanitization models, differential privacy algorithms, or agentic systems that must handle sensitive information responsibly—have been forced to work with datasets that are orders of magnitude too small and too narrow to support the kind of data-driven scaling that has transformed language modeling, vision, and other subfields. A researcher wanting to train a sanitization model that generalizes across medical, legal, financial, and communication contexts simply has not had a unified dataset to do so.

Why This Problem Is Becoming Urgent Right Now

The paper identifies a specific technological trend that makes this data scarcity problem increasingly critical: the rise of AI agents that persistently access highly sensitive personal information. The abstract explicitly names OpenClaw (Steinberger, 2026) and Gemini Agent (Google, 2025), and the introduction adds ChatGPT Health (OpenAI, 2026). These are not hypothetical systems—they are deployed or soon-to-be-deployed agents that read personal emails, process medical records, manage calendars, and interact with financial documents during inference time.

This creates a two-sided pressure that existing infrastructure cannot handle:

On the input side, there is a need for data sanitization and minimization (Zhou et al., 2025; Dou et al., 2024)—techniques that remove or abstract sensitive information before it reaches the agent's context window, so that the agent never sees, for instance, a user's exact home address when it only needs to know the city for a restaurant recommendation. This is a form of privacy-by-design: don't give the model information it doesn't need to complete the task.

On the output side, there is a need for post-hoc privacy techniques (Bagdasarian et al., 2024) that ensure models appropriately handle whatever personal information they do have access to—for example, not leaking a user's medical condition in a response to another user, or not memorizing and regurgitating sensitive details from one context into another.

The paper argues that both sides of this problem demand robust, generalizable methods. Yet the current state of the art is surprisingly weak. The authors cite specific evidence:

"current LLMs fail at even basic personally identifiable information (PII) detection (Shao et al., 2024; Pham et al., 2025)"

This is a damning assessment: if today's most capable models cannot reliably detect personal information, they certainly cannot be trusted to remove it or abstract it appropriately. And without large-scale, diverse training data that covers the many ways sensitive information appears across different document types and contexts, there is no clear path to improving this capability.

The urgency, then, comes from the collision of two trends: agents are being deployed into increasingly sensitive contexts, while the privacy-preserving techniques needed to make those deployments safe remain underdeveloped due to a fundamental data bottleneck.

Prior Approaches and Their Structural Limitations

The paper surveys three categories of prior work in Section 5 (Related Work) and Appendix F, identifying specific shortcomings in each.

PII Removal Datasets and Span Detection

The classical approach to privacy-preserving text processing is PII span detection: annotate spans of text that contain personally identifiable information (names, dates, addresses, account numbers) and train models to identify these spans for redaction or masking. Foundational corpora include:

MIMIC-II De-identification (Douglass et al., 2004): 2 million clinical notes with PII span annotations, but focused exclusively on clinical text and providing only deletion labels without rewritten alternatives.
The Text Anonymization Benchmark (TAB) (Pilán et al., 2022): 1,200 legal documents (court cases) with fine-grained span-level labels, highly curated but tiny in scale and single-domain.
Gretel Synthetic Financial PII (Gretel.ai, 2024): 55,900 synthetic financial documents with token-level PII spans, broadening domain coverage but still providing only deletion labels.
Automated Privacy Info Annotation (Zeng et al., 2025a): 154,000 records of LLM interactions with privacy-phrase spans, larger scale but limited to short query-response pairs.

The structural limitations across this body of work are consistent:

Scale is too small. The largest of these datasets (MIMIC-II) is larger than Privasis, but MIMIC-II is not publicly accessible in the same way—and crucially, it provides only span labels for deletion, not abstraction pairs or rewritten alternatives.
Domains are narrow. Each dataset covers a single domain (clinical, legal, financial, or LLM chat logs). A model trained on clinical notes cannot be expected to handle text messages, calendars, or legal contracts.
Annotation scope is restricted to predefined PII categories. These datasets typically label names, dates, IDs, and contact information, but cannot capture the broader range of sensitive information users might contextually want to remove—a particular medical condition, a specific financial transaction, the identity of a family member. The paper distinguishes between "PII spans" and "other sensitive spans," noting that most existing datasets only cover the former.
They provide only deletion, not abstraction. Masking or deleting an attribute (replacing a date with "[DATE]" or a name with "[NAME]") often destroys textual utility. What users often need is abstraction: "March 3rd, 2024" → "early March 2024" → "recently." Only a handful of datasets (Self-Disclosure, NAP²) provide abstracted rewrites, and those are limited in scale and domain.

Data Minimization and Abstraction

A separate line of work focuses on rewriting text to reduce identifiability while preserving meaning—going beyond strict PII to abstract or soften details that are not legal identifiers but remain sensitive. Key examples include:

Self-Disclosure Corpus (Dou et al., 2024): 4,800 Reddit posts with 19 disclosure types and paired rewrites that make posts safer without changing communicative intent. This is valuable because it demonstrates the abstraction paradigm, but it is limited to short social media posts in a single domain.
NAP² (Huang et al., 2025): A benchmark specifically designed to evaluate naturalness-preserving rewriting using delete and obscure/abstract strategies, with high-quality human rewrites. The paper uses this as an external generalization benchmark for Privasis-Cleaner.
LLM-based anonymization systems (Staab et al., 2025; Zeng et al., 2025b): These use language models themselves to perform adversarial anonymization or remove-then-restore pipelines, but they operate as methods rather than providing reusable datasets for training and evaluation.

The fundamental gap in this line of work is that the datasets are not large enough, diverse enough, or richly annotated enough to support training general-purpose sanitization models. Each prior dataset covers at most a few thousand records in one or two domains, with limited annotation granularity. A model trained on the Self-Disclosure Corpus learns to abstract sensitive details in Reddit posts—it does not learn to handle sensitive information in medical records, legal contracts, or financial statements.

Synthetic Data Generation

A third category of prior work focuses on methods for generating synthetic data, which is relevant because Privasis itself is synthetic. The paper distinguishes between:

Differentially private generation methods (Abadi et al., 2016; Mattern et al., 2022; Yue et al., 2023; McKenna et al., 2025): These train generative models on private data with DP-SGD and then release synthetic samples that approximate the private distribution while satisfying differential privacy guarantees. These methods are orthogonal to Privasis—they aim to protect real private data while enabling data release, whereas Privasis aims to create entirely synthetic private data for research and evaluation without any real private data as input.
Non-private generation methods (Wang et al., 2023; Gunasekar et al., 2023; Abdin et al., 2024; Kim et al., 2023; Jung et al., 2024): These use self-instruction, targeted prompting, or automated refinement to create synthetic datasets from seed data or reference trajectories. The paper's critique is pointed:

"Most methods typically rely on fixed prompts, seed data, or reference trajectories from data generators, which may constrain the diversity and novelty of the generated data"

The implication is that when you bootstrap from existing data or fixed templates, you inherit the biases and limitations of that starting point. A self-instruction approach seeded with a few medical record templates will produce variations on those templates, not genuinely novel document types or information combinations.

Privasis's key distinguishing claim is that it generates data entirely from scratch using auxiliary control variables (profiles with personal attributes, record types, background contexts) and diversity-preserving refinement, without predefined prompts or reference data beyond a public name database. This is what enables both the scale (1.4 million records) and the diversity (10 main categories spanning medical, legal, financial, government, personal, community, professional services, education, media, and recreation domains).

The Verification Gap: Frontier Models Are Not Good Enough

A critical piece of motivation appears in the paper's evaluation results (Section 4.3, Table 4), which the introduction previews. The authors test frontier LLMs—GPT-5, o3, DeepSeek R1, Qwen3-235B, Llama-4 Maverick—on the sanitization task using Privasis's vanilla test set. The finding is sobering:

"Sanitization is fundamentally a re-writing task, yet even the strongest frontier models fall short of perfect performance, with Full Success Record rates ranging from 64.4% (Qwen3-235B) to 70.3% (o3)."

This is on the vanilla test set—records where the authors' own decomposition-based pipeline (with GPT-4.1) achieves a perfect score. The models are not being asked to do something impossible—a structured pipeline with a less capable model can do it perfectly on this subset. Yet the frontier models routinely fail, and the failure mode is instructive:

"Although the Successful Attribute scores for most models are around 90%, their Successful Attribute/Record scores exceed 90%. This gap reveals a critical weakness: while models sanitize the majority of attributes, they routinely miss at least one per record. Such failures are unacceptable in privacy-sensitive settings, because one missed attribute is enough to compromise the entire record, no matter how many others were sanitized correctly."

This is the last-mile problem in text sanitization: doing 90% of the job correctly is not good enough. A single leaked attribute—one missed name, one unredacted date, one phone number left in a sanitized document—constitutes a privacy failure. And on the hard test set, performance collapses further: GPT-5 achieves only 13.1% full success, o3 drops to 11.7%, and Qwen3-235B falls to 10.3%. These numbers drive home the paper's central argument: the field needs better training data and better models, not just bigger models with general-purpose reasoning capabilities.

The Specific Failure Modes Are Revealing

Table 5 in the analysis (Section 4.4) breaks down the types of information leakage exhibited by different models. The dominant failure mode across all models is direct leak—sensitive information appears verbatim in the supposedly sanitized output. Across frontier models, direct leaks account for 53.7% (GPT-OSS-120B) to 75.3% (Qwen3-235B) of all failures. The models are not failing because they cleverly paraphrase sensitive details in ways that preserve inferability—they are simply not removing the sensitive text at all.

The paper also identifies specific attribute types that cause the most trouble (Table 10): name-related attributes (full name, last name, user handle) and dates. These are precisely the categories that span detection systems have targeted for decades, yet frontier LLMs still miss them. The error analysis examples in Appendix E are illustrative:

Direct leak example: GPT-5 successfully removes most occurrences of an employer name ("Royal Darwin Hospital") from the main body of a document but misses it in the header and email signature—suggesting a failure of contextual awareness rather than an inability to perform the removal operation itself.
Inference leak example: GPT-5 replaces all occurrences of a journal name with "[journal name]", but leaves the editor's email domain "jsal.org" intact, allowing the evaluator to reconstruct the journal name from the domain.
Proximity leak example: GPT-5 removes an explicit timestamp ("9:30 PM on 17 Sep"), but leaves sufficient contextual cues (day of week, relation to a blood draw time, date references) that the evaluator can reconstruct the approximate time.

These examples illustrate that sanitization requires not just pattern matching but deep semantic understanding of how information is distributed across a document, including in structural elements (headers, signatures), indirect references (email domains, contextual cues), and implicit entailments. Frontier models, despite their general reasoning capabilities, have not been trained to perform this specific type of fine-grained, consistency-requiring rewriting task—and the datasets to train them for it have not existed at sufficient scale.

How Privasis Positions Itself

The paper positions Privasis not as an incremental improvement over existing privacy datasets but as a category-defining resource that fills a vacuum. The positioning is articulated through several explicit design choices that differentiate it from prior work:

1. Scale by orders of magnitude. Privasis contains 1.4 million records, compared to the low thousands in most prior datasets with abstraction capabilities. The 55.1 million annotated attributes (~39 per record) provide supervision density that enables training models to recognize and handle the many forms sensitive information can take rather than a small predefined set.

2. Multi-domain coverage from a single pipeline. Rather than curating separate datasets for medical, legal, financial, and communication domains, Privasis generates all of them through a unified synthesis pipeline (Table 1), enabling models trained on it to generalize across domains. The 10 main categories and 42 subcategories span an intentionally broad range of document types.

3. Reference-free synthesis. The authors emphasize this repeatedly: Privasis is built "entirely from scratch" using only auxiliary control variables and a public name database. This eliminates the privacy risks associated with datasets that incorporate real-world reference data (even if anonymized), and it means the dataset can be released publicly without legal or ethical constraints tied to the provenance of the underlying data.

4. Context-sensitive, instruction-based sanitization. Prior datasets define a fixed set of PII categories (name, date, address, etc.) that should always be removed. Privasis-Sanitization supports user-specified sanitization targets through natural language instructions, reflecting the reality that sensitivity is contextual—a user might want to remove their medical condition but keep their age, or remove their exact address but keep their city. The paper's pipeline goes beyond fixed PII categories to cover "various information that users may contextually consider sensitive" (Section 3.1).

5. Multiple abstraction levels with retention targets. The sanitization pipeline supports both abstract and drop operations, and explicitly includes retention target attributes ( $\mathcal{K}$ ) that specify what information should be preserved. This creates a more realistic training signal than simple redaction: models learn to selectively remove some information while preserving other information in the same document, which is exactly what real-world privacy-preserving applications require.

6. Designed for training compact, locally-deployable models. The paper repeatedly emphasizes that sanitization models should be small enough to run on-device ("it is desirable for sanitization models to be small enough to run locally, so that sensitive text never needs to leave a user's device"). This motivates the choice to train 0.6B, 1.7B, and 4B parameter models rather than relying on API-based frontier models. The dataset is explicitly designed to support this use case, providing high-quality training triplets (original record, instruction, sanitized record) at a scale that makes supervised fine-tuning of small models effective.

The Guiding Philosophy: Privacy by Design Through Synthetic Data

Underlying the technical contributions is a philosophical position about how to advance privacy research without compromising privacy. The paper's approach is essentially: if we cannot use real private data for research, we should generate synthetic private data that is realistic enough, diverse enough, and large enough to serve as an effective substitute. The validation that generated profiles do not correspond to real people (Section 2.2, manual verification of 100 profiles, plus a larger check on 1,000 profiles with web-search-enabled GPT-5) is presented as evidence that this approach works—the synthetic data is privacy-safe because it is hallucinated from auxiliary variables rather than derived from real individuals, yet the resulting documents achieve "a level of naturalness and coherence comparable to human-written records" based on the blind human evaluation where 113/128 Privasis records were judged natural and coherent versus 111/128 from human-written datasets.

This philosophy directly addresses the chicken-and-egg problem that has plagued privacy research: you need data to develop privacy techniques, but the data you need cannot be shared because it is private. Privasis breaks this cycle by creating a privacy-safe substrate for research, enabling the development and evaluation of privacy-preserving methods without ever touching real sensitive data.

3. Technical Approach

3.1 Reader Orientation

The Privasis system is a synthetic data generation pipeline that produces 1.4 million text records containing rich private information, along with a companion sanitization parallel corpus and a set of compact sanitization models trained on that corpus. It solves the problem that privacy research has been bottlenecked by a lack of large-scale, diverse, publicly-available datasets containing sensitive information—you cannot share real private data, so you cannot build large training corpora for privacy-preserving techniques. The solution takes the shape of a two-stage pipeline: first, generate diverse privacy-rich text records entirely from scratch using only auxiliary control variables and a public name database (no real reference data); second, decompose those records and apply targeted, instruction-based sanitization to produce training triplets that teach small models how to selectively remove or abstract sensitive information from arbitrary text.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components:

Synthesis Pipeline (§2.1): Generates privacy-rich text records from scratch using auxiliary control variables—profiles with personal attributes (names, IDs, contact information), record types (e.g., "psychotherapy billing statement"), and background contexts—which are fed to LLMs to produce initial drafts, then iteratively refined using a weighted acceptance criterion combining LLM quality scoring and Vendi diversity metrics.
Sanitization Pipeline (§3.1): Takes each synthesized record and applies a decomposition-based target identification and rewriting procedure—records are split into manageable chunks, sensitivity targets are selected based on LLM-assigned weights, relevant chunks are identified and sanitized (either abstracted or dropped) using generated instructions, and the sanitized chunks are merged back into a complete sanitized record, producing training triplets of (original record, sanitization instruction, sanitized record).
Evaluation Framework (§4.2): A hierarchical evaluation system that checks for three types of information leakage—direct leak (verbatim appearance of target attribute), inference leak (evaluator LLM can predict the attribute from sanitized text), and proximity leak (evaluator's prediction from sanitized text is as close to the true value as prediction from original text)—while also measuring retention of non-target attributes.
Trained Sanitization Models (§4.1): Compact models (0.6B, 1.7B, 4B parameters) fine-tuned on the Privasis-Sanitization parallel corpus to perform instruction-following text sanitization, enabling local, on-device privacy-preserving processing without sending sensitive data to external servers.

Information flows as follows: auxiliary control variables (profile, record type, background context) → LLM generation of initial record drafts → iterative diversity-preserving refinement → filtered and annotated records stored in Privasis → these records fed into the decomposition-based sanitization pipeline → sanitization triplets stored in Privasis-Sanitization → triplets used to fine-tune compact models → models evaluated using the hierarchical evaluation framework against both vanilla and hard test sets.

3.3 Roadmap for the Deep Dive

First, the synthesis pipeline—how auxiliary control variables are instantiated, how they inform record generation, how iterative refinement with Vendi diversity scores works, and how attributes are annotated and records filtered—since this is the foundation upon which everything else is built.
Second, the sanitization pipeline—how records are decomposed into chunks, how sensitivity targets are selected and weighted, how abstraction and drop instructions are generated, and how retention targets are chosen—since this transforms the synthesized records into a training corpus.
Third, the model training setup—architectures, hyperparameters, and data composition—since this is what actually learns the sanitization task.
Fourth, the hierarchical evaluation framework—how direct leak, inference leak, and proximity leak are measured, and how retention is evaluated—since this is what determines whether the approach works.
Fifth, the design choices and their justifications—why specific decisions were made and what alternatives were considered—since understanding these reveals the paper's contribution to synthetic data generation methodology.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a dataset and systems paper whose core idea is that large-scale, fully synthetic, privacy-rich data can be generated from scratch using LLMs guided by auxiliary control variables and diversity-preserving refinement, and that this data can then be used to train compact models that outperform frontier LLMs on fine-grained text sanitization.

Synthesis Pipeline: Informed Initialization with Auxiliary Control Variables

The core challenge the synthesis pipeline addresses is that directly sampling complex, specific text containing private information from an LLM is difficult because LLMs "tend to favor high-probability, generic continuations rather than rare, highly specific instances of private information." The solution is a bottom-up approach: instead of asking the LLM to generate a document containing private information (which requires it to spontaneously invent specific names, dates, IDs, and contextual details in consistent, realistic combinations), the system provides explicit auxiliary control variables that specify what private information to include and what kind of document to produce. The LLM then generates text conditioned on these variables, guaranteeing that specific details appear while letting the model handle the surface-level writing.

A record $x$ is determined by two primary variables: semantic content ( $c$ ) and structural format ( $f$ ). These are themselves informed by three auxiliary variables that are generated sequentially:

Profile $i$ : Personal attribute values for a fictional individual. A first name is sampled from the US Social Security Administration applicant database, and an LLM is prompted to fill in a fixed set of attributes: last name, sex, ethnicity, citizenship, ID type, ID number, passport number, phone number, email, user handle, URL, and life event. For sex, ethnicity, and life event, predefined options are provided for the LLM to choose from, reducing variance while maintaining coverage. The prompt template is provided in Box A of the appendix.
Record type $d$ : A concise description of what the record is, derived from profile $i$ . The LLM is prompted to generate a list of candidate record types (e.g., "psychotherapy billing statement," "court summons and dismissal notice," "student loan account activity report") and one is randomly selected to promote diversity. The prompt is provided in Box A of the appendix. This random selection from a generated list is a critical design choice: rather than sampling from a fixed taxonomy of document types (which would limit diversity to the taxonomy designer's imagination), the LLM dynamically proposes record types conditioned on the profile, and randomness ensures that even for similar profiles, different record types can be selected across runs.
Background context $b$ : A description of the social context surrounding the record, derived from profile $i$ and record type $d$ . The LLM generates a list of candidate contexts (e.g., "Windsor awakens early on the morning of his medical check-up, feeling a mix of excitement and nervousness about finally seeing his cousin Amina after twelve years...") and one is randomly selected. The prompt is in Box A of the appendix. The background context serves to ground the record in a specific scenario—who is involved, what is happening, what emotional or situational factors are at play—which makes the resulting document more realistic and varied than a template-driven approach.

The record's semantic content $c$ is constructed as the concatenation of $i$ , $d$ , and $b$ . The format $f$ is generated given $d$ and $b$ —the LLM is prompted to generate a list of candidate formats (e.g., "Polite Friendly Structure – Warm Tone," "Standard Dutch Court Format – Authoritative Tone") and one is randomly selected. The prompt is provided in Box A of the appendix.

Finally, the initial draft $x_0$ is generated by the LLM given $c$ and $f$ , using the prompt in Box A of the appendix. Because the process flows bottom-up from explicit auxiliary variables—profile attributes → record type → background context → format → document—the variables serve as free annotations or metadata alongside the record, eliminating the need for post-hoc labeling.

The key property of this approach is that it factorizes the generation problem: the difficult part (creating consistent, realistic combinations of private attributes and document types) is handled by explicit variable instantiation and prompting, while the LLM handles only the surface-level text generation conditioned on those variables. This is why the approach works even without reference data—the LLM is not being asked to invent private information spontaneously, but rather to express pre-specified private information in natural language within a pre-specified document structure.

The initial draft $x_0$ may contain degenerate or overly generic content, while the goal is to produce records with realistic and concrete details. To improve quality, the system iteratively applies selective refinement. At each step $t$ , a candidate draft $x'_t$ is sampled and evaluated against the current draft $x_t$ . An LLM judge compares the two drafts on specificity (how concrete and detailed the information is) and realism (how plausible the document reads as a real-world record).

The challenge is that repeated refinement tends to produce mode collapse: if the LLM judge consistently prefers certain stylistic patterns or information structures, successive refinements converge toward those patterns, and the overall set of produced records becomes homogeneous. The paper explicitly notes:

"We find repeated refinement leads to a lack of variety converging to similar patterns."

To address this, the system uses the Vendi embedding score (Friedman & Dieng, 2023), which measures the diversity of a set in embedding space. The Vendi score is defined for a set of $n$ items with similarity matrix $K$ as the exponential of the Shannon entropy of the eigenvalues of $K/n$ . Intuitively, the score is higher when records cover a broader range of semantic directions (eigenvalues are more uniform, indicating the set spans more dimensions of the embedding space), and lower when they collapse to similar content (eigenvalues are concentrated, indicating all items cluster in a few directions).

The refinement procedure works as follows:

Maintain a collection of all final accepted records produced so far across the entire dataset generation process.
For each refinement evaluation for a new draft, randomly sample up to $n_P$ records from this collection to form a pool $P$ .
Compute the Vendi score of $P \cup \{x_t\}$ (pool with the current accepted draft).
Compute the Vendi score of $P \cup \{x'_t\}$ (pool with the candidate draft).
The diversity contribution of the new candidate is the difference: $\text{Vendi}(P \cup x'_t) - \text{Vendi}(P \cup x_t)$ .
Compute a weighted acceptance score:

$S(x'_t) = \alpha \cdot \text{LLMScore}(x_t, x'_t) + \beta \cdot \bigl(\text{Vendi}(P \cup x_t') - \text{Vendi}(P \cup x_t)\bigr)$

where $\text{LLMScore}(x_t, x'_t)$ is the LLM judge's quality comparison score (specificity and realism), $\alpha$ and $\beta$ are weights balancing quality and diversity, and the Vendi term is positive when the candidate increases diversity and negative when it decreases it.

What it computes: a single scalar score for a candidate draft that combines (a) whether the LLM judge considers it better than the current draft on specificity and realism, and (b) whether adding it to the pool of accepted records would increase or decrease the semantic diversity of the overall dataset. The candidate is accepted only if $S(x'_t) > \tau$ , with $\tau = 0.5$ .

Why this form: a pure quality-driven refinement (only the LLM score term) would cause the distribution of generated records to collapse toward whatever patterns the LLM judge happens to prefer, reducing overall dataset diversity. A pure diversity-driven approach (only the Vendi term) would accept records that are different but potentially low-quality. The weighted combination incentivizes the system to find candidates that are both better than the current draft and contribute to the overall diversity of the dataset, preventing mode collapse. The threshold $\tau = 0.5$ ensures that only candidates that meaningfully improve the combined score are accepted.

The refinement procedure is repeated for up to three steps. An ablation study in Appendix A.1 (Table 9) confirms the importance of both components: the full pipeline achieves a Vendi score of 57.3 and cosine similarity of 0.322, while removing the diversity-preserving term reduces the Vendi score to 50.1 and removing revision entirely drops it to 53.2 (with cosine similarity of 0.324). Removing the record type and format auxiliary variables causes a sharp drop in semantic diversity (cosine similarity increases to 0.408, Vendi score drops to 39.4), confirming that the structured bottom-up initialization is essential for covering diverse document types.

Synthesis Pipeline: Attribute Annotation, Grouping, and Filtering

After the final refinement step, the system extracts and annotates additional attributes that are present in the record but not explicitly captured in the profile $i$ . For example, while the profile might specify the individual's name and date of birth, a generated medical record might also mention a specific clinic name, doctor's name, medication dosage, or appointment reference number—these are extracted from the generated text and added as annotations in JSON format.

These attributes are then grouped into semantic clusters using an LLM. For example, in medical records, attributes like "clinic name," "pharmacy name," and "room number" might be clustered under "location," while "appointment date," "appointment time," and "follow-up date" might be clustered under "scheduling." This grouping produces contextual structure that can be leveraged in downstream sanitization tasks: when a user asks to remove all location information, the system can target the whole cluster rather than requiring individual attribute enumeration. The average record has 6.2 clusters, each with 6.3 attributes.

The filtering step removes records where any type of error occurred during generation (synthesis success rate approximately 94%), records with fewer than 64 words (66,894 excluded), profiles with age under 18 (1,180 excluded), and degenerate cases (1,232 excluded). The 64-word threshold is important because shorter records lack sufficient textual context to be useful for training sanitization models, and the under-18 exclusion ensures the dataset does not contain synthetic private information about fictional minors. The final dataset comprises 1,414,871 records with 55,092,084 annotated attributes, averaging approximately 39 attributes per record.

Synthesis Pipeline: Model Choice and Cost Analysis

The paper uses multiple LLMs for record generation to increase stylistic and distributional diversity: GPT-OSS-120B (67.9% of records), GPT-4.1-Mini (21.6%), Exaone-3.5-32B (7.2%), Qwen3-80B (1.1%), Llama-3.3-70B (1.1%), GPT-4.1 (0.7%), and other frontier models (<0.1%). The dominance of GPT-OSS-120B is justified by an analysis in Appendix A.1 (Table 8): across diversity metrics including MATTR, bigram diversity, Shannon entropy, cosine similarity, Vendi score, and output length, GPT-OSS-120B shows "consistently strong" performance without being the best on any single metric. It generates longer text than other models (611.8 words on average versus 521.9 for Gemini-2.5-pro or 477.6 for GPT-4.1-Mini), which is important for synthesizing realistic documents rather than short snippets. When factoring in cost, it "provides a substantially better price-performance ratio than frontier models... making it more suitable for large-scale generation."

The cost breakdown is provided in Appendix A: generating 10,000 records with GPT-4.1 costs approximately $900, while sanitizing 10,000 records costs about$ 1,100.

A notable positive finding in the appendix is multilingual extensibility: the authors tested generating Chinese and Korean records with Exaone 3.5 32B, Qwen3 80B, and GPT-4.1 through the same pipeline, and native-speaker human evaluation confirmed the generated records were "coherent, contextually appropriate, and diverse (with cosine similarity scores of 0.34 and 0.36, respectively), making them comparable to English ones." This suggests the synthesis pipeline is not English-specific but generalizes to other languages when paired with appropriate LLMs.

Sanitization Pipeline: Decomposition

The sanitization pipeline is designed to address a specific failure mode of monolithic text sanitization: when an LLM is asked to sanitize a long document (Privasis records average 527 words, with some exceeding 1,000 words), it "struggles with sanitizing long text records effectively" (Section 3). The solution is a decomposition-based approach that breaks records into manageable chunks, applies targeted sanitization per chunk, and then merges.

The record $x$ is recursively split into a set of chunks $\mathcal{C} = \{c_1, c_2, \dots\}$ using double newlines, EOS markers, or other natural boundaries (such as section breaks or paragraph separators). Splitting continues until each chunk satisfies $|c_i| \leq \tau$ , where $\tau = 512$ characters. The 512-character threshold was determined empirically (Appendix B, Figure 5): performance peaks at this size and degrades for both smaller chunks (128 characters lose surrounding context necessary for coherent sanitization) and larger chunks (2,048 characters cause models to struggle, similar to the monolithic case).

The variable-length decomposition is intentional: rather than splitting at fixed intervals (which would break sentences mid-thought), the system splits at natural boundaries, preserving local coherence within each chunk. For example, a list of medication prescriptions, a table of financial transactions, or a paragraph describing a medical history would typically remain intact within a single chunk.

Sanitization Pipeline: Target Selection

A key innovation of Privasis-Sanitization over prior PII removal datasets is that sensitivity is treated as contextual rather than predefined. Instead of a fixed set of PII categories (name, date, address, etc.) that should always be removed, the pipeline supports arbitrary user-specified sensitivity targets.

The procedure works as follows:

From the annotated attributes $\mathcal{A}$ of record $x$ , an LLM assigns a sensitivity weight $w_a$ to each attribute $a \in \mathcal{A}$ . The paper notes that this weighting prioritizes "highly sensitive information over relatively benign details that are difficult to sanitize (e.g., happy emotion)." The sensitivity weighting ensures that the training data reflects the reality that not all personal attributes are equally sensitive, and that sanitization models should learn to prioritize the most critical information.
Using these weights, a set of $n$ targets $\mathcal{T} = \{z_1, \dots, z_n\}$ is sampled. Targets may be individual attributes (e.g., "phone_number," "date_of_birth") or attribute groups (the clusters created during attribute annotation in §2.1, such as "Personal Identification Details" or "Financial Account Details"). The stochastic sampling (weighted by sensitivity) means that different training examples target different combinations of attributes, preventing the model from learning a fixed set of what to always sanitize and instead encouraging it to follow instructions.
Each target $z$ is randomly labeled with a sanitization operation $\ell_z \in \{\texttt{abstract}, \texttt{drop}\}$ . The distinction is important: drop means completely removing the information (replacing it with nothing or a generic placeholder), while abstract means replacing it with a less specific version that preserves some utility (e.g., "March 3rd, 2024" → "early March 2024" → "this spring" → "recently"). By training on both operations, the model learns to execute graded abstraction at multiple levels rather than only binary removal.

The paper's instruction-based, stochastically-targeted approach goes beyond PIIs to cover "various information that users may contextually consider sensitive." This is a significant conceptual and practical advance: in real-world privacy-preserving applications, what counts as sensitive is highly dependent on context—a user sharing a document with their doctor might want to keep medical details but remove financial information, while the same user sharing with their accountant wants the opposite. Fixed PII categories cannot capture this contextual variability.

Sanitization Pipeline: Chunk-Level Sanitization with Instruction Generation

For each target $z \in \mathcal{T}$ , the pipeline performs the following steps, with chunk operations running in parallel within the same target (but sequentially across different targets to maintain consistency):

Chunk identification: An LLM identifies the subset of chunks $\mathcal{C}_z \subseteq \mathcal{C}$ that are relevant to target $z$ . Not every target appears in every chunk of a long document, and this filtering step prevents the sanitization model from needing to check every chunk for every target.
Span extraction: For each relevant chunk $c \in \mathcal{C}_z$ , an LLM extracts spans $\mathcal{S}_{z,c}$ that correspond to target $z$ . These are the specific text segments that mention or reveal the target attribute.
Instruction generation: The instruction depends on the operation label:
- If $\ell_z = \texttt{abstract}$ , all chunks in $\mathcal{C}_z$ are concatenated and passed to the LLM to generate an abstraction instruction grounded in the full context. For example, the LLM might generate: "Abstract the specific date as 'in the coming months'" or "Generalize the exact medication dosage to a range." The concatenation of all relevant chunks ensures the instruction is coherent—it can reference the concrete details that appear across the document.
- If $\ell_z = \texttt{drop}$ , a fixed instruction is used: "Drop the information about $\{z\}$ from the text."
Sanitization application: The instruction $\mathsf{instr}_z$ is applied to each $c \in \mathcal{C}_z$ uniformly for consistency across chunks. This means that if a person's name appears in both the header and body of a document, both occurrences are sanitized in the same way.
Merging: After all targets have been processed, the sanitized chunks are merged to reconstruct the sanitized record $\tilde{x}$ .

The full procedure is formalized in Algorithm 1 in Appendix B.

Sanitization Pipeline: Final Instruction Generation and Retention Targets

After per-target sanitization is complete, a final step generates a coherent natural language instruction $\widehat{\mathcal{I}}$ based on all $\{\mathsf{instr}_z\}_{z \in \mathcal{T}}$ . This instruction aggregates the individual per-target instructions into a single user-facing directive, which is what the trained sanitization model receives as input alongside the original text.

To support scenarios where utility preservation is important (not just privacy protection), the system additionally includes a set of retention target attributes $\mathcal{K} = \{k_1, \dots, k_m\} \subseteq \mathcal{A}$ . These represent information that should be explicitly retained in the sanitized output—for example, keeping the city name while removing the street address, or keeping the medical diagnosis while removing the specific doctor's name. The model must learn to selectively remove some information while preserving other information in the same document, which is a harder task than pure redaction because it requires fine-grained discrimination between attributes that appear in close textual proximity.

The selection of $\mathcal{K}$ is designed to minimize interference with the sanitization process: attributes are chosen that have the lowest lexical overlap with the sanitization targets $\mathcal{T}$ , based on ROUGE scores. The paper explains the rationale in Appendix B:

"If the retention target attributes are too similar to the sanitization target attributes, they often end up containing or overlapping with the sanitization targets. In such cases, if the model were to sanitize correctly, it becomes desirable to sanitize the retention targets as well. This leads to worse performance of strong sanitizers."

In other words, if the system asks the model to "remove the street address but keep the city," and the street address contains the city name (e.g., "100 Main Street, Springfield"), then correctly removing the street address would also remove the city name, which conflicts with the retention instruction. By choosing retention targets with minimal lexical overlap with sanitization targets, the system constructs training examples where the sanitization and retention instructions are genuinely compatible rather than contradictory.

When grouped attributes are selected as targets, the final instruction $\widehat{\mathcal{I}}$ occasionally omits individual attribute names and only includes the group label—for example, "Please abstract all information related to locations while keeping the city" rather than "Please abstract the clinic address, session room, and the patient's address while keeping the city." This design choice "encourages contextual generalization to natural user requests" because real users are unlikely to enumerate every individual attribute they want sanitized; they are more likely to describe the category of information they want removed.

The final output of the pipeline is a triplet: (record x, instruction Î, sanitized record x̃), which directly supports supervised fine-tuning of instruction-following sanitization models.

Model Training

The sanitization models are fine-tuned to perform the mapping $(x, \widehat{\mathcal{I}}) \rightarrow \tilde{x}$ : given an original text record and a sanitization instruction, produce a sanitized version where target attributes are abstracted or removed while retention attributes are preserved.

The paper targets Qwen3 models at three parameter scales: 0.6B, 1.7B, and 4B parameters. The choice of lightweight models is motivated by the deployment constraint that "it is safer when sanitization models are run locally"—sensitive text should never need to leave a user's device, which means the sanitization model must be small enough to run on consumer hardware or edge devices.

The training dataset is a 37,000-example subset of Privasis-Sanitization (from the full 100,000 constructed examples), with 70.4% generated by GPT-OSS-120B and 29.6% by Qwen3-80B. The use of only two LLM sources for training data is a deliberate choice to test whether models trained on a subset of the generator distribution can generalize to records from other models in the test set.

Specific training hyperparameters from Appendix C:

For Qwen3 4B: Trained on 2 GPUs with micro-batch size 2 and 16 accumulation steps (global batch size 64), using AdamW optimizer with learning rate $1 \times 10^{-5}$ , weight decay 0.01, WarmupDecayLR scheduler with 50-step warmup and 1,000 total steps, bf16 precision, and ZeRO stage-2 optimization via DeepSpeed on NVIDIA H100s.
For Qwen3 0.6B: Trained on 4 GPUs with micro-batch size 1 and 20 accumulation steps (global batch size 80), using AdamW with learning rate $1 \times 10^{-5}$ , weight decay 0.1, WarmupDecayLR with 23-step warmup and 750 total steps, bf16 precision, and ZeRO stage-3 optimization.

The consistent learning rate of $1 \times 10^{-5}$ across model sizes is notable—it suggests the task does not require substantially different optimization dynamics at different scales, which is somewhat unusual and may reflect the relatively constrained nature of the sanitization task (the output format and vocabulary are similar to the input, unlike more open-ended generation tasks).

Hierarchical Evaluation Framework

The evaluation framework captures three progressively more subtle types of information leakage in sanitized text. This is a significant methodological contribution because it moves beyond simple string matching (which can miss semantically recoverable information) while avoiding the subjectivity of preference-based LLM judgments.

Level 1: Direct Leak. The system performs exact string matching to determine whether the target attribute value $z_{value}$ appears verbatim in the sanitized record. This is the most straightforward failure mode: the model was told to remove something and simply did not remove it (or removed some occurrences but missed others). The paper's analysis shows this is the dominant failure mode across all models (Table 5), accounting for 53.7% to 81.7% of all failures depending on the model.

Level 2: Inference Leak. If no direct match is found, the system tests whether the attribute can be inferred from the sanitized text. An evaluator LLM (GPT-OSS-120B) is prompted: "Please guess the [attribute key]. Make a guess even if it's not included in the given text." The prediction $\hat{z}_{sanitized}$ is checked for exact string match with the true value $z_{value}$ . If they match, the model has failed to prevent inference even though it removed the explicit text.

For example, if the original text said "the patient was treated at Royal Darwin Hospital" and the sanitized text replaces "Royal Darwin Hospital" with "[HOSPITAL NAME]" but leaves the hospital's phone number and department names intact, an evaluator with knowledge of hospital phone numbers might correctly guess which hospital it is. The exact string match criterion is conservative: it requires the evaluator to reproduce precisely the sanitized string, which means the evaluator's signal must be strong enough to recover the exact text, not just the semantic category.

The paper notes that the inference leak ratio is "significantly lower" than proximity leak because of this exact string matching requirement, and that the ratio "would likely increase if semantic entailment were used instead, since some attributes (e.g., lists or long string) are difficult to capture via exact string matching."

Level 3: Proximity Leak. If neither direct match nor inference match occurs, the system checks whether the sanitized text brings the evaluator closer to the true value than the original text would. The evaluator makes predictions from both the sanitized text ( $\hat{z}_{sanitized}$ ) and the original record ( $\hat{z}_{original}$ ). Another evaluator instance assesses which prediction is closer to the true attribute value $z_{value}$ . If $\hat{z}_{sanitized}$ is as close as, or closer than, $\hat{z}_{original}$ , this indicates a proximity leak—the sanitization has not meaningfully reduced the recoverability of the sensitive information.

This is the most subtle form of leakage and captures cases where the model removes explicit text but leaves enough contextual information that the sensitive attribute remains approximately inferable. For example, if the original text says "the last dose was taken at 9:30 PM on 17 September 2022, about 5 hours before the 2:30 AM blood draw" and the sanitized text removes "9:30 PM on 17 September 2022" but keeps "about 5 hours before the 2:30 AM blood draw," an evaluator can reconstruct the approximate time (9:30 PM) even though the exact string no longer appears.

The Successful Record metric requires that no target attributes exhibit any type of leakage (direct, inference, or proximity). A record is counted as successfully sanitized only if all targets pass all three checks. This is what makes the metric stringent: a single attribute with any form of leakage causes the entire record to count as a failure.

To prevent the trivial solution of outputting an empty string (which would avoid all leakage but destroy utility), the framework also evaluates Retention:

For each retention target $k \in \mathcal{K}$ , exact string matching checks whether $k_{value}$ appears verbatim in the sanitized record $\tilde{x}$ .
If no match is found, the evaluator LLM tries to infer the attribute ( $\hat{k}_{sanitized}$ ) from the sanitized text given the attribute key, checking for exact matches.
If still no match, the evaluator is directly queried whether each $k$ is present in the sanitized text given $\hat{k}_{sanitized}$ . A negative response is marked as a retention failure.

The Full Successful Record metric requires both no information leakage for any sanitization target AND preservation of all retention target attributes. This is the primary metric used to compare models because it captures the dual objective of privacy-preserving text sanitization: remove the sensitive stuff, keep everything else.

Vanilla vs. Hard test sets. The paper releases two test sets that probe different aspects of sanitization difficulty:

Vanilla set (1,042 records): Records on which the authors' own decomposition-based sanitization pipeline (using GPT-4.1) achieves a perfect Full Successful Record score. The instructions are typically direct and involve individual attributes rather than grouped attributes.
Hard set (1,149 records): Records where even the decomposition-based pipeline with GPT-4.1 fails to achieve a perfect score. These records are characterized by longer text (619.6 vs. 569.3 words on average), longer instructions (94 vs. 57.2 words), and a higher proportion of grouped attribute targets (87% in hard vs. 60% in vanilla) which require contextual target identification.

The difference is illustrated by examples in Appendix D (Boxes D): in the vanilla example, sanitization is straightforward because the record contains only a single occurrence of the attribute and the instruction is direct. In the hard example, the instructions require contextual understanding to determine which of several possible references to an attribute needs to be sanitized.

Robustness of proximity leak evaluation. The paper validates the proximity leak metric through two checks:

Human evaluation on all proximity leak failure cases ( $n = 140$ ) produced by the o3 model found 98% of them to be correct, indicating high precision of the automated evaluator.
Repeating proximity leak evaluation with two different evaluator models (GPT-OSS-120B and Qwen3-80B) over 5,000 cases achieved 97% inter-model agreement, indicating low susceptibility to evaluator-specific bias.

These validation results are important because they establish that the hierarchical evaluation framework, despite relying on LLM-based evaluators, produces reliable and reproducible judgments. The 97% inter-model agreement is particularly strong evidence: if the evaluation were sensitive to the specific evaluator model, agreement between different models would be substantially lower.

Design Choices and Their Justifications

Why auxiliary control variables rather than prompting for diversity? Direct prompting ("generate a diverse set of medical records with private information") would produce text conditioned only on the LLM's training distribution, which favors high-probability patterns and generic content. The auxiliary variable approach explicitly specifies the private information and document structure, forcing the LLM to incorporate specific details rather than reverting to generic templates. The ablation study (Table 9) confirms this: removing record type and format variables causes a sharp increase in cosine similarity (0.408 vs. 0.322), indicating that generated records become more semantically similar to each other.

Why Vendi score rather than other diversity metrics? The Vendi score operates directly on embedding-space representations and captures the effective dimensionality of the set—how many distinct semantic directions are spanned—rather than just pairwise similarity. Alternatives like average pairwise cosine similarity would only capture whether records are similar on average, not whether they cover a broad range of distinct topics. The Vendi score penalizes both clustering (many records in a few tight clusters) and uniformity (records spread across many narrow directions), favoring distributions that span many semantic dimensions with reasonable coverage of each.

Why decomposition with 512-character chunks? The empirical analysis in Appendix B (Figure 5) shows a clear performance peak at 512 characters, with degradation in both directions. The authors explain: "Smaller chunks (e.g., 128) lose the surrounding context necessary for sanitization, while overly large chunks (e.g., 2048) cause the models to struggle, similar to the vanilla case without our pipeline." The 512-character threshold represents a sweet spot that provides enough context for coherent sanitization (e.g., understanding which pronouns refer to which entities, or resolving cross-sentence references) while being short enough that the model can track all sensitive information within the chunk without losing attention to detail.

Why stochastic target selection with sensitivity weights? If the pipeline always sanitized the same set of attributes (e.g., always remove names and dates), the training data would teach the model to recognize a fixed set of patterns rather than to follow arbitrary instructions. By sampling targets based on sensitivity and varying which attributes are targeted, each training example teaches the model to pay attention to the instruction rather than learning a default set of what to remove. The sensitivity weighting ensures that the training distribution reflects practical priorities: models see more examples of sanitizing highly sensitive attributes (which is what users will most often request) while still learning to handle less sensitive ones when instructed.

Why ROUGE-based selection of retention targets? The choice to select retention targets with low lexical overlap with sanitization targets is an engineering solution to a deeper problem: natural language attributes are not orthogonal. If the system asks the model to remove "street address" and keep "city name," and the street address is "100 Springfield Avenue, Springfield," then removing the street address necessarily removes the city name. The ROUGE-based filtering avoids constructing these contradictory training examples. The acknowledgment that "current LLMs struggle with" semantic selection of genuinely orthogonal attributes (Appendix B) is a candid admission of a limitation: ideally, the system would choose retention targets that are semantically independent of sanitization targets, but in practice, lexical overlap serves as a computationally cheap proxy.

Why Qwen3 as the base model family? The paper does not explicitly justify this choice, but it is consistent with the goal of producing compact, locally-deployable models. Qwen3 models are available at multiple scales (0.6B, 1.7B, 4B, 8B, etc.) within a single architecture family, enabling fair comparison across scales. The choice of an open-weight model family also aligns with the paper's commitment to releasing all models publicly. The fact that base Qwen3 models perform poorly at sanitization (Qwen3-4B achieves only 53.65% Full Successful Record on the vanilla set; Qwen3-0.6B achieves 16.70%) demonstrates that the capability comes from fine-tuning on Privasis-Sanitization, not from any inherent sanitization ability in the base models.

Why three levels of leakage rather than just direct string matching? Direct string matching is a necessary but insufficient measure of sanitization quality. As the error analysis examples in Appendix E demonstrate, models can successfully remove explicit text while leaving enough information for inference. The three-level hierarchy captures progressively more subtle failures: direct leak (the model didn't even try), inference leak (the model tried but left recoverable information), and proximity leak (the model partially obscured but didn't sufficiently degrade the signal). This granularity is important for diagnosing why models fail, which informs how to improve them. The correlation between the three levels (direct leak is most common, followed by proximity leak, with inference leak being rarest due to the strict exact-match requirement) provides a consistent picture of the current state of sanitization technology.

Why train on only 37K of the 100K available examples? The paper does not explicitly address the 37K training subset size, but this likely reflects a practical constraint: fine-tuning on the full 100K examples might not provide proportional benefits given the cost, and 37K already represents a large increase over existing sanitization datasets (the Self-Disclosure Corpus has only 4.8K). The fact that models trained on this subset already outperform frontier LLMs suggests the dataset is rich enough that 37K examples provide sufficient coverage of the sanitization task distribution.

4. Key Insights and Innovations

Innovation 1: Privacy Datasets Can Be Generated Entirely from Scratch, Without Reference Data, at Million-Item Scale

The dominant assumption in privacy-preserving data generation prior to this work was that you needed something real to start from—real clinical records to imitate, real Reddit posts to rephrase, real financial documents to template, real private data to privatize via differential privacy. The paper's core intellectual move is to demonstrate that this assumption is false for the purpose of building research and evaluation datasets: you can generate diverse, realistic, richly-annotated private records at million-item scale using only auxiliary control variables and a public name database, with no reference to any actual private individual's data. The synthesis pipeline is not mimicking or transforming existing private data—it is constructing private-looking data from abstract specifications.

This is a fundamental shift, not an incremental improvement. Prior synthetic data work in privacy falls into two categories: (a) Differentially private generation (Mattern et al., 2022; Yue et al., 2023; McKenna et al., 2025), which requires access to real private data as input and adds noise to protect it—orthogonal to Privasis's goal of creating a public resource without touching real private data at all; and (b) Non-private generation via self-instruction or targeted prompting (Wang et al., 2023; Gunasekar et al., 2023; Kim et al., 2023), which bootstraps from seed data, predefined prompts, or reference trajectories. The paper's critique of the latter is pointed: relying on fixed prompts or seed data "may constrain the diversity and novelty of the generated data." If your seed data consists of 100 medical record templates, your generator will produce variations on those 100 templates, not genuinely novel document types.

The conceptual novelty is not any single algorithmic trick but the factorization of the generation problem into (1) explicit instantiation of auxiliary control variables that specify what private information to include, followed by (2) LLM surface-level text generation conditioned on those variables. The LLM is never asked to invent private information spontaneously (which it does poorly, favoring generic continuations); it is asked to express pre-specified information within a pre-specified document structure, which it does well. This factorization is what makes the reference-free approach viable: the hard combinatorial problem of creating consistent, realistic, diverse combinations of private attributes and document types is handled by the auxiliary variable system and diversity-preserving refinement, while the LLM handles only the relatively easier surface-form generation.

The evidence that this works is not just the scale (1.4M records) but the quality and diversity validation. The blind human evaluation (113/128 Privasis records judged natural and coherent vs. 111/128 from human-written datasets, Section 2.2) establishes that reference-free synthesis produces text comparable to human-authored documents. The diversity metrics in Table 2 are more revealing: Privasis subsets consistently achieve higher MATTR (0.807–0.823 vs. 0.700–0.794 for comparable human-written datasets), higher Shannon entropy, and lower cosine similarity, indicating that the synthetic data is more lexically diverse and less semantically redundant than real human-written data in the same domains. This is a surprising and non-obvious result—naïvely, one would expect synthetic data to be less diverse than human-authored data, not more. The diversity-preserving refinement with Vendi scores explains why: the system actively selects for records that increase semantic spread, which real-world data collection (constrained by what documents happen to exist and be shareable) does not do.

The finding matters beyond this paper because it suggests a general template for synthetic data generation in constrained domains: if you can specify the latent variables that determine what makes an item diverse and realistic (person profiles, document types, contexts), you can generate high-quality synthetic data without reference to real instances, provided you have a mechanism for preventing mode collapse during iterative refinement. This is a transferable methodological insight, not just a dataset release.

Innovation 2: Sensitivity Is Contextual, and Sanitization Models Should Learn from Instructions Rather Than Fixed Categories

The paper's second conceptual move is to reject the standard framing of text sanitization as PII category detection and removal in favor of instruction-following, context-sensitive rewriting. This is a reframing of what the sanitization task is, with significant downstream implications for how models are trained and evaluated.

The standard approach in PII removal datasets (MIMIC-II, TAB, Gretel, PANORAMA) is to define a fixed ontology of sensitive categories—name, date, address, phone number, ID number, etc.—annotate spans in text that match these categories, and train models to detect and remove those spans. This framing assumes sensitivity is a property of the category: all names are sensitive, all dates are sensitive, all addresses are sensitive. But real-world privacy needs are more nuanced. A user sharing a medical record with their doctor wants to keep the diagnosis and treatment details but might want to remove billing information. A user sharing a calendar with a colleague wants to keep meeting times and locations but might want to remove personal appointments. The same attribute (e.g., a date) can be sensitive in one context and essential in another.

Privasis-Sanitization's instruction-based approach represents a shift from category-based to intent-based sanitization. Rather than learning "always remove names and dates," models learn to "follow the instruction"—which might specify removing some names but keeping others, abstracting some dates while dropping others, or targeting entire semantic clusters (all location information) without enumerating individual attributes. The stochastic target selection with sensitivity weights (Section 3.1), the randomized abstract/drop labeling, and the inclusion of retention targets all serve to create a training distribution where the only reliable signal for what to do is the instruction itself—the model cannot fall back on learned heuristics about which categories are "always sensitive."

This is significant beyond the performance numbers because it aligns the training objective with the deployment objective. In deployment, a user will say "remove my financial information but keep my medical history" or "abstract all location details to city level"—they will not provide a list of PII categories. Models trained on fixed-category datasets learn a different mapping (text → redacted text based on predefined categories) than what users need (text + instruction → selectively rewritten text). The paper's framework trains exactly the mapping users need, which is why the compact Privasis-Cleaner models outperform frontier LLMs that have strong general reasoning but were not trained for instruction-following sanitization specifically.

Evidence for the importance of this reframing appears in the retention metrics (Table 4): frontier models systematically over-edit, destroying non-target attributes that should be preserved. GPT-5 retains only 93.4% of non-target attributes on the vanilla set, while Privasis-Cleaner-4B retains 99.2%. The frontier models are applying a sledgehammer—they recognize that the document contains private information and broadly rewrite it—while the trained models learn to perform surgical edits guided by the instruction. The quality of the instruction-following training data, with its explicit retention targets and instruction-aggregation step, is what enables this precision.

Innovation 3: Verbatim Leakage, Not Subtle Inference, Is the Dominant Failure Mode of Current Sanitization Systems

One of the paper's most striking empirical findings is also one of its most important diagnostic contributions: the dominant failure mode in text sanitization is not sophisticated inference from context but simple failure to remove the sensitive text at all. Table 5 breaks down information leakage by type across all evaluated models on the vanilla test set. Direct leak accounts for 53.7% (GPT-OSS-120B) to 81.7% (Qwen3-4B) of all sanitization failures. The models are not cleverly paraphrasing sensitive details in ways that preserve their inferability—they are simply not removing them.

This finding reframes the research priority for text sanitization. The intuitive concern about AI-powered sanitization is that it will be defeated by subtle inference: the model removes the explicit name but leaves enough context that a reader can figure out who is being discussed. The paper's evaluation framework explicitly tests for this (via the inference leak and proximity leak levels), and those failures do occur—but they are the minority. The majority of failures are at the most basic level of the task: the model was told to remove something and did not remove it.

The paper provides diagnostic granularity on why direct leaks occur through the error analysis in Appendix E. The employer name leak example (Box E) is particularly instructive: GPT-5 successfully removes the employer name "Royal Darwin Hospital" from the main body of a document but misses it in the email header and affiliation signature. The model can perform the removal operation—it does so correctly for 90% of occurrences—but it fails at contextual awareness, not recognizing that information can appear in structurally distinct parts of a document (headers, footers, signatures, metadata) that require different recognition strategies than running prose.

This finding has direct implications for training data design. If the dominant failure mode is not removing text that should be removed, then training data should emphasize diverse positional contexts—teaching models to recognize sensitive information in headers, signatures, metadata fields, tables, and lists, not just in paragraph text. The paper's decomposition-based pipeline naturally creates this diversity because it operates at the chunk level on documents that contain all these structural elements, but the diagnostic value of the finding extends beyond this paper: any future sanitization dataset or model should be evaluated on whether direct leaks remain the primary bottleneck, and if so, training should prioritize improving coverage of structurally diverse text segments over more sophisticated abstraction strategies.

The inter-model agreement on this pattern (all models show direct leak as the dominant failure mode, ranging from 53.7% to 81.7%) suggests this is not a quirk of specific architectures or training procedures but a fundamental property of current LLM behavior on this task. These models have been trained primarily on well-formed prose and dialogue; they have not been trained to attend to the structural elements of documents (headers, signatures, metadata fields) as potential locations of sensitive information. Specialized training on structurally diverse documents is the remedy, and Privasis provides the substrate for that training.

Innovation 4: Compact, Locally-Deployable Models Can Match or Exceed Frontier LLMs on Fine-Grained Rewriting Tasks—But Only with Sufficient Training Data Diversity

The paper's headline performance result—Privasis-Cleaner-4B achieves 72.5% full success rate on the vanilla test set, outperforming GPT-5 (70.1%) and o3 (70.3%)—is individually interesting but gains significance from why it happens. The base Qwen3-4B model, without fine-tuning, achieves only 53.65%. The 0.6B variant drops to 16.70%. Fine-tuning on a 37K-example subset of Privasis-Sanitization more than doubles the 0.6B model's performance (to 68.04%) and gives the 4B model a ~19 percentage point improvement.

This pattern challenges a growing assumption in the LLM deployment literature: that for complex, nuanced text-processing tasks, the best approach is to use the largest available frontier model via API. The paper demonstrates that for sanitization specifically—and by extension, potentially for other fine-grained, instruction-following rewriting tasks—a carefully constructed training dataset enables compact models to exceed the performance of models orders of magnitude larger, while also satisfying the crucial deployment constraint that sensitive text should never leave the user's device.

The conceptual contribution here is not simply "small models can be good" but rather the specific mechanism by which they become good: instruction-diverse training data at sufficient scale. The base Qwen3 models are not inherently good at sanitization—they are actively bad at it. The capability comes entirely from the training data. Frontier models like GPT-5 have seen vastly more total training data than Privasis-Cleaner-4B (which is fine-tuned on only 37K sanitization examples), but the frontier models' training data almost certainly contains very few examples of instruction-following text sanitization with explicit retention targets. General-purpose training on internet text teaches models to generate, summarize, and answer questions—it does not teach them to perform surgical removal of specific attributes from long documents while preserving everything else. The 37K examples in Privasis-Sanitization, despite being a tiny fraction of GPT-5's total training data, represent a higher density of task-relevant signal than the frontier models have ever seen.

The zero-shot generalization result on NaP² (Table 7) provides external validation of this claim. Privasis-Cleaner-4B, never trained on NaP², achieves the same leak ratio (10%) as a model fine-tuned directly on NaP², while the NaP²-fine-tuned model scores only 32% on Privasis's Full Successful Record metric compared to 72.5% for Privasis-Cleaner-4B. This asymmetry—the Privasis-trained model generalizes to NaP² better than the NaP²-trained model generalizes to Privasis—is strong evidence that training data diversity, not just task-specificity, determines generalization quality. NaP² is small (4.8K records) and single-domain; Privasis is large (1.4M records) and multi-domain. A model trained on the larger, more diverse dataset learns a more robust sanitization capability that transfers to narrower domains, while the reverse is not true.

This finding has implications beyond privacy. For any fine-grained text rewriting task where a compact on-device model is desirable—document redaction, controlled simplification, style transfer with content preservation, selective summarization—the lesson is that investing in diverse, instruction-rich training data at scale may yield better returns than scaling model parameters or relying on general-purpose frontier models. The paper does not claim this generalizes to all tasks, but it establishes a clear existence proof for sanitization, and the mechanism (task-relevant signal density in training data versus model scale) is plausibly transferable.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation uses the Privasis-Sanitization test set, constructed from records generated by four frontier models not used in the training data synthesis: Gemini-2.5-pro, GPT-5, Llama-4-Maverick, and Qwen3-235B. This is split into two deliberately distinct subsets: a Vanilla test set of 1,042 records where the authors' own decomposition-based sanitization pipeline (using GPT-4.1) achieves a perfect Full Successful Record score, and a Hard test set of 1,149 records where even that pipeline fails. The hard set is characterized by longer records (619.6 vs. 569.3 words on average), longer instructions (94 vs. 57.2 words), and a higher proportion of grouped attribute targets (87% vs. 60% in vanilla) that require contextual target identification rather than simple individual-attribute matching. This two-tier design probes both whether models can handle straightforward sanitization (vanilla) and whether they can generalize to cases requiring deeper contextual reasoning (hard).
Base model(s). The trained sanitization models use the Qwen3 family at three scales: 0.6B, 1.7B, and 4B parameters. The choice is driven by the deployment constraint that sanitization models should run locally on-device ("sensitive text never needs to leave a user's device"), and Qwen3 provides open-weight models at compact scales suitable for consumer hardware. For comparison, the paper evaluates a range of frontier LLMs accessed via API: o3, DeepSeek R1, GPT-5, GPT-4.1, GPT-OSS-120B, LLaMA-4 Maverick, and Qwen3-235B, as well as the untuned base models Qwen3-4B and Qwen3-0.6B. The frontier models are prompted with the sanitization instruction and original text; details of the exact prompting format are not specified beyond what is implicit in the instruction structure described in Section 3.1.
Metrics. The primary evaluation metric is the Full Successful Record rate: the percentage of records where (a) no sanitization target attribute $z \in \mathcal{T}$ exhibits any of three leakage types—direct, inference, or proximity leak—AND (b) all retention target attributes $k \in \mathcal{K}$ are preserved in the sanitized output. This is a stringent all-or-nothing metric: a single missed attribute or a single retention failure causes the entire record to count as a failure. Supporting metrics include Successful Attribute (the percentage of all individual target attributes across the test set that are successfully sanitized without leakage), Successful Attribute/Record (the average within-record success rate, computed as the mean of per-record attribute success rates), Successful Record considering sanitization only (ignoring retention), and separate Sanitization Success and Retention Success rates. The three leakage types are checked hierarchically: direct leak via exact string matching of $z_{value}$ in the sanitized text, inference leak via an evaluator LLM (GPT-OSS-120B) attempting to predict $z_{value}$ from the sanitized text and attribute key, and proximity leak via comparing the evaluator's predictions from the sanitized versus original text to assess whether sanitization meaningfully degraded recoverability (§4.2).
Baselines. The paper uses seven frontier LLMs as off-the-shelf baselines: o3, DeepSeek R1, GPT-5, GPT-4.1, GPT-OSS-120B, LLaMA-4 Maverick, and Qwen3-235B. Each receives the same (original record, instruction) pairs and is evaluated under identical conditions. Additionally, untuned base models (Qwen3-4B and Qwen3-0.6B) serve as lower bounds to isolate the contribution of Privasis-Sanitization fine-tuning from any inherent sanitization capability in the architecture. For the generalization experiment (§4.4), a NaP²-fine-tuned model (4B, trained directly on the NaP² dataset from Huang et al., 2025) serves as a domain-specific baseline for comparison with Privasis-Cleaner-4B evaluated zero-shot on NaP².
Generation budget / compute accounting. There is no explicit compute budget comparison in this paper's evaluation, as the experiments compare output quality (sanitization success rates) rather than inference cost. The models are evaluated under a fixed task definition—given an input record and instruction, produce a sanitized output—and the compute cost of producing that output is not measured or constrained. This is a notable omission: the paper argues for compact models on the basis that they enable local deployment, but it does not quantify inference latency, memory usage, or FLOPs for the different model scales. The cost analysis in Appendix A focuses on generating the dataset, not on inference costs during deployment. For the generalization experiment on NaP², the evaluation protocol is adapted to match NaP²'s original framework rather than Privasis's hierarchical evaluation, using NaP²'s proximity-leak evaluation with GPT-OSS-120B as the evaluator.
Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported. The test sets are fixed (1,042 vanilla, 1,149 hard), and all models are evaluated on the same sets. The proximity leak evaluation is validated for robustness: human evaluation of 140 o3 proximity-leak failure cases found 98% agreement, and repeating proximity leak judgments with two different evaluator models (GPT-OSS-120B and Qwen3-80B) over 5,000 cases achieved 97% inter-model agreement, providing evidence that the evaluation framework is reliable and not sensitive to evaluator-specific biases. The paper does not report confidence intervals, error bars, or statistical tests for any of the success rate comparisons.

Main Quantitative Results

Overall Sanitization Performance on Vanilla and Hard Test Sets

Table 4 is the central results table, reporting performance across all models on both test sets. The headline results are:

On the Vanilla test set, Privasis-Cleaner-4B achieves a 72.5% Full Successful Record rate, outperforming all tested frontier LLMs: o3 (70.3%), GPT-5 (70.1%), DeepSeek R1 (69.6%), GPT-4.1 (70.1%), GPT-OSS-120B (67.7%), LLaMA-4 Maverick (67.2%), and Qwen3-235B (64.4%). The margin over the best frontier model (o3) is modest at 2.2 percentage points but is achieved by a model with roughly two orders of magnitude fewer parameters. Even the smallest trained model, Privasis-Cleaner-0.6B at 68.0%, surpasses GPT-OSS-120B, LLaMA-4 Maverick, and Qwen3-235B, while the untuned Qwen3-0.6B manages only 16.7%—an improvement of 51.3 percentage points from fine-tuning on 37K examples.

On the Hard test set, performance collapses for all models: GPT-5 leads at 13.1%, followed by Privasis-Cleaner-4B at 12.4%, o3 at 11.7%, DeepSeek R1 at 11.2%, and GPT-4.1 at 12.2%. The rankings shift slightly—GPT-5 regains a narrow edge over Privasis-Cleaner-4B (13.1% vs. 12.4%) after being surpassed on the vanilla set—but the absolute numbers are grim across the board, with all models failing on roughly 87–89% of hard records.

A critical pattern in Table 4 is the gap between sanitization success and full success rates. On the vanilla set, the Successful Record (sanitization only, ignoring retention) scores are consistently higher than Full Successful Record scores: Privasis-Cleaner-4B achieves 72.8% sanitization success vs. 72.5% full success—a negligible difference—while GPT-5 achieves 73.9% sanitization success vs. 70.1% full success, a 3.8 percentage-point gap, and LLaMA-4 Maverick shows a 6.4 point gap (73.6% vs. 67.2%). The frontier models systematically destroy non-target attributes that should be retained. The Retention Success columns quantify this: GPT-5 retains only 93.4% of non-target attributes on the vanilla set, while Privasis-Cleaner-4B retains 99.2%. Frontier models over-edit—they sanitize broadly rather than surgically following the instruction.

The Successful Attribute and Successful Attribute/Record metrics tell a complementary story. On the vanilla set, most models achieve high per-attribute success rates (GPT-5: 90.2% Successful Attribute, 91.8% Successful Attribute/Record; o3: 91.5%, 91.6%), but the Successful Record rate is much lower (around 70%). This reveals the last-mile problem: models correctly sanitize most individual attributes but reliably miss at least one per record. On the hard set, per-attribute success drops substantially (GPT-5: 78.8% Successful Attribute, 75.3% Successful Attribute/Record), and the record-level success collapses to 13.1%, indicating that hard records contain more attributes overall and the probability of missing at least one compounds.

Leakage Type Analysis

Table 5 breaks down sanitization failures by leakage type on the vanilla test set. The dominant failure mode across all models is direct leak: the sensitive attribute appears verbatim in the supposedly sanitized output. GPT-OSS-120B has the lowest direct leak ratio at 53.7%, suggesting it is best at identifying what needs to be removed, but its Successful Record score (67.7%) is only middling—it identifies the targets but fails to actually sanitize them effectively. Qwen3-235B has the highest direct leak ratio at 75.3%, reflecting a broader limitation in the Qwen3 model family (the untuned Qwen3-4B is even worse at 81.7%). Privasis-Cleaner-4B achieves 64.8% direct leak, the third-lowest after GPT-OSS-120B and o3 (60.4%).

Inference leak ratios are low across the board (3.1%–8.4%) because of the strict exact string matching requirement. The paper acknowledges this likely underestimates true inference risk: "The inference leak ratio would likely increase if semantic entailment were used instead, since some attributes (e.g., lists or long string) are difficult to capture via exact string matching." Proximity leak accounts for the remainder of failures (15.2%–37.9%), with GPT-OSS-120B showing the highest proximity leak ratio (37.9%), consistent with the interpretation that it successfully removes explicit text but leaves sufficient contextual information for approximate recovery.

The pattern is consistent: models that are better at removing explicit text (lower direct leak) tend to have higher proximity leak, suggesting a tradeoff between removing surface strings and leaving contextual inference pathways open.

Domain and Attribute Difficulty Analysis

Table 6 reports the top 8 domain categories where each model fails most often on the vanilla set. Business & Finance and Health & Wellness are consistently the most challenging categories across models, though the order varies. o3 struggles disproportionately with Health & Wellness (26.5% of its failures) compared to other models; DeepSeek R1 and LLaMA-4 Maverick find Business & Finance hardest (22.7% and 21.5%, respectively). Privasis-Cleaner-4B shows a notably more balanced distribution across categories—its top category (Business & Finance at 15.2%) accounts for a smaller fraction of total failures than any frontier model's top category, suggesting more uniform capability across domains rather than extreme strength in some and weakness in others.

Table 10 (Appendix E) identifies the specific attributes that cause the most failures. Name-related attributes (full name, last name, user handle) and dates (event_date) dominate the failure lists across all models. For o3, full_name accounts for 2.7% of all attribute failures, last_name for 1.8%, and event_date for 1.1%. The pattern is similar for other models, with employer, id_type, and contact_email also appearing frequently. The consistency across models suggests these attribute types are inherently difficult—names can appear in multiple forms (first only, last only, full, with titles, in signatures, in headers), and dates have many formats—rather than reflecting model-specific weaknesses.

Zero-Shot Generalization to NaP²

Table 7 reports the cross-dataset generalization experiment. Privasis-Cleaner-4B, evaluated zero-shot on NaP² (never trained on NaP²), achieves a leak ratio of 10.0%, matching the performance of a model explicitly fine-tuned on NaP² (also 10.0%). However, when evaluated on Privasis's Full Successful Record metric, the NaP²-fine-tuned model achieves only 32.0%, compared to Privasis-Cleaner-4B's 72.5%. The asymmetry is striking: the Privasis-trained model generalizes to NaP² as well as a NaP²-specialized model, but the NaP²-specialized model cannot handle the diversity of Privasis. This is the paper's strongest evidence that training on a large, diverse, multi-domain sanitization dataset produces more robust generalization than training on a smaller, single-domain dataset, even when evaluated on that specific domain.

The evaluation protocol for NaP² uses NaP²'s own proximity-leak framework (GPT-OSS-120B judging whether the sanitized or original text is closer to the sensitive information) rather than Privasis's full hierarchical evaluation, so the 10.0% leak ratio is not directly comparable to the Privasis leakage metrics in Table 5. The paper does not report Privasis-Cleaner's performance broken down by leakage type on NaP², nor does it report the NaP²-fine-tuned model's leakage type breakdown on Privasis, which would provide more granular insight into where the generalization asymmetries arise.

Why Frontier Models Underperform Relative to Trained Compact Models

The results in Table 4, combined with the retention metrics and leakage analysis, paint a specific picture of frontier model failure. It is not that frontier models cannot understand sanitization instructions—they clearly can, as shown by their 88–92% per-attribute success on the vanilla set. The failures are concentrated in two areas:

1. The last attribute in each record. Per-attribute success rates of ~90% combined with ~70% record success implies models are missing roughly one attribute per record on average. The failure is not random across attributes but systematic: the models correctly sanitize most attributes but reliably fail on a small subset. The error analysis examples suggest this subset often involves attributes appearing in structurally distinct parts of documents (headers, signatures, metadata) or attributes requiring disambiguation from similar non-target attributes.

2. Over-editing of non-target content. The retention gap—frontier models retaining only 89–95% of non-target attributes vs. 99% for Privasis-Cleaner—indicates a lack of surgical precision. Frontier models approach sanitization as a broad rewriting task: they identify the document as containing sensitive information and rewrite it to be safer, often removing or altering non-target content in the process. The instruction-following training on Privasis-Sanitization, with its explicit retention targets, teaches models to make precise, localized edits rather than wholesale rewriting.

The inference and proximity leak rates (Table 5) suggest that when frontier models do successfully remove explicit text, they leave varying amounts of inferable context. GPT-OSS-120B, despite the lowest direct leak rate (53.7%), has the highest proximity leak rate (37.9%)—it is good at surface removal but poor at blocking inference. This is consistent with a model that has strong general text comprehension but hasn't been trained specifically to consider downstream inferability when making edits.

Ablation Studies and Robustness Checks

Synthesis pipeline component ablation (Appendix A.1, Table 9): Removing auxiliary control variables for record type and format causes a sharp increase in cosine similarity (0.408 vs. 0.322 full pipeline) and a drop in Vendi score (39.4 vs. 57.3), confirming that the structured bottom-up initialization is the primary driver of semantic diversity. Removing the revision step entirely reduces the Vendi score to 53.2 and lowers Shannon entropy (7.07 vs. 7.14), showing that iterative refinement adds diversity beyond what initialization alone provides. Removing the diversity-preserving term during refinement (keeping only the LLM quality score) causes the Vendi score to drop to 50.1 while MATTR and bigram diversity remain similar (0.805, 0.863 vs. 0.810, 0.890), indicating that quality-only refinement causes semantic mode collapse without necessarily reducing lexical diversity. The full pipeline achieves the highest Vendi score (57.3) and lowest cosine similarity (0.322), validating the combined approach.

Chunk size for decomposition (Appendix B, Figure 5): Performance peaks at 512 characters and degrades in both directions. Smaller chunks (128 characters) lose the surrounding context necessary for coherent sanitization—the model cannot resolve cross-sentence references or understand which pronouns refer to which entities. Larger chunks (2,048 characters) cause models to struggle similarly to the monolithic case, suggesting an effective attention or context-tracking bottleneck at longer lengths. The paper reports this as an empirical finding without mechanistic analysis of why 512 characters is optimal, but the monotonic degradation on both sides suggests a genuine tradeoff rather than an artifact of a particular model or evaluation setup.

Evaluator robustness checks (§4.4): Human evaluation of 140 proximity-leak failure cases from o3 found 98% agreement with the automated evaluator, indicating the proximity leak metric has high precision. Re-running proximity leak evaluation with two different evaluator models (GPT-OSS-120B and Qwen3-80B) on 5,000 cases achieved 97% inter-model agreement, indicating the evaluation is not brittle to the choice of evaluator LLM. These checks are important because the hierarchical evaluation framework relies on LLM-based judgments for inference and proximity leak detection, and without these validations, the results could be attributed to evaluator bias rather than genuine model performance differences.

Profile verification for privacy safety (§2.2): Manual investigation of 100 sampled profiles using Gemini-2.5-Pro Deep Research found that 5 profiles were incomplete due to off-topic responses, 15 returned multiple potential matches that were manually disambiguated (all confirmed as non-matches after checking attributes), 3 cases reported exact name/sex/nationality matches but showed major discrepancies in age and contact information, and the remaining profiles were all reported as fabricated. A larger check on 1,000 profiles with web-search-enabled GPT-5 found none judged as real. None of the generated URLs were accessible. The authors also ran the verification tool on themselves as a sanity check, and it correctly identified all authors as real individuals, increasing confidence in its ability to distinguish fabricated from real profiles.

Model family comparison for synthesis (Appendix A.1, Table 8): Across seven models evaluated on the synthesis task, GPT-OSS-120B shows consistently strong performance without being best on any single metric, generates the second-longest text (611.8 words vs. GPT-5's 1,168.1), and provides substantially better cost-performance ratio than frontier models. GPT-5 generates the longest text (1,168.1 words) but at frontier-model API cost. LLaMA-4 Maverick and LLaMA-3.3-70B show notably lower bigram diversity (0.8669, 0.8665 vs. 0.8970–0.9162 for other models), suggesting they produce more repetitive text. Exaone-3.5-32B shows competitive diversity metrics (MATTR 0.8036, cosine similarity 0.2931) despite being the smallest model tested for synthesis (32B).

Multilingual extensibility (Appendix A): Testing the synthesis pipeline in Chinese and Korean with Exaone 3.5 32B, Qwen3 80B, and GPT-4.1 produced records that native-speaker reviewers confirmed were "coherent, contextually appropriate, and diverse (with cosine similarity scores of 0.34 and 0.36, respectively), making them comparable to English ones." This is not a formal evaluation—no quantitative metrics beyond cosine similarity are reported, and sample sizes are not specified—but it provides suggestive evidence that the pipeline is language-agnostic when paired with appropriate LLMs.

Retention target selection strategy (Appendix B): The paper explains the choice of ROUGE-based lexical overlap to select retention targets but does not report an ablation comparing this approach to alternatives (e.g., semantic similarity-based selection, random selection, or no retention targets). The paper acknowledges that "current LLMs struggle with" semantic selection of orthogonal attributes, making lexical overlap a practical proxy, but this is identified as a limitation rather than validated as optimal. An ablation measuring retention success rates under different retention target selection strategies would strengthen the claim that the ROUGE-based approach is necessary rather than merely convenient.

Critical Assessment

Claim: Privasis-Cleaner-4B outperforms frontier LLMs on text sanitization

What the experiments actually show: On the vanilla test set, Privasis-Cleaner-4B achieves 72.5% Full Successful Record vs. 70.3% for o3 and 70.1% for GPT-5 (Table 4). On the hard test set, GPT-5 regains the lead at 13.1% vs. 12.4% for Privasis-Cleaner-4B. The claim of "outperforms" holds on the vanilla set by a narrow margin (2.2 percentage points over the best frontier model) but does not hold on the hard set, where GPT-5 leads by 0.7 percentage points.

The broader pattern in Table 4 is that trained compact models are competitive with frontier LLMs, not uniformly superior. Privasis-Cleaner-4B outperforms some frontier models (GPT-OSS-120B, LLaMA-4 Maverick, Qwen3-235B) by meaningful margins, essentially ties with others (GPT-4.1, DeepSeek R1), and is slightly ahead of the best (o3, GPT-5) on vanilla but slightly behind on hard. The fair characterization is that fine-tuning on Privasis-Sanitization elevates a 4B model into the performance range of frontier LLMs for this specific task, not that it definitively surpasses them. The paper's title claim that compact models "outperform" GPT-5 and Qwen3-235B is accurate for those specific comparisons but overstates the relationship to o3 and GPT-5, where performance is nearly identical.

A missing experiment that would strengthen this claim: evaluating frontier LLMs when fine-tuned on the same Privasis-Sanitization training data. If frontier models fine-tuned on Privasis-Sanitization substantially outperform the compact models, it would indicate that model scale still matters when training data is controlled—the compact models only appear competitive because frontier models are evaluated zero-shot. Conversely, if frontier models fine-tuned on Privasis-Sanitization perform similarly to the compact models, it would suggest the training data is the binding constraint, not model capacity.

Claim: Frontier models struggle with fine-grained sanitization, revealing a gap between general reasoning and task-specific rewriting

What the experiments actually show: This claim is strongly supported. The 64–70% Full Successful Record rates for frontier models on the vanilla set (Table 4), on records where the authors' own pipeline achieves 100%, is compelling evidence of a genuine capability gap. The retention metrics reinforce this: frontier models over-edit, destroying non-target attributes at rates of 5–11% vs. less than 1% for trained models. The direct leak dominance (Table 5) shows that failures are concentrated at the most basic level—not removing text that should be removed—rather than at sophisticated inference-blocking. The error analysis examples (Appendix E) provide concrete instances of frontier models failing at contextual awareness (missing employer name in headers) and failing to block inference pathways (leaving email domains that reveal masked journal names). These are not failures of general reasoning ability but failures of task-specific training: frontier models have not been trained to perform surgical text editing guided by precise instructions with explicit retention constraints.

A limitation in the evidence: the paper does not evaluate whether frontier model performance improves with few-shot examples or more carefully engineered prompts. The prompting format for frontier models is implicit (the instruction structure from the Privasis-Sanitization triplets) but not systematically varied. It is possible that different prompting strategies—chain-of-thought, explicit enumeration of targets, step-by-step verification—would substantially improve frontier model performance, potentially closing or reversing the gap with the fine-tuned compact models. The absence of prompt engineering ablation means the frontier model results represent a lower bound on what they could achieve with optimal prompting, not necessarily their ceiling.

Claim: The hierarchical evaluation framework captures multi-level leakage beyond simple string matching

What the experiments actually show: The framework is well-motivated and the three-level hierarchy is conceptually sound. The validation checks—98% human agreement on 140 proximity leak cases, 97% inter-model agreement across two evaluators on 5,000 cases—establish that the evaluation produces reliable, reproducible judgments. This is strong evidence that the framework is methodologically sound.

A limitation: the inference leak level uses exact string matching on the evaluator's prediction, which the paper acknowledges is conservative and "would likely increase if semantic entailment were used instead." This means the inference leak numbers in Table 5 (3.1–8.4%) underestimate the true risk of attribute inference from sanitized text. The proximity leak level partially compensates for this by capturing cases where the evaluator can approximately reconstruct the attribute even if the exact string doesn't match, but it does not capture cases where the evaluator can narrow the attribute to a small set (e.g., identifying the hospital as one of three possibilities rather than the exact one). A semantic entailment evaluation—judging whether the sanitized text logically entails or strongly implies the attribute value, even if not exactly—would be more comprehensive but also more subjective and harder to validate. The paper's choice of conservative exact matching for inference leak and comparative proximity evaluation is a reasonable pragmatic tradeoff, but the framework should be understood as providing a lower bound on true inference risk.

Claim: Training on Privasis yields superior generalization due to scale and diversity (NaP² experiment)

What the experiments actually show: Table 7 provides evidence of asymmetric transfer: Privasis-Cleaner-4B matches NaP²-fine-tuned performance on NaP² (10.0% leak ratio) while dramatically outperforming it on Privasis (72.5% vs. 32.0%). This is consistent with the diversity hypothesis—training on a larger, more diverse dataset produces more robust capabilities—but the experiment is a single pairwise comparison between two datasets. Stronger evidence would require demonstrating a monotonic relationship: as training data diversity increases (e.g., by training on increasingly diverse subsets of Privasis), does NaP² generalization systematically improve? Or does performance on specific domains saturate after a certain diversity threshold? The current experiment establishes existence (the Privasis-trained model transfers well) but not mechanism (whether diversity per se, total training examples, domain coverage, or some other factor drives the transfer).

A further limitation: the NaP² leak ratio of 10.0% is not decomposed by leakage type, so it is unclear whether the Privasis-Cleaner's failures on NaP² follow the same pattern (direct leak dominant) as on Privasis, or whether the domain shift introduces new failure modes. Additionally, NaP²'s proximity-leak evaluation protocol (binary judgment of whether sanitized or original is closer to sensitive information) is coarser than Privasis's three-level hierarchy, making the comparison of absolute numbers across datasets potentially misleading.

Claim: The dataset enables training compact, locally-deployable models

What the experiments actually show: The paper demonstrates that models as small as 0.6B parameters can achieve non-trivial sanitization performance (68.0% Full Successful Record, Table 4) after fine-tuning on Privasis-Sanitization, compared to 16.7% for the untuned base model. The 4B model achieves performance competitive with frontier LLMs. This establishes that the task is learnable at compact scales given appropriate training data, which is a necessary condition for the local deployment argument.

What is missing: The paper provides no measurements of inference latency, memory footprint, throughput, or energy consumption for these models. The claim that they are "compact enough for local deployment" is made on the basis of parameter count alone, without evidence that a 4B model can run with acceptable latency on consumer hardware (laptops, phones) for the typical record lengths in Privasis (average 527 words). For the 0.6B model, local deployment is more plausible given its size, but its performance (68.0% Full Successful Record on vanilla, 9.3% on hard) may not be sufficient for practical use. A deployment-feasibility analysis—latency benchmarks on representative hardware, memory requirements, batch processing capability—would substantially strengthen this claim.

Overall Strengths of the Experimental Design

The evaluation framework is thorough in its measurement of leakage types and its inclusion of both sanitization and retention metrics. The two-tier test set design (vanilla vs. hard) provides a meaningful difficulty gradient, and the hard set's characteristics (longer records, more grouped attributes, longer instructions) are well-documented. The evaluator robustness checks are a genuine strength—many papers using LLM-based evaluation do not validate against human judgment or test inter-evaluator agreement, and the 97–98% agreement rates here are reassuring. The inclusion of base model baselines (untuned Qwen3-4B and 0.6B) cleanly isolates the effect of fine-tuning from architectural capability. The NaP² generalization experiment, while limited to a single external dataset, provides the only evidence of out-of-distribution robustness and is methodologically important.

Overall Weaknesses of the Experimental Design

The test sets are fixed at around 1,000 records each. No confidence intervals or statistical tests are reported, so it is unclear whether the 2.2 percentage-point gap between Privasis-Cleaner-4B and o3 (72.5% vs. 70.3%) is statistically significant or within sampling variance. The paper evaluates only one model family (Qwen3) for fine-tuning, so the generalization of the finding to other compact architectures (Llama, Mistral, Phi) is unknown. No ablation of training data size is reported—does performance continue to improve with more than 37K examples, or has it saturated? The prompt engineering for frontier models is not systematically explored, and LLM-based evaluations (for inference and proximity leak) are used in the evaluation framework, creating a potential circularity where LLMs evaluate LLMs on a task they themselves perform poorly at—the evaluator robustness checks partially address this but do not eliminate the concern, since the evaluator models (GPT-OSS-120B, Qwen3-80B) are themselves frontier or near-frontier LLMs and may share failure modes with the models being evaluated.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Problem Is Unresolved—Deployment Requires Knowing Which Records Are "Hard"

The assumption or constraint: The paper's sanitization pipeline and evaluation framework are built on records where the difficulty of sanitization (vanilla vs. hard) is known in advance. The vanilla test set consists of records "on which our sanitization pipeline achieves a perfect Full Successful Record score" (Section 4.2), while the hard set consists of records where it fails. In a real deployment, a user submitting a document for sanitization does not know whether it falls into the vanilla or hard category. The paper makes no attempt to predict record difficulty from features of the input text or instruction, and does not discuss how a deployed system would handle the 87–89% failure rate observed on hard records across all models.

The consequence: A deployed Privasis-Cleaner model would operate with unknown and uncalibrated risk. On records resembling the vanilla distribution, the model achieves 72.5% Full Successful Record; on records resembling the hard distribution, it achieves only 12.4%. A user submitting a single document has no way to know which regime applies to their record, and therefore no way to estimate the probability that their sanitized output will contain a leak. For privacy-sensitive applications—the explicit motivation for the work—this uncertainty is unacceptable: a system that silently fails on 87% of hard records cannot be responsibly deployed without either (a) a reliable difficulty estimator that warns users when the model is likely to fail, or (b) a human-in-the-loop verification step that checks the model's output. Neither component exists in the current system.

The problem is compounded by the composition of the hard set. The paper notes that the hard set has "longer records (619.6 vs. 569.3 words) and longer instructions (94 vs. 57.2 words), reflecting higher complexity" (Section 4.2). A real user who needs to sanitize a long, complex record—exactly the kind of record where privacy protection is most critical—is most likely to encounter the regime where all models, including Privasis-Cleaner, perform near the floor.

What evidence exists in the paper: Table 4 directly shows the 60-percentage-point performance gap between vanilla and hard sets for Privasis-Cleaner-4B (72.5% vs. 12.4%). The hard set is explicitly characterized as having 87% grouped attribute targets vs. 60% in vanilla, and longer average record and instruction lengths (Section 4.2). The paper provides no model, heuristic, or feature analysis for predicting whether a given record falls into the hard regime, nor does it measure how often real user-submitted documents would fall into each category.

Mitigation status: Not addressed. The authors do not acknowledge this as a deployment limitation in the main text. The discussion of the hard set focuses on it as an evaluation challenge for benchmarking models, not on the practical problem of how a user would know whether their sanitization request is likely to succeed. Section 6 (Conclusion) states that the work will "stimulate research in privacy-preserving generation, controllable sanitization, and agentic systems processing sensitive data" but does not identify difficulty estimation or failure prediction as a specific direction for future work.

6.2 Sanitization Performance on Hard Records Is Near the Floor—The Approach Does Not Solve the Hardest Problems

The assumption or constraint: The paper's achieved performance—Privasis-Cleaner-4B at 72.5% Full Successful Record on the vanilla set—is strong but bounded. On the hard test set, performance collapses across all models to 10–13% Full Successful Record (Table 4). The hard set is not an adversarial or out-of-distribution collection; it is drawn from the same synthesis pipeline and sanitization process as the vanilla set, simply comprising records where the decomposition-based pipeline itself fails. This means the hard set represents records that are intrinsically difficult to sanitize correctly under the current paradigm—they require contextual target identification across longer documents with more abstract instructions—not records that are qualitatively different from the training distribution.

The consequence: The approach provides no meaningful solution for the category of records where privacy protection is arguably most important: long, complex documents with many sensitive attributes that require nuanced, context-dependent sanitization decisions. A medical record, legal contract, or financial statement that spans multiple pages and contains dozens of potentially sensitive attributes is exactly the kind of document that poses the greatest privacy risk—and exactly the kind of document where Privasis-Cleaner is least reliable. The 12.4% success rate on hard records means that in approximately 7 out of 8 such cases, the model will leak at least one sensitive attribute or destroy at least one attribute that should be retained.

This is not a failure of the training methodology per se—the paper demonstrates that fine-tuning on Privasis-Sanitization dramatically improves over base models. Rather, it reveals a fundamental difficulty ceiling for the single-pass instruction-following approach: when a document is long, the instruction is complex, and targets are specified at the group level rather than individually, current models (including frontier LLMs) cannot reliably track all constraints and execute all required edits without errors. The decomposition-based pipeline that generates the training data can handle these cases because it operates sequentially with explicit chunk-level identification, span extraction, and per-target instruction generation—a structured process that the fine-tuned model is asked to replicate in a single forward pass, which is inherently harder.

What evidence exists in the paper: Table 4 shows the 10–13% Full Successful Record rates on the hard set for all models. The hard set composition (87% grouped attributes, longer records and instructions) is described in Section 4.2. The error analysis in Appendix E provides concrete examples of the kinds of failures that occur, but does not analyze whether these failures cluster on particular types of hard records or are distributed uniformly.

Mitigation status: The paper does not propose any mitigation for the hard-record performance collapse. The decomposition-based pipeline that generates the training data achieves perfect performance on the vanilla set (by construction) and fails on the hard set, meaning that even the multi-step structured approach has a ceiling. The paper does not analyze whether iterative application of the sanitization model, chained decomposition, or human-in-the-loop verification could recover performance on hard records. This is left entirely to future work.

6.3 The Synthesis Pipeline Relies on Frontier LLMs as Generators—Creating a Circular Dependency Where the Dataset's Quality Is Bounded by the Models It Aims to Improve Upon

The assumption or constraint: Privasis is generated primarily by GPT-OSS-120B (67.9% of records) and other LLMs including GPT-4.1-Mini (21.6%), Exaone-3.5-32B (7.2%), and smaller contributions from Qwen3-80B, Llama-3.3-70B, and GPT-4.1 (Section 2.2). The synthesis pipeline uses these LLMs to generate profiles, record types, background contexts, formats, and the actual record text. The quality, diversity, and realism of the resulting dataset are therefore bounded by the capabilities of the generating LLMs. The sanitization pipeline that constructs Privasis-Sanitization also relies on LLMs (GPT-4.1 and GPT-OSS-120B) for target identification, span extraction, instruction generation, and chunk-level sanitization.

This creates a circular dependency: the dataset is intended to train models that outperform frontier LLMs on text sanitization, but the dataset itself is constructed using those same frontier LLMs (or near-frontier models). Any systematic errors, biases, or blind spots in the generating models are baked into the training data. If GPT-OSS-120B systematically fails to recognize employer names in email headers (which the error analysis in Appendix E shows even GPT-5 does), then the training data generated by GPT-OSS-120B may contain similar failures, and models trained on that data will learn to replicate those failures.

The consequence: There is a hidden ceiling on the performance of models trained on Privasis that is determined by the capabilities of the generating models. The Privasis-Cleaner models can surpass GPT-5 on the vanilla test set (Table 4) because the training data provides high-quality examples of sanitization that GPT-5 has not seen during its own training. But they cannot surpass the generating models on capabilities that the generating models themselves lack. If the generating models cannot correctly handle a particular type of sanitization (e.g., identifying attributes in document footers, or blocking inference through indirect references), the training data will not contain correct examples of that capability, and the trained models will not learn it.

The paper provides some evidence of this ceiling: the hard test set consists of records where the decomposition-based pipeline (using GPT-4.1) fails. All models trained on data generated by similar pipelines also fail on these records, suggesting that the training data does not contain examples of how to successfully handle these hardest cases. The paper's approach is essentially distillation from LLM-based pipelines into compact models—effective when the pipeline succeeds, but providing no path to exceed the pipeline's capabilities.

What evidence exists in the paper: The paper acknowledges the generator dependency explicitly in Section 2.2, noting that multiple models are used "both [to increase] stylistic and distributional diversity and [to show] that our pipeline generalizes across LLMs." But it does not analyze whether the trained sanitization models exhibit error patterns correlated with the generating models' weaknesses. The hard test set construction (records where the pipeline fails) implicitly acknowledges the ceiling, but the paper does not frame this as a fundamental limitation of the synthesis approach.

Mitigation status: The paper mitigates generator-specific biases by using multiple models in the synthesis pipeline (Section 2.2), which reduces the risk that a single model's idiosyncratic failures dominate the training distribution. The construction of the test set from four frontier models not used in training data generation (Gemini-2.5-pro, GPT-5, Llama-4-Maverick, Qwen3-235B) partially addresses the circularity concern for evaluation purposes—the test data is out-of-distribution relative to the training data generators. However, these mitigations do not address the fundamental ceiling: if all available LLMs share a systematic weakness (e.g., failing to attend to document structural elements like headers and footers), then no combination of generators can produce training data that teaches that capability. The paper does not propose methods for identifying or compensating for generator blind spots.

6.4 Evaluation Is on a Single Task Family (Instruction-Following Text Sanitization) with a Single Dataset—Generalization to Other Privacy Tasks Is Unproven

The assumption or constraint: All experimental evaluation in the paper focuses on a single downstream task: instruction-following text sanitization using the Privasis-Sanitization dataset. The NaP² generalization experiment (Table 7) tests transfer to a different sanitization dataset, but it remains within the same task family (selective removal/abstraction of sensitive attributes from text). The paper claims that Privasis will "broaden and accelerate research in areas where processing sensitive social data is inevitable" (Abstract) and enumerates potential applications including "improved sanitization models to differential privacy techniques, and agentic systems that must operate responsibly on sensitive information" (Section 1), but provides no experimental evidence that Privasis is useful for any task other than the specific sanitization formulation it was designed to support.

The consequence: The paper's broader claims about Privasis as a general-purpose resource for privacy research are aspirational rather than empirically supported. Specific open questions include:

Differential privacy: Can synthetic records from Privasis be used to evaluate the privacy-utility tradeoffs of DP training algorithms? The paper suggests this in Section 1 but provides no DP experiments. Privasis records are fully synthetic with known ground-truth attributes, which could make them useful for measuring attribute inference risk under DP models—but this requires validation that the synthetic data's statistical properties are representative enough to produce meaningful DP evaluations.
PII detection and span labeling: Privasis includes 55.1 million annotated attributes with per-record JSON structures, which could support training span detection models. However, the paper evaluates only end-to-end sanitization (rewriting), not span-level detection accuracy, and does not report whether models trained on Privasis improve PII detection on real-world benchmarks.
Agentic privacy: The introduction motivates Privasis by citing AI agents that process sensitive personal information, but the paper provides no experiments with agent architectures, no evaluation of whether sanitization via Privasis-Cleaner actually prevents downstream privacy violations in agent contexts, and no integration with any agent framework.

The NaP² generalization result (Table 7) demonstrates that Privasis-trained models transfer to another sanitization dataset, which is encouraging but narrow—both Privasis and NaP² address text sanitization, and the experiment shows that training on a larger, more diverse sanitization dataset improves performance on a smaller one, which is consistent with standard scaling expectations and does not demonstrate transfer to qualitatively different privacy tasks.

What evidence exists in the paper: The only generalization experiment is the NaP² zero-shot evaluation (Section 4.4, Table 7). All other results are on Privasis-Sanitization's vanilla and hard test sets. The paper does not evaluate on any standard PII detection benchmarks (e.g., i2b2, TAB), any differential privacy benchmarks, or any agent-based privacy evaluation frameworks (e.g., ConfAIde, PrivacyLens). The diversity analysis (Table 2) compares Privasis to human-written datasets in terms of lexical and semantic diversity, but this measures similarity of statistical properties, not usefulness for downstream tasks.

Mitigation status: Not addressed. The paper's conclusion lists future work areas including "privacy-preserving generation, controllable sanitization, and agentic systems processing sensitive data" but does not commit to specific multi-task evaluations. The release of the dataset is framed as enabling these evaluations by the broader community, which is a reasonable position for a dataset paper—the authors cannot evaluate on every possible downstream task—but the gap between the claimed breadth of applicability and the narrow experimental validation should be acknowledged more explicitly.

6.5 The Privacy Safety Guarantee Is Empirical and Negative—No Formal Guarantee Against Re-Identification

The assumption or constraint: The paper claims that Privasis is privacy-safe because the generated profiles are synthetic and do not correspond to real individuals. This claim is supported by empirical verification: manual investigation of 100 sampled profiles using Gemini-2.5-Pro Deep Research, and a larger automated check of 1,000 profiles with web-search-enabled GPT-5 (Section 2.2). None of the investigated profiles were found to match real people. The paper also notes that generated URLs were inaccessible, and that in cases where name/sex/nationality matched a real person, other attributes (age, contact information) showed major discrepancies.

This is an empirical negative result—"we searched and did not find matches"—not a formal guarantee. It demonstrates that the specific profiles checked do not correspond to identifiable real individuals at the time of checking, but it does not prove that future profiles generated by the same pipeline will always be safe, that the profiles are indistinguishable from fictional ones under a more powerful adversary, or that combinations of attributes across multiple records could not be linked to reconstruct a plausible synthetic identity that happens to match a real person.

The consequence: The paper's claim that Privasis is "privacy-safe yet privacy-rich" (Section 1) depends on the strength of this empirical verification. The risk is not that Privasis contains real private data—it is generated from scratch—but that the synthetic data might be realistic enough to create re-identification risks of its own. Consider: a synthetic medical record contains a specific combination of demographic attributes, medical conditions, medications, and provider names. If that combination happens to match a real person's actual medical profile (by chance, because the generator sampled attributes from distributions that overlap with real population characteristics), then the synthetic record could be used to infer or corroborate information about that real person, even though it was not generated from their data. The paper's verification checks for exact matches between full profiles and real individuals, but does not assess the risk of partial matches, attribute-level re-identification, or the possibility that an adversary could use multiple synthetic records to infer population-level statistics about sensitive attributes.

The paper also generates records using names sampled from the US SSN applicant database, which contains names of real individuals. The paper's verification found that generated profiles with these names did not match the real individuals who hold those names on other attributes, but the use of real names from a public database creates a superficial link to real people that could cause confusion or concern if the synthetic records were mistaken for real records.

What evidence exists in the paper: The manual verification of 100 profiles (Section 2.2) and the automated check of 1,000 profiles are described in moderate detail. The paper reports that 5 of the 100 manual checks were incomplete, 15 returned multiple potential matches that were manually disambiguated and confirmed as non-matches, and 3 reported exact name/sex/nationality matches with major discrepancies in other attributes. The sanity check on the paper's authors (Gemini-2.5-Pro correctly identified all authors as real, demonstrating the verification tool's capability) is a good practice. The paper does not report any analysis of partial attribute overlap between generated profiles and real individuals, does not assess differential privacy guarantees, and does not discuss the risk of synthetic records being used adversarially.

Mitigation status: The paper explicitly states that "all personal identifiers—including names, addresses, and Social Security numbers—are entirely fictitious, and any resemblance to real persons (living or deceased), business entities, or locations is purely coincidental" (Ethics Statement). The verification procedures are described as confirming this. The paper commits to releasing the dataset with restrictions for "non-commercial research and evaluation purposes only" and explicitly prohibits "attempts to re-identify individuals or to misuse the data for fraudulent or harmful activities" (Ethics Statement). These are standard ethical safeguards for synthetic data releases, but they are usage restrictions, not technical guarantees. The paper does not propose or implement any formal privacy framework (differential privacy, k-anonymity, plausible deniability) to bound the re-identification risk.

6.6 The Cost and Infrastructure for Reproducing or Extending the Dataset Are Substantial and Not Fully Characterized

The assumption or constraint: The paper reports API costs for generating and sanitizing records with GPT-4.1: approximately $900 per 10,000 records for generation and$ 1,100 per 10,000 records for sanitization (Appendix A). At 1.4 million records, the total cost for Privasis using GPT-4.1 would be approximately $280,000 ($ 126,000 for generation + $154,000 for sanitization), though the actual cost is lower because most records were generated with the cheaper GPT-OSS-120B. The paper does not provide a total cost breakdown across all models used, does not estimate the computational cost of the diversity-preserving refinement (which requires repeated LLM calls and Vendi score computation), does not report the infrastructure requirements (GPU hours, storage) for the filtering, annotation, and attribute extraction steps, and does not estimate the cost of reproducing the dataset at similar scale with current models.

The consequence: The paper's primary contribution is a dataset and a methodology for generating similar datasets. For the methodology to be adoptable by other researchers—to "broaden and accelerate research in areas where processing sensitive social data is inevitable" (Abstract)—the cost of reproduction needs to be understood. The reported costs suggest that generating a million-scale dataset with this pipeline requires tens to hundreds of thousands of dollars in API credits, plus the computational infrastructure for fine-tuning models, running diversity scoring, and managing the multi-step pipeline at scale. This puts reproduction out of reach for most academic research groups unless they have access to substantial compute grants or industry resources.

The cost barrier also affects extensibility. The paper demonstrates that the pipeline generalizes to Chinese and Korean (Appendix A), but generating a million-scale multilingual version of Privasis would require similar API investment. The paper shows that multiple models can be used in the synthesis pipeline (Table 8), but the cost-performance tradeoffs between models are only partially characterized—GPT-OSS-120B is identified as having the best price-performance ratio, but absolute costs are only given for GPT-4.1. A researcher wanting to generate a domain-specific variant (e.g., legal documents only, or medical records only) cannot easily estimate the required budget.

What evidence exists in the paper: The cost estimates in Appendix A are specific to GPT-4.1 and cover only the generation and sanitization steps. There is no total cost for the full Privasis dataset, no breakdown of costs by pipeline component (profile generation, record type generation, background context generation, format generation, record generation, iterative refinement, attribute annotation, filtering), and no estimate of the human engineering effort required to design, tune, and operate the pipeline. The paper uses 37K of 100K constructed sanitization examples for training (Section 4.1), suggesting that the sanitization pipeline produced more data than was used—but the cost of generating the unused 63K examples is not discussed.

Mitigation status: The paper commits to releasing all code, data, and models (Section 6), which mitigates the reproduction cost for researchers who only need to use the existing dataset. For researchers who want to extend or modify the pipeline—generate new domains, add languages, or increase scale—the cost barrier remains. The paper does not discuss strategies for reducing generation costs (e.g., using smaller models for profile generation, reducing the number of refinement steps, or amortizing diversity scoring across batches) or provide a cost-scaling analysis that would help researchers budget for extensions.

7. Implications and Future Directions

How This Work Changes the Landscape

Privasis represents a methodological reframing, not a paradigm shift. The paper does not introduce a new privacy technique, a new model architecture, or a new theoretical framework for sanitization. What it does—and this is genuinely consequential—is demonstrate that the fundamental bottleneck in privacy research is not a lack of clever algorithms but a lack of training data at sufficient scale, diversity, and annotation richness. The paper's most important contribution is empirical: it shows that when you give a compact model (4B parameters) access to 37,000 diverse, instruction-rich sanitization examples, it matches or exceeds frontier LLMs that are orders of magnitude larger and have been trained on trillions of tokens of internet text. The base Qwen3-4B model scores 53.65% Full Successful Record; fine-tuning on Privasis-Sanitization lifts it to 72.50% (Table 4). The capability comes from the data, not the model scale.

This reframing shifts the conversation from "how do we build better privacy-preserving algorithms?" to "how do we build better privacy-preserving training data?" It is an implicit argument that the field has been algorithm-rich but data-poor, and that progress in the former has been held back by starvation in the latter. This is a familiar pattern in AI—ImageNet enabled the deep learning revolution in vision not because AlexNet was algorithmically novel (convolutional neural networks existed for decades) but because it provided training data at a scale and diversity that made the algorithms work. Privasis makes an analogous move for privacy research: it provides the ImageNet for text sanitization, the large-scale, richly-annotated substrate that enables data-driven approaches to finally work.

The paper also resolves a tension that has been implicit in the privacy literature: the conflict between data minimization and data availability. Prior work on data minimization (Dou et al., 2024; Huang et al., 2025) demonstrated that abstraction and selective rewriting could preserve utility while reducing privacy risk, but these techniques could only be developed and evaluated on the tiny datasets that were publicly available—a few thousand Reddit posts, a few hundred clinical notes. The tension was that you need data to build privacy-preserving systems, but the data you need cannot be shared because it is private. Privasis breaks this cycle by synthesizing the data you need without touching real private data. The synthesis pipeline is reference-free: it generates fictional individuals, fictional records, and fictional contexts from auxiliary control variables and a public name database, with no real private data as input. The verification that generated profiles do not correspond to real people (Section 2.2, 1,000-profile check) establishes that this approach is not merely theoretically possible but practically achievable at scale.

A more subtle landscape change concerns what counts as a privacy failure. The paper's hierarchical evaluation framework—direct leak, inference leak, proximity leak—operationalizes a three-level definition of sanitization failure that is more nuanced than the binary "PII present vs. absent" that dominates prior work. The finding that direct leak is the dominant failure mode across all models (53.7% to 81.7% of failures, Table 5) should redirect research attention: the urgent problem in text sanitization is not sophisticated inference attacks but the simple failure to remove text that should obviously be removed. This is a diagnostic contribution that clarifies where effort should be invested. Improving models' ability to handle structural document elements (headers, signatures, metadata) and to track attributes across long contexts is more immediately valuable than developing adversarial robustness to inference attacks—because the models are not yet at the point where inference attacks are the binding constraint.

The paper also makes certain research directions less attractive. It provides strong evidence that general-purpose reasoning capability (as embodied in frontier LLMs like GPT-5 and o3) does not automatically translate to fine-grained text sanitization competence. The 64–70% Full Successful Record rates for these models on the vanilla test set (Table 4), on records where a structured pipeline achieves 100%, indicate that scale and general intelligence are not sufficient. This suggests that efforts to solve sanitization by waiting for ever-larger models, or by prompt-engineering frontier LLMs, are unlikely to close the gap without task-specific training data. The finding that Privasis-Cleaner-0.6B (68.04%) outperforms Qwen3-235B (64.40%) despite being roughly 400× smaller drives this point home: a small model trained on the right data beats a large model trained on the wrong data.

Follow-Up Research This Work Enables

Difficulty prediction for deployed sanitization models. The paper's most significant unaddressed deployment problem is that users have no way to know whether their sanitization request will succeed. The vanilla test set (72.5% success for Privasis-Cleaner-4B) and the hard test set (12.4% success) represent qualitatively different difficulty regimes, but the paper provides no mechanism for predicting which regime a given input falls into. A strong follow-up would train a lightweight classifier—perhaps a linear probe on top of the sanitization model's hidden representations, or a small BERT-style model—to predict whether a given (record, instruction) pair is "likely to succeed" based on features that correlate with the vanilla/hard distinction: record length, instruction length, proportion of grouped attribute targets, number of distinct attribute occurrences, lexical overlap between instruction and record, etc. The paper already provides the labeled data for this (vanilla vs. hard test sets with known outcomes for the pipeline and all evaluated models). The evaluation would measure: (a) the classifier's AUROC for predicting Privasis-Cleaner-4B's success/failure per record, (b) whether a simple threshold on predicted success probability can filter out records where the model is likely to fail, creating a "high-confidence" subset with substantially higher success rate, and (c) calibration—whether the predicted success probability matches the empirical success rate. A positive result (e.g., a classifier that identifies a subset of hard records where Privasis-Cleaner achieves >50% success) would make the deployment picture substantially more practical. A negative result (no features predict success better than chance) would reveal a fundamental brittleness that the current approach cannot address without architectural changes.

Iterative, interactive sanitization with human-in-the-loop verification. The paper's decomposition-based pipeline succeeds on hard records by operating sequentially: identify relevant chunks, extract spans, generate per-target instructions, apply sanitization, merge. The fine-tuned model is asked to do all of this in a single forward pass, which is inherently harder. A natural extension is to give the model multiple passes: first pass identifies what needs to be sanitized (outputting a structured plan: target attributes, their locations, proposed operations), second pass executes the sanitization, and a third pass verifies consistency. This decomposes the single-pass generation problem into a plan-execute-verify loop that more closely mirrors the successful pipeline. The experiment would compare: (a) single-pass Privasis-Cleaner (current), (b) two-pass (identify then sanitize, with the identification output fed as additional context), (c) three-pass with self-verification, and (d) a variant where the verification pass flags uncertain edits for human review. The key metric is hard-set performance—does iterative processing close the gap between the 12.4% single-pass success rate and the pipeline's (unknown but higher) ceiling? A related direction: train the model to output confidence scores for each edit, enabling selective human review of low-confidence sanitization decisions, which would make the system deployable even at current success rates by ensuring that the 87% of hard records where the model fails are flagged for manual inspection rather than silently leaked.

Training data scaling laws for fine-grained text rewriting tasks. The paper uses 37,000 training examples and achieves strong results, but provides no analysis of how performance scales with training data size. Does Full Successful Record continue to improve with 100K, 500K, or 1M examples? Does the hard-set performance benefit disproportionately from more data (suggesting that the hard cases are learnable given enough examples) or does it saturate early (suggesting a fundamental ceiling)? A systematic scaling study—training Privasis-Cleaner on {5K, 10K, 20K, 37K, 75K, 100K} examples and measuring both vanilla and hard-set performance—would characterize the data efficiency of the approach and inform whether further investment in dataset construction is worthwhile. This would also test the paper's implicit claim that data diversity, not just data quantity, drives performance: if examples are added from new domains and instruction types, does performance improve more than adding examples from already-well-covered domains? The paper's finding that Privasis-Cleaner generalizes to NaP² better than a NaP²-specialized model (Table 7) suggests domain diversity matters, but a controlled experiment varying domain coverage would make this causal rather than correlational.

Cross-architecture generalization: does the benefit of Privasis training transfer to other model families? The paper fine-tunes only Qwen3 models. Qwen3-235B shows notably poor zero-shot sanitization performance (64.40% Full Successful Record, the worst among frontier models; 75.3% direct leak ratio, the highest). This raises the possibility that the Qwen3 architecture or pretraining mixture has specific weaknesses for this task that make the fine-tuning gains look larger than they would be for other architectures. A replication study fine-tuning Llama-3.2-3B, Mistral-7B, and Phi-3-mini on the same Privasis-Sanitization data would reveal: (a) whether the base model's zero-shot performance predicts fine-tuned performance (if Qwen3's low base performance is due to fixable architectural quirks, fine-tuning might close the gap; if it reflects a deeper limitation, Qwen3 fine-tuned might underperform other architectures fine-tuned on the same data), (b) whether the optimal training data size is architecture-dependent, and (c) whether the generalization to NaP² is robust across architectures. This is important because the paper's deployment argument—compact, locally-deployable models—depends on the approach working across multiple hardware-efficient architectures, not just Qwen3.

Combining Privasis-Cleaner with the decomposition-based pipeline at inference time. The paper treats the decomposition-based pipeline as a data generation mechanism and the fine-tuned model as the deployed system, but never combines them. A hybrid approach: at inference time, decompose the input record into chunks using the same τ = 512 character splitting, run Privasis-Cleaner on each chunk independently (with the full instruction as context), then merge. This would address the long-context degradation that Figure 5 shows for chunk sizes >512 characters, and might improve hard-set performance by reducing the cognitive load on the model (each forward pass only needs to track attributes within a 512-character window rather than across a 600+ word document). The experiment would compare: (a) single-pass Privasis-Cleaner on the full record (current), (b) chunk-then-sanitize with Privasis-Cleaner, (c) chunk-then-sanitize with a frontier LLM, and (d) the full decomposition-based pipeline as an upper bound. The key question is whether the decomposition overhead (splitting, generating chunk-level contexts, merging) is worth the accuracy improvement, and whether the fine-tuned model can maintain cross-chunk consistency (e.g., abstracting the same name the same way across chunks) without the explicit per-target instruction generation that the pipeline uses.

Extending the evaluation framework to semantic entailment for inference leak. The paper acknowledges that the inference leak metric's exact string matching requirement "would likely increase if semantic entailment were used instead" (Section 4.4). A methodological follow-up would replace the exact-match inference leak check with a natural language inference (NLI) evaluation: given the sanitized text as premise and a hypothesis like "The patient was treated at Royal Darwin Hospital," does the premise entail the hypothesis? This would capture cases where the sanitized text logically implies the sensitive attribute without containing the exact string—the email domain example in Appendix E is a perfect case: "[journal name]" doesn't match "Journal of Something" exactly, but "jsal.org" in the editor's email entails the journal's identity. The experiment would use an off-the-shelf NLI model (e.g., a fine-tuned DeBERTa) to score entailment for each sanitized attribute, then compare inference leak rates under exact-match vs. entailment-based criteria across all models from Table 5. A positive result (entailment-based inference leak is substantially higher than exact-match inference leak) would quantify the hidden risk that the current evaluation misses and would motivate training models to block semantic inference, not just surface string removal. A negative result (entailment-based and exact-match rates are similar) would validate the current evaluation as sufficiently conservative.

Practical Applications and Downstream Use Cases

On-device data minimization for AI agent pre-processing. The paper explicitly motivates Privasis with the rise of AI agents (OpenClaw, Gemini Agent, ChatGPT Health) that process sensitive personal information at inference time. The deployment architecture implied by the paper is: before a user's email, calendar, medical record, or text message thread reaches the agent's context window, a local Privasis-Cleaner model strips or abstracts sensitive attributes based on the user's instructions. The user might specify "remove all financial account numbers and exact addresses, but keep city names and appointment times" and the model executes this before the cleaned text is sent to the cloud-based agent. The paper's results support the feasibility of this architecture: Privasis-Cleaner-0.6B achieves 68% Full Successful Record at a model size that could plausibly run on a laptop or high-end phone. The 99.2% retention rate for non-target attributes (vs. 93.4% for GPT-5, Table 4) means the sanitized text remains useful for downstream tasks—the agent still sees appointment times, city names, and diagnostic categories, just not the specific identifiers and fine-grained details that create privacy risk. The key practical challenge not addressed by the paper is latency: sanitizing a 527-word record (the Privasis average) with a 0.6B model on consumer hardware needs to complete in under a second to avoid degrading the user experience of an interactive agent. Benchmarking this latency across hardware tiers (phone, laptop, desktop with GPU) would determine whether the approach is deployable today or requires further model optimization.

Automated redaction for legal and medical document sharing. The paper's domain categories (Table 1) heavily represent legal, medical, and financial records—precisely the document types that organizations routinely need to redact before sharing with external parties. A hospital sharing de-identified patient records with researchers, a law firm producing discovery documents with privileged information removed, a bank sharing anonymized transaction records with auditors—these are high-stakes, high-volume workflows where manual redaction is expensive and error-prone. Privasis-Cleaner offers a path to automation: a 4B model that can be run on-premise (no data leaves the organization), fine-tuned on examples that span the relevant document types, and instructed in natural language rather than requiring regex rules for each new redaction policy. The paper's 72.5% Full Successful Record on the vanilla set is promising but not sufficient for high-stakes legal or medical use—a single leaked attribute in a court filing or a shared patient record could have serious consequences. The practical deployment path would involve the iterative or human-in-the-loop extensions discussed above: the model proposes redactions, a human reviewer verifies (or spot-checks) the output, and the model's confidence scores guide which documents receive full manual review. The paper's finding that direct leak is the dominant failure mode (Table 5) is actually encouraging for this use case: direct leaks are easier for human reviewers to spot than proximity leaks, and a verification pass that checks for verbatim sensitive strings would catch the majority of failures.

Synthetic data generation for privacy-preserving machine learning research. Beyond the specific sanitization task, Privasis itself is a resource for the broader privacy research community. Researchers developing differential privacy algorithms need datasets with known ground-truth attributes to measure privacy-utility tradeoffs—how much noise must be added to protect a specific attribute, and how does that noise degrade downstream task performance? Privasis provides 55.1 million annotated attributes across 1.4 million records in a publicly-releasable format with no real privacy risks. A differential privacy researcher could: (a) train a text generation model on Privasis with DP-SGD at various privacy budgets (ε values), (b) measure the model's ability to generate text that preserves or leaks specific attributes (using the paper's own hierarchical evaluation framework), and (c) characterize the privacy-utility Pareto frontier in a setting where ground-truth attribute values are known exactly. This is currently difficult because DP research on text either uses synthetic data with limited attribute annotations, or uses real private data that cannot be shared to reproduce results. Privasis removes both barriers. The paper's multilingual extensibility (Appendix A, Chinese and Korean records validated by native speakers) suggests this could extend to cross-lingual DP research, where the interaction between language structure and privacy guarantees is underexplored.