People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text

🎯 Pitch

Surprisingly, frequent ChatGPT users can nearly perfectly spot AI-written articles without any training—their five-person majority vote misclassifies only 1 out of 300 texts, crushing even commercial detectors. This expert intuition holds up against evasion tricks like paraphrasing and humanization that break automated tools.

1. Executive Summary

This paper empirically studies how well humans can detect text generated by modern commercial LLMs—GPT-4O, CLAUDE-3.5-SONNET, and O1-PRO—in the presence of evasion tactics like paraphrasing and humanization, using a dataset of 300 paired non-fiction English articles. The central finding is that annotators who frequently use LLMs for writing tasks constitute a population of expert annotators whose untrained majority vote achieves near-perfect detection accuracy, misclassifying only 1 of 300 articles and significantly outperforming nearly all commercial and open-source automatic detectors evaluated, including under adversarial conditions. Qualitative coding of expert explanations reveals that they rely on a diverse taxonomy of clues—vocabulary overuse (e.g., “delve,” “crucial”), formulaic sentence structure, grammatical perfection, and more complex phenomena like originality and tone—rather than any single heuristic. The paper establishes that human experts rival the best commercial detector (Pangram, also achieving 99.3% overall TPR with 0% FPR for the majority vote) while providing interpretable, paragraph-length explanations, though prompt-based LLM detectors attempting to mimic this expert reasoning struggle with humanized articles and fail to match human or closed-source performance, confirming that current automatic methods remain brittle specifically when generation quality increases.

2. Context and Motivation

The Core Problem: Automatic Detectors Are Brittle and Unexplainable

The fundamental problem this paper tackles is the unreliability of automatic AI-generated text detectors in real-world deployment scenarios. While the proliferation of LLMs like ChatGPT has made AI-generated text ubiquitous—appearing in academic essays, news articles, social media posts, and creative writing—the tools designed to identify such text suffer from three critical weaknesses that the paper catalogs in its opening paragraph (Section 1):

Low detection rates under adversarial conditions. Automatic detectors can be easily circumvented through simple post-hoc modifications. When users apply paraphrasing—rewriting AI-generated text at the sentence level to preserve meaning while altering surface form—detection accuracy plummets (Krishna et al., 2023; Sadasivan et al., 2024). More sophisticated humanization techniques, which explicitly instruct LLMs to mimic human writing patterns (avoiding overused vocabulary, introducing stylistic variation, incorporating colloquialisms), degrade detector performance even further. The paper's own benchmarking (Table 2, Section 2) quantifies this: open-source detectors like Binoculars achieve 100% TPR on vanilla GPT-4O output but drop to 6.7% on humanized O1-PRO articles; Fast-DetectGPT falls from 100% to 23.3% on the same comparison. Even commercial detectors like GPTZero degrade from 100% to 46.7% TPR when humanization is applied.

Limited explainability. Most automatic detectors output a scalar score or binary label without providing any justification for their decision (Lipton, 2018). This opacity is particularly problematic in high-stakes settings: before accusing a student of plagiarism or flagging a legal document as potentially AI-generated, an educator or legal professional needs to understand why the detector made its determination. As the paper notes in Section 3, automatic detectors "generally lack" the detailed, freeform explanations that human annotators can provide, which "limit[s] their trustworthiness in critical settings." A score of 0.87 tells you nothing about what specific features of the text triggered the classification.

Distribution shift and model-specific brittleness. The paper's experiments reveal that detectors trained or calibrated on one model's output often fail on another's. This is evident in the prompt-based detector results (Table 2, bottom): GPT-4O-2024-11-20 achieves 100% TPR on vanilla GPT-4O and CLAUDE-3.5-SONNET articles but drops to 6.7% on humanized O1-PRO. CLAUDE-3.5-SONNET as a detector (Table 23, Appendix E.3) fails to detect content generated by itself when humanization is applied. This brittleness reflects a deeper problem: detectors may be overfitting to superficial artifacts of specific model–prompt combinations rather than learning generalizable signatures of AI writing.

Why This Matters: Real-World Stakes

The paper's motivation extends beyond academic benchmarking to address genuine societal concerns where detection reliability has concrete consequences:

Academic integrity. As LLMs become increasingly accessible to students, the ability to distinguish human-written from AI-generated essays is essential for maintaining educational standards. Zhu et al. (2024) document the surge in LLM use by secondary students, making detection a pressing concern for educators. A false positive—accusing a student of using AI when they did not—can have severe consequences for academic careers. A false negative—failing to detect AI-generated work—undermines the purpose of assessment.

Misinformation and fake content. Malicious actors can use LLMs to generate large volumes of fake news articles, propaganda, or deceptive content at minimal cost (Gameiro et al., 2024). Reliable detection is a necessary (though not sufficient) component of combating such disinformation campaigns. The paper's focus on non-fiction articles (Section 2, "Article selection") is motivated partly by this threat: such articles are a primary vector for misinformation and do not require specialized knowledge to evaluate.

The arms race dynamic. The detection problem is inherently adversarial. As LLM capabilities advance—the paper explicitly notes the release of OpenAI's O1 model during their experimental timeline (Section 2.4)—generated text becomes more sophisticated and harder to distinguish from human writing. Meanwhile, users actively develop evasion tactics. This creates an ongoing arms race where detectors must continuously adapt to new models and new obfuscation strategies. Understanding the upper bound of detection capability—whether through humans or machines—is essential for calibrating expectations about what is achievable.

Prior Approaches and Their Shortcomings

The paper positions itself against several strands of prior work, each with identifiable limitations:

Pre-ChatGPT human detection studies. Earlier work on human detection of AI-generated text (Ippolito et al., 2020; Brown et al., 2020; Clark et al., 2021) was conducted primarily with GPT-2 and GPT-3 era models. These studies generally concluded that "naïve annotators do not reliably detect AI-generated texts" (Section 5). However, the paper identifies a critical gap: these studies predated both the widespread public deployment of LLMs and the emergence of a population of users who interact with LLMs daily for writing tasks. The annotators in these earlier studies were essentially encountering AI-generated text for the first time—a scenario that no longer reflects reality for many people. Additionally, these studies did not systematically investigate whether some annotators perform well, even though Ippolito et al. (2020) and Dugan et al. (2022) noted individual variation in detection ability.

Human evaluation methodology issues. Karpinska et al. (2021) demonstrated a significant problem in prior human evaluation of generated text: Mechanical Turk annotators often "read" long-form texts in seconds before providing evaluations, meaning their judgments were based on superficial skimming rather than careful analysis. The paper explicitly cites this issue in justifying their own methodology (Section 2, footnote 10): "Collecting explanations mitigates issues raised by Karpinska et al. (2021), whose Mechanical Turk annotators 'read' long-form texts in a matter of mere seconds before evaluation." By requiring paragraph-length explanations with highlighted spans, the paper ensures annotators actually engage with the text.

Automatic detection approaches. The paper categorizes existing automatic methods into two families (Section 5):

Perplexity-based methods (Mitchell et al., 2023; Bao et al., 2023; Hans et al., 2024) operate on the insight that LLM-generated text tends to lie in high-probability regions of the generating model's output distribution. Binoculars (Hans et al., 2024) computes cross-perplexity using two different language models; Fast-DetectGPT (Bao et al., 2023) estimates conditional probability curvature by sampling perturbations of the candidate text. These methods are zero-shot (requiring no training data) but are fundamentally vulnerable to paraphrasing since changing surface tokens while preserving meaning shifts the probability landscape. The paper's results confirm this: Binoculars drops from 100% to 60% TPR on paraphrased GPT-4O (Table 2).
Trained classifiers (Solaiman et al., 2019; Emi and Spero, 2024; Verma et al., 2023) learn to discriminate between human and AI text using labeled training data. Pangram (Emi and Spero, 2024) represents the state of the art here, using a Transformer classifier with hard negative mining and iterative synthetic data generation. However, these classifiers require access to representative training data from the target models and can suffer from distribution shift when deployed on new models or in the presence of evasion tactics.

The explainability deficit. A critical shortcoming the paper identifies across all automatic methods is the lack of interpretable outputs. Neither perplexity scores nor classifier probabilities tell users why a text was flagged. This is not merely a usability issue—in high-stakes domains, unexplainable decisions are fundamentally untrustworthy. Lipton (2018) articulated this general problem for machine learning systems; the paper applies this critique specifically to AI-generated text detection.

Paraphrasing and humanization as understudied threats. While prior work established that paraphrasing degrades detector performance (Krishna et al., 2023; Sadasivan et al., 2024), and some studies examined humanization attacks in the context of automatic detectors (Wang et al., 2024a,b; Lu et al., 2023; Shi et al., 2023), the paper identifies a gap: no prior work had systematically studied how humanization affects human detectors, nor had it evaluated human detection performance on text from frontier reasoning models like O1-PRO. This gap is practically significant because humanization is likely to become more common as users become more sophisticated, and as frontier models produce increasingly persuasive text.

How This Paper Positions Itself

The paper does not propose a new detection algorithm or claim that humans should replace automatic detectors. Instead, it establishes an empirical upper bound for detection performance and uses that bound to diagnose why and where automatic methods fail. This positioning is evident in several key design choices:

Establishing the human ceiling. By recruiting annotators who frequently use LLMs for writing tasks and testing them without any training or feedback, the paper asks: what is the best performance achievable by a motivated, experienced human working with full interpretability? The answer—majority vote of five experts misclassifies only 1 of 300 articles, rivaling the best commercial detector—provides a benchmark against which automatic methods can be measured. This is not a practical deployment strategy (the paper explicitly acknowledges that manual review "cannot scale to high-volume, low-stakes tasks" in Section 3), but rather a diagnostic tool for understanding what signals remain detectable even in sophisticated generated text.

Using human explanations to diagnose automatic detector failures. The paper's qualitative coding of expert explanations into a taxonomy of clues (Table 3, expanded in Table 17) serves a diagnostic purpose. By showing that experts rely on complex phenomena like originality, tone, and quotation naturalness—not just surface-level vocabulary patterns—the paper implicitly explains why automatic detectors that primarily capture statistical regularities in token distributions fail on humanized text. The humanization prompt (Section 2.5) was explicitly designed to remove the vocabulary and sentence-structure signatures that experts identified, yet experts still succeeded by relying on harder-to-automate dimensions like creativity and factual specificity.

Probing the limits of LLM-as-detector. The prompt-based detection experiments in Section 4 test whether LLMs can replicate expert reasoning when given access to the same guidebook that experts developed. The negative result—prompted LLMs achieve only 78% TPR (GPT-4O with CoT + guide) compared to 99.3% for the expert majority vote—demonstrates that current LLMs cannot fully internalize the nuanced detection criteria that humans apply, even when those criteria are explicitly described. This finding reinforces the paper's central claim: there exists a meaningful gap between what automatic methods (including LLM-based ones) can detect and what expert humans can detect.

Reconciling conflicting prior findings. The paper's distinction between "expert" and "nonexpert" annotators provides a potential resolution to the apparent contradiction in prior work. Studies that found humans perform near chance (Brown et al., 2020; Clark et al., 2021) may have predominantly sampled from the nonexpert population, while studies that noted some high-performing individuals (Ippolito et al., 2020; Dugan et al., 2022) may have inadvertently included experts. The paper's Experiment 1 (Section 2.1) directly demonstrates this: within the same batch of 60 articles, nonexperts achieved an average TPR of 56.7% and FPR of 51.7% (essentially random), while the five experts achieved an average TPR of 92.7% and FPR of 4.0% (Table 1). The key variable is not general human ability but specific experience with LLM writing patterns.

The adversarial framing. The paper positions detection as an adversarial problem where both the generator and the detector are evolving. The five experiments are explicitly ordered by increasing difficulty: vanilla GPT-4O (Experiment 1), a different model family (CLAUDE-3.5-SONNET, Experiment 2), paraphrasing (Experiment 3), a frontier reasoning model (O1-PRO, Experiment 4), and finally humanization specifically targeting expert-identified signatures (Experiment 5). This progressive escalation mirrors the real-world arms race and tests whether human expertise generalizes across these increasingly challenging conditions. The fact that the expert majority vote remains near-perfect throughout—only misclassifying one article in Experiment 4—is the paper's strongest evidence for the robustness claim.

Summary of the Gap

In concise terms: the field knew that automatic detectors were brittle and unexplainable, and it knew that "average" humans were poor detectors of pre-ChatGPT AI text. What it did not know—and what this paper establishes—is that a specific, identifiable population of LLM-experienced users constitutes highly robust detectors whose performance matches the best commercial systems and whose explanations reveal detection dimensions that current automatic methods fail to capture. This reframes the detection problem from "Can we build a better classifier?" to "Can we build classifiers that attend to the same signals that expert humans use?"

3. Technical Approach

3.1 Reader Orientation

This paper constructs a human annotation pipeline for evaluating how well people can detect AI-generated text. It is not building a new automatic detector. Instead, it systematically measures human detection performance across five progressively harder experiments—varying the LLM used, the presence of paraphrasing, and the presence of humanization—while collecting rich qualitative data about what clues annotators rely on.

The system solves the problem of establishing an empirical upper bound for AI text detection by identifying a specific population (frequent LLM users performing writing tasks) and measuring their accuracy, robustness, and decision-making process in a controlled setting where authorship is known but annotators receive no training or feedback.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental apparatus has five major components:

Paired Article Corpus — 300 non-fiction English articles (150 human-written, 150 AI-generated) organized as minimal pairs sharing title, subtitle, and approximate length, controlling for content so that authorship is the only varying factor.
LLM Generation Pipeline — a prompt-based system that produces AI-generated counterparts for each human article using three model families (GPT-4O, CLAUDE-3.5-SONNET, O1-PRO) with optional evasion modifications (paraphrasing and humanization).
Annotator Recruitment and Screening — a two-stage process that first identifies annotators with relevant LLM experience through surveys, then screens them on a 5-question qualification test (requiring ≥80% accuracy) followed by a full 60-article batch (requiring ≥90% accuracy to qualify as "experts").
Annotation Interface — a tool that presents articles (without titles, to prevent obvious pairing), collects binary labels, 5-point confidence ratings, highlighted spans, and paragraph-length free-form explanations.
Analysis Framework — a taxonomy of 16 clue categories (vocabulary, sentence structure, grammar, originality, quotes, etc.) used to code all 1,790 expert explanations, plus benchmarking against automatic detectors (Pangram, GPTZero, Binoculars, Fast-DetectGPT, RADAR) and prompt-based LLM detectors.

Information flows as follows: a human article enters the corpus → its title and subtitle are fed to the LLM generation pipeline to produce an AI counterpart → both articles in the pair are distributed to annotators in randomized order → annotators read each article, highlight suspicious spans, and provide a label, confidence score, and explanation → explanations are coded into the taxonomy → performance is compared against automatic detectors on the same articles.

3.3 Roadmap for the Deep Dive

The explanation follows the experimental pipeline from data construction through evaluation, because each stage's design constrains what the next can measure:

First, the paired article corpus construction — how minimal pairs are created, what publications are used, and why this design enables clean comparisons.
Second, the LLM generation pipeline — the exact prompts, the five experiment configurations, and the evasion tactics (paraphrasing, humanization).
Third, the annotator recruitment and screening protocol — how "experts" are identified, what distinguishes them from "nonexperts," and the justification for the sample size.
Fourth, the annotation interface and task design — what annotators see, what they provide, and how the process prevents contamination between pairs.
Fifth, the explanation coding taxonomy — the 16 categories, how they were developed, and how GPT-4O was used to scale coding across all explanations.
Sixth, the automatic detector benchmarking — which detectors are evaluated, how thresholds are set, and the comparison framework.
Seventh, the prompt-based detector experiments — the four configurations tested and how they relate to the human expert findings.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical measurement paper whose core idea is that frequent LLM users constitute a naturally occurring population of expert detectors whose majority vote defines the practical upper bound for AI text detection performance, and whose explanations reveal detection dimensions that automatic methods fail to capture.

Paired Article Corpus Construction

The foundation of all experiments is a corpus of 300 paired non-fiction English articles, where each pair consists of one human-written article and one AI-generated article sharing the same title, subtitle, and approximate length. This minimal-pair design ensures that any difference annotators detect must stem from authorship rather than topic or format, which is critical for isolating the signals of AI writing from confounding variables.

Human-written article sourcing. The 150 human-written articles are drawn from eight American publications (Section 2, "Article selection"): Associated Press, Discover Magazine, National Geographic, New York Times, Reader's Digest, Scientific American, Smithsonian Magazine, and Wall Street Journal. The publication date range spans from October 2022 to December 2024 (Table 5). All articles are nonfiction pieces of fewer than 1,000 words with consistent formatting appropriate to their source. The authors explicitly state that they purchased all articles in the corpus, ensuring legal compliance.

Why these publications? The selection prioritizes professional editing and proofreading, meaning human-written articles represent high-quality, polished English prose rather than the informal, error-ridden text found on social media or unedited platforms. This creates a harder detection task: if AI-generated text is being passed off as professional journalism, the baseline human text is already grammatically correct and stylistically refined, removing obvious giveaways like typos or inconsistent formatting. The authors note that they "restrict our study to English nonfiction articles of fewer than 1K words" and that they "cannot make claims about human detection performance in other domains."

Section and length distributions. Articles span diverse sections within each publication—Science, Animals, Environment, History & Culture, Travel, Mind & Brain, Technology, etc. (Figure 7). The mean article length across experiments ranges from approximately 625 to 816 words (Table 6), with standard deviations between roughly 107 and 234 words, indicating substantial variation. This variation matters because it prevents annotators from using length as a crude heuristic (e.g., "short articles are AI-generated").

Why minimal pairs? The pairing design follows the methodology of contrast sets and minimal pair evaluation established in prior NLP benchmarking work (Gardner et al., 2020; Warstadt et al., 2020; Karpinska et al., 2022, 2024). By constructing pairs where "only authorship [is] the varying factor," the paper eliminates the possibility that annotators are simply distinguishing between different topics or writing styles inherent to different subject matter. Without this control, a human detector might appear accurate simply because AI-generated articles happened to cover different topics than human-written ones. The pairing ensures that any systematic difference must be attributable to the generation process itself.

Article length matching. The generation prompt includes a word count target based on the corresponding human article's length (Table 8: "write a corresponding article of around YOUR WORD COUNT words"). Table 6 confirms that the resulting AI-generated articles have similar mean lengths to their human counterparts, though with some variation. For example, in Experiment 1 (GPT-4O), human articles average 652.5 words (SD 170.9) while AI articles average 625.1 words (SD 126.9)—close enough that length alone cannot reliably distinguish them.

LLM Generation Pipeline

The AI-generated counterparts are produced by prompting commercial LLMs with the title, subtitle, desired length, and publication source of each human article. The generation pipeline varies systematically across five experiments to probe increasingly difficult detection scenarios.

Base generation prompt (Experiments 1, 2, and 4). The core prompt template (Table 8) is:

"You are given the following title and subtitle of a general article from the [SECTION] section of [PUBLICATION] and asked to write a corresponding article of around [WORD COUNT] words. Include quotations from relevant experts and make sure the article is concise and easily understandable to a lay audience."

The prompt intentionally instructs the model to include expert quotations and target a lay audience, which mirrors the conventions of the source publications. This makes the AI-generated articles superficially plausible as journalism—they include attributed quotes, maintain a consistent voice, and address the stated topic.

Temperature settings. For Experiments 1-3, the authors set temperature=0 when using the API, producing deterministic outputs for a given prompt. For Experiments 4-5 (O1-PRO), they used the ChatGPT Pro web interface because the O1 API was not yet available at the time of data collection. The authors note that OpenAI has fixed temperature=1 for O1 model access, meaning O1-PRO outputs are non-deterministic even with identical prompts.

Model selection rationale. The three models represent different points in the capability spectrum:

GPT-4O (2024-08-06) — the most widely used LLM, both in general and by the expert annotators specifically, making it the natural baseline for "what people actually encounter."
CLAUDE-3.5-SONNET — a competitor from a different organization (Anthropic), testing whether detection skills transfer across model families or overfit to GPT-4O-specific artifacts.
O1-PRO — OpenAI's frontier reasoning model released during the experimental timeline, representing a paradigm shift due to its test-time scaling capabilities (Zhong et al., 2024). Critically, none of the experts had prior exposure to O1-PRO, making it a genuine test of generalization to unfamiliar model behavior.

The five experiments (Section 2.1–2.5). Each experiment consists of exactly 60 articles (30 human, 30 AI-generated), with no overlap in human-written articles across experiments (the 150 total human articles are split into 5 batches of 30). The experiments form a progressive difficulty curve:

Experiment 1 (GPT-4O, no evasion): The baseline. 30 human articles paired with 30 GPT-4O-generated articles using the standard prompt. This establishes whether any annotators can detect vanilla AI output.
Experiment 2 (CLAUDE-3.5-SONNET, no evasion): Same prompt structure, different model. Tests generalization across model families.
Experiment 3 (GPT-4O, paraphrased): AI-generated articles undergo an additional round of sentence-level paraphrasing before being presented to annotators. Tests robustness to the simplest and most common evasion tactic.
Experiment 4 (O1-PRO, no evasion): Tests whether experts can detect output from a model with substantially different generation characteristics (enhanced reasoning, more varied vocabulary, more natural structure) that they have never encountered before.
Experiment 5 (O1-PRO, humanized): The hardest configuration. Articles are generated by O1-PRO with an explicit prompt designed to evade the specific detection signatures identified by experts in Experiments 1-4. Tests whether removing known "AI tells" is sufficient to fool expert detectors.

Paraphrasing Implementation (Experiment 3)

The paraphrasing attack follows the sentence-level approach from PostMark (Chang et al., 2024), with modifications for the article domain. Understanding the mechanics is important because it reveals why paraphrasing fails to fool experts: it operates at the surface level while leaving deeper structural patterns intact.

Sentence-by-sentence procedure. The paraphrasing is performed using GPT-4O with temperature=0. For the first sentence of an article, only that sentence is provided to the paraphraser with the instruction (Table 9):

"Paraphrase the given sentence. Only return the paraphrased sentence in your response. Make it seem like a human wrote the article and that it is from the [SECTION] section of [PUBLICATION]."

For all subsequent sentences, the already-paraphrased portion of the article is included as context (Table 10):

"Previous context: [PARAPHRASED ARTICLE SO FAR]\nSentence to paraphrase: [CURRENT SENTENCE]\nYour paraphrase of the sentence:"

Why context matters. Including the paraphrased prefix ensures that the paraphraser maintains coherence and flow across sentence boundaries. Without this context, each sentence would be paraphrased independently, potentially introducing inconsistencies in terminology, reference chains, or narrative progression. The context window forces the paraphraser to make decisions consistent with what has already been written, which is important because—as the expert explanations later reveal—consistency patterns are themselves detection signals.

Named entity substitution. An additional modification addresses a vulnerability the experts had already exploited in Experiments 1 and 2: LLMs' tendency to generate the same fictional names. The authors "extract named entities from the human-written article and instruct GPT-4O to include these names in its generated article." This means the paraphrased articles now reference real people, places, and organizations from the corresponding human article, removing the "Sarah Thompson/Emily Carter" signature that experts had identified. The paper notes that "63.3% of GPT-4O and 70% of CLAUDE-3.5-SONNET articles include either the name Emily or Sarah" (Section 3), so this substitution targets a highly reliable but superficial clue.

What paraphrasing does NOT change. Critically, sentence-level paraphrasing preserves: (1) the overall structure and ordering of ideas in the article, (2) the presence and placement of quotations, (3) the argumentative arc (introduction → body → conclusion), (4) the ratio of narrative to exposition, and (5) meta-patterns like "always ending paragraphs with a quote" that experts identified. The paper's results show that experts continue to detect these structural signatures even when surface vocabulary is varied, which is why paraphrasing fails as an evasion tactic against expert humans.

Humanization Implementation (Experiment 5)

Humanization represents a qualitatively different evasion approach than paraphrasing. Rather than modifying surface tokens of an existing generation, humanization targets the generation process itself to produce text that avoids known AI signatures from the start. The paper's humanization method is novel and deserves careful explanation because it represents one of the first systematic attempts to evade human (not just automatic) detectors.

Design philosophy: target the detector's decision process. The humanization prompt (Table 13) is constructed from a "guidebook" that was developed by paying the five expert annotators $45 each to list the specific clues they look for during detection. The authors then manually organized these disparate clues into a structured document (Table 11) with sections covering vocabulary, grammar, sentence structure, formatting, tone, introductions, conclusions, content, factuality, and creativity. This guidebook is the adversarial target: the humanizer is explicitly instructed to produce text that would evade a detector using this guidebook.

Prompt structure. The humanization prompt (Table 13) contains four components:

The detection guide itself (Table 11) — a detailed taxonomy of AI writing signatures with examples, including a list of over 100 specific "AI vocabulary" items to avoid (Table 12): nouns (tapestry, testament, realm, landscape, quest, roadmap), verbs (delve, embrace, empower, resonate, navigate, foster, underscore), adjectives (crucial, pivotal, vibrant, profound, transformative, robust, seamless), adverbs (additionally, moreover, aptly), and phrases ("not only... but also," "it's not about... it's about," "paving the way," "when it comes to," "in a world where").
Example articles — human-written and AI-generated articles from Experiments 1-4, drawn from the same publication as the target article. For instance, when generating a humanized New York Times article, all human-written New York Times articles and their AI-generated counterparts from prior experiments are provided as reference. This gives the humanizer concrete examples of what the target publication's style looks like and what previous AI generations got wrong.
Explicit behavioral guidelines — 14 numbered instructions that operationalize the guidebook into generation constraints, such as:
- "Avoid formulaic introductions that elaborate on scene-setting or time"
- "Avoid generic, vague, and forward-looking conclusions, and do not summarize the article in the conclusion"
- "Add references to darker topics when appropriate"
- "Always opt for specifics over broad details. For example, mention correct statistics instead of broad claims"
- "Make sure that none of the claims made in your article are factually incorrect or implausible"
The standard article generation prompt (same as Experiments 1-4).

Why generate humanized articles from scratch rather than humanizing post-hoc? The paper describes an iterative development process (Appendix B.2) where the authors first tried three approaches that modified already-generated text:

Generate-then-humanize: Ask O1-PRO to first generate an article, then edit it to "sound more human." The first author's qualitative analysis found this "result[ed] in an effect similar to paraphrasing"—surface changes without deep structural modification.
Step-by-step humanization: Iteratively prompt the model to humanize the article, focusing on different elements at each step (first creativity, then tone, then grammar, then vocabulary). This was "better" but "the numerous calls to LLMs was adding more characteristics of AI," creating a recursive problem where each editing pass introduced new detectable patterns.
Two-step humanization: Give the humanizer the detection guide, ask it to list everything identifiable as AI-generated in the article and suggest edits, then apply those edits. This "yielded the best results" among post-hoc approaches.

The final approach—generating the article in a humanized fashion in a single step—was chosen because the authors found that "many of the largest identifiable traits of AI were due to the base generation" itself, not just surface features. By changing the generation process, the humanizer can make deeper structural decisions (e.g., choosing not to include a concluding paragraph at all, varying sentence length, avoiding the "three examples" pattern) that post-hoc editing cannot easily achieve.

What makes this a strong adversarial test? The humanization prompt targets the specific detection dimensions that experts identified. If experts were merely pattern-matching against a fixed list of vocabulary items and structural templates, the humanized articles should fool them completely—the prompt explicitly instructs the model to avoid all of those patterns. The fact that experts still maintain high accuracy on humanized articles (expert majority vote TPR of 100% in Experiment 5) implies they are detecting signals that are not captured by their own articulated guidebook, operating at a level of tacit knowledge that they cannot fully verbalize.

Only O1-PRO is used for humanization. The authors state (Section 2.5, footnote 18) that they "only use O1-PRO for this experiment as the previous experiments show it was more difficult to detect than GPT-4O and CLAUDE-3.5-SONNET." This is a deliberate worst-case choice: combining the hardest-to-detect base model with the strongest evasion technique creates the most challenging possible test.

Annotator Recruitment and Screening

The paper's most critical empirical finding—that expert annotators perform near-perfectly—depends entirely on how "expert" is defined and how experts are identified. The recruitment protocol is designed to isolate LLM writing experience as the key variable distinguishing high from low performers.

Initial pilot study. The paper began with an exploratory study (described in Appendix C) where five annotators with varying backgrounds and LLM familiarity judged the same batch of 60 articles (Experiment 1). The results revealed a stark bimodal distribution: four annotators who "rarely or never use LLMs for writing tasks" performed near chance (average TPR 56.7%, FPR 51.7%; Table 1), while one annotator who "uses LLMs daily to edit LLM-generated text" performed almost perfectly (TPR 96.7%, FPR 3.3%). This single observation motivated the entire expert recruitment strategy.

Expert identification protocol. To identify additional high-performing annotators, the authors implemented a two-stage screening process (Appendix A.1):

Targeted recruitment: They posted a job on Upwork seeking native English speakers with experience "editing LLM-generated content," who "frequently used LLMs," and had "a professional background in editing or writing." Ten candidates were recruited who met these criteria.
Qualification test (5 questions): Candidates completed a 5-question sample drawn from articles that "at most 2 out of 5 of the original annotators got correct"—specifically selecting the hardest-to-classify articles to create a discriminating test. Candidates needed at least 4 out of 5 correct (80% accuracy) to proceed. Five out of ten candidates passed this stage.
Full batch screening (60 questions): The five candidates who passed the 5-question test then completed the full 60-article batch from Experiment 1. Candidates who achieved at least 90% correct (54 out of 60) were considered "experts" and retained for all subsequent experiments. Four out of five candidates passed this threshold.

The final expert pool thus consists of the one original high-performer plus four newly identified experts. Their demographics (Table 4) confirm the intended profile: all hold at least a Bachelor's degree, work as editors/writers/proofreaders/language teachers, and use LLMs (ChatGPT, Claude, Copilot, Llama, Huggingface models) at least weekly—three use them daily.

The four nonexperts (Experiment 1 only). The remaining four annotators from the initial pilot serve as a contrast group. Their profiles differ sharply from the experts: one had never used AI, two had used it only a few times, and none reported using LLMs for writing tasks. Their occupations included transcription, finance, and general writing (not editing-specific). The purpose of including them in Experiment 1 is not to evaluate them as detectors but to demonstrate that LLM writing experience is the discriminating factor, not general intelligence, education, or English proficiency.

Sample size justification. The paper explicitly addresses the small sample size (five experts) in Section 2:

"While a larger sample size would strengthen population-level claims; prohibitive costs limit us to the current sample size."

The cost breakdown supports this: "It cost us $865 USD per annotator for five experiments." With five experts completing 300 articles each (60 per experiment), at$ 2 per article plus bonuses, the total annotation cost reaches $4,900 USD. The within-subjects design—where each annotator judges both human and AI articles—"reduces variability from individual differences and requires fewer annotators" (Allen, 2017), partially mitigating the small N concern. However, the paper is candid that population-level generalization is limited: they are establishing an upper bound for a specific population, not making claims about all LLM-experienced individuals.

Compensation and ethics. Annotators were paid " $2 per candidate text, resulting in an hourly rate of about$ 15–$30 USD." The IRB review (UMass IRB #5927) deemed the study exempt. All annotators provided informed consent, and those who wished to be acknowledged by name explicitly agreed (the Acknowledgments section lists five annotators with links to their Upwork profiles).

Annotation Interface and Task Design

The annotation task is designed to collect rich, multi-modal judgments while preventing annotators from using superficial heuristics or becoming aware of the human-AI pairing structure.

What annotators see. Each annotator is presented with one article at a time, in a web interface (Figure 6). The article title is not displayed—this is a deliberate design choice to "prevent obvious linkage of AI-generated and human-written text pairs" (Appendix A), since paired articles share the same title and subtitle. Annotators see only the article body and are asked to provide four pieces of information:

Binary label: A judgment of whether the article is "human-written" or "AI-generated."
Confidence rating: A Likert scale from 1 (least confident) to 5 (most confident).
Highlighted spans: The annotator can select specific text spans in the article that serve as evidence for their judgment. Figure 1 shows an example where the annotator has highlighted specific phrases like "running a hand through the frost on his jacket sleeve" and "He shrugged as if that were the most ordinary idea, then laughed."
Paragraph-length explanation: A free-form text explanation of their decision-making process. Figure 1 includes the example: "Lots of the quotes felt realistic, but many of the quotes did not need a narration alongside it such as with 'He shrugged as if that were the most ordinary idea, then laughed.' … could have been shortened to get more facts in about what people in Alaska face... Also, it got sentimental and corny at times too."

Why require explanations? The explanation requirement serves two purposes. First, it ensures that annotators actually read the articles carefully—the paper explicitly cites Karpinska et al. (2021), who found that "Mechanical Turk annotators 'read' long-form texts in a matter of mere seconds before evaluation." Requiring paragraph-length explanations with specific highlighted evidence makes superficial skimming impossible. Second, the explanations are the primary data for the qualitative analysis that forms the paper's second major contribution (the taxonomy of detection clues). Without them, the paper could only report that experts are accurate, not how they achieve that accuracy.

Randomization and blinding. Articles from all experiments are shown to annotators in a randomized order. Annotators are unaware of which articles are human-written and which are AI-generated, and they are not told that articles come in human-AI pairs. They receive no feedback on which articles they labeled correctly or incorrectly at any point during the study. This is critical because it means their detection ability is entirely self-developed—they are not being trained by the experiment, merely measured.

Within-subjects design. Each annotator judges both human-written and AI-generated articles within each experiment. This design "reduces variability from individual differences and requires fewer annotators" (Allen, 2017) compared to a between-subjects design where different annotators would judge different subsets. The trade-off is potential carryover effects—seeing an AI-generated article first might influence judgment of a human article second—but the authors mitigate this through randomization and by not revealing the pairing structure.

Task duration and logistics. Based on self-reported time and spreadsheet records, annotators could "read and label between 8 and 12 articles per hour." A batch of 60 articles thus required approximately 6-8 hours of work. Annotators were given one week to complete each batch, with more time allowed when needed. The interface included a save feature so annotators could work in multiple sessions.

Explanation Coding Taxonomy

The paper's qualitative contribution—understanding what experts detect—is operationalized through a systematic coding of all 1,790 expert explanations into a taxonomy of 16 clue categories (Table 3, with the full version in Table 17).

Taxonomy development process. The coding scheme was developed through an iterative human-in-the-loop process rather than being pre-specified. The first two authors "individually annotated a sample of 25 expert explanations" drawn as a stratified sample (one comment per expert per experiment). They then "came to an agreement on final human labels, refining categories as needed" (Appendix D.1). This bottom-up approach ensures the categories reflect what experts actually say rather than what the authors hypothesize they might say.

The 16 categories (with their definitional essence and overall frequency across all explanations, from Table 17):

Vocabulary (53.1%): LLMs use specific words and phrases more often than human writers, resulting in repetitive, unnatural, or overly complex wording. This is the single most common category, appearing in over half of all explanations.
Sentence Structure (35.9%): AI-generated sentences follow predictable patterns—"not only ... but also" constructions, lists of exactly three items, complex sentences with multiple dependent clauses—while humans vary sentence length and structure more.
Grammar & Punctuation (24.8%): AI text is usually grammatically perfect and avoids dashes, ellipses, and other informal punctuation; human text contains minor errors and more varied punctuation.
Originality (23.7%): AI writing is "safe," straightforward, and lacks surprises, humor, or creative analogies; human writing takes risks and engages the reader.
Quotes (22.3%): AI-generated quotes are overly formal, lack the nuanced variability of real speech, and mirror the article's main text too closely in style and vocabulary.
Clarity (19.5%): AI text over-explains or includes irrelevant details ("telling rather than showing"), lacking the concise flow of good human writing.
Formatting (15.0%): AI formatting is overly consistent—fully capitalized headings, bolded lists with identical structures, paragraphs of similar length.
Conclusions (13.1%): AI conclusions are repetitive, overly optimistic summaries ("testament," "beacon of hope"); human conclusions end more abruptly and less tidily.
Formality (12.3%): AI text lacks filler words, contractions, slang, or abbreviations, favoring fully spelled-out terms and a polished tone unless explicitly instructed otherwise.
Names & Titles (11.7%): LLMs generate the same fictional names (Emily, Sarah) and use consistent titles (always "Dr."); human writing includes unique names, real brands, and varied forms of address.
Tone (9.3%): AI tone is consistently neutral or positive, lacking the emotional range and implicit bias of human writing.
Introductions (7.3%): AI introductions are generic or focus on scenic details without providing key background; human introductions have more compelling and varied hooks.
Factuality (7.2%): AI text contains factual inconsistencies or hallucinations; human journalism is factually verifiable.
Topics (3.1%): AI avoids darker or more mature topics (violence, graphic descriptions, controversial subjects).
Other (2.6%): Clues that do not fit any category, often based on overall impression or intuition.
(The paper lists 15 explicit categories plus "Other"; the numbering here follows the table.)

Scaling via GPT-4O. After developing the taxonomy on 25 manually annotated examples, the authors used GPT-4O-2024-11-20 to code all 1,500 expert explanations (300 articles × 5 experts) using the prompt in Table 18. The prompt provides definitions for all categories and asks the model to identify which categories apply to each explanation, whether the annotator associates that category with AI-generated or human-written text, and the specific quote supporting the classification. The total cost of this automated coding was $6.02 USD.

Validation. The authors validated the automatic coding by first ensuring the prompt "was able to classify the majority of the sample explanations in alignment with the authors' labels" on the 25 manually annotated examples before applying it to the full dataset. This is a necessary sanity check, though the paper does not report quantitative inter-annotator agreement metrics between the human-coded and GPT-4O-coded labels.

What the taxonomy enables. The coded explanations enable two key analyses:

Per-experiment clue frequency shifts (Figure 3): How does the distribution of clue categories change as the generation method becomes more sophisticated? For example, vocabulary mentions drop from 69.8% of correct GPT-4O explanations to 42.3% for humanized O1-PRO explanations, suggesting that vocabulary becomes less diagnostic as humanization improves.
Per-annotator clue preferences (Figures 10-14): Which categories does each expert rely on most? This reveals complementary detection strategies that explain why the majority vote outperforms any individual expert.

Automatic Detector Benchmarking

To contextualize human performance, the paper benchmarks five automatic detectors on the same 300 articles, using recommended thresholds wherever possible.

Detector selection. The five detectors span the range from commercial closed-source to open-source research systems:

Pangram (Emi and Spero, 2024): A closed-source commercial detector using a Transformer classifier trained iteratively with hard negative mining and synthetic data. The paper evaluates two variants: the base Pangram model and "Pangram Humanizers," a newer version specifically trained to detect humanized data (Masrour et al., 2025). Both are accessed via API, and the paper uses the API's own binary labels, "ignoring the 'Possibly AI' label."
GPTZero (Tian and Cui, 2023): A closed-source commercial detector that runs a classifier sentence-by-sentence across the document. Accessed via API with the provided binary labels.
Binoculars (Hans et al., 2024): An open-source method computing cross-perplexity between two different language models. The paper evaluates in both recommended modes: "Accuracy" mode and "Low FPR" mode, which use different threshold settings provided by the original authors.
Fast-DetectGPT (Bao et al., 2023): An open-source method that samples and scores perturbations of the candidate text to estimate conditional probability curvature. The paper thresholds this at a false positive rate of 5%, calibrated on a held-out development set of 40 human-written articles.
RADAR (Hu et al., 2023): An open-source classifier trained adversarially against a paraphraser. Also thresholded at FPR=5% on the same development set.

Additional detectors tested but reported in Appendix Table 23 include e5-lora (one of the top-performing models on the RAID benchmark; Dugan et al., 2024), which "was unable to correctly classify any AI-generated articles at a FPR of 5%."

Thresholding procedure. For methods without recommended thresholds (RADAR, Fast-DetectGPT, e5-lora), the paper uses a held-out set of 40 human-written articles to calibrate a threshold that achieves 5% FPR. The specific thresholds used are: RADAR at 0.6051510572, Fast-DetectGPT at 0.96, and e5-lora at 0.8963184953 (Appendix E.1).

Evaluation metrics. The paper reports True Positive Rate (TPR)—the percentage of AI-generated articles correctly identified—and False Positive Rate (FPR)—the percentage of human-written articles incorrectly flagged as AI. These are computed per-experiment (60 articles each) and aggregated across all 300 articles. The authors note that prior work often reports "TPR at a fixed low FPR such as 1%" or "AUROC," which "we cannot easily do with human annotators" given the discrete nature of human judgments and the small sample size.

Why these particular detectors? The selection is not exhaustive but covers the major methodological families: perplexity-based (Binoculars, Fast-DetectGPT), adversarially trained classifiers (RADAR), commercial production systems (Pangram, GPTZero), and—in the prompt-based experiments—LLM-as-detector. This diversity allows the paper to claim that expert humans match or exceed all categories, not just a particular approach.

Prompt-Based Detector Experiments

Section 4 tests whether LLMs can be prompted to replicate expert human detection by providing them with the same guidebook that experts developed. This is not intended as a practical detector but as a diagnostic experiment: if LLMs with access to the expert guidebook cannot match human experts, then the experts are detecting something beyond what can be articulated in the guidebook.

Four configurations (Table 2, bottom section, and detailed in Appendix E.2):

Zero-Shot: The LLM receives only the candidate text and a simple prompt asking "Is the above candidate text HUMAN-WRITTEN or AI-GENERATED?" (Table 19). This measures the LLM's baseline detection ability without any guidance.
Zero-Shot + CoT (Chain-of-Thought): The LLM is asked to "concisely describe the features of the candidate text that exemplify either AI or human writing" before providing its answer (Table 20). This tests whether explicit reasoning improves detection, as it does for many reasoning tasks.
Zero-Shot + Guide: The LLM receives the full detection guide (Table 11) and is asked to determine authorship "based on the information in the provided guide" (Table 21). This tests whether the guide alone—without step-by-step reasoning—improves performance.
Zero-Shot + CoT + Guide: The full configuration: LLM receives the guide, is asked to reason about the features, then provide a judgment (Table 22). This most closely mimics the human expert process.

Detector LLMs tested. Two models are evaluated across all configurations:

GPT-4O-2024-11-20: Selected for its high TPR in preliminary tests.
O1-2024-12-17: Selected for its low FPR in preliminary tests.

Additional preliminary testing (Table 23) included GPT-4O-2024-08-06 (which showed "extremely varied performance" depending on the version) and CLAUDE-3.5-SONNET (which "failed to detect content generated by itself").

Why these experiments matter. The prompt-based detector results are not proposed as a practical alternative to commercial detectors—they are substantially worse and more expensive (O1 detection cost $100.60 USD for 300 texts vs. commercial API costs). Instead, they serve as an ablation: if providing the guidebook to an LLM produces near-human performance, then the experts' advantage reduces to their articulated knowledge, which could be automated. The fact that even the best configuration (GPT-4O + CoT + Guide) achieves only 78% overall TPR compared to the expert majority vote's 99.3% suggests that expert detection relies on tacit knowledge that resists verbalization into a guidebook.

Cost comparison context. The paper reports detection costs to ground the human-vs-machine comparison in economic reality: human experts cost approximately $2 per article (excluding bonuses and screening), the Pangram API costs per-query commercial rates, and the O1 prompt-based detector costs roughly$ 0.34 per article ( $100.60 for 300 texts). The economic calculus depends on scale: for high-stakes, low-volume applications (plagiarism investigations, legal document verification),$ 2 per article with human-level accuracy and explainability may be justifiable; for high-volume moderation, only automatic methods are practical.

Cross-Experiment Design Decisions

Several methodological choices span all experiments and deserve explicit justification:

No training or feedback. The paper repeatedly emphasizes that experts receive "no additional training" and "no feedback on which articles they labeled incorrectly." This is methodologically crucial because it means the measured performance represents naturally occurring detection ability, not ability developed through experimental participation. If experts improved over the course of the experiments, the later results would conflate innate ability with learning effects. The paper's claim that expert detection is robust to model changes and evasion tactics would be weaker if experts had been trained on earlier experiments.

No within-experiment improvement tracking. The paper does not report whether annotator accuracy improves from earlier to later batches within an experiment, nor whether performance changes across the five experiments. This is both a limitation (we cannot assess learning effects) and a strength (the cross-experiment comparisons are not confounded by practice effects within each experiment).

The article title blinding. By not showing article titles, the paper prevents annotators from using the most obvious linkage cue: if you see two articles with the same title, you know one is human and one is AI. Even without explicit knowledge of the pairing structure, seeing a repeated title would alert annotators that something unusual is happening. This design choice means annotators evaluate each article as if it were encountered in isolation, which better reflects real-world detection scenarios (you don't typically have a human-written reference article on the exact same topic).

Use of the ChatGPT Pro interface for O1-PRO. The authors note (Appendix B.2) that "the API for O1 was not yet available" during data collection, forcing them to use the web interface. This introduced two uncontrolled variables: (1) O1-PRO outputs are non-deterministic (OpenAI fixes temperature=1 for O1), and (2) the Pro interface may apply additional system-level formatting or safety filters not present in the API. The paper acknowledges this limitation implicitly by noting the API's unavailability but does not discuss how it might affect reproducibility.

4. Key Insights and Innovations

Innovation 1: Expertise in AI Detection Is a Function of Familiarity, Not General Aptitude — and It's Untrained

The paper's most fundamental conceptual move is reframing human AI-text detection from a general human ability question ("Can humans detect AI text?") to a population-specific expertise question ("Which humans can detect AI text, and what distinguishes them?"). Prior work (Brown et al., 2020; Clark et al., 2021) treated human detection as a monolithic capability, sampling annotators without regard to their LLM experience and concluding that humans perform near chance. This paper demonstrates that those conclusions were an artifact of sampling from the wrong population.

What makes this distinctive is not the finding that some people are better detectors than others—individual variation has been noted before (Ippolito et al., 2020; Dugan et al., 2022)—but rather the operationalization of expertise as a specific, measurable background characteristic (frequent LLM use for writing tasks) and the demonstration that this characteristic alone, without any task-specific training or feedback, produces near-perfect performance. The four nonexperts in Experiment 1 (Table 1) averaged 56.7% TPR and 51.7% FPR—essentially random guessing with a slight bias toward labeling everything as AI—while the five experts, selected purely on the basis of their LLM writing experience and screened only for baseline accuracy, achieved 92.7% average TPR and 4.0% FPR on the same articles. The experts received no training, no feedback, and no instruction on what to look for. Their ability is entirely self-developed through their daily interactions with LLMs.

This reframing has significant implications beyond the paper's empirical results. It suggests that the human detection ceiling is not fixed but scalable with experience: as more people use LLMs regularly for writing, the population of potential expert detectors grows. It also provides a resolution to the contradictory findings in prior work: studies that found humans perform near chance (Brown et al., 2020; Clark et al., 2021) likely drew predominantly from the nonexpert population, while those that noted individual high-performers (Ippolito et al., 2020) may have inadvertently included experts. The paper's explicit sampling and screening protocol makes this population distinction legible and replicable.

The finding that nonexperts are not just inaccurate but overconfident (average confidence 4.03 out of 5, Table 1) adds a further layer: nonexperts are not aware of their own limitations. This has practical implications for crowdsourced detection efforts, where confidence-weighting nonexpert judgments would not solve the problem—the annotators genuinely believe they are performing well while achieving near-chance accuracy.

A subtle but important aspect: the experts' expertise transfers across model families. Four of five experts maintained high performance on CLAUDE-3.5-SONNET despite most having no prior experience with Claude (Section 2.2), and four of five remained robust to O1-PRO (Section 2.4), a model they had never encountered before. This suggests they are detecting something fundamental about how LLMs construct text—not just memorizing the specific vocabulary quirks of GPT-4O. The partial exception, Annotator 2, whose TPR dropped to 80% on CLAUDE-3.5-SONNET (Table 2) and who "did not have any prior experience using Claude," demonstrates that cross-model generalization is strong but not universal: the specific vocabulary distributions of different models do require some calibration that comes from exposure.

Innovation 2: Human Explanation as a Diagnostic Instrument for Characterizing Detector Brittleness

The paper's second distinctive contribution is methodological: it demonstrates that systematic qualitative coding of human expert explanations can serve as a diagnostic instrument for understanding why automatic detectors fail, not just that they fail. This transforms the paper from a simple benchmark comparison into a tool for guiding detector improvement.

The taxonomy of 16 clue categories (Table 3, expanded in Table 17) is not merely a descriptive catalog. It is an adversarial target that the paper uses to probe the limits of both human and automatic detection. By constructing a humanization prompt (Experiment 5, Section 2.5) that explicitly instructs the generator to avoid every category of clue the experts had identified, the paper creates a direct test: if experts succeed on humanized articles, they are detecting signals not captured by their own verbalized knowledge. The fact that the expert majority vote achieves 100% TPR on humanized O1-PRO articles (Table 2) while the humanization prompt was specifically designed around the experts' own guidebook (Table 11) is revealing. It implies that expert detection relies on tacit pattern recognition that resists full articulation—the experts can do detection better than they can explain how they do it.

This diagnostic function extends to the prompt-based detector experiments (Section 4). By providing the same guidebook to LLMs and measuring their performance, the paper disentangles two potential explanations for the human-machine performance gap: (1) humans have better articulated knowledge, or (2) humans have access to detection signals that cannot be captured in a text guidebook. The results support the latter: even with the guidebook and chain-of-thought reasoning, GPT-4O achieves only 78% overall TPR (Table 2), and this drops to 3.3% on humanized O1-PRO articles—worse than random. The guidebook helps on easier configurations (GPT-4O, CLAUDE-3.5-SONNET) but provides minimal benefit on the hardest cases. This suggests that current LLMs lack the perceptual machinery to operationalize expert detection criteria, even when those criteria are explicitly described. It is not a knowledge gap but a capability gap.

The paper's use of the taxonomy to track clue category shifts across experiments (Figure 3) provides additional diagnostic granularity. For example, vocabulary mentions drop from 69.8% of correct GPT-4O explanations to only 42.3% for correct humanized O1-PRO explanations, while originality and clarity become relatively more important. This quantifies which detection dimensions become more vs. less diagnostic as generation quality improves, providing a roadmap for which signals automatic detectors should prioritize in adversarial settings.

This methodological contribution is significant beyond the paper's specific findings because it establishes a template for future detector evaluation: rather than simply reporting TPR/FPR numbers, incorporate human expert explanations to understand structurally where and why detectors fail. The guidebook itself (Table 11, with its categorized list of over 100 specific AI vocabulary items and structural patterns) is a concrete artifact that future work can use, extend, or test detectors against.

Innovation 3: The Empirical Demonstration That Evasion Tactics Have a Detectability Floor — Even When Targeting the Detector's Own Criteria

The paper's most striking empirical finding is that humanization specifically designed to evade expert-identified signatures fails to fool expert majority judgment. This is not obvious. A reasonable prior would be that if you ask experts what they look for, then explicitly instruct the generator to avoid all of those things, the resulting text should be undetectable. The fact that it is not—expert majority vote achieves 100% TPR on humanized O1-PRO articles (Experiment 5, Table 2)—constitutes a significant finding about the nature of AI-generated text.

What makes this an innovation rather than simply a robustness result is the adversarial methodology: the humanizer was constructed from the experts' own detection criteria. The guidebook (Table 11) was compiled directly from the experts' lists of clues (for which they were paid $45 each), organized by the authors, and then fed back to O1-PRO with explicit instructions to produce text that would evade a detector using that guidebook. This is as close to a controlled adversarial experiment as is possible in this domain: the detector's stated decision criteria become the generator's avoidance targets.

The fact that this fails—and fails at the aggregate majority-vote level—implies one of two things: either (1) the guidebook is incomplete and experts are detecting signals they did not articulate, or (2) current LLMs, even when explicitly instructed to avoid specific patterns, are incapable of fully doing so because those patterns are deeply embedded in their training distribution. The paper's evidence supports both interpretations. The guidebook's incompleteness is demonstrated by the fact that experts continue to find new tells in humanized articles—for instance, they noted that humanized articles "use titles for these people (e.g., Dr., Prof.) much more frequently than in human-written articles" (Section 2.5), a pattern no one had included in the original guidebook. The generator's inability to fully avoid AI patterns is demonstrated by the persistent presence of detectable structural signatures even after humanization.

The dropout of Annotator 3 on humanized O1-PRO (TPR of 0%, Table 2) actually strengthens this innovation: it shows that humanization did succeed at removing some detection dimensions (particularly formality-based signals, on which Annotator 3 relied most heavily—Figure 12 shows they mentioned formality in 66.7% of O1-PRO and 83.3% of humanized O1-PRO explanations, compared to nearly 0% in earlier experiments). However, the other four experts shifted their attention to different dimensions, compensating for the loss of the formality signal. The ensemble compensates for individual vulnerabilities, and the vulnerabilities themselves are informative about which signals are successfully removed by humanization.

This finding has broader implications for the detection arms race. It suggests that text generation, even when adversarially optimized against known detection criteria, leaves detectable traces that are detectable by humans with sufficient domain-specific experience. It establishes an empirical floor on the detectability of AI-generated text by expert humans, at least for the current generation of models. Whether this floor will persist as models improve is an open question, but the paper provides a methodology for tracking it.

Innovation 4: The Expert Ensemble as a Practical Detection Instrument — Matching Commercial Systems While Providing Interpretability

The paper's most practically impactful finding is that a small ensemble of untrained expert humans matches the best commercial detector (Pangram) while providing detailed, interpretable explanations that automatic systems lack. The expert majority vote misclassifies only 1 of 300 articles overall (Table 2, "OVERALL" column), tying Pangram Humanizers at 99.3% TPR with 0% FPR. This parity with a production commercial system—achieved without any training data, model development, or engineering—is remarkable and reframes the cost-benefit calculus for human-in-the-loop detection in high-stakes settings.

What distinguishes this from a simple "humans are good at this" observation is the ensemble structure. Individual experts vary substantially in their performance and their detection strategies. Annotator 3 achieves only 59.3% overall TPR, completely failing on O1-PRO articles (16.7% TPR on non-humanized, 0% on humanized; Table 2). Yet the majority vote is near-perfect because different annotators fail on different articles and rely on different clue categories (Figures 10-14 in Appendix D.3). Annotator 1 uniquely detects "AI names" (the Emily/Sarah pattern), Annotator 4 analyzes article flow and specificity of detail, Annotator 5 focuses on quotation naturalness. Their errors are uncorrelated, meaning the majority vote compensates for individual weaknesses.

This ensemble property is not an accident of the specific five annotators recruited—it emerges naturally from the diversity of detection strategies that experts develop through their individual experiences with LLMs. The paper's qualitative analysis (Section 3, "Annotators don't always focus on the same clues") demonstrates this diversity concretely, and the individual heatmaps (Appendix D.3) quantify it. The implication is that any sufficiently diverse group of LLM-experienced writers will exhibit complementary detection strengths, making ensemble detection robust by construction.

The practical significance of this finding extends beyond the paper's numbers. In high-stakes detection scenarios—academic plagiarism hearings, legal document verification, journalism fact-checking—the combination of near-perfect accuracy and paragraph-length explanations with highlighted evidence spans (Figure 1) provides a level of due process and transparency that no automatic detector currently offers. The paper explicitly connects this to Lipton's (2018) critique of model interpretability: a score of 0.87 from a black-box classifier is fundamentally less trustworthy than a human's explanation that "the quotes feel fake because every expert speaks the same way and it's too homogenous with the text" (Table 3). The economic analysis— $865 per annotator for five experiments, yielding approximately$ 2 per article—establishes that this level of scrutiny is economically viable for low-volume, high-stakes applications, even if it cannot scale to forum moderation or bulk content filtering.

The paper also demonstrates that expert performance can likely be improved further through training. The statement that "our experts are untrained at this detection task, and they could likely improve their individual performance if provided with feedback on their errors" (Section 3) is supported by the observation that different experts focus on non-overlapping clue categories. If Annotator 3 were made aware of the "AI names" pattern that Annotator 1 uses, or if Annotator 1 were trained to attend to quotation naturalness as Annotator 5 does, individual TPR would likely increase, raising the ensemble ceiling even higher. This training dimension is unexplored in the current paper but is a natural extension of its findings.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses a custom corpus of 300 paired non-fiction English articles, consisting of 150 human-written articles sourced from eight American publications (Associated Press, Discover Magazine, National Geographic, New York Times, Reader's Digest, Scientific American, Smithsonian Magazine, and Wall Street Journal; Table 5) and 150 AI-generated counterparts produced by prompting commercial LLMs with the same title, subtitle, desired length, and publication source as each human article. The 300 articles are evenly divided across five experiments of 60 articles each (30 human, 30 AI-generated), with no overlap in human-written articles across experiments, and article lengths are controlled to be comparable within pairs (Table 6 reports mean lengths ranging from approximately 625 to 816 words across experiments, with standard deviations between roughly 107 and 234 words).
Base model(s) for AI generation. The AI-generated articles are produced using three model families across the five experiments: GPT-4O (2024-08-06) for Experiments 1 and 3, CLAUDE-3.5-SONNET for Experiment 2, and O1-PRO for Experiments 4 and 5. These models are selected to represent different points in the capability spectrum—from the most widely used LLM (GPT-4O), to a competitor from a different organization testing cross-model generalization (CLAUDE-3.5-SONNET), to a frontier reasoning model that none of the expert annotators had prior exposure to (O1-PRO, released during the experimental timeline). For the prompt-based detector experiments (Section 4), GPT-4O-2024-11-20 and O1-2024-12-17 serve as the detector LLMs.
Metrics. The primary evaluation metrics are True Positive Rate (TPR)—the percentage of AI-generated articles correctly classified as AI-generated—and False Positive Rate (FPR)—the percentage of human-written articles incorrectly classified as AI-generated. These are reported per-experiment (each with 30 AI and 30 human articles) and aggregated across all 300 articles. For human annotators, the paper reports both individual annotator TPR/FPR and the majority vote (at least three out of five experts agreeing), since the ensemble is the primary detection instrument. For automatic detectors, thresholds are set either using the detector's recommended settings (Pangram, GPTZero, Binoculars) or calibrated to achieve 5% FPR on a held-out development set of 40 human-written articles (Fast-DetectGPT, RADAR, e5-lora; specific threshold values reported in Appendix E.1). The paper explicitly notes that it cannot report AUROC or TPR at fixed low FPR for human annotators "given the discrete nature of human judgments and the small sample size" (Section 2).
Baselines. The paper compares human expert performance against five automatic detectors: Pangram (Emi and Spero, 2024), a closed-source commercial Transformer classifier evaluated in both its base and "Humanizers" variants; GPTZero (Tian and Cui, 2023), a closed-source commercial sentence-level classifier; Binoculars (Hans et al., 2024), an open-source cross-perplexity method evaluated in both "Accuracy" and "Low FPR" modes; Fast-DetectGPT (Bao et al., 2023), an open-source perturbation-based method; and RADAR (Hu et al., 2023), an open-source adversarially trained classifier. Additionally, the paper evaluates prompt-based LLM detectors in four configurations (zero-shot, zero-shot + chain-of-thought, zero-shot + guide, zero-shot + CoT + guide) using GPT-4O-2024-11-20 and O1-2024-12-17 as detector models. The four nonexpert annotators from Experiment 1 (Table 1) serve as a human baseline, achieving an average TPR of 56.7% and FPR of 51.7%, essentially random performance.
Generation budget / compute accounting. For the LLM generation pipeline, costs are reported per model: GPT-4O generations cost an estimated $1.85 USD, O1-PRO cost$ 3.81 USD, and CLAUDE-3.5-SONNET cost $0.51 USD for all generations across experiments (Appendix B.2, footnote 29). For the prompt-based detection experiments, O1 detection on all 300 articles cost$ 100.60 USD and GPT-4O-2024-11-20 detection cost $25.24 USD (Appendix E.3). Human annotation cost$ 2 per candidate text plus bonuses, totaling $4,900 USD for 1,790 annotations across all experiments, which translates to an hourly rate of approximately$ 15–$30 USD (Section 2). The paper does not use "generations" as a compute metric for fair comparison between humans and automatic detectors; instead, comparisons are made at the article level (all methods evaluate the same 300 articles), with the cost analysis provided separately in economic terms.
Cross-validation / statistical protocol. The paper does not employ cross-validation for strategy selection or hyperparameter tuning, as there is no model training involved for the human detection experiments. For automatic detector thresholding, a held-out development set of 40 human-written articles is used to calibrate thresholds for methods without recommended settings (RADAR, Fast-DetectGPT, e5-lora), targeting a 5% FPR. Annotator screening uses a two-stage qualification process: a 5-question test requiring at least 80% accuracy, followed by a full 60-article batch requiring at least 90% accuracy to qualify as an expert (Appendix A.1). The within-subjects experimental design—each annotator judges both human-written and AI-generated articles, with randomized order and without knowledge of the pairing structure—reduces variability from individual differences and requires fewer annotators (Allen, 2017). The paper reports per-annotator metrics and majority vote results but does not compute confidence intervals or statistical significance tests for the comparisons between human and automatic detection performance.

Main Quantitative Results

Experiment 1: Baseline Detection of GPT-4O Articles

The initial experiment on 60 articles (30 human, 30 GPT-4O-generated with no evasion) establishes the fundamental performance gap between nonexperts and experts that motivates the entire study (Table 1, Section 2.1). The four nonexpert annotators—those who rarely or never use LLMs for writing tasks—perform essentially at chance: their average TPR is 56.7% and average FPR is 51.7%, meaning they correctly identify AI-generated articles barely more than half the time while incorrectly flagging human-written articles at a similar rate. Despite this poor performance, these annotators report high confidence in their decisions (average 4.03 out of 5), indicating they overestimate their detection ability.

In contrast, the five expert annotators—recruited based on their frequent use of LLMs for writing tasks and screened through the qualification protocol—achieve an average TPR of 92.7% and average FPR of 4.0% on the same 60 articles (Table 1). The expert majority vote correctly determines authorship of all 60 articles, achieving 100% TPR and 0% FPR (Table 2, "GPT-4O" column). Individual expert performance on this batch ranges from 86.7% to 96.7% TPR, with FPRs between 0% and 6.7% (Table 2, rows for Annotators 1–5). This near-perfect aggregate performance on vanilla GPT-4O output establishes the human upper bound for the simplest detection scenario.

The nonexpert confusion reveals specific patterns in their errors (Section 2.1). Nonexperts mistakenly fixate on vocabulary choice, treating any unusual or "fancy" word as evidence of AI authorship, while experts are familiar with the specific words and phrases that LLMs overuse (e.g., testament, crucial). Nonexperts incorrectly believe that human authors are more likely than AI to form grammatically-correct sentences, while experts recognize that humans actually make more grammatical errors than LLMs. Nonexperts also attribute any text written in a neutral tone to AI, producing many false positives on human-written articles that maintain journalistic objectivity.

Experiment 2: Cross-Model Generalization to CLAUDE-3.5-SONNET

When the generation model is switched from GPT-4O to CLAUDE-3.5-SONNET—a model from a different organization that most experts had no prior experience with—expert performance remains largely unchanged (Table 2, "CLAUDE" column, Section 2.2). The expert majority vote again achieves 100% TPR and 0% FPR, correctly classifying all 60 articles. Individual TPRs range from 80.0% to 100%, with FPRs between 0% and 30% (the 30% FPR outlier is Annotator 2, discussed below). Two annotators achieve perfect individual scores (100% TPR, 0% FPR). The average TPR across experts remains high, demonstrating that detection expertise is not overfitting to GPT-4O-specific artifacts but generalizes across model families.

The one notable degradation is Annotator 2, whose individual TPR drops from 96.7% on GPT-4O to 80.0% on CLAUDE-3.5-SONNET (Table 2). The paper attributes this to Annotator 2's specific detection strategy: they "focuses more heavily on the existence of 'AI vocab' in the article than the other annotators, and they were not familiar with the distribution of words that are overused by Claude" (Section 2.2). This suggests that while the general structural and stylistic signatures of AI text transfer across models, the specific vocabulary distributions are model-dependent, and annotators who rely disproportionately on vocabulary cues may need exposure to new models to calibrate their internal "AI word list." Notably, three other annotators who were also unfamiliar with Claude maintained TPRs of at least 96.7%, indicating that their detection strategies relied less on model-specific vocabulary knowledge.

Experiment 3: Robustness to Paraphrasing

The paraphrasing attack—sentence-level rewriting using GPT-4O to vary surface tokens while preserving meaning, with added named entity substitution to replace fictional AI names with real names from corresponding human articles—has minimal effect on expert detection (Table 2, "GPT-4O PARAPHRASED" column, Section 2.3). The expert majority vote again achieves 100% TPR and 0% FPR on all 60 articles. Individual TPRs range from 86.7% to 100%, with FPRs between 0% and 10%. Overall TPR and FPR "remain almost unchanged in this experiment compared to the first two" (Section 2.3), suggesting that paraphrasing is not an effective evasion tactic against expert human detectors.

The paper's quantitative coding of expert explanations reveals a counterintuitive pattern for this experiment: vocabulary is mentioned in 88% of correct explanations about paraphrased articles, compared to only 69.8% for non-paraphrased GPT-4O articles (Figure 3, discussed in Section 3). A close reading of the explanations reveals that this increase does not mean paraphrasing introduced more AI vocabulary—rather, experts are flagging that even after paraphrasing, certain words and phrases remain telltale. The sentence-level rewriting fails to remove all instances of the vocabulary patterns that experts recognize, and in some cases, the paraphrasing process may surface new instances of AI-preferred synonyms when replacing the original AI vocabulary. Additionally, quotations are mentioned in 33.8% of paraphrased article explanations (Figure 3), a much higher rate than other configurations, because experts noticed that paraphrased articles consistently place quotes in the same format and style (e.g., only at the end of each paragraph), a structural signature that paraphrasing does not alter.

Experiment 4: Detection of Frontier Reasoning Model (O1-PRO)

The release of O1-PRO during the experimental timeline provided a unique test: since none of the experts had prior exposure to this model, how well would their detection skills transfer to a fundamentally different generation paradigm with enhanced reasoning capabilities? The results show strong but slightly degraded performance (Table 2, "O1-PRO" column, Section 2.4). The expert majority vote achieves 96.7% TPR and 0% FPR, correctly classifying 59 of 60 articles. This marks the first—and only—article misclassified by the expert majority vote across all five experiments.

The single misclassified article is "New Telescope Could Potentially Identify Planet X" (Table 14), a Discover Magazine article about the search for Planet Nine. Analyzing the experts' explanations, the three annotators who voted "human-written" for this article "most focused on vocabulary and sentence structure" and "did not find enough clues in those categories to judge the article as AI-generated" (Section 2.4, footnote 17). This suggests that O1-PRO is more successful at varying its vocabulary and sentence patterns than previous models, making it harder for annotators who rely primarily on those dimensions.

Individual performance varies more on O1-PRO than in previous experiments. Four of five experts maintain TPRs between 90.0% and 100%, with FPRs between 0% and 10%. However, Annotator 3 experiences a dramatic performance drop: their TPR falls to 16.7%, essentially failing to detect any O1-PRO articles. The paper identifies the cause as Annotator 3's over-reliance on formality cues. Annotator 3 "relied too much on signs of informality (e.g., contractions, slang usage, usage of just and actually) as a sign of human writing" (Section 2.4). Figure 12 (Appendix D.3) quantifies this: Annotator 3 mentions formality in 66.7% of their O1-PRO explanations, compared to nearly 0% in Experiments 2 and 3. O1-PRO, because of its enhanced reasoning and stylistic flexibility, produces text that is more informal than previous models—using contractions and colloquialisms—which Annotator 3 misinterprets as evidence of human authorship.

Average expert confidence drops to 4.21 out of 5 in this experiment, compared to 4.39, 4.38, and 4.48 in Experiments 1, 2, and 3 respectively (Section 2.4, Figure 2). Verbal feedback from experts confirmed the increased difficulty posed by O1-PRO articles. The paper notes that experts "shift focus" in their explanations for O1-PRO, "more frequently commenting on identifying characteristics that make text sound 'human'" rather than looking primarily for AI signatures (Section 2.4). For example, experts point to how humans repeatedly use the word says, while AI tries to use descriptive synonyms like notes and explains—a pattern they had not articulated as explicitly in prior experiments.

Experiment 5: Robustness to Adversarial Humanization

The most challenging experiment combines O1-PRO generation with an explicit humanization prompt constructed from the experts' own detection guidebook (Section 2.5). Despite being specifically designed to evade the exact detection criteria that experts had articulated, the humanized articles fail to fool the expert majority vote: the ensemble again achieves 100% TPR and 0% FPR on all 60 articles (Table 2, "O1-PRO HUMANIZED" column).

Individual performance, however, reveals selective vulnerability. Annotator 3's performance collapses completely: their TPR on humanized O1-PRO is 0%, meaning they labeled every AI-generated article as human-written. Their FPR is 3.3%, indicating they labeled nearly every article—both human and AI—as human-written. This is consistent with their over-reliance on formality cues (Figure 12 shows formality mentioned in 83.3% of their humanized O1-PRO explanations), which the humanization prompt explicitly targets by instructing the model to use contractions, colloquialisms, and informal language patterns.

The other four experts maintain strong performance, with individual TPRs of 86.7% to 100% (Table 2). Annotator 1 experiences a notable increase in false positives (FPR of 23.3%, up from 6.7% in Experiment 4), suggesting that some human-written articles now appear AI-generated to them after humanization—perhaps because the humanization prompt's instructions result in articles that are unnatural in a different way, being too deliberately informal.

Confidence drops substantially in this experiment: 15.1% of annotations have a confidence rating of 1 (the lowest), compared to much lower rates in prior experiments (Figure 2). This confidence degradation, combined with the perfect majority vote, demonstrates that experts recognize the increased difficulty of the task while still arriving at correct aggregate judgments. The paper notes that the "clues used by experts remained consistent from prior experiments, showing that humanization does not completely remove the 'AI signature' from texts" (Section 2.5). Specifically, experts found that humanized articles use titles for named individuals (e.g., Dr., Prof.) much more frequently than human-written articles—a pattern that was not in the original guidebook and thus not targeted by the humanization prompt.

Aggregate Performance Across All Experiments

When results are aggregated across all 300 articles (Table 2, "OVERALL" column), the expert majority vote achieves 99.3% TPR and 0% FPR, misclassifying only a single article (the O1-PRO "Planet X" article from Experiment 4). This ties Pangram Humanizers for the highest overall TPR among all methods evaluated. Among automatic detectors:

Pangram Humanizers achieves 99.3% TPR and 2.7% FPR across all experiments, matching the expert majority vote in TPR but with a non-zero FPR. Its performance dips slightly on humanized O1-PRO (96.7% TPR, 10% FPR), where the expert majority vote is perfect.
Pangram (base) achieves 98.0% TPR with 2.0% FPR, also faltering on humanized O1-PRO (90.0% TPR, 6.7% FPR).
GPTZero performs well on easier configurations (100% TPR on GPT-4O and paraphrased GPT-4O) but degrades substantially on O1-PRO (76.7% TPR) and humanized O1-PRO (46.7% TPR).
Open-source detectors show sharp degradation under adversarial conditions. Binoculars (Accuracy mode) achieves 100% TPR on GPT-4O but drops to 60.0% on paraphrased GPT-4O, 73.3% on O1-PRO, and 6.7% on humanized O1-PRO. Fast-DetectGPT (FPR=0.05) drops from 100% on GPT-4O to 56.7% on paraphrased, 86.7% on O1-PRO, and 23.3% on humanized. RADAR (FPR=0.05) fails catastrophically, achieving only 15.3% overall TPR, with 0% TPR on CLAUDE-3.5-SONNET and humanized O1-PRO.
Prompt-based detectors (Table 2, bottom section) fail to match either human experts or closed-source detectors. The best GPT-4O configuration (Zero-Shot + CoT + Guide) achieves 78.0% overall TPR with 10.7% overall FPR, performing competitively on GPT-4O and CLAUDE-3.5-SONNET (100% TPR on both for several configurations) but collapsing on humanized O1-PRO (3.3% TPR for GPT-4O detectors, 0% for O1 detectors). O1-based detectors are more conservative, achieving lower FPRs (0.6% overall for the best configuration) at the cost of much lower TPR (54% for the best O1 configuration).

The key performance pattern across all experiments is that detection difficulty is not monotonic with model capability but interacts strongly with evasion tactics. Open-source and prompt-based detectors fail primarily when generation quality increases (O1-PRO) or when explicit humanization is applied, while commercial detectors and human experts maintain robustness across these dimensions. The expert majority vote is the only method besides Pangram Humanizers to achieve near-perfect performance on the most challenging configuration (humanized O1-PRO).

Explanation Coding Results

The qualitative coding of 1,790 expert explanations into the 16-category taxonomy (Table 17) provides quantitative characterization of what experts detect and how this shifts across experiments (Figure 3, Section 3). Vocabulary-related clues are mentioned in 53.1% of all explanations, making it the single most common category. Sentence structure (35.9%), grammar and punctuation (24.8%), originality (23.7%), and quotes (22.3%) round out the top five. The distribution of clue categories shifts systematically as generation sophistication increases:

For vanilla GPT-4O articles, vocabulary is mentioned in 69.8% of correct explanations and sentence structure in 40.5% (Figure 3, "GPT-4o" column). These surface-level features are the primary giveaways.
For paraphrased GPT-4O, vocabulary mentions increase to 88% of correct explanations, which the paper interprets as annotators noting that vocabulary patterns persist despite rewriting. Quotes are mentioned unusually frequently (33.8%).
For CLAUDE-3.5-SONNET, the distribution is similar to GPT-4O, with vocabulary at 72.1% and sentence structure at 37.2%, confirming that experts detect the same categories of signals across model families.
For O1-PRO, vocabulary drops to 57.1% of correct explanations, while originality (28.6%) and formatting (19.0%) become relatively more important. Experts shift from looking for specific overused words to evaluating higher-level properties like creativity and structural patterns.
For humanized O1-PRO, vocabulary drops further to 42.3% of correct explanations, while clarity (21.2%) and formatting (19.2%) gain relative importance. The diminished role of vocabulary is expected since the humanization prompt explicitly targets vocabulary overuse, forcing experts to rely more on structural and qualitative dimensions.

Figure 9 shows the clue distribution for incorrect expert explanations, revealing which categories lead experts astray. For O1-PRO articles, formality is a major source of misdirection (Annotator 3's reliance on informality as a human signal), while for human-written articles incorrectly flagged as AI, vocabulary (31% of false positive explanations) and sentence structure are the most common mistaken triggers—typically when a human author happens to use a word on the "AI vocab" list or employs a balanced sentence structure that resembles AI patterns.

The per-annotator heatmaps (Figures 10–14 in Appendix D.3) quantify individual differences in detection strategy. Annotator 1 uniquely relies on names and titles (mentioning this category across all experiments), while Annotator 5 most frequently mentions quotes. Annotators 2 and 3 emphasize grammar and punctuation, while Annotator 4 focuses on clarity and flow. This diversity of cue usage explains why the majority vote outperforms any individual expert: different annotators fail on different articles because they attend to different, complementary signal dimensions.

Ablation Studies and Robustness Checks

Prompt-based detector configurations (Section 4, Table 2, bottom section): The paper systematically varies whether the detector LLM receives the detection guide and whether it is asked to reason step-by-step before answering. For GPT-4O-2024-11-20, adding the guidebook improves overall TPR from 71.3% (zero-shot) to 76.0% (zero-shot + guide) and 78.0% (zero-shot + CoT + guide), though FPR also increases (from 6.7% to 8.7% and 10.7%). Chain-of-thought alone (zero-shot + CoT) degrades performance to 42.0% TPR, suggesting that explicit reasoning about features without the guidebook leads the model to overthink and misclassify. For O1-2024-12-17, all configurations underperform GPT-4O in TPR, with the best being zero-shot + CoT at 54.0% TPR with 4.0% FPR. The guidebook paradoxically reduces O1's TPR (from 42.2% zero-shot to 36.7% with guide), indicating that O1 is worse at applying the human-derived detection criteria.

Detector model version sensitivity (Table 23, Appendix E.3): The paper reports that GPT-4O-2024-08-06 performs substantially worse than GPT-4O-2024-11-20 on paraphrased and O1-PRO articles. For example, GPT-4O-2024-08-06 in zero-shot achieves only 26.7% TPR on paraphrased GPT-4O and 3.3% on O1-PRO, compared to 100% and 56.7% for GPT-4O-2024-11-20. This version sensitivity—within the same model family—highlights the brittleness of prompt-based detection and the danger of treating "GPT-4O" as a stable detector.

CLAUDE-3.5-SONNET as detector (Table 23): CLAUDE-3.5-SONNET in the zero-shot + CoT + guide configuration achieves only 53.3% overall TPR, and notably "failed to detect content generated by itself"—achieving 86.7% TPR on GPT-4O articles but only 43.3% on its own CLAUDE-3.5-SONNET generations. This counterintuitive result (a model failing to detect its own output) suggests that CLAUDE-3.5-SONNET's internal representations do not encode self-detection signals in a way accessible through prompting, and that detection ability does not automatically follow from generation capability.

e5-lora as a detector (Table 23): The paper tests e5-lora, one of the top-performing models on the RAID benchmark (Dugan et al., 2024), and finds that at a calibrated FPR of 5%, it "was unable to correctly classify any AI-generated articles"—achieving 0% TPR across all five experiments. This negative result demonstrates that strong performance on existing benchmarks does not guarantee transfer to the specific article domain, model families, and evasion tactics tested in this paper.

Annotator qualification threshold sensitivity (Appendix A.1): The screening protocol applies two thresholds: ≥80% on the 5-question test and ≥90% on the full 60-article batch. Five of ten candidates passed the first threshold; four of those five passed the second. This means that among candidates who met the profile (frequent LLM users with editing/writing backgrounds), 40% ultimately qualified as experts. The paper does not explore how sensitive the results are to these specific thresholds—for example, whether lowering the full-batch threshold to 85% would include additional annotators with similar ensemble properties, or whether the 90% threshold is necessary for the near-perfect majority vote.

Within-subjects design as robustness check (Section 2, Appendix A): By having each annotator judge both human-written and AI-generated articles in randomized order without knowledge of the pairing, the design prevents annotators from using relative comparisons within pairs and ensures that detection judgments are made on each article independently. The article title blinding further removes the most obvious linkage cue. The fact that annotators maintain near-perfect performance despite these constraints strengthens the claim that their judgments are based on intrinsic text properties rather than experimental artifacts.

Critical Assessment

The experiments provide strong evidence for the paper's central descriptive claim: annotators who frequently use LLMs for writing tasks can detect AI-generated text with near-perfect accuracy at the majority-vote level, even under paraphrasing and humanization, and their performance matches or exceeds that of the best commercial automatic detectors on the tested articles. The quantitative results in Table 2 are unambiguous on this point—99.3% overall TPR with 0% FPR for the expert majority vote across 300 articles, compared to 99.3% for Pangram Humanizers (the only matching detector) and substantially worse for all other methods. The five-experiment structure systematically escalates difficulty and demonstrates that expert performance is not an artifact of a particular model or an easily removed surface feature.

However, the experiments support this claim only for the specific population, domain, and models tested, and several aspects of the experimental design limit the strength of the conclusions that can be drawn.

The critical limitation is the sample size and its implications for generalizability. Five expert annotators evaluating 300 articles is sufficient to establish that some humans can perform this task at near-perfect levels, but it does not establish that this ability is widespread among frequent LLM users, nor does it characterize the distribution of detection ability within this population. The screening process selected the top four of ten candidates who met the experience profile—a 40% qualification rate. Whether the 60% who did not qualify would perform similarly to the four nonexperts (near chance) or somewhere in between is unknown. The paper's claim that "annotators who frequently use LLMs for writing tasks excel at detecting AI-generated text" (Abstract) is supported only for the subset of such annotators who can pass a 90%-accuracy screening test. The unscreened population might have substantially lower average performance.

The test set itself is small for claims about individual experiment performance: 30 human and 30 AI articles per experiment means that a single misclassification changes TPR or FPR by 3.3 percentage points. The expert majority vote's perfect scores (100% TPR, 0% FPR) in four of five experiments are based on correctly classifying all 60 articles. While impressive, confidence intervals around these point estimates would be wide—with only 30 AI articles per experiment, a 100% TPR could be consistent with a true detection rate anywhere from approximately 88% to 100% (using the rule of three for zero observed failures). The aggregate 99.3% TPR across 150 AI articles (one miss) is more stable but still has a confidence interval extending below 98%.

The domain restriction to non-fiction English articles from professional publications means that the findings may not transfer to other text types. The paper acknowledges this limitation explicitly: "While we cannot make claims about human detection performance in other domains (e.g., scientific papers or social media posts)" (Section 2). The choice of professionally edited, grammatically correct human-written articles as the baseline actually makes the detection task harder in some respects (no spelling errors or informal grammar to use as human signals) but potentially easier in others: professional journalism follows genre conventions that LLMs may over-regularize in detectable ways. Detection of AI-generated social media posts, student essays, creative fiction, or technical documentation might involve different cue distributions and different expert performance levels. The pilot study on Reddit stories (Appendix C) hints at this: nonexperts performed better on stories than articles (69.3% vs. 56.7% average TPR; Table 16 vs. Table 1), and the paper notes that many differences were "superficial due to the very informal nature of reddit," suggesting that domain-specific expectations strongly influence detection strategies.

The humanization approach, while well-motivated, has several limitations that affect the interpretation of Experiment 5. First, the humanization prompt was constructed from the experts' guidebook, which captures their articulated knowledge—but as the paper itself demonstrates, experts detect signals beyond what they can verbalize. The humanization prompt therefore targets only a subset of the actual detection cues, making it a weaker adversarial test than it first appears. A more powerful humanization approach might involve iterative adversarial generation where the humanizer receives feedback from the same experts on which articles were detected and why, creating a true arms race within the experiment. The paper's finding that humanization fails to fool experts is thus partly an artifact of the specific (non-iterative, non-adaptive) humanization method used.

Second, the humanization prompt was only applied to O1-PRO, which was selected because it "was more difficult to detect than GPT-4O and CLAUDE-3.5-SONNET" (Section 2.5, footnote 18). While this creates the hardest possible test, it means we cannot distinguish whether humanization's limited effectiveness is a property of the humanization technique or of O1-PRO's generation characteristics. Humanization applied to GPT-4O or CLAUDE-3.5-SONNET might be more or less effective—GPT-4O might be worse at following the complex avoidance instructions, making humanization less successful, or its baseline output might have more detectable signatures that humanization fails to fully remove.

Third, the humanization experiment uses the same guidebook that was provided to the prompt-based detectors in Section 4. The fact that prompt-based detectors with the guidebook fail on humanized articles (3.3% TPR for GPT-4O + CoT + Guide; Table 2) while human experts succeed (100% majority vote TPR) is presented as evidence that experts detect signals beyond the guidebook. However, this comparison is confounded by the detector models' baseline capabilities: GPT-4O-2024-11-20 is not the same model as O1-PRO (the humanizer), and its ability to apply the guidebook's criteria may be limited by its own perceptual limitations, not by the completeness of the guidebook. A cleaner comparison would test whether the expert annotators themselves, when given only the guidebook and asked to judge articles without reading them carefully (or when time-limited), can match their own unconstrained performance—this would measure how much of their expertise is actually captured by the guidebook.

The paper does not explore whether expert detection ability can be taught, which is a significant gap given the practical implications. If expert-level detection requires years of daily LLM interaction, then the population of available experts is limited. If, however, a training program based on the paper's taxonomy of clues could rapidly upskill nonexperts, the practical deployability of human detection would be much broader. The paper notes that experts "could likely improve their individual performance if provided with feedback on their errors" (Section 3) but does not test this, and the cross-annotator variation in clue usage (Figures 10–14) strongly suggests that exposing each expert to the clues used by others would improve individual performance. This is a missed opportunity: a simple experiment where annotators review the guidebook and see examples of detected patterns, then re-evaluate a held-out set, would have tested whether explicit training closes the gap between experts and nonexperts.

The economic comparison between human experts and automatic detectors is underdeveloped. The paper reports human costs ( $2 per article,$ 4.9K total) and API costs for the prompt-based detectors ( $100.60 for O1,$ 25.24 for GPT-4O on 300 articles), but does not report the per-article cost of the commercial detectors (Pangram, GPTZero), which would be the most relevant comparison for practical deployment decisions. Without these numbers, the claim that "hiring expert human annotators to perform detection is a viable strategy, particularly in high-stakes settings" (Section 1) is only partially supported: human detection is clearly effective, but whether it is cost-effective relative to Pangram (which matches its performance) depends on Pangram's pricing, which is not provided.

The paper's claim about explainability as a human advantage is well-illustrated but not experimentally tested. The taxonomy of explanations (Table 3) demonstrates that humans can provide rich justifications for their decisions, but the paper does not test whether these explanations are actually faithful—that is, whether the cues annotators report using are the cues they actually used. The finding that humanization targeting the reported cues does not eliminate detection suggests some degree of mismatch between reported and actual cue usage, but the paper does not systematically compare highlighted spans to explained reasoning or test whether annotators' judgments change when specific cues are experimentally manipulated. Establishing that human explanations are causally connected to their decisions, rather than post-hoc rationalizations, would require additional experiments (e.g., systematically inserting or removing specific cue types and measuring the effect on judgments).

The ensemble diversity finding is an observation, not a designed experiment. The paper shows that different annotators use different cues (Figures 10–14) and that the majority vote outperforms individuals. However, it does not systematically test whether this diversity is necessary for the ensemble's performance or merely a feature of this particular group. Would any five experts show complementary errors, or did the authors happen to recruit a particularly well-diversified set? Would three experts be sufficient, or is five necessary? These questions matter for practical deployment, and the current data—while suggestive—cannot answer them definitively.

A missing baseline: untrained human detection with the guidebook. The paper demonstrates that experts succeed largely without training, but it does not test whether giving the guidebook to nonexperts would elevate their performance. This would be the most direct test of whether the guidebook captures actionable detection knowledge: if nonexperts with the guidebook approach expert-level performance, then the expertise is primarily explicit knowledge that can be transferred; if they do not, then the expertise is primarily tacit and experiential. The prompt-based detector experiments partially address this for LLMs but not for humans.

Aggregation across experiments is straightforward but hides performance heterogeneity. The "OVERALL" column in Table 2 combines five experiments with substantively different generation methods. Pangram Humanizers' 99.3% TPR and 2.7% FPR overall masks that its TPR drops to 96.7% and FPR rises to 10% on humanized O1-PRO. The expert majority vote's 99.3% overall TPR masks that it misclassified one article—the O1-PRO Planet X article—a fact that would be invisible if only overall numbers were reported. The paper's transparent reporting of per-experiment breakdowns mitigates this concern, but the headline comparison (99.3% for both experts and Pangram Humanizers) obscures that they fail on different configurations: experts misclassify one non-humanized O1-PRO article, Pangram Humanizers misclassify some humanized O1-PRO articles and generate false positives.

The paraphrase experiment's named entity substitution confounds the paraphrase effect. In Experiment 3, in addition to sentence-level paraphrasing, the authors "extract named entities from the human-written article and instruct GPT-4O to include these names in its generated article" (Section 2.3). This means the paraphrased articles differ from the non-paraphrased articles in two ways: surface token variation and named entity realism. The improved detection performance on paraphrased articles (vocabulary cues mentioned in 88% of correct explanations vs. 69.8% for non-paraphrased GPT-4O articles; Figure 3) might reflect either the persistence of AI vocabulary despite paraphrasing or the named entity changes making other AI signatures more salient. The paper attributes the effect to vocabulary persistence, but the confound makes this interpretation less certain. A cleaner design would have applied paraphrasing alone without named entity changes, or included a separate condition with only named entity changes.

Despite these limitations, the experiments collectively establish a robust qualitative finding that cannot be dismissed as a statistical artifact: on professionally written non-fiction English articles, a small group of LLM-experienced humans can detect AI-generated text with accuracy that substantially exceeds most automatic methods and matches the best commercial detector, and this ability persists across model families and in the presence of evasion tactics specifically designed to counter their own articulated detection criteria. The individual-level variation (Annotator 3's failure on O1-PRO, Annotator 2's degradation on CLAUDE-3.5-SONNET) actually strengthens this conclusion by demonstrating that the aggregate result is not driven by a single exceptional annotator but by a genuinely complementary ensemble. The paper's primary contribution is not the precise numerical claim (99.3% TPR) but the demonstration that such performance is achievable by an identifiable, recruitable population without specialized training—a finding that reframes the detection problem from an algorithmic challenge to a human-in-the-loop design problem.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in the Headline Efficiency Gains

The paper's central practical claim—that expert humans match the best commercial detector while providing explainability—rests on an implicit assumption that the cost of identifying expert annotators is separable from the cost of deploying them. This assumption breaks down when the screening and qualification process is considered part of the detection pipeline.

The expert identification protocol described in Appendix A.1 requires a two-stage screening process: a 5-question qualification test followed by a full 60-article batch, with candidates needing ≥90% accuracy on the latter to qualify. Of ten candidates who matched the target profile (frequent LLM users with editing/writing backgrounds), only four passed both thresholds. The paper does not report how many total candidates were initially considered or screened out before the ten, nor does it account for the cost of this screening in the $4.9K total annotation budget. The screening itself consumes at minimum 65 articles of evaluation per candidate, which represents a substantial fixed cost for any deployment that needs to assemble a new expert panel.

The consequence is that the reported $2-per-article detection cost is a **marginal cost** that excludes the fixed cost of panel assembly. For a one-time evaluation of 300 articles, the true cost includes the screening overhead for all candidates (both successful and unsuccessful). In the paper's case, the ten-candidate screening alone required 50 qualification questions (10 candidates × 5 questions) plus 240 full-batch evaluations (4 successful candidates × 60 articles, though the paper implies all five who passed the 5-question test completed the full batch, which would be 300 evaluations). If screening costs are comparable to annotation costs ($ 2 per evaluation), the panel assembly cost could easily exceed the detection cost for a small-scale deployment. For large-scale, ongoing detection efforts, this fixed cost is amortized, but the paper's claim that expert detection is "a viable strategy, particularly in high-stakes settings" (Section 1) would be strengthened by a break-even analysis showing how many articles must be evaluated before the fixed screening cost is recovered relative to commercial detector API fees.

The paper partially acknowledges the screening requirement by describing the protocol in detail (Appendix A.1), but it does not incorporate screening costs into any cost-effectiveness analysis, nor does it discuss how screening requirements would scale if expert panels needed to be assembled repeatedly (e.g., for different domains, languages, or time periods). The paper does not suggest any method for reducing screening costs, such as using the guidebook (Table 11) as a pre-screening knowledge test or developing a shorter qualification instrument.

The Findings Are Unvalidated Beyond a Single Domain, Single Language, and Single Set of Publications

The paper restricts its study exclusively to non-fiction English articles from eight American publications, with all articles professionally written, edited, and proofread to publication standards (Section 2, "Article selection"). The authors acknowledge this limitation explicitly: "While we cannot make claims about human detection performance in other domains (e.g., scientific papers or social media posts)" (Section 2). However, the consequences of this restriction are more severe than a simple "not yet tested" caveat—there are structural reasons to expect domain dependence in both the signals experts detect and their ability to detect them.

The consequence is that the paper's quantitative findings—99.3% TPR for the expert majority vote across 300 articles—cannot be assumed to transfer to other text types without independent validation, and the direction of transfer (better or worse performance) is unknown a priori. Several domain-specific factors that likely influence detection performance include:

Genre convention strength: Professional journalism follows stable genre conventions (inverted pyramid structure, attributed quotations, neutral tone, length constraints) that LLMs may over-regularize in detectable ways. Student essays, creative fiction, social media posts, or technical documentation have different—and in many cases less standardized—conventions, potentially making both human and AI text more variable and thus harder to discriminate.
Baseline human text quality: The human-written articles in this study are grammatically polished and stylistically consistent. In domains where human text is more error-ridden or informal (social media, messaging, unedited amateur writing), the "grammatical perfection" and "formality" cues that experts rely on (Table 3: 24.8% and 12.3% of explanations, respectively) would be even more diagnostic—human detection might be easier. Conversely, in domains where human text is highly formulaic (legal documents, technical reports, certain types of business writing), the distinction between human and AI formulaicity might blur, making detection harder.
Publication diversity: The eight publications, while representing different sections and audiences, all share a baseline of professional American English journalism. Different publication ecosystems (e.g., British broadsheets, non-native English publications, specialized trade journals) might exhibit different human writing patterns that experts familiar with American journalism would misattribute to AI, or vice versa.

The pilot study on Reddit stories (Appendix C, Table 16) provides suggestive but inconclusive evidence of domain dependence. Nonexperts performed better on stories than articles (62.5% vs. 56.7% average TPR, excluding the expert annotator), and the paper notes that "many of the differences spotted were superficial due to the very informal nature of reddit"—LLMs did not naturally produce the ALL CAPS, abbreviations ("u" for "you"), and R-rated content common on that platform. This suggests that domain mismatch between the LLM's training distribution and the target domain's conventions can create easily detectable signals, but it also implies that detection strategies developed on professional journalism may not transfer to informal domains where the "AI signature" looks different.

The paper provides no evidence about detection performance in other domains, languages, or publication ecosystems. The limitation section acknowledges the language restriction ("Our study is limited to articles in American English") but does not discuss the domain restriction's implications for the generalizability of the taxonomy of clues (Table 3), which is presented as a general detection framework despite being derived entirely from annotations on one text type. A practitioner deploying expert detection in a new domain would need to re-validate the entire taxonomy and potentially re-screen annotators for domain-specific expertise.

The Humanization Adversarial Test Underestimates the True Adversarial Threat

The humanization experiment (Experiment 5, Section 2.5) is the paper's primary evidence that expert detection is robust to evasion tactics that specifically target known detection signals. However, the adversarial methodology has a fundamental limitation that undermines the strength of this conclusion: the humanizer was given only the experts' articulated detection criteria, not access to the experts themselves for iterative feedback.

The consequence is that the paper's claim that "humanization does not completely remove the 'AI signature' from texts" (Section 2.5) is demonstrated only against a one-shot, non-adaptive adversary. In a genuine adversarial setting—for instance, a student trying to evade an instructor known to use expert detection, or a disinformation actor targeting a specific review process—the adversary would have access to the detector's decisions and could iteratively refine their evasion strategy. Each generation-detection cycle would reveal which signals the detector is still picking up on, allowing the adversary to target those remaining signals in the next round. The paper's one-shot humanization prompt cannot adapt to the specific signals that the experts actually used (as opposed to what they said they would use in their guidebook contributions).

The paper's own data demonstrates the gap between articulated and actual detection cues. The guidebook (Table 11) was compiled from experts' lists of clues, yet when humanization specifically targeted everything in the guidebook, experts still maintained 100% majority-vote TPR (Table 2, Experiment 5). This implies that experts detect signals beyond what they can verbalize—and by extension, a one-shot humanizer targeting only verbalized signals will always leave some detectable traces. An iterative adversary who receives feedback on which of their generated articles were detected (and why) could probe for these unverbalized signals and develop countermeasures against them.

The paper's development process for the humanization prompt (Appendix B.2) itself illustrates this point: the authors tried multiple approaches (generate-then-humanize, step-by-step humanization, two-step humanization) and iteratively improved their method based on the first author's qualitative assessment of the outputs. This is an adversarial development process, but it stopped at the point where the first author—not the expert panel—could no longer detect the articles. The experts might have been able to provide feedback that would have further improved the humanizer, closing the loop that a real adversary would exploit.

More concretely, a fully adversarial evaluation would involve:

Iterative rounds: Experts evaluate humanized articles, their explanations are fed back to the humanizer, and the humanizer generates new articles targeting the newly identified signals.
Targeted individual attacks: Since different experts use different cues (Figures 10–14), an adversary could target the specific vulnerabilities of each expert (e.g., generating articles that exploit Annotator 3's over-reliance on formality cues while still fooling the others).
Adaptive majority-vote attacks: An adversary who knows that the detection system uses majority vote could design articles specifically to split the expert panel (convincing two experts while failing to convince three), which the paper's one-shot humanizer does not attempt.

The paper does not acknowledge this limitation of its adversarial methodology. The humanization experiment is presented as strong evidence of robustness (Section 2.5: "Despite our best efforts to generate articles that our experts would find undetectable... the expert majority vote is again perfect"), without discussing how "our best efforts" falls short of what a motivated, feedback-driven adversary could achieve. The paper also does not suggest future work on iterative adversarial evaluation, which would be the natural next step for establishing a detection ceiling under genuine adversarial pressure.

The Small Number of Expert Annotators Prevents Population-Level Generalization

The paper's headline result—99.3% TPR with 0% FPR for the expert majority vote across 300 articles—is based on exactly five annotators who passed a rigorous screening process selecting the top 40% of candidates who met the experience profile. The authors are transparent about this constraint: "While a larger sample size would strengthen population-level claims; prohibitive costs limit us to the current sample size" (Section 2). However, the implications of this constraint for the paper's practical claims are more significant than a simple "more data would be better" caveat.

The consequence is that the paper cannot characterize the distribution of detection ability among frequent LLM users who match the expert profile. The five experts are the upper tail of a screening process, not a random sample from the target population. Key unanswered questions that affect practical deployability include:

What is the expected performance of an unscreened frequent LLM user? If a practitioner posts a job for "LLM-experienced writers to detect AI text" without the paper's two-stage screening, what accuracy should they expect? The paper cannot answer this because it only reports the screened (expert) and unscreened nonexpert (people who don't use LLMs for writing) extremes. The performance of unscreened frequent LLM users—the population from which experts are drawn—is unknown.
How many candidates must be screened to assemble a panel of five experts? The paper's screening process yielded a 40% qualification rate (4 of 10 candidates passed), but this rate is estimated from a small sample and may not generalize. A practitioner needs to budget for screening failures. If the true qualification rate in the broader population is 20% rather than 40%, assembling a five-expert panel requires screening 25 candidates at 65 evaluation units each (1,625 total evaluations just for screening), dramatically changing the cost calculus.
What is the minimum panel size for reliable detection? The paper uses five experts but does not test whether three or four would suffice, or whether seven would be substantially better. The ensemble diversity analysis (Figures 10–14) shows that experts rely on complementary cues, but it does not quantify how much redundancy exists or how performance degrades as panel size decreases. For a practitioner, adding each additional expert costs $865 per five experiments (Appendix A.1), and knowing the marginal benefit of the$ N$th annotator matters for resource allocation.
Are the five experts representative of their population? They are all native English speakers from the US, UK, and South Africa, all hold university degrees, and most work as editors or proofreaders (Table 4). This is a narrow demographic and professional profile. Whether experts with different backgrounds—non-native English speakers who use LLMs for writing, technical writers rather than editors, younger users who grew up with LLMs—would exhibit similar detection patterns is unknown.

The paper acknowledges the cost constraint and the within-subjects design's partial mitigation ("reduces variability from individual differences and requires fewer annotators"; Section 2), but does not discuss how practitioners should interpret the small- $N$ results for their own deployment decisions. The individual-level variation observed even among these five screened experts—Annotator 3's TPR dropping from 86.7% to 0% across experiments; Annotator 2's TPR dropping to 80% on CLAUDE-3.5-SONNET (Table 2)—demonstrates that even within a screened panel, performance is heterogeneous, and the majority vote's near-perfection depends on error independence that may not replicate with a different set of five experts.

The paper also does not address the temporal stability of expert performance. The five experiments were conducted sequentially as models were released (Section 2.4 notes that O1-PRO was released "while we were in the middle of conducting experiments"). If expert detection ability degrades over time as models improve—which the increased difficulty of O1-PRO (lowered confidence, Figure 2) and the single majority-vote failure in Experiment 4 suggest—then a panel assembled today may not maintain its accuracy against next year's models. This is a fundamental unsolved problem for human-in-the-loop detection that the paper's cross-sectional design cannot address.

The Prompt-Based Detector Comparison Is Confounded by Model Capability Differences

Section 4 attempts to determine whether LLMs can be prompted to replicate expert detection by providing them with the same guidebook that experts developed. The finding that GPT-4O with the guidebook achieves only 78% overall TPR—compared to the expert majority vote's 99.3%—is presented as evidence that "current LLMs lack the perceptual machinery to operationalize expert detection criteria" and that the gap reflects a capability deficit rather than a knowledge deficit. However, this conclusion is confounded by the fact that the detector LLMs and the generator LLMs share overlapping architectures and training paradigms, creating a self-detection problem that is fundamentally different from human detection.

The consequence is that the prompt-based detector experiments conflate two distinct questions: (1) Can an LLM detect AI-generated text by applying human-derived criteria? (2) Can an LLM detect text generated by itself or its close relatives? The paper's results show that detector performance is highly sensitive to the specific model version (GPT-4O-2024-11-20 substantially outperforms GPT-4O-2024-08-06 on paraphrased and O1-PRO articles; Table 23), and that CLAUDE-3.5-SONNET "failed to detect content generated by itself" (Appendix E.3), achieving only 43.3% TPR on CLAUDE-generated articles. These version and self-detection effects suggest that something beyond the guidebook's utility is driving the prompt-based results—likely the detector model's internal representations of its own (or related models') output distributions.

Humans do not face a self-detection problem because they are not the same system as the generator. An expert human detecting GPT-4O output is making a cross-system judgment based on learned patterns. GPT-4O detecting GPT-4O output is making a within-system judgment that may be influenced by the model's own generative priors, calibration, and representational similarity between generation and detection pathways. The paper's comparison between human experts and prompt-based detectors therefore compares fundamentally different detection scenarios, and the conclusion that the performance gap "is not a knowledge gap but a capability gap" (Section 4 interpretation) may partly reflect the intrinsic difficulty of self-detection rather than a general limitation of LLMs as detectors.

A cleaner experimental design would use detector LLMs that are not from the same developer as the generator LLMs—for example, using CLAUDE-3.5-SONNET to detect GPT-4O output and vice versa, systematically crossing detector and generator model families. The paper reports some cross-model results (Table 23 shows CLAUDE-3.5-SONNET detecting GPT-4O outputs at 86.7% TPR, substantially better than its 43.3% TPR on its own outputs), but does not structure the main prompt-based detector experiments (Section 4, Table 2 bottom) around this cross-model design. The primary comparison—GPT-4O and O1 detecting GPT-4O, CLAUDE-3.5-SONNET, and O1-PRO outputs—confounds detector model architecture, training data, and self-detection effects.

The paper does not acknowledge this confound. The prompt-based detection results are interpreted as evidence about "LLMs' ability to mimic human expert detectors" (Section 4) without discussing whether the generator-detector model relationship might explain some of the performance patterns. The conclusion that "this simple prompting strategy is competitive with existing detectors on easier configurations but struggles to detect humanized articles" (Abstract, Section 4) is factually supported by the data but may understate prompt-based detection's potential if detector and generator models were systematically decoupled.

The Expert Ensemble Advantage Depends on Error Independence That May Not Generalize

The expert majority vote's near-perfect performance relies on the observation that different experts make errors on different articles, so the ensemble compensates for individual weaknesses. The paper demonstrates this empirically for its specific five-expert panel (Figures 10–14 show different cue usage patterns; Table 2 shows individual TPRs ranging from 59.3% to 97.3% while the majority vote achieves 99.3%). However, the paper does not establish whether this error independence is a natural property of diverse expert panels or an artifact of this particular group's composition.

The consequence is that a practitioner assembling their own expert panel cannot assume that majority voting will produce the same near-perfect results. Error independence requires that experts' mistakes are uncorrelated—that the articles which fool Annotator 1 are different from those which fool Annotator 2. If errors are correlated (e.g., if all experts are fooled by the same subset of especially convincing humanized articles), the majority vote provides no benefit over individual judgments. The paper provides descriptive evidence of cue diversity (the per-annotator heatmaps in Appendix D.3) but does not compute formal measures of error correlation or agreement (e.g., Fleiss' kappa, pairwise correlation of error patterns) that would quantify the degree of independence.

Several factors could produce correlated expert errors in practice:

Shared blind spots from similar training: The five experts all have editing/writing backgrounds and use similar LLMs (predominantly ChatGPT; Table 4). If there is a class of AI-generated text that is particularly convincing to editors—for example, articles that mimic the terse, fact-forward style of wire service copy—all five experts might share this blind spot, producing correlated false negatives.
Capability ceiling effects: The single article misclassified by the majority vote—the O1-PRO "Planet X" article (Experiment 4, Table 14)—demonstrates that some articles genuinely resist expert detection, even in ensemble. If future models produce a larger fraction of such articles, the majority vote's accuracy would degrade sharply, not gradually, because the errors would be correlated (everyone is fooled by the same hard articles).
Practice and fatigue effects over time: The paper does not analyze whether expert error patterns change over the course of each 60-article batch. If all experts experience fatigue toward the end of a batch and become more error-prone on later articles, their errors would be temporally correlated even if they are using different detection strategies.

The paper partially addresses the diversity question by demonstrating that experts focus on different clue categories (Section 3, "Annotators don't always focus on the same clues"), but this is a description of strategy diversity, not a proof of error independence. Two experts could attend to entirely different cues yet still be fooled by the same articles if those articles are strong on both dimensions. For instance, an article that uses realistic names and natural-sounding quotations would fool both Annotator 1 (who focuses on names) and Annotator 5 (who focuses on quotes), even though their strategies are different.

The paper does not suggest any method for ensuring error independence when assembling a new expert panel, such as screening for cue diversity rather than just accuracy, or pre-testing candidate experts on a calibration set to identify complementary strengths. This is a notable gap for a paper whose primary practical recommendation is to use majority voting among expert annotators.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the detection conversation from "Can we build a better classifier?" to "Can we operationalize what expert humans already do?" That is a reframing, not just an incremental improvement. Prior work treated human detection as a monolithic, largely ineffective capability (Brown et al., 2020; Clark et al., 2021) and invested effort primarily in automatic methods—perplexity-based detectors, trained classifiers, and watermarking schemes. This paper demonstrates that the failure of "average" humans masked the existence of a naturally occurring expert population whose untrained majority vote achieves 99.3% overall TPR with 0% FPR across 300 articles, matching the best commercial detector (Pangram Humanizers, also 99.3% TPR) while providing paragraph-length explanations that automatic systems cannot produce. The magnitude of this shift is substantial: it is not that humans are "surprisingly good" at detection, but that a specific, identifiable, and recruitable population is near-perfect in the tested domain, even against adversarial humanization.

The reframing resolves a long-standing contradiction in the literature. Studies that concluded humans cannot reliably detect AI-generated text (Brown et al., 2020; Clark et al., 2021) and studies that found LLMs cannot self-correct reasoning (Huang et al., 2023) were implicitly sampling from the wrong population. The paper's Experiment 1 makes this explicit: within the same batch of 60 articles, nonexperts perform at chance (56.7% TPR, 51.7% FPR) while experts achieve 92.7% average TPR and 4.0% FPR (Table 1). The variable is not general human ability but specific experience with LLM writing patterns. This resolves the apparent contradiction by establishing that previous null results were an artifact of treating "human" as a monolithic category rather than stratifying by LLM familiarity.

The paper also resolves a tension around what makes AI text detectable. Prior work on automatic detection emphasized statistical regularities—probability curvature (Mitchell et al., 2023; Bao et al., 2023), cross-perplexity (Hans et al., 2024), or classifier-learned features (Solaiman et al., 2019; Emi and Spero, 2024). The paper's taxonomy of 16 clue categories (Table 3, expanded in Table 17) demonstrates that expert humans rely on a much richer set of signals, including originality (23.7% of explanations), clarity/flow (19.5%), quotation naturalness (22.3%), and tone (9.3%)—dimensions that are difficult to operationalize as token-level probabilities. The finding that humanization targeting the experts' own articulated vocabulary and sentence-structure criteria fails to fool the majority vote (Experiment 5, 100% TPR) shows that these harder-to-automate dimensions carry genuine detection signal, not just post-hoc rationalization. This shifts the research priority: rather than building detectors that get better at finding the same surface-level features, the goal should be to build detectors that attend to the kinds of features experts use.

The paper also changes the economic framing of detection. The cost analysis—approximately $2 per article for expert human annotation versus commercial API pricing for automatic detectors—establishes that human-in-the-loop detection is not a research curiosity but a viable deployment strategy for high-stakes, low-volume settings. The demonstration that expert performance matches Pangram, a production commercial system, without any training data, model development, or engineering, implies that for applications where explainability and trustworthiness are paramount (academic integrity hearings, legal document verification, high-stakes journalism), the cost of human review may be justifiable even when automatic alternatives exist.

The research directions that become more attractive after this work are those that treat the expert human as an upper bound to be approached, rather than an obstacle to be bypassed. This includes: building detectors that operationalize the expert taxonomy (not just vocabulary, but originality, flow, and quotation quality); developing training protocols to increase the supply of expert annotators; and creating human-AI collaborative detection systems where automatic detectors flag suspicious texts and humans provide final judgment with explanations. The directions that become less attractive are those that invest heavily in marginal improvements to perplexity-based or classifier-based methods without addressing the explainability gap or the specific failure modes on humanized and frontier-model text that the paper documents (Binoculars at 6.7% TPR on humanized O1-PRO; Fast-DetectGPT at 23.3%; RADAR at 0%; Table 2).

Methodologically, the paper establishes a template for detector evaluation that goes beyond TPR/FPR benchmarking. The systematic use of human expert explanations as a diagnostic instrument—coding them into a taxonomy, tracking clue distribution shifts across generation methods (Figure 3), and using the resulting guidebook as both a prompt-based detector input (Section 4) and an adversarial target (Experiment 5)—provides a richer evaluation framework than scalar metrics alone. Future detector papers can adopt this template: report not just whether a detector succeeds or fails, but on which detection dimensions it fails, by comparing its error patterns to the clue categories that expert humans use on the same texts.

Follow-Up Research This Work Enables

Training expert detectors: Can the expert taxonomy be taught to nonexperts to close the performance gap? The paper establishes that LLM writing experience is the discriminating factor between experts and nonexperts (Table 1: 56.7% vs. 92.7% average TPR on the same articles), but it does not test whether explicit training can substitute for experience. A direct experiment would take a cohort of nonexperts (infrequent LLM users), provide them with the guidebook (Table 11), show them annotated examples of detected AI patterns with explanations, and measure their post-training performance on a held-out set of articles from the five experimental configurations. The key question is whether the expert-non-expert gap is primarily a knowledge gap (experts know what to look for, and this knowledge can be transferred) or an experiential gap (experts have developed tacit pattern recognition through thousands of hours of LLM interaction that cannot be compressed into a guidebook). The paper's finding that prompt-based detectors with the guidebook underperform humans (78% vs. 99.3% overall TPR; Table 2) suggests that some of the expertise is tacit, but humans may be better able to internalize and apply the guidebook's criteria than LLMs. A strong follow-up would measure whether trained nonexperts approach expert-level TPR, and if not, which clue categories (from the 16-category taxonomy) they fail to operationalize despite explicit instruction.

Iterative adversarial evaluation: How robust are expert humans to an adversary who receives feedback on which articles were detected? The paper's humanization experiment (Section 2.5) is a one-shot adversarial test: the humanizer receives the experts' articulated detection criteria and generates articles targeting those criteria, but receives no feedback on whether the resulting articles actually fooled the experts. A genuine adversarial evaluation would close this loop. The experiment would involve multiple rounds: experts evaluate humanized articles, their explanations are coded into the taxonomy, the specific clue categories mentioned in correct detections are fed back to the humanizer as new avoidance targets, and the humanizer generates a new batch of articles. This iterates until either the experts' TPR drops below some threshold or the humanizer reaches a performance floor. This would establish the true adversarial detection ceiling—what remains detectable when the adversary has perfect information about what the detector is picking up. The paper's observation that experts detect signals not captured in their own guidebook (e.g., the "titles for named individuals" pattern discovered during Experiment 5) suggests that iterative adversarial probing would continuously surface new detection dimensions, and tracking which dimensions emerge and in what order would characterize the detection signal hierarchy.

Cross-domain validation: Does the expert taxonomy transfer to scientific papers, student essays, and social media posts? The paper's entire analysis is restricted to professional non-fiction articles from eight American publications (Section 2). The taxonomy of 16 clue categories (Table 3) is presented as a general detection framework, but it was derived entirely from annotations on one text type. A systematic cross-domain study would replicate the five-experiment structure (GPT-4O, CLAUDE-3.5-SONNET, paraphrasing, O1-PRO, humanization) in three new domains: (1) scientific abstracts or papers (where human text is highly technical and follows IMRaD conventions), (2) student essays (where human text varies widely in quality and may contain genuine errors), and (3) social media posts (where human text is informal, abbreviated, and platform-specific). For each domain, the study would recruit domain-appropriate experts (scientists who use LLMs for writing, teachers who grade student work, heavy social media users), measure their detection performance, code their explanations into the existing taxonomy, and test whether the same or different clue categories dominate. The paper's pilot data on Reddit stories (Appendix C, Table 16) hints that domain conventions strongly influence detection strategies—nonexperts performed better on stories than articles (62.5% vs. 56.7% TPR) because LLMs' failure to mimic Reddit informality was highly diagnostic. A cross-domain study would clarify whether the taxonomy is universal or domain-specific, and whether expert detection ability transfers across domains or requires domain-specific calibration.

Operationalizing the expert taxonomy in automatic detectors: Can a detector be trained to recognize "originality," "quotation naturalness," and "clarity"? The paper's most important diagnostic finding is that expert humans rely heavily on detection dimensions (originality at 23.7% of explanations, clarity at 19.5%, quotation naturalness at 22.3%; Table 17) that current automatic detectors do not explicitly model. A concrete research direction is to build detectors that target these dimensions. For quotation naturalness, one could fine-tune a model to distinguish real interview quotes from LLM-generated quotes by training on a dataset of attributed quotations from journalism paired with LLM-generated quotations on the same topics. For originality, one could adapt creativity evaluation metrics from the NLG literature (e.g., measures of semantic divergence from expected completions) as detection features. For clarity/flow, one could use discourse coherence metrics or information density measures. The evaluation would test whether detectors incorporating these features improve TPR on humanized O1-PRO articles—the configuration where vocabulary-based detection fails (vocabulary drops to 42.3% of correct explanations; Figure 3) and where current open-source detectors collapse (Binoculars: 6.7%, Fast-DetectGPT: 23.3%, RADAR: 0%; Table 2). A strong result would be an open-source detector that matches Pangram Humanizers' 96.7% TPR on humanized O1-PRO by explicitly modeling the non-vocabulary dimensions that experts use.

The minimum viable expert panel: How many experts are needed, and how should they be selected for error independence? The paper's majority vote uses five experts and achieves near-perfect performance, but it does not establish whether this number is necessary or sufficient. A practical follow-up would systematically vary panel size (1, 2, 3, 4, 5, 7, 9 experts) and measure majority-vote TPR/FPR at each size, using bootstrapping over the existing five-expert annotations to simulate smaller panels and collecting additional annotations to simulate larger ones. The analysis would also characterize the relationship between panel diversity (measured by pairwise disagreement rates or by the entropy of clue category usage across annotators) and ensemble performance: does error independence naturally emerge from recruiting annotators with different professional backgrounds (editors vs. writers vs. teachers), or is explicit cue-diversity screening needed? The paper's per-annotator heatmaps (Figures 10–14) show that the five experts naturally specialize in different clue categories, but this may be an artifact of this particular group rather than a general property. A strong follow-up would provide a practical protocol for panel assembly: screen N candidates on a calibration set, select the subset whose error patterns are maximally decorrelated, and predict the resulting ensemble accuracy.

Longitudinal tracking: Does expert detection ability degrade as models improve, and can it be maintained through deliberate practice? The paper's cross-sectional design captures a single snapshot of expert performance against then-current models. The sequential release of O1-PRO during the experimental timeline (Section 2.4) provides a natural quasi-experiment: experts had no prior exposure to this model, and their confidence dropped (average 4.21 vs. 4.39–4.48 in prior experiments; Figure 2) while one expert's TPR collapsed to 16.7% (Annotator 3; Table 2). This suggests that model improvement can erode detection ability, but the paper cannot distinguish temporary disorientation from permanent capability loss. A longitudinal study would follow a panel of experts over 12–24 months, testing them on each major model release (GPT-5, Claude-4, Gemini-Ultra, etc.) with and without providing feedback on their errors between sessions. The key measurements would be: (1) the rate at which baseline (no-feedback) TPR declines across model generations, (2) whether periodic feedback and calibration sessions arrest this decline, and (3) whether experts who actively use each new model for writing tasks maintain detection ability better than those who do not. This would establish whether expert detection is a stable skill or one that requires continuous updating—with direct implications for the sustainability of human-in-the-loop detection as a deployment strategy.

Practical Applications and Downstream Use Cases

Academic integrity panels for high-stakes plagiarism cases. The paper provides direct evidence that a small panel of LLM-experienced annotators can detect AI-generated text with 99.3% accuracy while providing detailed, quotable explanations for their judgments (Table 2, Section 3). This maps directly onto the needs of academic integrity hearings, where the consequences of false positives (wrongly accusing a student) and false negatives (failing to detect AI-generated work) are both severe, and where explainability is a procedural requirement. A university could maintain a pool of 5–10 trained expert reviewers (faculty who regularly use LLMs for writing, writing center staff, or trained graduate students) who evaluate contested submissions. At approximately $2 per evaluation based on the paper's cost structure, a panel review of a thesis or dissertation would cost roughly$ 10–$20 in annotator time—trivial compared to the stakes. The paper's finding that the majority vote corrects for individual errors (Annotator 3's 0% TPR on humanized O1-PRO does not affect the ensemble; Table 2) means the panel provides built-in redundancy. The explainability requirement—panelists must provide paragraph-length justifications with highlighted evidence, following the paper's annotation protocol (Figure 1)—ensures due process and creates a record that can be reviewed if the decision is appealed.

Journalism verification for high-impact stories. News organizations investigating potentially fabricated or AI-generated articles can deploy expert review as a verification step before publication or before issuing corrections. The paper's domain—non-fiction articles from professional publications—is exactly the setting where this would apply. The finding that expert detection transfers across model families (CLAUDE-3.5-SONNET detected at 100% majority-vote TPR despite most experts having no prior Claude experience; Table 2, Experiment 2) means the panel does not need to know which LLM was used, and the robustness to paraphrasing (Experiment 3, 100% majority-vote TPR) means surface-level rewriting will not fool the review. The cost structure is favorable for this use case: verifying a single high-stakes investigative piece might cost $10–$ 20 for a five-expert review, which is negligible for a newsroom. The explanations provide specific, citable evidence ("the quotes feel fake because every expert speaks the same way"; Table 3) that can be included in editor's notes or corrections, making the verification process transparent to readers.

Training data filtration for LLM fine-tuning. When curating training data for instruction fine-tuning or preference optimization, practitioners need to exclude AI-generated text to avoid model collapse or reward hacking. The paper demonstrates that expert humans can identify AI-generated text even after paraphrasing and humanization—precisely the kinds of text that would be most dangerous as training contamination. A data curation pipeline could route a random sample of candidate training documents to an expert panel for quality control, using the experts' judgments to estimate the contamination rate of the full dataset and to train or calibrate automatic filters. The experts' explanations provide a taxonomy of contamination types (Table 3) that can guide the development of automated filtration rules. At $2 per document, sampling even 1% of a million-document corpus would cost$ 20,000—substantial but potentially justified for high-stakes training runs where data quality directly affects model safety and performance.

Calibration and evaluation of automatic detectors. The paper demonstrates that all open-source detectors and most commercial detectors fail on specific configurations—Binoculars at 6.7% TPR on humanized O1-PRO, GPTZero at 46.7%, RADAR at 0% (Table 2). Detector developers can use expert human annotations as a ground-truth benchmark for evaluating and improving their systems, particularly for the hardest cases (humanized frontier-model output) where automatic methods currently fail. The paper's full dataset of 1,790 annotations with explanations is released, providing a ready-made evaluation set. A detector developer could measure not just overall TPR/FPR but per-clue-category performance: does the detector fail specifically on articles that experts flagged for "originality" or "quotation naturalness"? This diagnostic granularity, enabled by the coded explanations, allows targeted improvement rather than blind hyperparameter tuning. The paper's finding that even the best detector (Pangram Humanizers) generates false positives on some humanized articles (10% FPR on humanized O1-PRO; Table 2) while the expert majority vote does not (0% FPR) provides a concrete accuracy target for detector development.