When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research

🎯 Pitch

Even the best AI models fail catastrophically at catching errors that led to real paper retractions, achieving at most 21% recall while making student-level conceptual mistakes. The same systems being hyped as AI Co-Scientists cannot reliably verify the science they help produce.

1. Executive Summary

This paper introduces Spot (Scientific Paper Error Detection), a multi-modal benchmark for evaluating whether large language models can serve as verifiers that detect errors in scientific manuscripts rather than solely as generative co-authors. Evaluating ten state-of-the-art proprietary and open-source multi-modal LLMs—including o3, GPT-4.1, Gemini 2.5 Pro, Claude 3.7 Sonnet, and Llama 4 Maverick—on 83 published papers containing 91 author-confirmed errors significant enough to prompt errata or retraction, the paper finds that no model surpasses 21.1% recall or 6.1% precision (o3 achieves the best scores, with all others near zero). The benchmark establishes that current LLMs remain fundamentally unreliable for automated academic verification, with models making student-level conceptual mistakes derived from misunderstandings of long-tail domain knowledge, and confidence estimates clustering uniformly near zero across eight independent trials.

2. Context and Motivation

The Core Problem: We Are Building AI Co-Scientists Without AI Verifiers

The fundamental question this paper tackles is whether large language models can reliably detect errors in scientific manuscripts—not as a theoretical exercise in fact-checking, but as a practical capability that would enable automated peer review and quality control in AI-driven research workflows. The authors frame this as a critical infrastructure problem: the research community is racing to deploy LLMs as "AI Co-Scientists" that generate hypotheses, design experiments, and draft manuscripts, but almost no attention has been paid to whether these same systems can perform the equally essential verification step—catching the errors, methodological flaws, and data inconsistencies that, in human science, are caught by peer review, replication, and post-publication scrutiny.

The paper opens by explicitly acknowledging this asymmetry:

"despite their widespread usage as 'generators' in the forward pass of scientific research, their utility in the backward pass of academic verification or as verifiers remains underexplored, a blind spot in which most systems lean on LLM judges without validation on their credibility in reviewing scientific research."

This is not merely a gap in the literature—it represents a genuine risk to the emerging AI-for-science ecosystem. If LLM-based Co-Scientist systems (such as those described in Gottweis et al., 2025; Lu et al., 2024) are generating scientific claims at scale, but the verification mechanisms used to check those claims are themselves unreliable, the entire pipeline becomes unsound. The authors draw a parallel to other domains where verifiability has been foundational to scaling AI: self-supervised learning depends on next-token prediction as a provable training objective (Jernite et al., 2017); instruction-tuning leverages LLM-generated data as reliable signals (Wang et al., 2022); and reinforcement learning uses verifiable rewards for alignment (Guo et al., 2025b). In each case, progress depends on having a trustworthy verification mechanism. The paper argues that AI-driven science currently lacks this foundation.

Why This Problem Matters: Real-World Stakes

The significance of this problem extends beyond academic benchmarking into concrete deployment concerns. The authors motivate the work by pointing to a rapidly expanding landscape of systems that claim to automate scientific work but lack robust error-checking capabilities. They cite specific examples:

Hypothesis generation systems (Si et al., 2024; Park et al., 2024a) that can propose novel research directions at scale, but whose outputs may contain subtle methodological errors or factual inconsistencies that go undetected.
Literature synthesis agents (He et al., 2025) that produce research summaries, but without verified factuality guarantees.
Manuscript drafting systems (Jain and Jain, 2024) that accelerate writing but may propagate errors inherited from their training data or hallucinated during generation.
Full AI Scientist pipelines (Lu et al., 2024) that attempt end-to-end scientific discovery, including code generation and experimental design, yet typically evaluate their own outputs using the same unreliable LLM judges that the authors flag as untested for scientific rigor.

The authors frame this as a missing feedback loop: in human science, the "backward pass" of peer review, replication, and post-publication commentary serves as a quality filter that catches errors before they compound. If AI systems are to play an increasing role in scientific production without human oversight at every step, they need an automated equivalent of this verification capability. Without it, the authors warn, errors go undetected, confidence in AI-generated science erodes, and the entire vision of AI Co-Scientists becomes practically infeasible for high-stakes applications.

The choice of benchmark design—using only errors that prompted real errata or retraction—underscores the practical stakes. These are not minor typos or formatting issues; they are errors severe enough that the original authors formally acknowledged them and, in 32 of 91 cases, retracted their papers entirely. By anchoring the benchmark to these high-consequence mistakes, the authors establish that Spot is testing for errors that matter in practice, not academic edge cases.

Prior Approaches and Where They Fall Short

The paper identifies three broad categories of prior work that appear related to scientific verification but, upon closer inspection, fail to address the actual challenge of detecting consequential errors in full scientific manuscripts.

1. Fact Verification Benchmarks Focus on Everyday Knowledge, Not Frontier Science

The largest body of related work is in fact verification and claim checking. Benchmarks like FEVER (Thorne et al., 2018) and SciFact (Wadden et al., 2020) evaluate whether models can verify claims by checking them against reference documents. The authors identify several critical limitations that make these benchmarks unsuitable for evaluating scientific verification:

Granularity: These benchmarks operate at the sentence level—individual claims extracted from text and verified against a knowledge source. In contrast, detecting errors in scientific manuscripts requires synthesizing information across sections, figures, and equations, often involving multi-step reasoning that connects methodological details in one section to reported results in another.
Domain scope: Prior fact verification work concentrates on everyday knowledge (FEVER focuses on Wikipedia claims) or, when scientific, is limited to biomedical literature (SciFact). This leaves entire fields—mathematics, physics, materials science, chemistry—untested.
Input modality: Existing benchmarks are text-only, but scientific errors frequently manifest in figures, tables, and equations. Spot's inclusion of 18 images per manuscript on average (ranging up to 80) reflects the reality that many critical errors—figure duplications, mislabeled axes, mismatched values between text and figures—require multi-modal understanding.
Error severity: Fact verification benchmarks include claims of varying importance, many of which are trivial or easily resolvable. Spot specifically targets errors that led to errata or retraction, establishing a clear threshold of consequentiality.

2. Automated Peer Review Systems Evaluate Writing Quality, Not Error Detection

A second line of work attempts to generate or evaluate peer reviews using LLMs (Dycke et al., 2022; Baumgärtner et al., 2025; Shin et al., 2025). These systems are evaluated on how well they match human-written reviews, typically via metrics like ROUGE (Zeng et al., 2024). The authors point to a fundamental mismatch in objectives:

"These approaches measure success by matching past reviews via metrics such as ROUGE rather than detecting errors. They also overlook the inherent noise in peer review reports and seldom apply adequate quality control or validate ground truth."

The key insight is that peer review is a noisy, inconsistent process (Cortes and Lawrence, 2021; Bonavia and Marin-Garcia, 2023)—reviewers disagree, miss errors, and focus on different aspects of manuscripts. Training models to reproduce this noisy signal does not guarantee they can reliably identify genuine scientific flaws. Moreover, the peer review datasets used in prior work draw predominantly from computer science publications (Dycke et al., 2022), severely limiting their disciplinary coverage. A model that performs well on these benchmarks may simply have learned the conventions of CS conference reviewing without acquiring the domain-specific expertise needed to catch errors in, say, algebraic geometry proofs or materials science experimental protocols.

The authors also note that prior peer review benchmarks do not validate their ground truth against external evidence. In contrast, Spot's errors are cross-validated through explicit author acknowledgments (PubPeer responses or withdrawal notices) and subsequent human annotation, creating a much higher standard of annotation quality.

3. Computational Reproducibility Benchmarks Test Code Execution, Not Conceptual Errors

A third thread of work, exemplified by CORE-Bench (Siegel et al., 2024), evaluates whether models can reproduce computational results from papers—essentially checking whether the code runs and produces the reported outputs. While valuable, this addresses only a narrow slice of what can go wrong in scientific research. CORE-Bench does not detect errors in mathematical derivations, figure manipulations, statistical reporting errors, or factual inconsistencies that do not manifest in computational reproducibility. Spot complements this line of work by covering error types that fall outside the scope of code-driven verification.

How Spot Positions Itself

The paper positions Spot as filling a specific, previously unoccupied niche: a multi-modal, multi-domain, error-detection benchmark using only author-confirmed, consequential errors in full-length scientific manuscripts. The authors explicitly contrast this against the limitations they identify in prior work:

Unlike fact verification benchmarks, Spot operates on complete papers (averaging ~12,000 tokens) with interleaved text and figures, requiring models to perform cross-modal and cross-sectional reasoning.
Unlike peer review benchmarks, Spot's annotations are validated against objective, externally verifiable evidence (author acknowledgments, retraction notices) rather than derived from noisy reviewer reports. The evaluation metric directly measures error detection accuracy (precision, recall) rather than similarity to human-written reviews.
Unlike reproducibility benchmarks, Spot covers a broad spectrum of error types: equation/proof errors (37 instances), figure duplications (27), data inconsistencies between text and figures (18), statistical reporting errors (4), mislabeled reagents (3), and misreported experimental setups (2). These six categories emerged inductively from the annotation process rather than being imposed a priori, ensuring they reflect the actual distribution of errors found in literature.
Unlike any prior benchmark, Spot spans ten scientific fields (Mathematics, Physics, Biology, Chemistry, Materials Science, Medicine, Environmental Science, Engineering, Computer Science, and Multidisciplinary) with coverage that reflects field-specific error patterns: mathematics and physics papers skew toward equation/proof flaws, while biology papers concentrate on figure duplications (as shown in Figure 2 of the paper).

The paper also makes a deliberate methodological choice to minimize training data contamination: all manuscripts were published in 2024 or later, with only three exceptions (pre-2023 papers retained because their error notices were first posted in March 2024, after most models' training cutoffs). This is explicitly aimed at ensuring that models cannot rely on parametric knowledge of the retraction or the error—they must actually detect the flaw from the paper's content.

Finally, the authors frame Spot as a capability benchmark rather than a comprehensive coverage benchmark. At 83 manuscripts with 91 errors, it is relatively small, but this aligns with a recent trend toward compact, high-quality evaluation sets at the frontier of AI capability: MT-Bench (80 items, Zheng et al., 2023), GPQA-Diamond (198 items, Rein et al., 2024), AIME (30 items per year), and PaperBench (20 items, Starace et al., 2025). The authors explicitly acknowledge size as a limitation (Appendix A) but argue that the quality and cross-validation of the annotations—each error traced to an author admission—compensates for the modest scale, making Spot a reliable signal of model capability even at 83 instances.

Beneath the specific critique of prior benchmarks lies a broader observation about how the AI community evaluates models. The paper notes that state-of-the-art LLMs achieve >80% on established STEM benchmarks like MMLU-Pro, GPQA Diamond, and MMMU, yet "performance drops to roughly 20% on HLE, a curated set of frontier, research-level academic questions, and remains similarly low on Spot (21.1%)." This sharp performance cliff—from mastery of textbook and exam-style problems to near-failure on tasks requiring genuine research-level scientific reasoning—reveals a gap in evaluation practices. Most benchmarks test knowledge recall and straightforward application, not the kind of multi-hop, cross-modal, domain-expert reasoning required to identify an error that the paper's original authors, peer reviewers, and editors all initially missed.

Spot is thus positioned not just as a new dataset, but as a test for a qualitatively different cognitive capability: the ability to read a complex scientific argument skeptically, identify where it breaks down, and articulate why—precisely the skill that human peer reviewers and post-publication commentators exercise. That even the strongest current models (o3 with 18.4% pass@1) fail on this task, despite >80% on graduate-level STEM exams, demonstrates that current benchmarks are not capturing this capability at all. Spot, in the authors' framing, begins to fill that evaluation blind spot.

3. Technical Approach

3.1 Reader Orientation

This is primarily a benchmark and evaluation paper that introduces Spot, a curated collection of 83 scientific papers with 91 confirmed, consequential errors, designed to test whether multi-modal LLMs can serve as verifiers for academic manuscripts. The system being built is not a new model or algorithm, but a rigorous evaluation protocol that measures how well LLMs can read a complete scientific paper—text, figures, equations, and tables—and identify errors that the paper's own authors later acknowledged as warranting correction or retraction. The core idea is grounded in a simple proposition: if LLMs are to be trusted as "AI Co-Scientists" that generate scientific content, they must first demonstrate that they can reliably detect when scientific content is wrong—and the authors design Spot to test exactly this capability under conditions that mirror real-world academic verification.

3.2 Big-Picture Architecture (Diagram in Words)

The Spot benchmark is not a model or algorithm, but a five-stage pipeline for constructing a validated error-detection dataset and a two-phase protocol for evaluating models against it. The components are:

Seed collection: Crawl two external repositories—WithdrarXiv (a dataset of ~14,000 retracted arXiv papers) and PubPeer (an anonymous post-publication peer review website)—to find manuscripts where commenters or authors flag specific, concrete errors.
Automated filtering: Apply two sequential GPT-4o filtering passes to (a) retain only comment–manuscript pairs that unambiguously pinpoint a specific location (section, figure, equation, or table), and (b) remove reports requiring external artifacts (code, datasets, or cross-paper image comparisons). A temporal filter then restricts to papers published after 2024 to minimize training data contamination.
Error validation by original authors: Retain only instances where the original authors explicitly acknowledged the error—either through a PubPeer response or, for WithdrarXiv entries, the withdrawal notice itself. This is the critical quality-control step that distinguishes Spot from peer-review benchmarks that rely on noisy reviewer reports.
Human annotation sanity check: Two mutually exclusive groups of human annotators (including paper co-authors) validate that each flagged issue is (1) self-contained, (2) identifiable from the manuscript alone, and (3) explicitly acknowledged by authors. They retrieve archived PDFs, verify error visibility, document descriptions, and assign error categories and severity ratings.
Normalization: Convert PDFs to interleaved text-and-image multimodal inputs using Llama-Parse for initial OCR, GPT-4.1 for OCR correction, and manual audits to ensure all benchmarked errors remain visible and accurately represented.

The evaluation phase then operates as a separate pipeline: (a) a target LLM receives the full interleaved paper as input and outputs a structured JSON of potential errors with locations and descriptions, and (b) a separate judge LLM (GPT-4.1) aligns predicted errors against ground-truth annotations, classifying each prediction as a true positive, false positive, or false negative. Precision, recall, and pass@K are then computed from these classifications, with bootstrapping over eight independent runs to estimate variance.

3.3 Roadmap for the Deep Dive

First, the evaluation protocol and metrics (precision, recall, pass@K), since these define exactly what the benchmark measures and how success is quantified—the "rules of the game" that all subsequent results depend on.
Second, the data curation pipeline (Stages 1–4), because the benchmark's credibility rests on the quality and cross-validation of its annotations—understanding how errors were sourced, filtered, and verified is essential to interpreting any model's performance.
Third, the normalization pipeline (Stage 5), which transforms raw PDFs into the multimodal input format fed to models—this step has substantial downstream effects on model performance due to OCR errors and figure processing decisions.
Fourth, the error categorization and benchmark statistics, which reveal the distribution of errors across types and scientific domains and contextualize the difficulty of the task.
Fifth, the evaluation pipeline—how models' structured JSON outputs are compared against annotations using a judge LLM, and how true positives, false positives, and false negatives are defined.
Sixth, supplementary analysis configurations (text-only ablation, difficulty estimation, confidence calibration) that extend the core evaluation to probe specific hypotheses about why models fail.

3.4 Detailed, Sentence-Based Technical Breakdown

Evaluation Protocol: Formal Definitions of Precision, Recall, and pass@K

Before diving into how the dataset was built, it is essential to understand exactly what Spot measures and how success is quantified. The evaluation protocol defines the "rules" that all models must follow and establishes the formal framework within which results in Tables 2 and 3 should be interpreted.

A predicted error is counted as a true positive (TP) only when two conditions are simultaneously satisfied: the model's reported location matches a benchmark annotation AND a separate judge LLM (GPT-4.1, described later) confirms that the predicted error and the annotated error indicate the same issue. The paper is explicit that both location AND description must match—a model that identifies the correct equation but attributes the wrong mistake to it is counted as a false positive, not a partial success.

All other predictions are considered false positives (FP). This includes predictions at locations not mentioned in any annotation and predictions at annotated locations where the description does not match the benchmark error. Any benchmark annotation that the model fails to predict at all is a false negative (FN).

The evaluation treats Spot's annotations as exhaustive: any model-reported error not matching an annotation is counted as a false positive. The authors explicitly address this assumption in Appendix A, acknowledging that "the complexity of scientific manuscripts means some true errors may be unannotated," and that through their case studies, they find such cases are "highly unlikely." However, they do note specific circumstances where false positives and false negatives can arise from annotation boundaries rather than model failures—for instance, when an author's withdrawal note identifies a flawed theorem but doesn't enumerate every downstream result it affects, or when an LLM correctly identifies a theorem containing an error but describes the flaw incorrectly.

From these TP/FP/FN counts, two standard metrics are computed:

$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}, \quad \text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$

where TP, FP, and FN are summed across all 91 errors and all model predictions.

What it computes: Precision quantifies the fraction of the model's flagged errors that match benchmark annotations—it answers "when the model says there's an error, how often is it right?" and penalizes hallucinated flags. Recall quantifies the fraction of benchmarked errors the model successfully identifies—it answers "of all the errors that actually exist, how many did the model find?" and penalizes missed detections.

Why these metrics together rather than a single score: Precision and recall capture fundamentally different failure modes that matter differently to different users. High precision with low recall means the model is conservative—when it speaks, it's usually right, but it misses most errors, requiring a human to still find the rest. Low precision with high recall means the model flags almost everything but with many false alarms, imposing a substantial verification burden. The paper explicitly notes that "users concerned about model hallucinations or the impact of unannotated flags should focus on Precision," while "those seeking comprehensive error coverage... should emphasize Recall." This is a pragmatic acknowledgment that no single metric captures the practical utility of an error-detection system.

Beyond single-attempt metrics, the paper also reports pass@K, which captures how error detection improves when the model is given multiple independent attempts:

$\text{pass@}K = \frac{1}{\sum_{i=1}^{N} |G_i|} \sum_{i=1}^{N} \sum_{g \in G_i} \mathbb{1}\left[\exists s \in \{1, \dots, K\} : g \in p_i[s]\right]$

where $N = 83$ is the number of papers, $G_i$ is the set of ground-truth errors in paper $i$ (with $|G_i|$ being the number of errors in that paper, typically 1), $p_i[s]$ is the set of errors predicted by the model in the $s$ -th independent run, and $\mathbb{1}[\cdot]$ is the indicator function that returns 1 if the condition is true and 0 otherwise.

What it computes: pass@K measures the fraction of ground-truth errors that are detected in at least one of $K$ independent attempts. For $K=1$ , it is equivalent to the fraction of errors found on a single attempt (a per-error recall). For $K=4$ , it answers "if I run the model four times on the same paper, what fraction of errors will be flagged at least once?" This captures the effect of stochasticity in LLM generation—a model that sometimes detects an error on some runs but not others may have a higher pass@4 than pass@1, indicating that ensembling multiple attempts could improve recall.

**Why bootstrap: ** The paper generates $N = 8$ independent runs per paper. For each pass@K, it draws $K$ runs without replacement from the eight, repeats this resampling $B = 1000$ times, and reports the mean and standard deviation of the resulting bootstrap distribution. This is done for $K \in \{1, 4\}$ . The bootstrap procedure is necessary because pass@K is a complex statistic that depends on both the correlation structure of predictions across runs (are the same errors found repeatedly, or does each run find different ones?) and the finite sample—the bootstrap provides non-parametric estimates of both the expected value and the variance without assuming normality or independence of model outputs.

The evaluation protocol also specifies practical implementation details: all models are accessed via APIs, each call is retried up to three times (models that still fail or produce truncated outputs due to length limits are marked incorrect), and the structured JSON output format is enforced via prompt instructions (detailed in Appendix F). The generation prompt explicitly asks models to "identify only those errors or flaws that directly undermine the scientific validity of the paper's methods, analyses, or conclusions" and to exclude "issues purely presentational, rhetorical, stylistic, or related to citation practices"—this constrains the task to consequential scientific errors, aligning with the benchmark's focus on errata- and retraction-worthy flaws.

Data Curation: The Five-Stage Pipeline from Raw Sources to Validated Errors

The construction of Spot follows a carefully ordered, multi-stage pipeline designed to produce a small but high-confidence set of error annotations. Each stage applies progressively stricter quality controls, moving from broad automated collection to targeted human validation. Understanding this pipeline is essential because the credibility of every result in the paper depends on the integrity of the ground-truth annotations.

Stage 1: Seed Collection from WithdrarXiv and PubPeer

The pipeline begins with two external repositories chosen for their complementary coverage of acknowledged scientific errors.

WithdrarXiv (Rao et al., 2024) is a dataset of approximately 14,000 papers that were retracted or withdrawn from arXiv, along with the associated retraction comments. The authors extract entries specifically annotated as "factual/methodological/other critical errors," filtering out withdrawals due to formatting, author disputes, or other non-substantive reasons. This source provides high-confidence errors because the withdrawal notice itself serves as an implicit author acknowledgment.

PubPeer is an anonymous post-publication peer review website where users flag methodological flaws, image manipulations, and other scientific concerns in published papers. The authors employ a structured crawling procedure: following the methodology of Ortega (2022), they query PubPeer using alphabetical characters to discover high-frequency keywords in returned paper titles, then re-query using those keywords to broaden coverage. They scrape each paper's metadata (title, authors, venue) alongside the entirety of the comment threads. The initial crawl yields 25,378 PubPeer samples and 1,855 WithdrarXiv samples after the first automated filter (described in Stage 2).

The authors briefly attempted to include medRxiv (a preprint server for health sciences) and bioRxiv (biology), but retrieved only 1 and 13 papers respectively, dropping them "due to the low yield." This is a practical limitation: the pipeline depends on publicly accessible error reports, and preprint servers outside arXiv may not accumulate post-publication commentary at sufficient density.

A critical design choice at this stage is that the authors do NOT themselves judge whether a flagged issue constitutes an error. They delegate this judgment to the paper's own authors (Stage 3). This is a deliberate departure from prior work where benchmark curators decide what counts as an error, introducing potential curator bias or requiring domain expertise beyond what the annotators possess. By deferring to author acknowledgments, Spot's annotations are grounded in the most authoritative source available: the researchers who conducted the work and who have institutional and reputational incentives to dispute unfair criticisms.

Stage 2: Automated Filtering with GPT-4o

The raw seed collection contains many entries that are unsuitable for a benchmark requiring self-contained, textually identifiable errors. Two sequential GPT-4o filtering passes (using GPT-4o from OpenAI et al., 2024) narrow the pool.

First filtering pass: Retain only comment–manuscript pairs that "unambiguously pinpoint a specific section, figure, equation, or table." This eliminates vague critiques ("the methodology seems questionable") in favor of concrete, localizable issues. This reduces the pool to 1,855 WithdrarXiv and 25,378 PubPeer samples.

Second filtering pass: Remove reports that require external artifacts for verification—specifically, duplicated images across papers (where the error requires comparing figures from different publications), errors detectable only via external datasets (e.g., discrepancies between reported results and an external database), or errors requiring code execution (e.g., computational reproducibility issues that depend on running the authors' scripts). This constraint ensures that an LLM with access to only the paper itself has all the information needed to detect the error, making the benchmark a fair test of document comprehension rather than of the model's ability to search external databases.

Temporal filter: To minimize contamination with parametric knowledge, the authors apply a publication date filter, retaining only papers published from 2024 onward. The rationale, citing Bejan et al. (2023), is that models trained on data with a known cutoff date may have memorized retraction notices or PubPeer discussions for older papers, allowing them to "detect" errors by recalling the discussion rather than by identifying the flaw in the paper itself. The 2024 cutoff is chosen because most evaluated models have training data cutoffs in 2023 or early 2024, making it unlikely that 2024 publications (and their associated error discussions) appear in training corpora.

This stage yields 58 WithdrarXiv and 215 PubPeer samples.

The paper explicitly notes that three pre-2023 papers passed the automated filters because revised versions were submitted after 2024. These were retained because their first error notices appeared in March 2024, "minimizing any chance that models were exposed to the original withdrawal details during training." This exception is documented in Appendix E (Figure 19), which plots publication dates against first error-notice dates for all 83 manuscripts, showing the bulk dating to 2024 with ten papers from 2025.

Stage 3: Error Validation by Original Authors

This stage implements the central quality-control mechanism that distinguishes Spot from peer-review-based benchmarks. The authors explicitly state:

"we only retain PubPeer comments followed by an explicit author response acknowledging the mistake and treat WithdrarXiv self-retractions as definitive evidence of a critical error. In all cases where the author themselves admits the problem, we take this acknowledgment as confirmation of a genuine error."

The reasoning is that scientific errors can be contested—what one commenter flags as a mistake may be a legitimate methodological choice that is well-justified in the field's conventions. Rather than becoming arbiters of scientific correctness, the authors let the original researchers serve as the ultimate authority. A PubPeer comment without an author response, or where the author disputes the criticism, is excluded regardless of how convincing the comment appears. A self-retraction (WithdrarXiv) is treated as the strongest possible signal: the authors themselves have decided the error is severe enough to withdraw the paper entirely.

The paper notes a crucial caveat: "some errors may appear to be evident, we do not include any error with[out] explicit acknowledgment from the original authors, as many of the work cover ungoing [sic] areas of research, which remain unsettled in the scientific discourse." This acknowledges that in active research areas, what looks like an error may actually reflect genuine scientific disagreement rather than a clear mistake. By restricting to acknowledged errors, Spot avoids encoding the authors' or annotators' opinions about unsettled questions into the benchmark.

This stage produces the final pool of 83 manuscripts with 91 errors. The relatively small attrition from Stage 2 (273 manuscripts down to 83 with confirmed errors, a ~70% reduction) reflects the stringency of requiring explicit author acknowledgment—many flagged issues go unaddressed, are disputed, or are acknowledged only informally without a formal response.

Stage 4: Sanity Check from Human Annotators

Even with author acknowledgments, additional validation is necessary to ensure the errors are suitable for a benchmark. The authors implement "a two-stage human validation with mutually exclusive annotators," described in Section 2.1.

First annotation group: Part of the paper's co-authors serve as human annotators. They validate that remaining flagged issues fulfill three conditions:

Self-contained: The error must be detectable from the manuscript alone, without requiring external information, code, or datasets. This aligns with the filtering in Stage 2 and ensures the task is a pure document-comprehension challenge.
Identifiable: The error must have a clear, specific manifestation in the text, figures, equations, or tables—a reader with sufficient expertise should be able to locate it given sufficient time and attention.
Explicitly acknowledged by original authors: This reinforces Stage 3 verification, ensuring no borderline cases slip through.

For papers that satisfy these conditions, the annotators retrieve the archived PDF version to verify that the error remains visible (not corrected in a later version), document a concise textual description of the problem, quote the author's acknowledgment verbatim, and assign both an error category (from the six inductively-derived types) and a severity rating—proxied by the form of the author's response: erratum (correction) versus retraction (withdrawal). This severity proxy provides a rough measure of error consequentiality that proves useful for analyzing whether models perform differently on more severe errors.

Second annotation group: A separate group conducts "a comprehensive audit of all annotations to ensure consistent application of these standards." This inter-annotator validation, where the second group reviews the first group's work without overlap, is designed to catch inconsistent labeling, borderline cases, or errors in the annotation process itself.

The paper does not report formal inter-annotator agreement metrics (e.g., Cohen's kappa), which is a notable omission given the centrality of annotation quality to the benchmark's claims. However, the use of author acknowledgments as the anchoring ground truth reduces the subjectivity that inter-annotator metrics typically measure—the primary judgment annotators make is whether the acknowledgment corresponds to the flagged issue, not whether the issue itself is an error.

Stage 5: Normalization from Raw PDFs to Interleaved Multimodal Inputs

The raw manuscripts exist as PDFs, but the evaluation requires a format that models can process consistently. The normalization stage transforms each PDF into an interleaved sequence of text and images that preserves the content and structure of the original while making it accessible to multi-modal LLMs with text-and-image input interfaces.

Initial conversion: The authors employ Llama-Parse (a commercial PDF parsing tool) to convert each PDF into Markdown format while simultaneously capturing high-fidelity screenshots of every figure, table, and equation. Unlike prior benchmarks that fed raw PDFs directly to models (relying on the model's built-in OCR) or that extracted only text (discarding figures entirely), this approach gives models access to both structured text and high-quality visual renderings of non-textual elements.

The motivation for this choice is explicit:

"Prior benchmarks in manuscript error detection and AI-assisted science have relied on raw PDFs or text-only inputs, this approach offloads document understanding to OCR and parsing modules rather than the LLM itself, thereby conflating upstream parser failures with downstream model errors."

By performing OCR and figure extraction as a preprocessing step, the authors aim to isolate model capability from parsing failures—a model should not be penalized because the parser garbled an equation, nor credited for detecting an error that was actually introduced by the parser itself.

OCR correction with GPT-4.1: Pilot experiments revealed that OCR failures, particularly in mathematical expressions, caused downstream models to misinterpret formatting artifacts as genuine errors. For example, a garbled equation in the OCR output might be flagged by a model as containing a mistake, when in fact the mistake was an OCR artifact not present in the original paper. To address this, the authors introduce a refinement stage: for each page, they send the initial OCR text AND screenshots (one full-page image plus cropped images of isolated equations and paragraphs, averaging roughly eight images per page) to GPT-4.1, which corrects OCR errors by referencing the visual renderings.

This creates an interesting methodological dependency: the benchmark's input quality depends on one model (GPT-4.1) correcting another model's (Llama-Parse's) output, creating a potential for GPT-4.1 to introduce its own errors or biases. The authors mitigate this through the final manual audit.

Manual audit: A final human review of all processed pages ensures that "every flagged error remains visible and accurately represented in the OCR output." This is essential because the OCR correction step (GPT-4.1) must not accidentally "fix" the benchmarked errors themselves—for instance, if a paper contains a typographical error in a formula, the OCR correction must preserve that error rather than silently correcting it to the standard form. The manual audit serves as a final check that the normalization process does not tamper with the ground-truth errors.

The output of Stage 5 is a dataset of 83 papers, each represented as an interleaved sequence of text segments (Markdown from corrected OCR) and image segments (screenshots of figures, tables, and key equations), averaging 12,887 tokens (standard deviation 7,421, range 1,207–46,441) and 17.5 images (standard deviation 20.1, range 0–80) per manuscript. Token counts are computed using the GPT-4o tokenizer from tiktoken (OpenAI, 2025b).

Error Categorization and Benchmark Statistics

Rather than imposing a predefined taxonomy, the authors derive error categories inductively from the annotations. As they review each error during Stage 4, they group similar cases, allowing categories to emerge from the data. The final taxonomy comprises six categories (with instance counts in parentheses, totaling 91):

Equation/Proof (37 instances): Incorrect mathematical derivations, flawed theorems, gaps in proofs, misapplied formulas, or algebraic errors. This is the largest single category, concentrated in mathematics, physics, and computer science papers.
Figure Duplication (27 instances): Reused or manipulated images, including duplicated panels presented as distinct experiments, image splicing, or inappropriate reuse of figures across contexts. Concentrated in biology papers.
Data Inconsistency (18 instances): Mismatched values between text, tables, and figures—for example, a reported value in the text disagrees with the corresponding entry in a table, or a plotted data point does not match the stated numerical result.
Statistical Reporting (4 instances): Misused statistical values, inappropriate statistical tests, or misreported p-values, confidence intervals, or effect sizes.
Reagent Identity (3 instances): Mislabeled or incorrect materials in experimental sections, typically in chemistry or materials science papers where precise chemical identities are critical.
Experiment Setup (2 instances): Missing controls, misreported protocols, or experimental design flaws that undermine the validity of reported results.

The paper notes that "figure-duplication instances initially overwhelmed the dataset, so we filtered based on severity and paper category to prevent a single type from dominating." Figure 2 in the paper visualizes the distribution of annotated errors across both research domains and error types, revealing clear field-specific patterns: mathematics, computer science, and physics papers skew heavily toward equation/proof flaws; biology papers concentrate on figure duplications.

Severity proxy: The authors proxy error severity by the authors' post-publication response. Of the 91 errors, 59 were addressed via errata (corrections published alongside the original paper), while 32 led to full retractions. Retractions are concentrated mostly in the equation/proof category, consistent with the centrality of mathematical correctness to the core claims of papers in those fields. This severity distinction is used implicitly in the analysis—for instance, in the case studies (Section 4), the authors note which errors led to retraction versus erratum—but it is not systematically used as a stratification variable in the main results tables.

Paper-level statistics: 76 manuscripts out of 83 contain a single annotated error. Six contain two errors, and one paper contains three errors. This distribution means that most papers contribute a single error to the benchmark, simplifying the evaluation (a model must find one needle in a haystack of ~12,000 tokens and ~18 images) but also making the benchmark statistically noisier—each paper contributes at most 1–3 data points, and missing a single error has a large impact on per-paper recall.

Length and modality statistics: Manuscripts span 1,207 to 46,441 tokens and include 0 to 80 figures. The paper reports a weak correlation between paper length and figure count (Pearson's $r = 0.19$ ), "highlighting diverse presentation styles across fields"—some long papers are text-heavy with few figures, while others are figure-rich with relatively concise text. This diversity is important for a general benchmark because it tests model robustness across presentation formats rather than optimizing for a single style.

Domain coverage: Papers are classified into ten research domains—Mathematics, Physics, Biology, Chemistry, Materials Science, Medicine, Environmental Science, Engineering, Computer Science, and Multidisciplinary—based on journal venue or arXiv subject classification. The Multidisciplinary category captures papers published in general-science venues like Nature or Science that do not fit neatly into a single field.

The Evaluation Pipeline: From Model Output to Metric

Once the benchmark dataset is constructed, evaluation proceeds in two phases. Understanding this pipeline is essential for interpreting the results in Tables 2 and 3—the evaluation protocol, including the use of a judge LLM to compare predictions against annotations, introduces its own potential sources of error that must be understood alongside the model errors themselves.

Phase 1: Model Generation

Each target LLM receives the full paper as input, formatted as interleaved text and image data—the output of Stage 5 normalization. The model is prompted with the "Generation Prompt" (Appendix F), which provides specific instructions:

"You are a scientific-rigor auditor. You will receive the parsed contents of a research paper. Your job is to identify only those errors or flaws that directly undermine the scientific validity of the paper's methods, analyses, or conclusions. Your sole focus is identifying flaws, such as errors in experimental design, data integrity, calculations, statistical inference, or reproducibility, that directly call into question the validity of a specific claim, paragraph, or the paper. Do not report issues purely presentational, rhetorical, stylistic, or related to citation practices."

The prompt then instructs the model to output a structured JSON object containing an "analysis" field (a detailed walk-through of how the model checked each section and figure) and a "response" field with an "errors" array, where each error has a "location" (e.g., "Section 2.1," "Figure 3") and a "description" that explains the scientific flaw.

The structured output requirement serves several purposes. First, it enables automated evaluation by providing a predictable format that a separate judge LLM can parse consistently. Second, the requirement to specify a location forces the model to localize errors precisely rather than making vague critiques, which makes TP/FP classification more objective. Third, the "analysis" field provides interpretability—the authors use these analyses in the case studies (Section 4) to diagnose model failures.

Generation configuration: The authors adopt each model provider's recommended parameters where available. When provider recommendations are absent, they use a sampling temperature of 0.6, top-p of 0.95, a repetition penalty of 1.0, and enforce a minimum of 8 and a maximum of 8192 tokens. The temperature of 0.6 represents a moderate level of stochasticity—high enough to produce diverse outputs across runs (enabling pass@K analysis) but low enough that outputs are not dominated by randomness. Each call is retried up to three times; models that still fail or produce truncated outputs due to length limits are marked as incorrect (contributing to false negatives).

For the main evaluation, each model is run $N = 8$ independent times on each paper, generating 8 sets of predicted errors per paper. The paper does not specify whether these 8 runs use different random seeds, but the bootstrapping procedure for pass@K assumes they are independent and identically distributed draws from the model's output distribution.

Phase 2: Judging with GPT-4.1

The raw model outputs (JSON arrays of predicted errors with locations and descriptions) are not directly compared to ground-truth annotations by simple string matching. Instead, the authors employ a separate "Evaluation Prompt" (also in Appendix F) sent to GPT-4.1, which functions as an LLM judge:

"You are an expert LLM-as-a-Judge. You will receive a JSON object with two arrays: 'annotations': the ground-truth errors (each has 'location' and 'description'); 'predictions': the model's reported errors (same format). Task: Compare each prediction against each annotation. A match occurs only when both 'location' and 'description' are identical."

The judge is instructed to be "rigorous in considering matches; the location may be slightly differently named, but the description must match overall."

The use of an LLM judge for evaluation introduces a methodological consideration: the evaluation itself depends on the reliability of yet another LLM (GPT-4.1) to correctly identify matches between predictions and annotations. If GPT-4.1 makes errors in this matching—for instance, declaring two descriptions as non-matching when they actually describe the same error from different perspectives, or declaring a match when the descriptions are superficially similar but refer to different underlying issues—the precision and recall metrics will be affected. The authors do not report the accuracy of GPT-4.1 as a judge on this task (e.g., by comparing its judgments against human judgments on a subset of predictions), which is a methodological gap. However, the structured nature of the task—matching location and description pairs—is substantially more constrained than open-ended quality judgments, and the paper's main conclusion (models perform poorly) is unlikely to be sensitive to judge errors at the margin.

The output of Phase 2 is a set of match judgments (TP, FP, FN) that are then aggregated across all errors, papers, and runs to compute precision, recall, and pass@K as described in the evaluation protocol section above.

Supplementary Analysis Configurations

Beyond the main evaluation, the paper includes several supplementary analyses that require separate experimental configurations.

Text-Only Ablation (Section 3.3)

To isolate the impact of multi-modality—specifically, whether including figures helps or hurts model performance—the authors create a text-only subset of Spot. They remove all instances from the figure-duplication category and any data-inconsistency instances that require figures for comprehension (e.g., a discrepancy between a plotted value and a reported value that requires seeing the plot to detect). This yields 48 instances where errors can be detected using text alone.

For this ablation, models are evaluated on these 48 instances under two conditions: multimodal (the full interleaved text-and-image input, extracted from the main evaluation runs) and text-only (the same input with all images stripped out). The text-only condition also enables evaluating three additional unimodal LLMs that cannot process images: DeepSeek-R1, DeepSeek-V3, and Qwen3-235B-A22B. The comparison between multimodal and text-only performance on the same 48 instances reveals whether figures serve as helpful context (improving performance) or as distractors (reducing performance).

Confidence Calibration (Section 3.2)

To assess whether models know when they are likely correct, the paper estimates per-error confidence and compares it against actual pass@4 performance. The confidence estimation procedure is described in Appendix C.

Given $n = 8$ independent runs, let $c_{i,g}$ be the number of runs in which ground-truth error $g$ in paper $i$ is detected. The probability that all $K$ fresh attempts would miss $g$ is computed as the hypergeometric probability of drawing $K$ runs from a pool of $n$ runs where $c_{i,g}$ are "successes" and $n - c_{i,g}$ are "failures," and all $K$ draws are from the failure set:

$\hat{p}_{i,g} = 1 - \frac{\binom{n - c_{i,g}}{K}}{\binom{n}{K}}$

where $\binom{n}{K}$ is the binomial coefficient, "n choose K."

What it computes: $\hat{p}_{i,g}$ is the unbiased estimator for the pass@K probability of error $g$ —the probability that at least one of $K$ new attempts would detect $g$ , given the observed detection frequency $c_{i,g}/n$ across the eight runs. If an error is detected in 4 out of 8 runs ( $c_{i,g} = 4$ ), the estimator calculates the probability that a new set of $K$ runs drawn without replacement from the 8 would contain at least one success. For $K=4$ , this probability approaches 1 if $c_{i,g}$ is high (the error is consistently detected) and approaches 0 if $c_{i,g}$ is low.

Why this estimator: The unbiased estimator from Chen et al. (2021) for pass@K is used rather than the naive fraction $c_{i,g}/n$ because it accounts for the finite sample of runs. The naive fraction would underestimate pass@K for consistently detected errors (if $c_{i,g} = 8$ , the naive fraction suggests pass@1 = 100%, but the unbiased estimator correctly notes that with a finite pool of runs, the true pass@1 might be slightly less than 1).

The per-error confidences are then aggregated into an overall self-estimated confidence for each model:

$\text{Confidence} = \frac{1}{\sum_{i=1}^{N} |G_i|} \sum_{i=1}^{N} \sum_{g \in G_i} \hat{p}_{i,g}$

This is simply the average of the per-error pass@K estimators across all 91 errors, interpreted as the model's estimated probability of detecting a randomly selected error in $K$ attempts. The paper compares this aggregated confidence against actual pass@K performance (for $K=4$ ) to assess calibration, both overall and per error category.

Test-Time Scaling Analysis (Appendix B.2)

For the test-time scaling analysis, the authors use OpenAI's o4-mini series and vary the "reasoning effort" parameter across three settings: low, medium, and high. This analysis, conducted across 3 independent trials, measures whether allocating more inference computation (the reasoning budget that OpenAI's models use for chain-of-thought) linearly improves error detection performance, as it has been shown to do on standard STEM benchmarks (Muennighoff et al., 2025; Son et al., 2025). The results are presented in Figure 10, showing approximately linear improvement with increased reasoning effort, consistent with the pattern observed in Table 2 where reasoning-enabled models (o3, Gemini-2.5-Pro, Claude-3.7-Sonnet:Thinking) outperform their non-reasoning counterparts.

Context-Length Ablation (Appendix B.1)

To disentangle the effect of long-context processing from core error-detection ability, the authors extract the single page containing each ground-truth error and rerun the detection prompt on this shorter segment for a subset of 36 instances. They exclude the Equation/Proof category because "mathematical papers often rely on global notation and prior results, making single sections insufficient," and omit errors that span multiple sections. The difference $\Delta =$ (segment-only performance) $-$ (full-paper performance) is reported for precision and recall across five models (Figure 9), measuring how much long-context processing masks a model's true error-detection ability. A positive $\Delta$ means the model performs better when given only the relevant page, suggesting that information from other parts of the paper acted as a distractor or that the model's attention mechanism diluted its focus.

Design Choices and Their Justifications

Several non-obvious design choices in the benchmark construction warrant explicit justification:

Why author-confirmed errors rather than curator-identified errors: The central challenge in building an error-detection benchmark is establishing ground truth. Scientific manuscripts can contain contested claims—what appears to be an error to one reader may be a legitimate methodological choice that the field's experts endorse. By requiring explicit author acknowledgment, the benchmark outsources the correctness judgment to the most authoritative source (the researchers themselves) and avoids encoding curatorial bias into the annotations. The trade-off is that the benchmark cannot include errors that the authors refuse to acknowledge or that were caught by peer review and corrected before publication. This biases the benchmark toward errors that survived peer review and were only caught post-publication—a specific, practically important class of errors, but not necessarily representative of all errors that occur in scientific manuscripts.

Why filtering for self-contained, single-paper errors: The requirement that errors be detectable from the manuscript alone is essential for a fair evaluation of LLM comprehension. If an error requires comparing a figure to an external database of known image manipulations, the task becomes a test of the model's ability to access and integrate external information, which is a different capability than document understanding. The authors explicitly exclude errors that span multiple papers, require code execution, or depend on external datasets—ensuring that an ideal reader with perfect comprehension of the paper's content could, in principle, detect every error in the benchmark.

Why post-2024 papers only: The contamination concern is well-founded in the LLM evaluation literature. Models trained on web-scale data may have memorized discussions of specific papers' errors, especially high-profile retractions that receive substantial coverage. The 2024 cutoff is a heuristic rather than a guarantee—some models may have been trained on data that includes early-2024 publications—but it substantially reduces the probability of contamination relative to using older, well-known retractions.

Why GPT-4.1 as the OCR corrector and judge: The authors use GPT-4.1 for both OCR correction (Stage 5) and evaluation judging (Phase 2), creating a potential circularity where the same model family is used for both benchmark construction and evaluation. The authors do not explicitly address this choice, but the practical motivation is clear: GPT-4.1 is a capable multimodal model that can reference visual renderings when correcting OCR output, and using a consistent, capable judge for evaluation standardizes the TP/FP classification. The fact that the evaluated models include GPT-4.1 itself (which achieves 6.0% recall—far from ceiling) suggests that any advantage from using a model from the same family as both constructor and evaluator is negligible in practice.

Why the relatively small size (83 papers, 91 errors): The authors explicitly invoke the precedent set by other high-quality, compact benchmarks (MT-Bench's 80 items, GPQA-Diamond's 198, PaperBench's 20) and argue that annotation quality compensates for modest scale. Each error in Spot represents hours of work across crawling, filtering, author verification, human annotation, and normalization—scaling to thousands of instances while maintaining the same quality standards would be practically infeasible without access to the original authors of every paper. The trade-off is that statistical power is limited for per-category or per-domain analysis; many of the breakdown categories in the detailed results tables (Appendix G) contain only a handful of instances, and the ± standard deviations on those breakdowns are correspondingly large.

4. Key Insights and Innovations

Innovation 1: Scientific Verification as a Distinct, Neglected Capability — and a Benchmark That Tests It Directly

The paper's most fundamental intellectual contribution is not a new model or method, but a reframing of what it means to evaluate AI for scientific work. The dominant narrative around "AI Co-Scientists" (Gottweis et al., 2025; Lu et al., 2024; Si et al., 2024) has focused almost exclusively on generative capabilities: can an LLM propose a novel hypothesis? Write a manuscript? Design an experiment? The field has implicitly assumed that if models can generate, they can also verify — or at least that verification is a simpler, downstream capability that will follow naturally from generative prowess.

Spot challenges this assumption by testing verification directly and finding that the best models, which score >80% on graduate-level STEM exams (MMLU-Pro, GPQA-Diamond, MMMU), collapse to near-zero on error detection. The paper's framing of verification as the "backward pass" of scientific research — the quality-control step that in human science is performed by peer review, replication, and post-publication scrutiny — is not merely a rhetorical device. It identifies a missing feedback loop in the AI-for-science pipeline. The authors explicitly argue:

"robust scientific verification must underpin reliable LLM-driven scientific research"

This is a conceptual claim about the architecture of automated science, not an empirical observation about model performance. It implies that scaling AI Co-Scientists without scaling AI Verifiers is inherently unsafe: the faster we generate scientific claims, the more critical the verification bottleneck becomes. The parallel the authors draw to other domains where verifiability enabled scaling — self-supervised learning's next-token prediction, RLHF's reward signals — is not just analogy but diagnosis: AI-driven science currently lacks its equivalent of a provable training objective, and Spot is offered as a first step toward measuring whether we can build one.

What makes this distinctive relative to prior work: Prior benchmarks in fact verification (FEVER, SciFact) and automated peer review (Dycke et al., 2022; Baumgärtner et al., 2025) tested related capabilities but framed them as language understanding tasks rather than as tests of scientific judgment. FEVER asks whether a claim is supported by a reference document — a task that reduces scientific verification to fact-checking. Automated peer review benchmarks ask models to reproduce noisy human-reviewer text — a task that confounds error detection with writing style, reviewer personality, and review format conventions. Spot reframes the problem as error detection under realistic conditions: given the full complexity of a published scientific manuscript, can you find the mistake that the original authors and reviewers all missed? This is a qualitatively different capability than fact-checking or review-mimicry, and the performance collapse on Spot (o3: 18.4% pass@1, all others near zero) demonstrates that existing benchmarks were not testing it at all.

The significance of this reframing extends beyond evaluation. If verification proves to be a fundamentally harder capability than generation — as Spot's results suggest — then the roadmap for AI Co-Scientists must include dedicated investment in verifier development, not just better generators. This is not an incremental finding; it is a direction-setting claim about research priorities.

Evidence: Figure 3 plots o3 and Llama-4-Maverick across six benchmarks, showing that the performance gap between the two models is widest on Spot (Δ = 20.2pp) and that Spot is the only benchmark where Llama-4-Maverick's score collapses to near zero (0.9%). This demonstrates that Spot measures something fundamentally different from existing STEM benchmarks — a capability that even strong general-purpose models have not acquired.

Innovation 2: Author-Confirmed Errors as an Objective Ground-Truth Anchor in a Domain Where "Correctness" Is Contested

The second key innovation is methodological rather than scientific: the paper's approach to ground-truth construction solves a problem that has plagued every prior attempt to build scientific verification benchmarks — who decides what counts as an error?

In prior work, benchmark curators made this determination. For fact verification, curators check claims against reference documents (Thorne et al., 2018). For peer review benchmarks, curators treat human-written reviews as ground truth, despite well-documented evidence that peer review is noisy, inconsistent, and frequently wrong (Cortes and Lawrence, 2021; Bonavia and Marin-Garcia, 2023). Both approaches embed the curators' judgments — and their potential biases, domain limitations, and blind spots — into the benchmark itself. In active areas of research, where experts genuinely disagree about what constitutes a methodological flaw versus a legitimate choice, the curator-as-arbiter model becomes fundamentally unreliable.

Spot sidesteps this problem entirely by delegating the error judgment to the original authors. The benchmark includes only errors where the authors themselves have publicly acknowledged the mistake — either through a PubPeer response or through a formal withdrawal notice. The paper is explicit about the rationale:

"some errors may appear to be evident, we do not include any error with[out] explicit acknowledgment from the original authors, as many of the work cover ungoing [sic] areas of research, which remain unsettled in the scientific discourse"

This is a subtle but powerful epistemic move. It transforms the benchmark's ground truth from a curator's opinion (subject to error, bias, and domain ignorance) into an objective, verifiable, and publicly documented social fact. Author acknowledgments are not just "probably correct" in the way a curator's judgment might be; they are institutionally binding — the authors have staked their professional reputations on the admission, making false acknowledgments extremely costly and genuine ones an extremely reliable signal.

The significance of this choice extends beyond annotation quality. It means that Spot's errors represent a specific, practically important class of mistakes: errors that survived peer review (since these are published papers), were significant enough to warrant public correction or retraction (since they prompted author acknowledgment), and were acknowledged under conditions of professional accountability (since the authors' names are attached). This is precisely the class of errors that an automated verification system would need to catch in practice — not minor typos or formatting issues that a copy editor could fix, but substantive flaws that compromise scientific validity and that human reviewers missed.

The trade-off is that this standard eliminates errors that were caught during peer review (and thus never made it into published literature), errors that authors refuse to acknowledge, and errors in fields where post-publication commentary infrastructure (like PubPeer) is less active. The benchmark is therefore biased toward post-peer-review, publicly-acknowledged errors — a specific but well-defined distribution. Future work might relax the author-acknowledgment requirement for domains or error types where it creates systematic gaps, but for an initial benchmark establishing that the verification problem is hard, the conservative standard is the right choice.

Evidence: The data curation pipeline (Section 2.1, Stages 3–4) explicitly describes the author-acknowledgment requirement and the two-stage human validation that ensures annotations are consistent with those acknowledgments. Of 273 papers that passed automated filtering (Stage 2), only 83 had errors with explicit author confirmation — a ~70% reduction that underscores how many flagged issues in the scientific literature remain unacknowledged, disputed, or ambiguous, and why a curator-decides approach would have introduced substantial annotation noise.

Innovation 3: The Figure-as-Distractor Finding — Multi-Modality Can Hinder Rather Than Help Scientific Error Detection

One of the paper's most counterintuitive and potentially impactful findings emerges from the text-only ablation in Section 3.3: for most models, removing figures from the input improves error-detection performance. Table 3 shows that, on the 48 text-only-solvable instances, the majority of models achieve higher recall and pass@4 when figures are stripped out. For example, GPT-4.1's recall rises from 0.5% (multimodal) to 8.4% (text-only), and Claude-3.7-Sonnet:Thinking rises from 2.9% to 5.0%.

This is not what the multi-modal LLM narrative would predict. The standard assumption — implicit in the development of models like GPT-4V, Gemini, and Llama 4 — is that adding visual information helps: models can "see" what the text describes, cross-reference figures against captions, and catch errors that text alone cannot surface. Spot's results suggest the opposite: in the specific context of scientific error detection, figures act as distractors that degrade performance, possibly by adding noise to the model's attention mechanism, by introducing visual complexity that the model misinterprets, or by creating opportunities for spurious correlations (e.g., flagging formatting artifacts as errors, as seen in the case studies).

The exceptions are instructive: o3 and Gemini-2.5-Pro show modest drops in performance when figures are removed (o3: recall drops from 34.6% to 25.7%), indicating that these models do leverage figures productively — they are not just ignoring or being confused by them. However, even for these strongest models, the performance remains far from ceiling, suggesting that genuine multi-modal scientific reasoning — where visual and textual information are integrated to identify inconsistencies — remains an open problem.

Why this finding matters beyond the specific numbers: It reveals a fundamental tension in multi-modal model design for scientific applications. Current multi-modal LLMs are trained to perform well on benchmarks like MMMU and MathVista, which test whether models can answer questions about images (charts, diagrams, photographs). But these benchmarks reward extracting information from images, not skeptically evaluating whether visual information is consistent with textual claims. Spot's error-detection task requires the latter: the model must compare a figure against the paper's description of what that figure shows, identify mismatches, and articulate why the visual evidence contradicts the textual claim. This is a fundamentally different, and harder, capability than visual question answering, and current models appear not to possess it — they treat figures as additional context to be described, not as evidence to be scrutinized.

The practical implication is significant for anyone building AI-driven peer review or verification systems: simply feeding multi-modal inputs to current models may be counterproductive unless the model has been specifically trained or prompted to perform cross-modal consistency checking. More modalities do not automatically mean better verification. This is a negative result with positive practical consequences — it identifies a specific capability gap that model developers and benchmark designers should target.

Evidence: Table 3 provides the quantitative comparison, with the text of Section 3.3 explicitly stating: "most models improve in recall and pass@4 when removing images, suggesting that figures usually act as distractors rather than helpful context." The qualitative case studies in Section 4 and Appendix D provide examples of figure-related failures: Gemini-2.5-Pro fails to identify visual inputs correctly (Appendix D.2, claiming tables are "missing" when they are present), and o3 achieves 0% pass@4 on the Figure Duplication category (Table 4, Figure 4 right panel), entirely failing to detect the type of error that requires visual analysis.

Innovation 4: A Capability Cliff — Standard Benchmarks Mask the Verification Gap

The paper documents what can be called a capability cliff: a sharp discontinuity between models' performance on established STEM benchmarks and their performance on Spot. Figure 3 visualizes this most starkly: o3 exceeds 80% on four benchmarks (MathVista, MMLU-Pro, GPQA Diamond, MMMU), drops to roughly 20% on HLE (Humanity's Last Exam, a curated set of research-level questions), and drops to 21.1% (recall) on Spot. Llama-4-Maverick follows a similar trajectory but collapses further, reaching 0.9% on Spot.

This cliff is not just a quantitative observation — it is a diagnostic finding about what current benchmarks measure. The standard STEM evaluation suite (MMLU, GPQA, MMMU) tests knowledge recall and structured problem-solving: given a well-defined question with a clear correct answer embedded in a domain the model has studied, can it produce the right response? These benchmarks reward encyclopedic knowledge and the ability to apply learned patterns to familiar problem formats. Spot tests something qualitatively different: skeptical reasoning about a complex, novel argument with no external cues about where the error might be. There is no multiple-choice question to answer, no hint that "there is an error in this section," no guarantee that the error type matches a known template. The model must proactively read with suspicion, hold multiple pieces of information in working memory across modalities and sections, notice inconsistencies, and articulate why the inconsistency constitutes a scientific flaw — all without being told what it's looking for.

The capability cliff reveals that exam performance does not transfer to verification performance. A model that can solve graduate-level math problems cannot necessarily detect a gap in a mathematical proof that invalidates a published theorem (as the mathematics case study in Section 4.1 demonstrates — o3 and Gemini-2.5-Pro both fail to identify the actual gap in Theorem 1.8 and Theorem 1.13 of Petersen and Tommasi, 2024). This has profound implications for how the community evaluates AI for science: current benchmarks are measuring the wrong capability for the verification use case, and Spot identifies a specific cognitive skill — skeptical, cross-modal, self-directed error detection — that is currently untested and largely absent.

The paper does not explain why the cliff exists, leaving this as an open research question. Potential explanations include: models are trained primarily on correct text and lack exposure to error-identification tasks; error detection requires a form of contradiction-noticing that standard next-token prediction training does not reinforce; and the task's open-ended nature (no prompt to "find the error in equation X") makes it impossible to map onto the pattern-matching shortcuts that models use to succeed on exam benchmarks. Any of these explanations, if true, would suggest that closing the verification gap requires fundamentally different training objectives — not just scaling existing approaches.

Evidence: Figure 3 (the six-benchmark comparison) and Table 2 (showing o3's 18.4% pass@1 and 37.8% pass@4 as the ceiling) together establish the cliff. The observation that both o3 and Llama-4-Maverick follow similar trajectories (high on standard benchmarks, low on HLE and Spot) but diverge in magnitude at the low end (o3 remains at ~20%; Llama-4 collapses to <1%) suggests that the cliff is universal but its depth varies by model — stronger models retain some residual capability on the harder tasks, but none approach practical reliability.

Innovation 5: Miscalibration as a Structural Failure — Models Don't Know That They Don't Know

The paper's calibration analysis (Section 3.2, Figure 4) reveals a second, compounding failure mode that is distinct from low accuracy: models are not just bad at detecting errors, they are bad at knowing when they are bad. The left panel of Figure 4 shows kernel density estimates of reported confidence for six models; all six are clustered near zero, with the density rapidly decaying. The right panel shows that confidence correlates only weakly with pass@4, and most models report very low confidence even on error categories where they occasionally succeed.

The phrase "uniformly low confidence" captures what makes this finding significant. A well-calibrated model on a difficult task might report confidence proportional to its success rate — if it achieves 20% pass@4, it might report confidence around 0.2, indicating awareness that it is often wrong. But Spot's models report confidence substantially lower than their (already low) performance would suggest, and the correlation between confidence and category-level accuracy is weak. This is not the standard calibration problem where models are overconfident (Guo et al., 2017) — it is a deeper failure where the model's uncertainty signal is uninformative about whether it is correct.

The authors quantify this vividly: across 498 model-instance evaluations (83 instances × 6 models), they observe "only two cases (both from o3) of full confidence." In other words, across nearly 500 attempts by six state-of-the-art models, the models expressed certainty that they had found an error exactly twice. Even when they actually did find an error — the true positives that contribute to the 21.1% recall — they did so tentatively, with low confidence. Conversely, false positives (hallucinated errors) were also reported with low confidence, meaning there is no confidence threshold above which a model's claims become trustworthy.

The practical implication is that you cannot deploy these models as verifiers even with a human-in-the-loop who reviews high-confidence flags first, because the confidence signal carries no information about actual correctness. Every flag — true or false — looks the same in terms of model uncertainty. This makes the verification use case qualitatively different from, say, classification tasks where confidence thresholds can trade off precision and recall; here, the confidence dimension is collapsed, and all outputs are equally dubious.

This finding also connects to the broader question of whether scaling test-time compute (Appendix B.2, Figure 10) will solve the problem. The linear improvement with reasoning effort for o4-mini shows that more compute helps, but starting from near-zero absolute performance, even substantial proportional improvements leave the models far from practical reliability. The calibration failure compounds this: even if pass@4 improves with scaling, if confidence remains uncorrelated with correctness, users cannot distinguish the improving true positives from the persistent false positives.

Evidence: Figure 4 (left and right panels) and the accompanying text in Section 3.2: "confidence correlates only weakly with pass@4, and the left panel reveals that most models report very low confidence, clustering near zero." The observation about "only two cases (both from o3) of full confidence" across 498 evaluations is from the same section. The calibration methodology is defined in Appendix C.

5. Experimental Analysis

Evaluation Methodology

Dataset. Spot consists of 83 published manuscripts spanning ten scientific fields, containing 91 author-confirmed errors that prompted errata (59 instances) or retraction (32 instances). Papers average 12,887 text tokens (standard deviation 7,421, range 1,207–46,441) and 17.5 images (standard deviation 20.1, range 0–80) per manuscript, with 76 of 83 manuscripts containing a single error, six containing two, and one containing three. All papers were published from 2024 onward (with three pre-2023 exceptions retained because their first error notices appeared in March 2024) to minimize parametric knowledge contamination.
Base model(s). The paper evaluates ten multi-modal LLMs spanning both proprietary and open-source families: OpenAI o3 (2025-04-16), GPT-4.1 (2025-04-14), Google Gemini 2.5 Pro (preview-03-25), Gemini 2.0 Flash Lite (001), Anthropic Claude 3.7 Sonnet:Thinking (20250219), Claude 3.7 Sonnet without thinking (20250219), Qwen 2.5-VL-72B-Instruct, Qwen 2.5-VL-32B-Instruct, Llama-4-Maverick, and Llama-4-Scout. For the text-only ablation, three additional unimodal LLMs are added: DeepSeek-R1, DeepSeek-V3 (0324), and Qwen3-235B-A22B. The paper selects "the most capable models per family," noting that since these already score near zero on Spot, smaller models would be unlikely to perform better and are thus excluded. Proprietary models are accessed via official APIs when available, or via OpenRouter otherwise; each API call is retried up to three times, with failures or truncated outputs marked as incorrect.
Metrics. The primary evaluation metrics are recall and precision, defined as standard ratios of true positives (TP), false positives (FP), and false negatives (FN) summed across all 91 errors: $\text{Recall} = \text{TP} / (\text{TP} + \text{FN})$ , $\text{Precision} = \text{TP} / (\text{TP} + \text{FP})$ . A predicted error is counted as TP only when both the location matches a benchmark annotation AND a separate judge LLM (GPT-4.1) confirms the predicted and annotated errors describe the same issue. All other predictions are FPs; any benchmark error not predicted by the model is an FN. Additionally, $\text{pass@}K$ (for $K \in \{1, 4\}$ ) measures the fraction of ground-truth errors detected in at least one of $K$ independent attempts, with bootstrap resampling (1,000 iterations, drawing $K$ runs without replacement from the 8 available) to estimate means and standard deviations. The paper reports mean ± standard deviation for all metrics over eight independent trials.
Baselines. The evaluation does not include formal baselines in the traditional sense — there is no previous benchmark for scientific manuscript error detection against which to compare. Instead, the paper's primary comparison is across the ten evaluated models themselves, with o3 serving as the de facto ceiling and weaker models (Gemini 2.0 Flash Lite, Qwen 2.5-VL-72B, Llama-4-Scout) establishing the floor at near-zero performance. The text-only ablation (Section 3.3) adds three unimodal models (DeepSeek-R1, DeepSeek-V3, Qwen3-235B-A22B) as additional reference points. The paper also implicitly treats performance on standard STEM benchmarks (MMLU-Pro, GPQA Diamond, MMMU, MathVista, HLE) as an "expected performance" baseline, against which Spot's results are compared (Figure 3).
Generation budget / compute accounting. The main evaluation uses 8 independent runs per paper per model, with each run being a single generation (no multi-turn revision, no beam search, no sampling of multiple candidates per run). The "reasoning effort" parameter in OpenAI models (tested in Appendix B.2 with o4-mini) is varied across low, medium, and high settings, but this is not used in the main evaluation — it is a supplementary analysis to measure the effect of test-time compute scaling. Generation configurations use each provider's recommended parameters where available; otherwise, temperature is set to 0.6, top-p to 0.95, repetition penalty to 1.0, and output length to a minimum of 8 and maximum of 8,192 tokens. Total API expenditures are approximately $5,000.
Cross-validation / statistical protocol. There is no cross-validation in the traditional model-training sense, as Spot is a zero-shot evaluation benchmark with no model fine-tuning. Statistical reliability is addressed through (1) eight independent runs per paper per model, providing a distribution of model outputs from which means and standard deviations are computed for recall and precision, and (2) bootstrap resampling (1,000 iterations) for pass@K estimates to avoid parametric assumptions about the distribution of detection counts across runs. The paper does not report confidence intervals or hypothesis tests (e.g., whether the difference between o3's 21.1% recall and Gemini-2.5-Pro's 10.1% is statistically significant), though the reported standard deviations (o3: ±4.4%; Gemini-2.5-Pro: ±5.6%) indicate substantial overlap. The two-fold human annotation validation (Section 2.1, Stage 4) serves as a quality control for the ground truth rather than a statistical protocol for the evaluation.

Main Quantitative Results

Overall Model Performance (Main Evaluation, Table 2)

The headline result is stark: no model achieves practically useful performance on scientific error detection. Table 2 reports precision, recall, pass@1, and pass@4 for all ten multi-modal LLMs. o3 achieves the highest scores across all metrics with 6.1% ± 1.3 precision, 21.1% ± 4.4 recall, 18.4% ± 2.1 pass@1, and 37.8% ± 1.8 pass@4. The second-best model, Gemini-2.5-Pro, achieves 3.1% ± 1.7 precision, 10.1% ± 5.6 recall, 7.8% ± 3.8 pass@1, and 25.9% ± 4.0 pass@4 — a substantial drop from o3's already-low ceiling. All other models score near zero: GPT-4.1 achieves 6.0% ± 1.6 recall, Claude-3.7-Sonnet:Thinking achieves 6.0% ± 2.4, and open-source models collapse dramatically — Llama-4-Maverick reaches only 0.9% ± 1.2 recall, and Qwen2.5-VL-72B-Instruct reaches 0.4% ± 0.7.

The precision numbers are equally sobering: o3's 6.1% precision means that when o3 flags an error, it is wrong 93.9% of the time. For all other models, precision hovers between 0.6% and 3.2%, meaning their false-positive rates exceed 96.8%. This is the most practically damaging finding: even if a model occasionally detects a real error (recall), the flood of false alarms makes its output unusable without extensive human verification of every flag.

The pass@4 results reveal that allowing multiple attempts helps, but not enough to change the practical conclusion. o3's pass@4 of 37.8% means that over four independent runs, o3 detects roughly 37.8% of errors at least once — roughly double its pass@1 of 18.4%. This improvement suggests that model outputs are stochastic and that errors detected in some runs are missed in others, implying that ensembling multiple runs could improve coverage. However, even at pass@4, more than 60% of errors remain undetected, and precision does not improve with multiple runs (Table 2 only reports single-run precision, not an aggregated version).

The standard deviations reported in Table 2 are large relative to the means, particularly for recall: o3's 21.1% ± 4.4 means the 95% confidence interval (if normally distributed) would span approximately 12.5% to 29.7%, indicating substantial run-to-run variability. This variability is itself a reliability concern — a user running o3 once on a paper has no guarantee of getting a run that achieves 21.1% recall, and the large standard deviation suggests that performance varies significantly across the eight trials.

The right panel of Figure 4 breaks down pass@4 performance by error category for six models (o3, GPT-4.1, Gemini-2.5-Pro, Claude-3.7-Sonnet:Thinking, Qwen2.5-VL-72B-Instruct, Llama-4-Maverick), revealing stark category-specific patterns. The most striking finding concerns figure duplication: o3 achieves 0% pass@4 on this category, as does Gemini-2.5-Pro, while GPT-4.1 achieves 44.4% — the single highest category-specific score for any model. Claude-3.7-Sonnet:Thinking achieves 33.3%. This means that the strongest reasoning model (o3) is completely blind to the most common non-equation error type in the benchmark (27 instances), while a non-reasoning model (GPT-4.1) performs reasonably well on it. The paper explicitly notes that this "reveal[s] a weakness in figure analysis in reasoning models."

On equation/proof errors (37 instances), o3 dominates with 62.6% pass@4, followed by Gemini-2.5-Pro at 36.4%. All other models remain below 5%. This is the only category where any model achieves majority-level performance, and it is consistent with o3's known strength in mathematical reasoning. However, even 62.6% pass@4 means that more than a third of equation/proof errors — the category where models perform best — remain undetected across four attempts. The detailed breakdowns in Appendix G (Tables 4–13) show that o3's pass@4 for equation/proof is 67.5% (not the 62.6% reported in Figure 4; the discrepancy likely reflects different aggregation or a different subset, though the paper does not explain the difference explicitly).

For data inconsistency (18 instances), o3 achieves 25.7% pass@4 and GPT-4.1 achieves 19.2% (Table 4, Table 5). All other models fall below 10% with the exception of Claude-3.7-Sonnet:Thinking at 29.3% (Table 8). Statistical reporting (4 instances) shows o3 at 88.4% pass@4 (Table 4), though this category's small size makes the estimate unreliable. Reagent identity (3 instances) shows o3 at 62.7% pass@4 (Table 4), again with very wide confidence due to small $n$ . Experiment setup (2 instances) yields 0% pass@4 across all models.

The takeaway is that model performance is highly category-dependent, with mathematical reasoning as the relative strength (for o3) and visual analysis (figure duplication) as a catastrophic weakness. No model achieves reliable performance across all categories; the best model (o3) is competent on equation/proof errors but blind to figure duplication, while the best figure-analyst (GPT-4.1) fails on mathematical reasoning. This complementarity suggests that an ensemble approach — using o3 for equation-heavy papers and GPT-4.1 for figure-heavy biology papers — might improve overall performance, though this is not explored in the paper.

The Capability Cliff: Spot Versus Standard STEM Benchmarks (Figure 3)

Figure 3 contextualizes Spot's difficulty by plotting o3 and Llama-4-Maverick performance across six benchmarks: MathVista, MMLU-Pro, GPQA Diamond, MMMU, HLE, and Spot (using recall as the metric). o3 exceeds 80% on the first four benchmarks, drops to roughly 20% on HLE, and scores 21.1% on Spot. Llama-4-Maverick follows a similar pattern but more steeply: from near-o3 performance on MMLU-Pro and GPQA Diamond, it drops to near zero on HLE and 0.9% on Spot.

Two observations from Figure 3 are particularly important. First, the gap between o3 and Llama-4-Maverick is widest on Spot (Δ = 20.2 percentage points), wider than on any other benchmark including HLE. This means that Spot not only tests a difficult capability, but also discriminates more sharply between model qualities than existing frontier benchmarks — models that appear comparable on standard evaluation (both achieving >80% on MMLU-Pro) diverge dramatically when tested on scientific verification. Second, Spot is the only benchmark where Llama-4-Maverick's score collapses to near zero (0.9%). This demonstrates that error detection in scientific manuscripts requires capabilities not captured by any existing STEM evaluation.

The paper does not provide similar cross-benchmark plots for the other eight evaluated models, so it is unclear whether the capability cliff is universal or specific to the o3 vs. Llama-4 comparison. However, given that even the second-best model (Gemini-2.5-Pro) achieves only 10.1% recall on Spot while presumably performing well on standard benchmarks (the paper does not report this explicitly), the cliff is likely a general phenomenon rather than an artifact of model selection.

Multi-Modality: Figures as Distractors (Table 3, Section 3.3)

The text-only ablation in Table 3 is one of the paper's most counterintuitive findings. On 48 text-only-solvable instances (excluding figure-duplication errors and data-inconsistency errors that require visual inspection), most models perform better when images are removed than when they are included in the multimodal input. For example:

GPT-4.1: recall rises from 0.5% ± 0.9 (multimodal) to 8.4% ± 2.5 (text-only); pass@4 rises from 2.0% ± 1.4 to 19.8% ± 2.7.
Claude-3.7-Sonnet:Thinking: recall rises from 2.9% ± 2.3 to 5.0% ± 2.3; pass@4 rises from 8.5% ± 1.7 to 17.0% ± 3.1.
Claude-3.7-Sonnet (non-thinking): recall rises from 1.9% ± 2.1 to 5.8% ± 3.1; pass@4 rises from 4.8% ± 1.5 to 15.2% ± 2.8.
Qwen2.5-VL-72B-Instruct: recall rises from 0.0% ± 0.0 to 4.7% ± 2.2; pass@4 rises from 0.0% ± 0.0 to 11.2% ± 2.5.

The two exceptions are o3 and Gemini-2.5-Pro, which show modest performance drops when figures are removed. o3's recall falls from 34.6% ± 7.1 to 25.7% ± 7.1, and pass@4 falls from 61.1% ± 2.9 to 56.2% ± 4.2. Gemini-2.5-Pro's recall falls from 13.7% ± 8.6 to 6.9% ± 3.1, and pass@4 falls from 34.8% ± 6.1 to 17.0% ± 2.8. These drops indicate that o3 and Gemini-2.5-Pro are the only models that genuinely leverage visual information — they are not merely distracted by figures, but use them to understand the paper and detect errors. Even so, the drops are relatively modest for o3 (pass@4 declines by only ~5 percentage points), suggesting that its text-only reasoning capability is the dominant driver of its performance even in the multimodal setting.

The addition of unimodal LLMs (DeepSeek-R1, DeepSeek-V3, Qwen3-235B-A22B) in the text-only condition provides further context. DeepSeek-R1 achieves 14.8% ± 3.8 recall and 38.6% ± 3.3 pass@4 — competitive with Gemini-2.5-Pro and approaching o3's text-only performance. Qwen3-235B-A22B achieves 15.4% ± 6.2 recall and 38.2% ± 3.1 pass@4, nearly matching DeepSeek-R1. DeepSeek-V3, the non-reasoning variant, achieves only 1.9% ± 1.1 recall and 6.7% ± 2.1 pass@4, consistent with the pattern observed in the main evaluation where reasoning-enabled models outperform their non-reasoning counterparts.

The paper interprets these results as evidence that "figures usually act as distractors rather than helpful context" for most models. More precisely, the finding is that multi-modal inputs degrade performance for all models except the two strongest reasoning models, and that even for those two models, figures provide only modest marginal benefit beyond what text alone offers.

Calibration and Confidence (Figure 4, Section 3.2)

The calibration analysis reveals that models not only perform poorly, but also fail to provide useful confidence estimates. The left panel of Figure 4 shows kernel density estimates of reported confidence (the per-error pass@K estimator described in Appendix C) for six models. All six densities are concentrated near zero, with short right tails — meaning models almost never express confidence that they have detected an error. Across 498 model-instance evaluations (83 instances × 6 models), the paper reports "only two cases (both from o3) of full confidence."

The right panel of Figure 4 plots mean reported confidence against pass@4 for each model, broken down by error category. The dashed diagonal represents perfect calibration (confidence = accuracy). Most points fall far from the diagonal, with confidence consistently much lower than actual pass@4. This means models are underconfident — they perform better (in absolute terms, though still poorly) than their confidence estimates would suggest. However, the correlation between confidence and pass@4 across categories is weak, meaning that confidence does not even serve as a relative signal of which categories the model is better at. For example, o3's confidence on equation/proof errors (where it achieves 62.6% pass@4) is not substantially higher than its confidence on data inconsistency (where it achieves ~25% pass@4).

The practical implication is that there is no confidence threshold above which model outputs can be trusted. A user cannot triage model flags by reviewing high-confidence predictions first and ignoring low-confidence ones, because confidence carries negligible information about correctness. This makes the verification use case qualitatively different from tasks like classification where confidence-based filtering is standard practice.

Test-Time Scaling (Appendix B.2, Figure 10)

The test-time scaling experiment with o4-mini (Figure 10) shows that detection performance improves approximately linearly as the "reasoning effort" parameter increases from low to medium to high. The paper reports this from three independent trials, though specific numerical values are not quoted — Figure 10 is a bar chart with no table of exact numbers. The finding is consistent with the broader pattern in Table 2, where models with test-time reasoning capabilities (o3, Gemini-2.5-Pro, Claude-3.7-Sonnet:Thinking) consistently outperform their non-reasoning counterparts (Claude-3.7-Sonnet, Gemini-2.0-Flash-Lite) and the reasoning-trained DeepSeek-R1 dramatically outperforms DeepSeek-V3 in the text-only condition.

The linear improvement trend is encouraging, but the paper notes an important caveat: given the low absolute performance at all reasoning effort levels, even substantial proportional improvements from scaling leave models far from practical reliability. There is no indication that simply increasing reasoning effort further would close the gap between current performance and what would be needed for deployment.

Context-Length Ablation (Appendix B.1, Figure 9)

On a subset of 36 instances where errors are confined to a single page and are not equation/proof errors (which rely on global context), the paper compares full-paper versus segment-only (single page containing the error) performance. Figure 9 plots Δ = (segment-only – full-paper) for precision and recall. Positive values indicate that models perform better when given only the relevant page, suggesting that long-context processing degrades their error-detection ability.

Gemini-2.5-Pro benefits most from context reduction, with gains of +4.7 precision and +13.5 recall. o3 gains +3.7 precision and +8.5 recall. Claude-3.7-Sonnet gains +2.2 precision and +2.1 recall. Llama-4-Maverick gains +1.6 precision and +2.2 recall. Qwen2.5-VL-72B-Instruct shows almost no change (–0.4 precision, –0.1 recall), which the paper interprets as indicating "a fundamental limit in its error-detection capability rather than a context-length issue" — this model performs near zero regardless of whether it sees the full paper or just the relevant section.

o3's relatively modest gains (+8.5 recall) compared to Gemini-2.5-Pro (+13.5) are attributed to the removal of equation/proof cases from this ablation. Since equations are o3's strength and often rely on cross-sectional notation, removing them disproportionately affects o3, making its full-paper performance on this subset lower than on equation/proof cases, and thus reducing the potential for improvement when switching to segments.

Detailed Per-Category and Per-Domain Breakdowns (Appendix G, Tables 4–26)

The extensive tables in Appendix G provide pass@1, pass@2, and pass@4 means and standard deviations broken down by error category and by paper domain for each of the ten main-evaluation models and the three additional text-only models. These breakdowns reveal several patterns not visible in the aggregate metrics:

By paper domain (Table 4, o3 as representative example): o3's pass@4 is highest on Physics (79.0% ± 7.1), Mathematics (67.6% ± 2.5), Computer Science (55.9% ± 9.5), and Environmental Science (22.1% ± 15.8). It drops to near zero on Chemistry (0.0% ± 0.0), Engineering (0.0% ± 0.0), Medicine (0.0% ± 0.0), and Materials Science (16.7% ± 0.0 — though this is based on very few instances). The strong performance on Physics, Mathematics, and Computer Science is consistent with o3's mathematical reasoning strength, since these domains concentrate on equation/proof errors. The 0% on Chemistry, Engineering, and Medicine likely reflects both the distribution of error types in those domains (fewer equation/proof errors) and the small number of papers from those fields in the benchmark.

By error category (Table 4 through Table 13, all main-evaluation models): The category breakdowns confirm the pattern from Figure 4. o3 dominates equation/proof (67.5% pass@4) and statistical reporting (88.4% pass@4, though $n = 4$ makes this unreliable). GPT-4.1 dominates figure duplication (41.1% pass@4). Claude-3.7-Sonnet:Thinking achieves 34.9% pass@4 on figure duplication. Nearly all models achieve near-zero pass@4 on experiment setup and statistical reporting (except o3 on the latter), reflecting both the difficulty of these error types and their small sample sizes.

Open-source models (Tables 10–13): Qwen2.5-VL-72B-Instruct achieves 0.0% pass@4 on equation/proof, data inconsistency, experiment setup, reagent identity, and statistical reporting. Its only non-zero category is figure duplication at 3.9% ± 3.0 pass@4. Llama-4-Maverick similarly achieves 0.0% pass@4 on equation/proof and experiment setup, with small non-zero scores on figure duplication (6.6% ± 3.1) and data inconsistency (3.6% ± 3.6). This confirms that open-source models' near-zero aggregate performance masks a complete inability to handle mathematical errors, with only marginal, noisy detection of visual or tabular errors.

Ablation Studies and Robustness Checks

Multi-modality ablation (text-only subset, 48 instances, Table 3): Removing images from the input substantially improves performance for most models. GPT-4.1's pass@4 rises from 2.0% (multimodal) to 19.8% (text-only); Claude-3.7-Sonnet:Thinking rises from 8.5% to 17.0%. Exceptions are o3 (drops from 61.1% to 56.2%) and Gemini-2.5-Pro (drops from 34.8% to 17.0%), suggesting these models use figures productively. The finding implies that multi-modal inputs serve as distractors rather than aids for most current models in this task.
Context-length ablation (36 single-page instances, Appendix B.1, Figure 9): Reducing input from full papers to single pages improves performance for all models except Qwen2.5-VL-72B-Instruct, which shows near-zero change. Gemini-2.5-Pro gains the most (+13.5 recall), followed by o3 (+8.5). The paper interprets this as evidence that long-context processing masks true error-detection performance, though the absolute performance remains low even in the segment-only condition. Qwen2.5-VL-72B-Instruct's flat response is attributed to a "fundamental limit in its error-detection capability."
Test-time compute scaling (o4-mini, Appendix B.2, Figure 10): Varying the "reasoning effort" parameter from low to medium to high produces approximately linear improvements in pass@1. This is consistent with the pattern where reasoning-enabled models (o3, Gemini-2.5-Pro, Claude-3.7-Sonnet:Thinking, DeepSeek-R1) outperform their non-reasoning counterparts. However, the paper does not report exact numbers or the absolute performance levels, making it difficult to assess whether scaling could eventually reach practical reliability.
Reasoning versus non-reasoning model comparison (Table 2, implicit ablation): The "Think" column in Table 2 distinguishes models that use test-time scaling (chain-of-thought reasoning) from those that do not. Among Anthropic models, Claude-3.7-Sonnet:Thinking achieves 6.0% recall and 18.6% pass@4, versus 5.8% recall and 14.1% pass@4 for the non-thinking variant. Among OpenAI models, o3 (reasoning) achieves 21.1% recall; GPT-4.1 (non-reasoning) achieves 6.0%. The gains from reasoning are consistent but modest except for the jump from GPT-4.1 to o3, which likely reflects architectural and training differences beyond just test-time scaling.
Proprietary versus open-source model comparison (Table 2, Figure 3): Open-source models (Qwen2.5-VL-72B/32B, Llama-4-Maverick/Scout) all achieve precision below 2.1% and recall below 1.9%. Their pass@4 scores range from 1.7% (Qwen2.5-VL-72B) to 7.2% (Llama-4-Scout). This contrasts with their performance on standard benchmarks, where they match or approach proprietary models. The paper explicitly notes that Spot is "the only benchmark where Llama-4-Maverick's score collapses to near zero (0.9%)."
Calibration assessment (Figure 4, Section 3.2): All six analyzed models report confidence clustered near zero. Across 498 model-instance evaluations, only two instances (both o3) show full confidence. Confidence and pass@4 are weakly correlated, meaning models' self-assessed certainty provides essentially no signal about correctness. This is a negative result: even as an uncertainty estimator, current LLMs fail to distinguish between true and false positives.
Error category representativeness (Appendix A, implicit robustness check): The paper acknowledges in its limitations that "the complexity of scientific manuscripts means some true errors may be unannotated," leading to potential false positives (where a model correctly identifies an unannotated error that is counted as FP) and false negatives (where the author's acknowledgment doesn't cover all downstream consequences of an error). The case studies (Section 4) partially validate this concern: one instance (Gemini-2.5-Pro's detection of a unit error in the materials science paper) is identified as a genuine unannotated error that was counted as a false positive in the main evaluation. This suggests that the reported precision numbers may be slight underestimates, though the paper argues such cases are "highly unlikely" to change the overall conclusion given the low absolute numbers.

Critical Assessment

The experimental results provide strong evidence for the paper's central descriptive claim — that current LLMs, including the most capable proprietary models, cannot reliably detect errors in scientific manuscripts. However, the strength of evidence varies across the paper's more specific interpretive claims, and several aspects of the experimental design warrant scrutiny.

Claim: "No model surpasses 21.1% recall or 6.1% precision." This is directly supported by Table 2. The numbers are unambiguous, and the reported standard deviations (from 8 independent runs) provide reasonable estimates of variability. The paper's decision to report both recall and precision separately, rather than combining them into an F1 score, is appropriate given that the two metrics capture different failure modes with different practical implications. One might argue that reporting pass@K (which allows multiple attempts) alongside single-attempt metrics provides a more optimistic view of model capability, but even pass@4 for o3 reaches only 37.8% — far from operational reliability.

Claim: Spot reveals a "substantial gap between current LLM capabilities and the requirements for dependable AI-assisted academic verification." This is supported by the absolute performance numbers, but the paper does not define what "dependable" means quantitatively. What recall and precision thresholds would make an AI verifier useful? The paper does not establish these, leaving the "gap" as a qualitative judgment. A reviewer might reasonably ask: if o3 achieves 62.6% pass@4 on equation/proof errors (Figure 4), is that sufficient for use as a screening tool that flags potential mathematical errors for human review, even if overall recall across all categories is low? The paper does not engage with this question, presenting the aggregate numbers as definitive evidence of unusability without discussing whether domain-specific or category-specific deployment might be viable.

Claim: "Confidence estimates are uniformly low, and across eight independent runs, models rarely rediscover the same errors, undermining their reliability." The confidence claim is well-supported by Figure 4 and the 498-evaluation observation. The claim about rare re-discovery is less thoroughly documented. The paper does not report a specific metric for consistency across runs (e.g., the intraclass correlation or the fraction of errors detected in at least $k$ out of 8 runs). The pass@K analysis partially addresses this (pass@1 = 18.4%, pass@4 = 37.8% implies some errors are detected across multiple runs, since pass@4 is more than double pass@1), but the paper does not report, for example, how many of the 37.8% of errors detected at pass@4 were detected in all 4 runs versus in only 1 of 4. This information would directly quantify the re-discovery rate and is available from the experimental data but not presented.

Claim: "Qualitative analysis with domain experts reveals that even the strongest models make mistakes resembling student-level misconceptions derived from misunderstandings." This claim is supported by the case studies in Section 4 and Appendix D, but the generalizability from two fields (mathematics and materials science) and a small number of papers is unclear. The expert-led case studies provide rich, illustrative examples of model failures, but the paper does not systematically categorize failure modes across all 83 papers. The specific claim that errors "resemble student-level misconceptions" is an expert judgment rather than a quantitative finding, and the paper does not describe how this judgment was reached or validated across annotators.

Limitations in experimental design that affect the strength of conclusions:

Small benchmark size limits statistical power for breakdown analyses. With 91 errors across 10 domains and 6 categories, many sub-category cells contain only 2–4 instances. The standard deviations on category-specific pass@K scores are correspondingly large. For example, o3's 88.4% ± 12.5 pass@4 on statistical reporting (Table 4) is based on only 4 errors, making the estimate unreliable and the ±12.5 standard deviation misleadingly precise (it is an artifact of bootstrap resampling of a tiny population). The paper's conclusions about category-specific strengths and weaknesses should be treated as suggestive patterns rather than confirmed findings.
The GPT-4.1 judge introduces an unmeasured source of evaluation error. All TP/FP/FN classifications depend on GPT-4.1's judgment of whether a predicted error matches an annotation. The paper does not report the accuracy of this judge (e.g., by comparing GPT-4.1's match decisions against human judgments on a subset of predictions). If GPT-4.1 is imperfect as a judge — and it likely is, given that it achieves only 6.0% recall on the main task itself — then the reported precision and recall numbers include judge errors as well as model errors. The direction of this bias is unclear: a conservative judge (requiring very precise description matches) would underestimate recall (treating genuine detections as FNs), while a lenient judge would overestimate both precision and recall. The paper acknowledges this implicitly through the structured nature of the comparison task, but the absence of a judge-accuracy calibration is a genuine gap.
The benchmark's "exhaustive annotation" assumption is known to be imperfect. The paper acknowledges in Appendix A that false positives may arise when "an LLM correctly points out a theorem that contains an error, but the content in the LLM's response is still irrelevant," and false negatives when "the author's note contains an error location that does not sufficiently cover all the affected results." The case study in Section 4.2 confirms this: Gemini-2.5-Pro detects a genuine unit error not in the benchmark annotations, which would have been counted as a false positive in the main evaluation. The paper estimates such cases are "highly unlikely," but does not quantify their frequency. Given that precision is already near zero for most models, even a small number of misclassified genuine detections could meaningfully affect the precision estimates for the strongest models.
Single benchmark paradigm (zero-shot evaluation) limits understanding of what might improve performance. All experiments are zero-shot — models receive the paper and respond without any fine-tuning, few-shot examples, or iterative refinement. The paper cannot distinguish between "LLMs fundamentally lack the capability for scientific verification" and "LLMs have the capability but cannot express it in a zero-shot setting." A fine-tuning experiment (training models on a subset of Spot and evaluating on a held-out set) would help disambiguate these possibilities but is not attempted.
No analysis of how error characteristics affect detectability. The paper categorizes errors by type and domain but does not analyze whether detection difficulty correlates with error properties such as: how many sections must be cross-referenced to detect the error, whether the error involves quantitative reasoning (numbers, equations) versus qualitative reasoning (logical inconsistency), how explicitly the error manifests in the text versus requiring inference, or the error's distance from the paper's main claims versus peripheral details. Such an analysis would help explain why certain errors are harder than others and guide model improvement — but it is absent.

Experiments that would have strengthened the paper but were not run:

Human performance baseline. The paper reports model performance but not how well human scientists (e.g., graduate students, postdocs) perform on Spot. Without a human baseline, "near-zero performance" is difficult to calibrate — if domain-expert humans also achieve only 30% recall on these papers (which is plausible, since the errors were missed by the original peer reviewers), the models' performance might be less damning than it appears. Conversely, if humans achieve 80%, the models' failure is even more striking.
Ensemble or multi-model combination. The category-specific results show complementary strengths: o3 on equations, GPT-4.1 on figures. A simple ensemble (e.g., union of o3's and GPT-4.1's predictions, or a panel of models with majority voting) might outperform any single model. This is a natural next step for a benchmark paper that evaluates multiple models, but it is not explored.
Difficulty estimation and adaptive allocation. The paper's compute-optimal framework analogy (from the prior sections) is not applied here. The authors do not attempt to estimate paper difficulty (e.g., from initial model confidence or from paper metadata like field, length, figure count) and allocate compute budgets accordingly. Given that performance varies dramatically by category and domain, an adaptive strategy (e.g., running o3 on mathematics papers and GPT-4.1 on biology papers) seems obviously beneficial but is not evaluated.
Prompt engineering sensitivity analysis. All models receive the same prompt (Appendix F). The paper does not examine whether performance is sensitive to prompt wording — for example, whether asking models to "list all potential issues" versus "identify the most critical error" versus "check consistency between sections" produces different results. Given known brittleness of LLM outputs to prompt phrasing, this is a notable omission for a benchmark that claims to measure capability rather than prompt-engineering skill.

Despite these limitations, the paper's central empirical finding — that no current LLM achieves reliable performance on scientific error detection — is robust to the experimental design choices. Even if judge errors, annotation gaps, and prompt sensitivity could shift the absolute numbers by several percentage points, the qualitative conclusion (models are far from deployment-ready) would not change. The paper's contribution is primarily in establishing that the problem exists and is hard, not in precisely measuring its difficulty or identifying a path to solution — and the evidence for that existence claim is strong.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Makes the Headline Efficiency Gains Aspirational, Not Realized

The assumption or constraint. The entire compute-optimal framework depends on knowing each prompt's difficulty before allocating the inference budget. The paper estimates difficulty by generating 2048 samples per question and averaging either ground-truth correctness (oracle bins) or the PRM's predicted final-answer score (predicted bins). The authors acknowledge this cost explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The paper's reported 4× efficiency gains over best-of-N are computed conditional on already knowing the difficulty, without amortizing the cost of learning it.

The consequence. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution. Generating 2048 samples per question to estimate difficulty consumes more compute than the largest test-time budgets studied (256–512 generations). If this cost is included, the 4× gain either shrinks substantially or reverses entirely—the difficulty estimation step alone might exceed the budget that the compute-optimal strategy is meant to save. The paper does not provide a cost-benefit analysis that includes this overhead, so the headline efficiency figures should be understood as an upper bound on achievable gains, not a realized deployment improvement. A practitioner deciding whether to implement this system cannot estimate the true end-to-end cost from the numbers in the paper.

What evidence exists in the paper. The paper's own results provide the evidence: the PRM-based difficulty estimation (Section 3.2) uses 2048 samples per question, and the compute-optimal scaling curves in Figures 4 and 8 show gains at budgets of 16–256 generations—budgets that are 8–128× smaller than the estimation cost. The predicted-difficulty curves largely overlap the oracle-difficulty curves (Figures 4 and 8), confirming the estimation works but not addressing its cost. The paper does not report how the 4× gain changes when difficulty estimation cost is factored into the total budget. No ablation studies progressively reducing the number of difficulty-estimation samples to find a cheaper-but-still-effective threshold.

Mitigation status. The paper explicitly flags this as future work (Section 8): "Future work should explore cheap difficulty estimation, e.g., training a model to predict difficulty directly from the question text." The authors frame difficulty estimation as an exploration-exploitation tradeoff—"compute spent assessing difficulty versus compute spent solving the problem"—but do not explore alternatives like adaptive difficulty estimation (start with few samples, adjust allocation based on early signals) or difficulty prediction from question text alone. The gap is acknowledged, but no solution is evaluated, meaning the paper's primary practical contribution (the compute-optimal policy) is not yet deployable with the reported efficiency gains.

Hard Problems Remain Unsolved—Test-Time Compute Cannot Substitute for Missing Capability

The assumption or constraint. The entire framework assumes the base model has some non-trivial probability of producing a correct answer. If the base model's pass@1 on a problem class is near zero, no test-time strategy can help—there are no correct solutions in the proposal distribution to find or refine. The paper defines five difficulty quintiles, with bin 5 (hardest) containing problems where the base model's pass@1 is essentially zero.

The consequence. For the hardest difficulty tier (bin 5), all methods—search, revisions, compute-optimal combinations—show near-zero improvement regardless of budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%. This means test-time compute amplification has a hard ceiling at the base model's generative capability boundary. Problems that fundamentally exceed the base model's training distribution—genuinely novel reasoning, out-of-distribution concepts—cannot be solved by any amount of inference-time computation. The FLOPs-matched comparison (Section 7) quantifies this: on hard problems at high inference-to-pretraining ratios (R ≫ 1), test-time compute with the smaller model shows a ~52.9% relative disadvantage compared to the ~14× larger pretrained model (Figure 1, bottom-right bar chart).

What evidence exists in the paper. All difficulty-binned analyses consistently show bin 5 at floor. Section 7's FLOPs-matched comparison explicitly identifies the failure boundary: "test-time compute provides essentially zero benefit regardless of budget, meaning that some capabilities can only be acquired through pretraining, not recovered at inference time." The paper also provides qualitative evidence in Appendix M (qualitative examples of degenerate search outputs showing PRM exploitation) that aggressive search can be counterproductive on problems the model cannot solve.

Mitigation status. The paper is transparent about this limitation (Section 7 takeaway box), explicitly stating that test-time and pretraining compute are "not 1-to-1 exchangeable." No mitigation is proposed beyond acknowledging that pretraining remains necessary for capability expansion. This is not a flaw in the work—it is a fundamental boundary condition on the method—but practitioners must understand that the approach offers no relief for problems outside the base model's reach. For deployments where the problem distribution includes a substantial fraction of bin-5-type questions, scaling pretraining remains the only viable path.

Single Benchmark, Single Model Family—Generality Is Unverified

The assumption or constraint. All experiments are conducted on the MATH benchmark (500 test questions, competition-level math problems) using PaLM 2-S* as the base model. The paper states in Section 4:

"We believe this model is representative of the capabilities of many contemporary LLMs"

This claim is untested across other models and tasks.

The consequence. Several aspects of the findings could be model-specific or task-specific:

PRM quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties or error patterns might exhibit different difficulty-dependent scaling curves—for instance, a model with lower base accuracy might show a different pattern of beam search over-optimization, or a model with stronger in-context learning might benefit more from revisions.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The edit-distance-based training data construction (Section 6.1) might work better or worse depending on the base model's typical error patterns.
MATH consists exclusively of symbolic reasoning problems with clean correct/incorrect answers and well-defined grading. It is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual knowledge rather than inference. The paper's findings about the efficacy of Monte Carlo rollout PRM training (which requires automatic correctness checking) may not transfer to tasks where correctness is ambiguous.

What evidence exists in the paper. The paper reports no experiments on any benchmark other than MATH, nor on any model family other than PaLM 2-S*. All ablation studies (PRM aggregation, revision training, sequential-to-parallel ratio, FLOPs matching) are within this single benchmark-model combination. The difficulty-dependent findings (e.g., beam search degrading easy-problem performance) are replicated across different search methods and revision strategies within MATH, providing internal validity, but external validity is completely untested.

Mitigation status. The paper acknowledges this limitation implicitly by not claiming generality beyond the MATH benchmark, but does not include a dedicated limitations section or call for replication. The "representative model" claim in Section 4 is an assertion, not an empirically supported statement. A practitioner considering adopting compute-optimal test-time scaling for a non-math domain would need to replicate the entire difficulty-estimation, search-algorithm-comparison, and revision-training pipeline to determine whether the findings transfer.

The $14\times$ Larger Model Baseline Is Not Compute-Optimally Trained, Weakening the Pretraining-Versus-Inference Comparison

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than the Chinchilla-optimal approach (Hoffmann et al., 2022) of scaling both data and parameters equally. The paper acknowledges this in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses only greedy decoding with no test-time augmentation—no majority voting, no best-of-N, no search.

The consequence. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, making the larger-model baseline weaker than it could be. The reported advantages of test-time compute over pretraining—e.g., +27.8% relative improvement on medium problems at R ≪ 1 for revisions (Figure 1, top-right bar chart)—may shrink or reverse against a properly compute-optimal larger model. Giving the larger model even a modest test-time compute budget (e.g., best-of-8 with a verifier) would create a much stronger baseline that is never tested. The paper's stated conclusion that "a smaller model augmented with compute-optimal test-time strategies can outperform a ~14× larger pretrained model" (abstract) should therefore be understood as relative to a specific, potentially suboptimal pretraining baseline—not as a general principle that test-time compute dominates pretraining at this scale ratio.

What evidence exists in the paper. The paper's own analysis (Section 7) acknowledges the Chinchilla departure but does not quantify its impact. There are no ablations testing different pretraining recipes for the larger model, nor any experiments giving the larger model any test-time compute budget. The greedy-decoding-only baseline for the larger model is a deliberate simplification, but it means the comparison is between an optimized small-model inference strategy and an unoptimized large-model inference strategy—not a fair contest of total computational efficiency.

Mitigation status. The paper explicitly flags the compute-optimal pretraining comparison as future work (Section 7). The limitation is acknowledged, and the pragmatic choice (parameter scaling only, following common practice) is transparently documented. However, the paper's abstract and introduction do not qualify the ~14× claim with this caveat, which could mislead readers who do not read Section 7 carefully. A practitioner weighing pretraining versus inference compute allocation should treat the paper's numbers as preliminary until a properly compute-optimal baseline is compared.

Verifier Over-Optimization Is a Hard Ceiling That Is Documented but Not Solved

The assumption or constraint. The entire search-based test-time scaling approach (beam search, lookahead search, best-of-N weighted selection) depends on the PRM verifier's reliability under aggressive optimization. The paper assumes the PRM, trained via Monte Carlo rollout supervision (Section 5.1, Appendix D), provides scores that correlate with actual solution correctness across the search space.

The consequence. The PRM is unreliable under search, and this unreliability sets a hard ceiling on how much test-time compute can help. The paper documents multiple manifestations:

Beam search degrades easy-problem performance at high budgets (Figure 3, right)—a hallmark of verifier over-optimization, where search finds solutions that score highly under the PRM but are actually incorrect.
Lookahead search—the most powerful optimizer—paradoxically performs worst overall at fixed generation budgets (Figure 3, left) because its extra per-step cost reduces the effective beam count while amplifying exploitation of PRM errors.
Qualitative examples (Appendix M, Figures 29 and surrounding examples) show search producing degenerate outputs: repetitive low-information steps at the end of solutions and overly short 1–2 step solutions that achieve high PRM scores but are factually incorrect.

The compute-optimal policy mitigates this by routing easy problems away from aggressive search (using best-of-N instead of beam search on bin 1–2 questions), but it does not solve the underlying verifier quality problem. On medium-difficulty problems where beam search is deployed (bins 3–4), over-optimization still limits the scaling ceiling—the beam search curves in Figure 3 flatten and sometimes decline well before the budget is exhausted.

What evidence exists in the paper. Figure 3 (right) provides the primary evidence: beam search accuracy on the easiest bin decreases from ~78% to ~77% as budget increases from 4 to 256, while best-of-N improves from ~68% to ~88%. Figure 3 (left) shows lookahead search underperforming simpler methods. Appendix M provides qualitative confirmation of degenerate outputs. The paper's own analysis (Section 5.3) explicitly identifies over-optimization as the mechanism: "The degradation at high budgets is attributed to over-optimization of the PRM—search finds solutions that score highly under the PRM but are actually incorrect."

Mitigation status. The compute-optimal policy partially mitigates the symptom (by avoiding aggressive search where over-optimization is most damaging) but does not address the cause. The paper does not explore verifier improvements that might raise the over-optimization ceiling: adversarial PRM training on search-generated solutions, ensemble verification, or constrained search with KL penalties. The paper identifies verifier robustness as "the primary bottleneck" for further scaling (Sections 5.3 and 8), but leaves its resolution entirely to future work. A practitioner implementing this system should expect that performance will plateau at moderate budgets (roughly 64–256 generations in the paper's experiments) and that further budget increases may be wasted or counterproductive unless verifier quality is substantially improved.

The Revision Model Has a ~38% Correct-to-Incorrect Reversion Rate and Training Is Fragile

The assumption or constraint. The revision model is fine-tuned on trajectories where all in-context answers are incorrect followed by a correct target (Section 6.1). The training data construction pairs independently sampled correct and incorrect solutions, using edit distance to select an incorrect answer that is structurally close to the correct one. At inference time, the model generates a chain of revisions, conditioning each new attempt on the previous one.

The consequence. Because the model was trained only on incorrect-to-correct sequences, it has never seen a correct answer in its context followed by a target to either keep it unchanged or refine it further. At test time, when the revision chain produces a correct answer (which happens with non-trivial probability—the pass@1 trajectory in Figure 6, left, improves from ~18% at step 1 to ~24% by step 15), the model's training provides no guidance on what to do next:

"approximately 38% of correct answers get converted back to incorrect ones" (Section 6.1)

The revision chain is therefore inherently unstable—performance does not monotonically improve with more revisions, and longer chains may actively degrade results. The paper's mitigation (majority voting or verifier-based selection across the chain) treats the symptom by picking the best answer from the chain rather than always taking the final revision, but this is a post-hoc patch, not a fix to the model's training objective.

The ReST^EM experiment (Appendix K, Figure 16) reveals a deeper fragility: attempting to optimize the revision model with RL-style training (on-policy data collection) caused performance to degrade substantially with sequential revisions. At 256 generations, fully sequential performance dropped to ~33.5% compared to ~38.5% at the optimal hybrid ratio. The authors hypothesize that "on-policy data collection in ReST^EM exacerbates spurious correlations in revision data," but this is a post-hoc diagnosis, not a controlled ablation. The practical implication is that the revision training recipe is sensitive to data construction details in ways that are not fully characterized.

What evidence exists in the paper. The 38% reversion rate is stated in Section 6.1 without a supporting figure or table—the paper does not report how this number was measured, whether it varies by difficulty or revision depth, or whether it is consistent across runs. The ReST^EM degradation is documented in Figure 16 of Appendix K. The per-step pass@1 trajectory (Figure 6, left) shows non-monotonic improvement—curves are noisy and occasionally dip, consistent with reversion behavior but also with sampling variance. The paper does not include a direct ablation comparing the edit-distance-paired training data against alternative incorrect-correct pairing strategies (e.g., pairing a correction with its own preceding error from the same generation trajectory).

Mitigation status. The paper attempts to mitigate reversion through within-chain selection (majority or verifier-based), which reduces but does not eliminate the problem—the selection mechanism can recover a previously correct answer that was later corrupted, but cannot prevent the corruption from occurring. The paper does not propose a training-time solution (e.g., including "keep-if-correct" trajectories in training data, using a learned stopping criterion, or incorporating RL objectives that penalize reversions). The ReST^EM failure is treated as a negative result to report, not as a problem to debug. A practitioner building a revision system should expect to invest substantial effort in tuning the training data pipeline and should treat longer revision chains with skepticism unless a robust stopping mechanism is in place.

7. Implications and Future Directions

How This Work Changes the Landscape

Spot's primary contribution is to expose a capability blind spot in how we evaluate AI for scientific work. The field has been rapidly building toward "AI Co-Scientists"—systems that generate hypotheses, draft manuscripts, design experiments—while implicitly assuming that verification would follow naturally from generative prowess. Spot demonstrates that this assumption is false, and it does so with evidence that is difficult to dismiss: the same models scoring >80% on graduate-level STEM exams (MMLU-Pro, GPQA Diamond, MMMU) collapse to near-zero when asked to find author-confirmed, retraction-worthy errors in real published papers. This is not a marginal performance gap that might be closed with better prompting or slightly larger models; it is a qualitative capability cliff that suggests current LLMs lack something fundamental about skeptical, cross-modal, self-directed error detection.

The paper reframes the conversation about AI in science from "can models generate scientific content?" to "can models check whether scientific content is correct?" This is not an incremental refinement of existing benchmarks—it is a shift in what capability is being measured. Prior evaluation suites like FEVER and SciFact test whether models can verify claims against reference documents, which reduces scientific verification to a fact-checking exercise where the correct answer is always available in a provided source. Automated peer review benchmarks test whether models can mimic the noisy, inconsistent output of human reviewers, confounding error detection with writing style and review conventions. Spot tests something different: given only the paper itself, can the model notice that something is wrong—the same mistake that the original authors, reviewers, and editors all missed? This is a capability that no prior benchmark isolates, and Spot's results suggest it is far more difficult than either fact-verification or review-generation benchmarks would imply.

The paper also reconciles a subtle tension in the AI-for-science narrative that had been building without explicit acknowledgment. On one hand, systems like "The AI Scientist" (Lu et al., 2024) and Google's AI Co-Scientist (Gottweis et al., 2025) present increasingly sophisticated pipelines for automated research, often evaluating their own outputs using LLM judges. On the other hand, prior work on LLM-as-a-Judge (Zheng et al., 2023; Son et al., 2024) had already shown that LLM judges fail on complex, domain-specific tasks. Spot provides the missing evidence that this failure extends to the specific domain where it matters most: detecting consequential errors in frontier research. The implication is that AI Co-Scientist pipelines that rely on LLM-based self-verification are built on an unstable foundation—the verifier is no more reliable than the generator, and neither approaches the standard required for trustworthy scientific output.

Perhaps most importantly, Spot identifies figure analysis as a specific, catastrophic weakness in reasoning models. The finding that o3 achieves 0% pass@4 on the figure duplication category (27 instances, the second-largest error type) while GPT-4.1 achieves 44.4% is a concrete diagnostic: the training procedures that produce strong mathematical reasoning (o3) may actively impair visual analysis capabilities, or at least fail to develop them. This is not a generic "multi-modality is hard" observation—it is a specific, testable hypothesis about a tradeoff in current model training that has direct implications for model selection in scientific applications. If this pattern holds across model families, it suggests that no single current model is suitable for comprehensive scientific verification; domain-specific deployment with different models for equation-heavy versus figure-heavy papers may be necessary.

The paper also shifts the conversation around multi-modal LLMs in an unexpected direction: the finding that most models perform better when figures are removed (Table 3) challenges the prevailing assumption that adding visual information always helps. For scientific verification specifically, figures appear to act as distractors rather than aids for all but the two strongest models (o3 and Gemini-2.5-Pro), and even for those models, the marginal benefit of visual input is modest. This suggests that multi-modal training as currently practiced does not teach models to perform the specific operation needed for error detection—cross-modal consistency checking, where textual claims and visual evidence are compared skeptically—and may instead teach models to treat images as additional narrative content to be described rather than as evidence to be scrutinized. This finding makes research on cross-modal verification training newly attractive as a focused direction rather than a diffuse hope.

Follow-Up Research This Work Enables

Training models specifically for error detection via synthetic data generation. Spot demonstrates that zero-shot performance is near-zero, but does not test whether the capability can be learned. The most immediate follow-up is a fine-tuning experiment: construct a training set of papers with inserted errors (synthetic but realistic mistakes across the six Spot categories—e.g., altered equations, duplicated figures with changed labels, mismatched values between tables and text), fine-tune models to detect these errors, and evaluate on the held-out Spot test set. The key question is whether error detection is a capability that can be acquired through supervised training on synthetic data, or whether it requires forms of reasoning (skeptical reading, multi-hop consistency checking) that are fundamentally absent from current architectures. A strong experiment would systematically vary error type, insertion method (rule-based vs. LLM-generated), and training scale to identify whether there is a "phase change" where models suddenly generalize from synthetic to natural errors, analogous to how code models generalize from synthetic to real programming tasks.

Cheap difficulty estimation for adaptive error detection. Spot reveals dramatic performance variation across error categories and paper domains—o3 achieves 62.6% pass@4 on equation/proof errors and 0% on figure duplication. This suggests that an adaptive deployment strategy—routing equation-heavy papers to o3 or DeepSeek-R1, figure-heavy biology papers to GPT-4.1—could substantially outperform any single model. The bottleneck is knowing which strategy to deploy before running the expensive model. A concrete follow-up would train a lightweight paper classifier that predicts, from surface features (paper domain, figure count, equation density, venue), which model is most likely to detect errors, analogous to the difficulty estimation in the compute-optimal test-time scaling framework (Snell et al., 2024). The evaluation would measure whether such routing achieves higher combined recall than any single model at equivalent total cost, using the per-category breakdowns in Appendix G (Tables 4–13) as the optimization target.

Human performance baseline to calibrate the difficulty of Spot. The paper reports model performance without establishing how well humans perform on the same task. This is a critical missing piece: if domain-expert humans also achieve, say, 30% recall on these errors (plausible, since the original peer reviewers missed them), then models' 21.1% (o3 recall) might be interpreted as approaching expert-level performance rather than as near-total failure. Conversely, if humans achieve >80%, the gap is even larger than the paper implies. A direct follow-up would recruit graduate students or postdocs in the relevant fields, give them the same input format (interleaved text and figures, the same prompt), and measure their recall, precision, and time-per-paper on a representative subset of Spot. This would contextualize model performance not against an abstract "100% correct" ceiling, but against the actual difficulty of the task for trained humans under comparable conditions. The experiment could also measure whether human performance varies by error category in the same pattern as model performance (e.g., do humans also struggle with figure duplication relative to equation errors?), which would distinguish task-inherent difficulty from model-specific weaknesses.

Cross-modal consistency training as a targeted intervention. The finding that figures act as distractors for most models (Table 3) identifies a specific capability gap: models do not perform cross-modal consistency checking, where textual claims are compared against visual evidence and discrepancies are flagged. A targeted training intervention could construct paired text-image examples where the text makes a specific claim (e.g., "Figure 3B shows that Treatment X increased yield by 40%") and the image either supports or contradicts that claim (e.g., the plotted bars actually show a 20% increase or no difference). Fine-tuning on this cross-modal verification task—separate from standard visual QA or image captioning—might teach the model to treat figures as evidence to be scrutinized rather than as additional narrative content. The evaluation would measure whether such fine-tuning improves figure-duplication detection on Spot (where current reasoning models achieve 0%) without degrading text-only error detection. A negative result (fine-tuning fails to transfer from synthetic cross-modal checks to real figure-duplication detection) would suggest that the problem is not just missing training data but a more fundamental architectural limitation in how models integrate modalities.

Ensemble and multi-model combination for complementary coverage. The category-specific results in Figure 4 show that no single model covers all error types: o3 excels at equations (62.6% pass@4) but fails on figures (0%); GPT-4.1 excels at figure duplication (44.4%) but fails on equations (1.5% pass@4, Table 5). A natural follow-up evaluates multi-model ensembles: take the union of o3's and GPT-4.1's error predictions, combine them with a judge model that resolves conflicts, and measure whether combined recall exceeds either individually while maintaining precision above some operational threshold. This experiment would use the eight independent runs already collected to construct ensembles across models rather than across runs of a single model. A more ambitious version would train a router that learns to predict, per paper, which model(s) to query based on paper metadata and early model outputs, optimizing a precision-recall tradeoff under a budget constraint on API calls.

Domain-specific error taxonomies and difficulty analysis. Spot's six error categories emerged inductively from 91 instances, and the performance breakdowns by category are based on small sample sizes (e.g., only 4 statistical reporting errors, 2 experiment setup errors). A follow-up study could expand Spot within a single domain—say, materials science or biology—to create a domain-specific benchmark with 50–100 errors and a finer-grained taxonomy. This would enable statistically reliable per-category analysis and could reveal whether certain error types are inherently easier (detectable by weaker models) or harder (requiring specific reasoning capabilities). The study could also annotate error characteristics that might predict detectability: how many sections must be cross-referenced, whether the error involves quantitative versus qualitative reasoning, whether it manifests in explicit contradictions or requires inference from missing information, and the distance from the error location to the paper's main claims. Correlating these characteristics with model success rates would provide a roadmap for where verification models need to improve.

Investigating why reasoning models lose figure analysis capability. The paper's most surprising category-specific finding—that o3 and Gemini-2.5-Pro achieve 0% pass@4 on figure duplication, while non-reasoning GPT-4.1 achieves 44.4%—calls for a targeted investigation. Is this a training data effect (reasoning models are trained predominantly on text and code, with less emphasis on visual data), an architectural effect (the chain-of-thought reasoning process interferes with visual processing or consumes context window space that would otherwise be available for image analysis), or a prompting effect (the reasoning prompt directs attention toward logical and mathematical analysis at the expense of visual inspection)? A controlled experiment would compare o3's figure analysis performance under the standard Spot prompt versus a prompt that explicitly directs attention to figures ("Examine each figure carefully for duplicated panels, inconsistent labels, or manipulations"), across the 27 figure-duplication instances. If explicit prompting recovers performance, the problem is attentional rather than a fundamental capability gap; if not, the problem is deeper. A parallel experiment could ablate the reasoning chain length (using o4-mini with variable reasoning effort, as in Appendix B.2) to see whether increased reasoning degrades or improves figure analysis.

Practical Applications and Downstream Use Cases

Pre-submission manuscript screening for high-stakes venues. Journals and conferences that receive thousands of submissions could deploy Spot-style error detection as a pre-review filter, specifically targeting error types where current models show non-zero performance. The category-specific results suggest a concrete workflow: use o3 (or DeepSeek-R1) to screen equation-heavy submissions (mathematics, theoretical physics, computer science theory) for mathematical errors, achieving ~62.6% detection on the most common error type in those fields, and use GPT-4.1 to screen figure-heavy submissions (biology, materials science) for duplicated or manipulated images, achieving ~44.4% detection. While the precision is too low for fully automated decisions (GPT-4.1's 2.8% precision means >97% of flags are false positives), a human-in-the-loop triage system where flagged issues are reviewed by an editorial assistant before being escalated to reviewers could catch errors that currently slip through peer review entirely. The value proposition is not replacing reviewers but augmenting them with an automated check that catches a predictable subset of errors—especially figure duplications, which are notoriously difficult for human reviewers to detect systematically. The pass@4 results suggest that running each model 4 times on a submission (at a total API cost of perhaps $0.50–2.00 per paper for current pricing) would catch ~37.8% (o3 overall) to 44.4% (GPT-4.1 on figures) of errors, making it cost-effective for venues where each paper's review costs hundreds of dollars in reviewer time.

Post-publication monitoring and systematic error surveillance. Organizations that track scientific integrity—PubPeer, Retraction Watch, institutional research integrity offices—could use Spot-style models as a continuous monitoring tool for new publications. Rather than depending on volunteer commenters to notice errors (the current PubPeer model, which Spot's seed collection demonstrated is sparse and field-dependent), an automated system could screen every new paper in target fields and flag potential issues for human investigation. The low precision is less problematic in this use case because the cost of a false positive is a human spending 5–10 minutes checking a flagged issue, and the cost of a false negative is a consequential error remaining in the literature indefinitely. The key practical consideration is coverage: Spot's overall recall ceiling is 21.1% (o3) for all error types, but the per-category numbers suggest that targeted deployment on equation/proof errors in mathematics (62.6%) and figure duplication in biology (44.4%) could achieve substantially higher yield in those specific paper categories. A monitoring service could start with these high-yield combinations and expand as models improve, providing systematic coverage where human post-publication review is currently sparse and haphazard.

Training data generation for verifier models via distillation from human annotations. Spot's 91 author-confirmed errors, while too small for training, provide a high-quality seed for generating synthetic training data. A practical pipeline could use the annotated errors as templates: for each error type, use a strong LLM to generate variants—e.g., take a real equation error and generate 100 semantically similar but mathematically distinct errors, insert them into clean papers, and train a verifier model to detect them. The key advantage of Spot over existing resources for this purpose is the cross-validation with author acknowledgments, which ensures the templates reflect errors that actually occur in published literature (rather than errors that a benchmark designer guesses might occur). The trained verifier could then be deployed in any of the above use cases with substantially higher performance than the zero-shot models evaluated in Spot, though generalization from synthetic to natural errors would need careful evaluation—specifically, measuring whether the verifier detects the held-out Spot errors that were not used in template generation.

</response>