ReviewScore: Misinformed Peer Review Detection with Large Language Models

🎯 Pitch

One in four reviewer questions in top AI conferences can already be answered by the paper itself, and one in six weaknesses rests on incorrect premises. Even the best large language models achieve only moderate agreement with human experts when tasked with detecting these misinformed review points. This work introduces a systematic framework and an annotated dataset that reveals both the scale of the peer review quality crisis and the current limits of AI to solve it.

1. Executive Summary

This paper introduces REVIEWSCORE, a new evaluation metric that detects misinformed review points in peer review—questions that can already be answered by the submitted paper, and weaknesses containing incorrect premises regarding the paper. Using human expert annotation of ICLR 2021–2023 reviews on 40 papers, the authors verify that 15.2% of weaknesses and 26.4% of questions are misinformed, and they construct a dataset of 657 annotated review points (including 1,748 premises from 422 arguments) to benchmark LLM evaluation capability. To handle argumentative weaknesses, the paper proposes an automatic argument reconstruction engine (a feedback-loop system combining LLM generation with SAT solver validity checks and faithfulness verification to extract explicit and implicit premises from a weakness), and defines ADVANCED REVIEWSCORE as the aggregation of premise-level factuality scores rather than weakness-level judgments. Eight state-of-the-art LLMs achieve only moderate agreement with human experts (F1 scores of 0.4–0.5, kappa scores of 0.3–0.4), with a thorough disagreement analysis revealing that most errors stem from models' misunderstanding or incorrect reasoning, establishing that fully automated misinformed review detection remains challenging despite premise-level evaluation significantly outperforming weakness-level evaluation (3.40× higher F1 for ArgScore vs. Base ReviewScore with Gemini 2.5 Flash).

2. Context and Motivation

The Core Problem: A Degrading Peer Review System

The peer review system is under genuine strain. As the number of submissions to AI conferences has exploded—the paper notes this is an exponential increase—the demand for qualified reviewers has exploded with it, diluting the quality of the reviewing pool. The consequences, documented by prior work (Stelmakh et al., 2021; Cortes and Lawrence, 2021; Shah, 2022; Kim et al., 2025), are not subtle: review quality degrades, which directly undermines the integrity and reliability of the entire system that academic research depends on for validating new contributions.

The paper opens with this diagnosis not as background flavor but as the primary motivation. The problem is both acute (current reviews contain detectable misinformation, as the paper later quantifies at 15–26%) and structural (the supply of expert reviewers cannot scale with submission volume, so the problem will likely worsen unless systems are built to assist or compensate).

The Gap: No Clear, Objective, and Applicable Evaluation Criteria

Given that review quality is degrading, the natural response is to evaluate reviews—to measure their quality so that low-quality reviews can be identified, flagged, or filtered. The paper identifies a specific, unresolved tension in prior work on this question: a trade-off between applicability and specificity of evaluation criteria.

Vague but broadly applicable criteria. Goldberg et al. (2025) and Du et al. (2024) propose evaluation rubrics that can apply to almost every review—criteria like "understanding" and "substantiation." These are important dimensions of quality, but the paper argues they are vague and subjective. Two evaluators reading the same review may disagree sharply on whether the reviewer "sufficiently substantiated" their criticism, because substantiation is a matter of degree and interpretation, not a binary or bounded fact. This makes such criteria difficult to operationalize for automated detection.

Specific but narrowly scoped criteria. Guo et al. (2023), Sadallah et al. (2025), Purkayastha et al. (2025), and Ou et al. (2025) introduce more specific and objective criteria. For example, these works evaluate whether a review is grounded in the target paper—does the reviewer's claim actually reference something in the submission? This is a concrete, checkable property. However, the paper argues these approaches target a narrow scope of reviews. They typically focus only on weaknesses (not questions), they check whether claims are supported but not whether the supporting reasoning is factually correct, and they largely ignore the logical structure connecting multiple statements within a review point.

The unresolved gap, as the paper characterizes it: how do you define evaluation criteria that are simultaneously (a) specific and objective enough to reduce annotator subjectivity and enable automation, and (b) applicable to the full range of content found in real reviews, including both negative claims and open questions?

The Specific Blind Spot: Nobody Checks Whether Review Criticisms Are Factually Wrong

This is the paper's sharpest motivation. Prior work on automatic review evaluation, summarized in Table 1, largely focuses on groundedness—whether a reviewer's statements can be traced back to something in the paper. Guo et al. (2023), Sadallah et al. (2025), and Ou et al. (2025) all evaluate this property. But a review statement can be grounded (it references a specific section or claim in the paper) and still be factually incorrect (it misunderstands what that section says, or it makes an incorrect logical inference from correct premises).

Du et al. (2024) takes one step further by evaluating factuality at the sentence level within weaknesses. But the paper identifies a critical limitation of this approach:

"a sentence-level factuality cannot fully capture correctness of an underlying logic of weaknesses"

In other words, a weakness in a review is often not a single factual claim but an argument—a chain of reasoning where multiple statements (some explicit, some implicit) combine to support a conclusion. A review might contain individually true sentences that, when combined through flawed logic, produce an incorrect criticism. Or a review might contain a mix of true and false premises, making a verdict on the whole weakness ambiguous without understanding which premises carry the logical weight. Sentence-level factuality checks miss this structure entirely.

Shin et al. (2025) and Purkayastha et al. (2025) take yet another approach: classifying reviews into predefined "types" (facets or lazy-thinking patterns). But these classification schemas do not evaluate factuality or groundedness at all—a review might be structurally sound and fall into a reasonable "type" while still being completely wrong about the paper's content.

The paper thus identifies a genuine blind spot: no prior work checks whether the answers to reviewer questions can already be found in the paper, and no prior work reconstructs the full logical structure of critical arguments to evaluate whether the underlying premises are factually correct. These are specific, objective criteria (a premise either is or is not supported by the paper, a question either is or is not answerable) that apply across the major content types in real reviews.

Why This Matters: The Stakes of Misinformed Reviews

The paper grounds its motivation in real reviewer guidelines from major conferences, making the criteria feel less like an academic exercise and more like a formalization of existing community standards:

ACL 2023 Peer Review Policies state: "before writing a negative review, check whether your questions are already answered." (Boyd-Graber et al., 2023) This is an explicit instruction to reviewers to avoid the exact problem REVIEWSCORE measures—questions whose answers are already in the paper.
NeurIPS 2025 reviewer guidelines state: "reviewers should minimize the chance of misunderstandings during the reviewing process." (NeurIPS 2025 Program Chairs, 2025) This is directly relevant to weaknesses containing incorrect premises, which often arise from reviewer misunderstanding.

The paper is effectively saying: the community already agrees these are problems, but we have no systematic way to detect them. REVIEWSCORE is proposed as the measurement tool that aligns with these existing norms.

The human annotation results make the problem concrete: 15.2% of weaknesses and 26.4% of questions in the annotated ICLR reviews are misinformed. These are not rare edge cases—they represent a substantial fraction of review content. For authors receiving these reviews, approximately one in four questions could have been answered by reading the paper more carefully, and more than one in seven critical weaknesses rests on incorrect premises. This has downstream effects on paper acceptance decisions, author morale, and the overall efficiency of the reviewing process (authors spend rebuttal time addressing questions already answered in their papers).

The Untapped Connection: Logic and Critical Thinking for Review Evaluation

A subtle but important motivation running through the paper is the observation that peer review evaluation has not meaningfully engaged with the established literature on argument analysis from logic and critical thinking. The paper explicitly connects its work to this tradition:

An argument is classically defined as "a list of statements, one of which is the conclusion and the others are the premises" (Dutilh Novaes, 2022; Lau and Chan, 2025).
Argument reconstruction—the process of identifying and making explicit all premises (including implicit ones) that support a conclusion—is a well-established practice in critical thinking (Brun and Betz, 2016; Groarke, 2024; Dowden, 2024).
Prior computational work on argument reconstruction (Betz and Richardson, 2021) trained a T5 model for this task, but on synthetic or polished datasets where no additional context was needed to understand the arguments.

The gap the paper identifies here: real peer review arguments are unpolished, domain-specific, and require the full paper context to reconstruct faithfully. The Betz and Richardson (2021) approach cannot handle a review that says "the method lacks novelty because it merely combines existing techniques" without access to the referenced paper to understand what those techniques are and whether the reviewer's characterization is accurate. The paper's argument reconstruction engine (Section 3.2) is explicitly designed to bridge this gap: it takes both the review text and the full submitted paper as input, and it includes validity and faithfulness feedback loops (the former using a SAT solver for rigor, the latter using LLM judgment for semantic fidelity).

How This Paper Positions Itself

The paper's self-positioning has several layers:

It fills the applicability–specificity gap. REVIEWSCORE criteria (unanswerability of questions, factuality of premises) are argued to be specific enough for reliable annotation (as evidenced by the moderate inter-annotator agreement achieved despite the task difficulty) while applicable to the full range of review content—questions, simple claims, and complex arguments alike.
It shifts the evaluation unit from sentence to premise. By reconstructing arguments and evaluating factuality at the premise level, the paper addresses the limitation of sentence-level approaches (Du et al., 2024) that miss logical structure. This is not just a different granularity—it is a different type of evaluation, one that aligns with how arguments actually function (premises collectively entail conclusions, and the incorrectness of any premise can undermine the whole argument).
It separates the problem into two stages: reconstruction, then evaluation. By building an explicit reconstruction engine and then defining ADVANCED REVIEWSCORE as an aggregation over reconstructed premises, the paper modularizes what would otherwise be a single end-to-end judgment task. This modularity is what enables the key empirical finding: premise-level evaluation produces substantially higher human-model agreement than weakness-level evaluation (3.40× higher F1 for ArgScore vs. Base ReviewScore with Gemini 2.5 Flash), because reconstruction forces the model to make its logical commitments explicit before evaluating them.
It provides a benchmark, not (yet) a solution. The paper is careful to frame its LLM evaluation results as a measurement of the current capability gap, not as a deployment-ready system. The moderate F1 scores (0.4–0.5) and kappa scores (0.3–0.4) are presented as evidence that the task is genuinely hard and that current models are not yet reliable enough for fully automated review screening. This is a responsible positioning: define the problem, build the dataset, baseline the models, and identify what's still failing.

3. Technical Approach

3.1 Reader Orientation

REVIEWSCORE is a two-layer evaluation framework that detects when peer review points are misinformed—either because a question is already answered by the submitted paper, or because a weakness rests on factually incorrect premises. The core idea is that evaluating review quality requires more than checking whether criticisms reference the paper (groundedness); it requires checking whether the criticisms are actually correct (factuality), and for argumentative weaknesses, this means first reconstructing the full logical structure of the argument—including implicit premises—before evaluating each premise individually against the paper.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

Review Point Decomposition — an LLM breaks each review into independent review points (self-contained units of evaluation or inquiry), each classified as a question, claim (weakness without supporting reasons), or argument (weakness with supporting reasons).
Automatic Argument Reconstruction Engine — for argumentative weaknesses, a feedback-loop system extracts all explicit and implicit premises and a conclusion. It uses an LLM to propose reconstructions, a SAT solver to verify logical validity (premises must deductively entail the conclusion), and an LLM judge to verify faithfulness (the reconstruction must accurately represent the original argument). Invalid or unfaithful reconstructions trigger regeneration with feedback.
BASE REVIEWSCORE — a direct 5-point scoring of each review point: Factuality for weaknesses (claims and arguments) and Unanswerability for questions. This is the simpler, more naive evaluation that the paper shows suffers from annotator divergence.
ADVANCED REVIEWSCORE — for arguments, the reconstruction engine first extracts premises; then each premise's factuality is scored individually on a 5-point scale (Factuality) and optionally weighted by its untrivialness (a 0–2 scale measuring whether the premise is trivially true based on common knowledge of CS/AI undergraduates). These premise-level scores are aggregated into a single ArgScore using either logical conjunction (an argument is true iff all premises are true) or weighted average.
LLM Evaluation Pipeline — given a submitted paper (text and tables only, parsed from PDF) as the knowledge base, an LLM evaluates each review point's REVIEWSCORE according to the defined rubrics, selecting an appropriate knowledge source (the paper, annotator knowledge, or referred papers) for each premise.

Information flows as follows: a review enters the system → an LLM decomposes it into review points and classifies each type → for arguments, the reconstruction engine extracts premises and a conclusion → human annotators (for dataset construction) or LLMs (for automatic evaluation) score the factuality of claims, the factuality of individual premises, or the unanswerability of questions → aggregation (for arguments) produces the final REVIEWSCORE.

3.3 Roadmap for the Deep Dive

First, the formal definitions of BASE REVIEWSCORE and ADVANCED REVIEWSCORE (Definitions 3 and 4), since everything downstream depends on understanding what these scores represent and how they differ.
Second, the aggregation methods (logical conjunction and weighted average with untrivialness), which are the bridge between premise-level scores and argument-level scores in ADVANCED REVIEWSCORE.
Third, the automatic argument reconstruction engine (Section 3.2), which is the mechanism that makes ADVANCED REVIEWSCORE possible—without it, premise-level evaluation cannot happen for arguments.
Fourth, the human annotation process and dataset construction (Section 3.3), which provides the ground truth against which LLM evaluation is measured and includes the cross-validation and disagreement-resolution protocols needed for trustworthy labels.
Fifth, the insights from the initial group discussion (Appendix B) that motivated key design decisions—why certain criteria were chosen and why premise-level evaluation was developed.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a definitional and system-building paper whose core idea is that review quality can be measured by checking the factuality of underlying premises rather than holistic judgments, and that doing so requires a rigorous argument reconstruction pipeline to make implicit reasoning explicit.

BASE REVIEWSCORE: Direct Factuality and Unanswerability Scoring

The paper begins with the simpler formulation, BASE REVIEWSCORE, defined in Definition 3. The goal is to directly apply the concept of a misinformed review point (Definition 2) as a 5-point scale.

For weaknesses (WScore). A weakness $x \in W$ about a submitted paper $S$ is assigned a factuality score:

$\text{Factuality}_S(x) \in \{1, 2, 3, 4, 5\}$

The mapping is: scores 1–2 indicate "Misinformed" (the weakness is incorrect or contains incorrect premises), while scores 3–5 indicate "Not misinformed" for binary classification purposes. The rubric is not fully detailed in the main text (it is in Appendix H), but the essential logic is that a score of 1 ("Definitely False") means the weakness is clearly contradicted by the paper, while a score of 5 ("Definitely True") means the weakness is fully supported.

For questions (QScore). A question $x \in Q$ about paper $S$ is assigned an unanswerability score:

$\text{Unanswerability}_S(x) \in \{1, 2, 3, 4, 5\}$

Here, the scale runs in the opposite direction from factuality: score 1 means "Definitely Answerable by the paper" (the question is misinformed—the answer is already there), while score 5 means "Definitely Unanswerable by the paper" (the question is valid—the paper does not address it). For binary classification, scores 1–2 are again "Misinformed" and 3–5 "Not misinformed." This inverted scale for questions is important because it means "low score = misinformed" is consistent across both weakness and question types.

The formal definition. Base ReviewScore is expressed as a piecewise function:

$\text{ReviewScore}_{\text{base}}(x) = \begin{cases} \text{Factuality}_S(x) & \text{if } x \in W \\ \text{Unanswerability}_S(x) & \text{if } x \in Q \end{cases}$

where $W$ is the set of weaknesses and $Q$ is the set of questions in a review of paper $S$ .

What it computes: For any review point, base REVIEWSCORE returns a single integer from 1 to 5 indicating how misinformed it is. A low score (1–2) means the review point is problematic—either the weakness is factually wrong, or the question is already answered. A high score (3–5) means the review point is acceptable—either the weakness is factually correct, or the question genuinely requires clarification beyond what the paper provides.

Why this form: The 5-point scale with a mid-point (3 = "No Verdict") is chosen to accommodate uncertainty. Human annotators can express degrees of confidence rather than forced binary decisions, which is important because evaluating whether a complex technical criticism is "correct" often involves borderline cases. The piecewise definition separates the two different underlying evaluation dimensions (factuality for weaknesses, unanswerability for questions) while keeping the same numerical range, enabling a unified treatment of all review points.

The problem with BASE REVIEWSCORE. During the initial group discussion (Section 1 and Appendix B), the authors discovered that human annotators' factuality judgments on weaknesses frequently diverge. The cause is structural:

"a weakness often contains both correct and incorrect explicit premises or incorrect implicit premises, which hinders humans to reliably annotate factuality"

In other words, when a reviewer writes a criticism like "The method lacks novelty because it merely combines existing techniques A and B," the human evaluator must simultaneously decide: (1) does the paper actually combine techniques A and B? (2) does combining existing techniques constitute a lack of novelty? (3) is the reviewer implicitly assuming that all combinations of existing techniques are non-novel? Different annotators weigh these sub-questions differently when assigning a single holistic score, producing disagreement. This observation directly motivates the transition from BASE to ADVANCED REVIEWSCORE.

ADVANCED REVIEWSCORE: Premise-Level Factuality with Aggregation

To resolve the ambiguity in holistic weakness scoring, the paper introduces ADVANCED REVIEWSCORE, defined in Definition 4. The key structural change is that weaknesses are split into two subtypes—claims (weaknesses with no supporting reasons) and arguments (weaknesses with supporting reasons)—and arguments are decomposed before scoring.

The categorization. Let $C$ be the set of claims, $A$ be the set of arguments, and $Q$ be the set of questions in a review of paper $S$ . The distinction between claims and arguments is based on the presence of supporting reasons: a claim is a bare assertion about the paper ("The experimental results are insufficient"), while an argument includes explicit or implicit reasoning connecting premises to a conclusion ("The experimental results are insufficient because the authors only test on one dataset").

Argument reconstruction. For an argument $x \in A$ , let its conclusion be $C$ and its explicit and implicit premises be:

$P(x) = \{p_1, p_2, \ldots, p_k\}$

with the requirement that the premises deductively entail the conclusion, denoted $\{p_1, \ldots, p_k\} \vdash C$ . This is a formal logical requirement: the set of premises must be sufficient for the conclusion to follow necessarily. The reconstruction process that produces this set is the automatic argument reconstruction engine (Section 3.2, described in detail below).

Knowledge base selection. The paper defines a set of available knowledge bases $K = \{S, \text{annotator\_knowledge}, \text{referred\_papers}\}$ . A premise's factuality is evaluated against a specific knowledge base chosen by a selector function $\text{KB}: U \to K$ , where $U = C \cup \bigcup_{x \in A} P(x)$ is the set of all claims and premises. The rules are:

For claims $x \in C$ : the knowledge base is always the submitted paper $S$ ( $\text{KB}(x) = S$ ). Claims are assertions about the paper, so the paper is the only relevant authority.
For premises $p_i$ of an argument: the knowledge base can be any of $S$ , annotator knowledge, or referred papers, selected per-premise. A premise like "Transformer models use self-attention" would be evaluated against annotator knowledge (common knowledge before the paper's publication year), while a premise like "The paper claims a 3% improvement over baseline" would be evaluated against the submitted paper.

The factuality function. With the knowledge base selected, the factuality of each premise $p_i$ is scored:

$\text{Factuality}_{\text{KB}(p_i)}(p_i) \in \{1, 2, 3, 4, 5\}$

where the subscript $\text{KB}(p_i)$ indicates which knowledge base is used. This is the same 5-point scale as BASE REVIEWSCORE (1 = Definitely False, 5 = Definitely True, 3 = No Verdict). Claims similarly receive factuality scores $\text{Factuality}_S(x)$ against the paper.

The aggregation step. The ADVANCED REVIEWSCORE for an argument is defined as:

$\text{ReviewScore}_{\text{adv}}(x) = \text{Agg}\left(\text{Factuality}_{\text{KB}_i}(p_i) \mid p_i \in P(x)\right)$

where $\text{Agg}$ is an aggregation operator that combines multiple premise-level scores into a single argument-level score. For claims, the ADVANCED REVIEWSCORE is identical to the claim's factuality score $\text{Factuality}_S(x)$ . For questions, it is identical to BASE REVIEWSCORE's $\text{Unanswerability}_S(x)$ .

What it computes: ADVANCED REVIEWSCORE decomposes the problem of evaluating an argument into (1) extracting all premises, (2) scoring each premise's factuality against the appropriate knowledge base, and (3) combining those scores via an aggregation rule. The key computational difference from BASE REVIEWSCORE is that the intermediate step—premise-level scoring—is made explicit, which forces evaluators to surface their reasoning about which specific claims within the weakness are correct or incorrect.

Why this form: The premise-conclusion decomposition mirrors how arguments are analyzed in logic and critical thinking. By making premises explicit, the evaluation becomes more transparent and less subjective: two annotators may disagree on the overall strength of a weakness, but they can often agree on whether individual premises are true or false. The aggregation step then combines these more reliable atomic judgments into an overall score. This is why ADVANCED REVIEWSCORE is expected to show higher human-human and human-model agreement than BASE REVIEWSCORE—and the paper's empirical results (Table 3) confirm this, with 3.40× higher F1 for ArgScore vs. Base with Gemini 2.5 Flash.

The subtlety of the selector function. A key design choice is that the knowledge base for each premise is selected per-premise, not globally. This is critical because arguments about papers often mix premises that depend on the paper's content with premises that depend on general domain knowledge. For example, in an argument like "The method cannot handle long sequences because the attention complexity is quadratic," the premise "attention complexity is quadratic" is common knowledge (evaluable against annotator knowledge), while "the method uses attention" depends on the paper. Without per-premise knowledge base selection, an evaluator might incorrectly judge the domain-knowledge premise against the paper (the paper may not discuss quadratic complexity at all) and conclude it's unsupported, when in fact it's trivially true background knowledge.

Aggregation Methods: Logical Conjunction and Weighted Average

The paper proposes two aggregation operators $\text{Agg}$ for combining premise-level factuality scores into an argument-level ArgScore. Both are motivated by the literature on logical argument evaluation, but they address different practical concerns.

Logical conjunction aggregation. Following the standard definition in logic (Beall et al., 2024), an argument is true if and only if all of its premises are true. The paper operationalizes this for the binary classification setup:

A premise is defined as "true" if its factuality score is 3–5 (Not misinformed)
A premise is defined as "false" if its factuality score is 1–2 (Misinformed)
The argument is "true" iff all premises are true (i.e., no premise has score 1–2)
The argument is "false" otherwise (i.e., at least one premise is misinformed)

The logical conjunction produces a binary output (true/false) rather than a 5-point scale. In the binary classification evaluation (F1 scores, Table 3), this is the aggregation method used.

What it computes: For an argument with premises $\{p_1, ..., p_k\}$ , the aggregation checks whether every premise's factuality score is ≥ 3. If yes, the argument is "Not misinformed." If any premise is misinformed (score 1–2), the entire argument is "Misinformed."

Why this form: This aggregation captures the logical principle that an argument whose conclusion follows deductively from its premises can be no stronger than its weakest premise. If even one premise is false, the entire argument is unsound, regardless of how strong the other premises are. This is the correct normative standard for deductive arguments.

The fragility problem. Logical conjunction has a critical practical weakness: if a human annotator mistakenly labels a single premise as false (when it is actually true), or if an LLM makes a single premise-level error, that error propagates to the entire argument. Since arguments can have many premises (the dataset contains 1,748 premises across 422 arguments, averaging ~4.1 premises per argument), and each premise-level judgment has some probability of error, the probability that at least one premise is misjudged grows quickly with the number of premises. This makes the argument-level score brittle.

Weighted average aggregation. To mitigate the fragility of logical conjunction, the paper introduces a second aggregation method that maintains the 5-point scale:

$\text{Agg}_{\text{weighted}}\left(\text{Factuality}_{\text{KB}_i}(p_i) \mid p_i \in P(x)\right) = \frac{\sum_{i} w_i \cdot \text{Factuality}_{\text{KB}_i}(p_i)}{\sum_{i} w_i}$

where the weight $w_i$ is the untrivialness of premise $p_i$ .

Untrivialness scoring. Each premise is assigned an untrivialness score on a 0–2 scale (rubric in Appendix H, Figure 17):

0 (Definitely Trivial): The premise is trivially true based on common knowledge of CS/AI-majoring undergraduate students before the paper's publication year. For example, "Transformer models use self-attention mechanisms" would be trivially true—it's basic knowledge in the field.
1 (Probably Trivial): The premise is likely true based on common knowledge, but there may be some nuance.
2 (Definitely Not Trivial): The premise is not common knowledge and requires specific verification against the paper or other sources.

What untrivialness does in the weighted average: Premises with untrivialness 0 receive weight 0 in the aggregation—they are effectively dropped because they contribute no discriminating information (everyone agrees they're true). Premises with untrivialness 2 receive full weight. The idea is that trivial premises (like "if the method is novel, then the method is novel"—a tautology often added by the reconstruction engine to make arguments valid) should not influence the final score, because they are always true and never misinformed. Only substantive premises that could actually be wrong should drive the argument-level evaluation.

Why weight by untrivialness and not importance: The paper acknowledges that the ideal weighting would be by importance of each premise to the argument (how central it is to the criticism), but "it is difficult to measure the importance of premises." Untrivialness serves as a computationally simpler proxy: trivially true premises are typically logical scaffolding (conditional premises needed to make the argument formally valid) rather than substantive claims, so filtering them out approximates weighting by contribution to the actual criticism. However, the paper notes that this approximation has limitations—the manual analysis of human-model disagreements found that "humans and models have slightly misaligned criteria on whether a given premise is trivially true or not," which limits the benefits of the weighted average compared to logical conjunction.

The conditional premise handling. A subtle rule in the untrivialness scoring applies to conditional premises (premises of the form "If A, then B"). For such premises, the evaluator must assume the antecedent A is true and then judge whether the consequent B is trivially true. This is described in the prompt (Appendix H, Figure 20):

"For premises that are conditionals (If A then B), you must assume that the antecedent (A) is true and judge if the consequent (B) is trivially true or not."

This rule prevents circular reasoning: without it, an evaluator might judge "If A then B" as trivially false because A is false in the real world, when the logical structure of the argument only requires that B follows from A (which could be trivially true even if A is false).

The Automatic Argument Reconstruction Engine

This is the most technically complex component of the paper. Without it, ADVANCED REVIEWSCORE cannot be evaluated, because premises must be extracted from arguments before their factuality can be scored. The engine is described in Section 3.2 and detailed in Appendix C.

The core challenge. Given an argumentative review point (a paragraph of natural language criticism) and the full submitted paper (potentially 8+ pages of technical content), the engine must produce a valid and faithful premise-conclusion structure. "Valid" means the premises deductively entail the conclusion (no logical gaps). "Faithful" means the reconstruction accurately and completely captures the original argument's intended meaning, including implicit premises that the reviewer assumed but did not state.

Why direct LLM prompting fails. The paper first attempted to have an LLM directly reconstruct arguments given detailed instructions and the paper as context. This approach "mostly fails to generate valid and faithful reconstructions" (Section 3.2). The quantitative results in Appendix C (Table 5) quantify this: without feedback, the direct reconstruction achieves only 0.895 validity (about 10% of reconstructions are logically invalid) and 2.91/5 faithfulness (substantial deviations from the original argument), with only 36.9% of attempts producing a reconstruction that passes both criteria.

The failure modes, as illustrated in the qualitative examples (Appendix C, Figures 5–7), include: missing intermediate conclusions that bridge premises to the final conclusion (invalid), producing conclusions that are stronger or weaker than the original argument states (unfaithful), and including extra premises that are not part of the reviewer's actual reasoning.

The feedback-loop architecture (Figure 2a). To address these failures, the engine wraps the LLM-based reconstruction in two sequential feedback loops, illustrated schematically:

Step 1: Extract verbatim conclusion and reasons. Before attempting reconstruction, an LLM extracts the verbatim conjecture (main claim) and verbatim reason statements from the review point text (Figure 23 in Appendix H). This step isolates the explicit components of the argument from surrounding text (hedges, elaborations, transitional phrases) to provide clean input to the reconstruction step.

Step 2: Generate reconstruction with formalization. Given the verbatim conclusion, verbatim reasons, and the submitted paper, an LLM proposes a set of premises, intermediate conclusions, and a final conclusion. Critically, in the same inference call, the LLM also formalizes the reconstruction into First-Order Logic (FOL) formulas with defined variables and predicates. For example, the natural language premise "The paper claims avoiding inductive biases in transformers is an advantage" might be formalized as $B$ with a key mapping $B$ to that natural language statement. The LLM also generates a deductive proof showing how the formalized premises imply the formalized conclusion (Figure 24 in Appendix H).

What the formalization enables: By translating natural language argument components into FOL, the system creates a representation that a SAT solver can mechanically check for logical validity. This bypasses the known limitation that LLMs often cannot reliably verify logical entailment from natural language alone—they may generate plausible-sounding but logically flawed proofs.

Step 3: Validity check with SAT solver. An LLM extracts the necessary formalized premises from the full set (removing redundant ones), writes a Python program using the Z3 SMT solver (De Moura and Bjørner, 2008), and checks whether the premises deductively imply the conclusion. The Python program (code snippet in Figure 26, Appendix H) uses Z3 to check unsatisfiability of $\text{premises} \land \neg\text{conclusion}$ —if this conjunction is unsatisfiable, the argument is valid. The program also finds the minimal subset of premises needed for the proof.

Additionally, the LLM checks for circularity: whether the conclusion itself appears as one of the premises (a trivial but invalid "proof"). If the conclusion is among the necessary premises, the reconstruction is flagged as circular.

Step 4: Validity feedback. If the reconstruction is invalid (the SAT solver finds a counterexample) or circular, a rule-based natural language feedback signal is sent back to Step 2, and the LLM regenerates the reconstruction. There are two types of signals: a generic "the formalized premises do not imply the conclusion" message, and a specific "the proof is circular" message if the conclusion appears as a premise.

Step 5: Logical streamlining (deformalization). If the reconstruction is valid, the LLM translates the formalized premises (FOL formulas with keys) back into natural language (Figure 28 in Appendix H). This process, called logical streamlining in the critical thinking literature (Bowell and Kemp, 2014; Brun and Betz, 2016; Betz and Richardson, 2021), rephrases the premises to make their logico-semantic structure more transparent. For example, a formalized conditional premise $B \to A$ with key $B$ : "The paper claims X" and $A$ : "X is true" might be streamlined to "If the paper claims X, then X is true." This streamlining exposes the logical commitments that were implicit in the reviewer's argument.

Why streamlining matters: The formalized representation is machine-checkable but human-unreadable. The streamlined natural language version is what human annotators actually evaluate for faithfulness (Step 6) and what the LLM evaluates for premise factuality (in the ADVANCED REVIEWSCORE scoring). Streamlining ensures that what gets evaluated is both logically sound and semantically transparent.

Step 6: Faithfulness check. Given the original argumentative review point and the streamlined natural language premises and conclusion, an LLM judges whether the reconstruction is faithful (Figure 29 in Appendix H). The faithfulness criteria are explicitly defined:

Accuracy & Charity: The reconstruction should keep the author's intended meaning while eliminating irrelevancies—obeying the principle of charity (preferring the strongest sensible reading of ambiguous passages).
Completeness: All explicit premises, the main conclusion, and any indispensable implicit premises must be included.

The LLM judge outputs a binary "Yes/No" decision with step-by-step justification.

Step 7: Faithfulness feedback. If the reconstruction is unfaithful, the LLM judge's justification is fed back to Step 2 as natural language feedback, and the LLM regenerates the reconstruction to address the identified faithfulness issues.

Step 8: Termination. The loop continues (validity → streamlining → faithfulness → possible regeneration) until either (a) both validity and faithfulness are satisfied, or (b) 10 total iterations have been reached. If the loop terminates at the iteration limit without satisfying both criteria, the last reconstructed argument is returned. This hard limit prevents infinite loops when the LLM cannot produce a satisfactory reconstruction.

The series connection of the feedback loops. The validity and faithfulness checks are connected in series, not in parallel: only valid reconstructions are streamlined and checked for faithfulness. This design minimizes unnecessary LLM calls—an invalid reconstruction is rejected before the (more expensive) faithfulness evaluation occurs. The paper states this explicitly:

"To minimize model calls in practice, we connect these two loops in series so that only valid reconstructions are judged for their faithfulness."

Engine performance (Table 5). With Claude Sonnet 3.7 as the base LLM:

Validity with feedback: 1.00 (every reconstruction is logically valid by SAT solver verification)
Faithfulness with feedback: 4.47/5 (average score by human annotators)
Pass rate with feedback: 0.935 (93.5% of arguments produce a reconstruction that passes both criteria within the iteration limit)
Average iteration count: 3.09 ± 2.73 (the feedback loops are actively used, with typical arguments requiring 2–4 iterations)

Without the feedback loops, the same base model achieves only 0.895 validity, 2.91/5 faithfulness, and 0.369 pass rate. The feedback mechanism is essential, not incremental.

The base model dependence. The reconstruction quality depends on the capability of the underlying LLM. The paper notes (in Limitations) that upgrading the base model to Claude Sonnet 4 produces "considerable output quality improvement," suggesting the engine will improve as models advance. However, the core architecture—LLM proposing, SAT solver verifying, LLM judging faithfulness—is model-agnostic.

The cost tradeoff. The engine requires multiple LLM calls per argument (one call per iteration for reconstruction and formalization, plus calls for extraction, streamlining, and faithfulness checking). With an average of 3.09 iterations and multiple calls per iteration, the total cost per reconstructed argument is roughly 10–15 LLM calls. For the dataset construction, this cost is amortized across many arguments, but for real-time deployment it would be significant. The paper does not provide detailed cost analysis.

Representative example (Figure 2b). The paper provides a complete reconstruction example for an argument from a review of Dosovitskiy et al. (2020), the ViT paper. The original argument:

"The arguments about 'inductive biases' are confusing and self-contradictory. On one hand, the introduction section says that CNN generalize better due to the inductive biases such as translation equivariance and locality. On the other hand, the rest of the paper claims that avoid inserting inductive biases into the transformer is an advantage."

The reconstructed argument has 5 premises (P1–P5) and a conclusion (C):

P1: The paper claims avoiding inductive biases in transformers is an advantage.
P2: If the paper claims avoiding inductive biases in transformers is an advantage, then avoiding inductive biases is an advantage.
P3: Inductive biases are beneficial for vision tasks.
P4: If inductive biases are beneficial for vision tasks, then avoiding inductive biases is not an advantage.
P5: If avoiding inductive biases is an advantage and avoiding inductive biases is not an advantage, then the arguments about inductive biases in the paper are confusing and self-contradictory.
∴ C: The arguments about inductive biases in the paper are confusing and self-contradictory.

The formalization maps these to atomic propositions (A, B, C, D) and shows the deductive structure: P1 and P2 give A (avoiding inductive biases is an advantage); P3 and P4 give ¬A (avoiding inductive biases is not an advantage); P5 and the contradiction (A ∧ ¬A) give D (the paper's arguments are confusing). This example illustrates how the reconstruction makes explicit the implicit premise that "if the paper claims X, then X is an advantage" (P2), which the reviewer assumed but never stated. Whether P2 is factually correct is a separate question—the reconstruction's job is to surface it for evaluation.

Human Annotation Protocol and Dataset Construction

The human annotation process, described in Section 3.3 and detailed in Appendices D and E, is methodologically significant because the paper's empirical claims rest on the quality of these annotations. The paper takes several steps to ensure trustworthiness.

Annotator recruitment and paper selection. Fifteen graduate students studying AI were recruited as annotators. They were divided into five groups of three based on shared research interests. Each group selected eight papers from ICLR 2021–2023 that were relevant to all three members' expertise. The total dataset covers 40 papers (after excluding the 5-paper pilot study) and 155 reviews.

The annotators' expertise (Table 8, Appendix E.1) is substantial: an average of 3.93 publications in AI/ML per annotator (including arXiv preprints), and an average paper relevance score of 4.06/5, where 4 means "Strong relevance—adjacent/overlapping subtopic with regular use of the paper's methods or domain; can evaluate technical choices with minimal ramp-up." No annotator had relevance below 3 ("Moderate relevance"). However, there is inter-group variance: Group 2 averaged 5.33 publications and 4.38 relevance, while Group 3 averaged 1.33 publications and 3.96 relevance.

The annotation pipeline. The process is collaborative between LLMs (for preprocessing) and humans (for verification and scoring):

Review point extraction: An LLM decomposes each review into independent review points (Definition 1). Each review point is a self-contained unit—a single criticism or question that stands on its own semantically.
Type classification: The LLM classifies each review point as a claim, argument, or question. Humans verify this classification.
For questions: The human scores unanswerability on the 5-point scale (1 = Definitely Answerable, 5 = Definitely Unanswerable) and provides justification if needed.
For claims: The human scores factuality on the 5-point scale (1 = Definitely False, 5 = Definitely True) and provides justification if needed. This is the BASE REVIEWSCORE for claims.
For arguments (BASE REVIEWSCORE): The human scores factuality on the same 5-point scale, same as claims. This provides the baseline for comparison with ADVANCED REVIEWSCORE.
For arguments (ADVANCED REVIEWSCORE): The automatic argument reconstruction engine extracts premises. The human first scores reconstruction faithfulness on a 5-point scale (Figure 4, Appendix C.2: 5 = "Definitely faithful, no change is essential," 1 = "Definitely not faithful, more than two major errors exist"). If faithfulness is < 4, the argument is skipped for further annotation (the reconstruction quality is insufficient for reliable premise-level scoring). If faithfulness ≥ 4, the human proceeds to: (a) select a knowledge base for each premise (submitted paper, annotator knowledge, or referred papers), (b) score each premise's factuality on the 5-point scale, (c) score each premise's untrivialness on the 0–2 scale, and (d) justify any of these three decisions as needed.

Why faithfulness threshold of 4: The threshold means that human annotators only evaluate premise factuality when the reconstruction is "faithful, but one or two minor changes recommended" or better. This prevents premise-level scores from being contaminated by reconstruction errors—if the reconstruction misrepresents the argument, premise scores would be meaningless.

Inter-annotator agreement protocol. To control label quality and reduce annotator bias, the paper uses a multi-stage process:

Stage 1: Initial independent annotation. All three annotators in a group independently annotate the same review points.

Stage 2: Disagreement-aware recheck. Instances where the maximum and minimum scores differ by ≥ 2 are flagged as disagreements. Each annotator independently rechecks these instances without seeing others' labels, revising only if they identify clear mistakes in their original annotation.

Stage 3: Group discussion for remaining disagreements. After recheck, if disagreements persist, the group discusses the instances to reach consensus. The final labels are typically the median of the three annotations, but for instances where post-discussion labels were exceptionally adopted, those labels are used instead.

Annotator support infrastructure. The paper emphasizes several structural supports to ensure quality:

Detailed guidelines: Provided to all annotators (the rubrics in Appendix H, Figures 15–17, are part of these).
One-hour online orientation session: Before annotation begins.
Access to materials: Annotators can use any related materials—OpenReview discussions between authors and reviewers, web search, etc.—to inform their judgments.
Active communication channel: A group chat where annotators can ask questions and receive instant responses, with answers shared to all annotators to "build a global consensus among human annotators."

Inter-annotator agreement results (Table 9, Appendix E.2). Despite the difficulty of the task, the paper achieves:

Median Krippendorff's Alpha across groups: 0.489 for overall REVIEWSCORE
Range: 0.254 (Group 5, lowest) to 0.663 (Group 2, highest)
QScore (unanswerability of questions): 0.554 median (highest agreement)
ArgScore (premise factuality for arguments): 0.457 median
ClaimScore (factuality of simple claims): 0.392 median (lowest agreement—claims are the most subjective)

The pattern is revealing: questions show the highest agreement because answerability is more objective (the answer either is or isn't in the paper at some reasonable level of detail). Arguments show moderate agreement because premise-level factuality against specific knowledge bases is checkable. Claims show the lowest agreement because bare assertions (without supporting reasoning) are inherently more ambiguous—human evaluators bring different interpretations of what the claim means.

The inter-group variance and its cause. The paper's disagreement analysis identifies that "most disagreements come from human annotators with low paper relevance (i.e., anno_11 and anno_51)," the annotators with the lowest paper relevance scores in their groups. This is an important validity check: it suggests that the annotations are more reliable when annotators have strong domain expertise, and that the lower agreement in some groups is partly attributable to annotator-paper mismatch rather than fundamental ambiguity in the task. However, it also means the annotations are not perfectly reliable—there is irreducible noise from varying expertise levels.

Dataset scale (Table 6, Appendix D.3). The final dataset (main + pilot) contains:

45 papers (5 pilot + 40 main)
174 reviews
657 review points (84 pilot + 573 main)
422 arguments containing 1,748 premises

The dataset is modest by modern ML standards but substantial for a task requiring expert human annotation of domain-specific logical content. Each argument's premises, each labeled for factuality and untrivialness against a specific knowledge base, represents minutes to tens of minutes of expert time.

The pilot study. Before the main annotation, a 5-paper pilot was conducted with three annotators (Appendix D.2). One annotator was an author on one paper (self-evaluation), while the other two non-authored papers were common across annotators. This pilot served to calibrate the annotation guidelines, identify ambiguities in the rubrics, and estimate the time and difficulty of the task before scaling to 40 papers and 15 annotators.

Design Rationale: Why Premise-Level Evaluation

The transition from BASE to ADVANCED REVIEWSCORE is the paper's central technical contribution, and understanding why it works requires connecting three pieces of evidence:

The initial group discussion insight (Appendix B). When three graduate students independently analyzed reviews and justified their quality scores, two observations emerged:

Review points often contain mixed content—some parts are factually correct while others are misinformed. Human annotators struggled with "which parts in a review point they should weigh more to evaluate the review point," directly motivating the need for premise-level decomposition and explicit aggregation rules.
Only two quality criteria showed consistent multi-annotator agreement: unanswerable questions and misunderstood (factually incorrect) criticisms. Three other criteria—out-of-scope remarks, observations pointing out minor details, and unclear points—showed frequent single-annotator disagreement (one annotator flagged it, others did not). This empirical finding justifies the paper's focus on the two criteria that operationalize REVIEWSCORE.

The human-model agreement results (Table 3). ADVANCED REVIEWSCORE for arguments (using premise-level factuality with aggregation) substantially outperforms BASE REVIEWSCORE (weakness-level factuality). With Gemini 2.5 Flash, the improvement is 3.40× in F1 (0.137 → 0.466) and 3.24× in Kappa (0.124 → 0.402). The gap is consistent across models and metrics. Even without aggregation (just premise-level scores), ADVANCED REVIEWSCORE outperforms BASE REVIEWSCORE (F1: 0.395 vs. 0.137 for Gemini 2.5 Flash), confirming that the decomposition itself is beneficial—it forces the model to evaluate specific factual claims rather than making holistic judgments that are more susceptible to misinterpretation.

The disagreement analysis (Figure 3). The primary cause of human-model disagreement on arguments is "models' misunderstanding or incorrect reasoning" (53% of disagreements) and "models' predictions which correspond to minority of human-annotated scores" (37%). These are concentrated in the model's ability to correctly interpret and evaluate premises, not in the reconstruction step. By isolating the evaluation to individual premises, ADVANCED REVIEWSCORE makes these errors diagnosable and potentially fixable through improved premise-level evaluation, rather than being buried in an opaque holistic judgment.

The unsolved problem with claims. The paper's results expose a fundamental challenge: claims (bare assertions without supporting reasons) show substantially lower human-model agreement than arguments or questions. The disagreement analysis (Figure 3) shows that 36% of claim disagreements are caused by claims being "value-laden" or "lacking supporting reasons." Without the logical structure that premises provide, claims are inherently more subjective—"The experimental findings in this paper are somewhat trivial" can reasonably be judged differently by different evaluators depending on their standards for triviality. This suggests that the premise-level approach has a lower bound on applicability: it works for argumentative content but not for bare assertions, which remain a difficult problem.

Why not just improve models? A natural question is whether the gains from ADVANCED REVIEWSCORE could be achieved simply by using better LLMs. The paper's results suggest otherwise: even the strongest model tested (Claude Sonnet 4) achieves only 0.482 F1 and 0.341 Kappa for overall REVIEWSCORE, and the BASE-to-ADVANCED gap persists across all models. The decomposition is not compensating for weak models—it is addressing a fundamental ambiguity in the evaluation task that affects humans and models alike. When a human annotator scores a multi-premise weakness holistically, they must implicitly decide which premises matter most and how to weigh conflicting signals (one premise true, another false). Different humans do this differently. ADVANCED REVIEWSCORE makes these weighting decisions explicit and uniform through the aggregation rules, reducing a source of variance that better language modeling alone cannot fix.

4. Key Insights and Innovations

Innovation 1: The "Premise-Level Factuality" Diagnostic — Moving Review Evaluation from Holistic Judgment to Atomic Fact-Checking

The paper's most significant conceptual move is the insight that evaluating the quality of a critical review argument should not be done at the level of the argument itself, but at the level of its constituent premises. This sounds obvious in retrospect—of course an argument is only as strong as its premises—but prior work on automatic review evaluation had not operationalized it. Instead, the field operated under an implicit assumption that review quality could be meaningfully assessed through holistic properties: whether a review is "substantiated" (Goldberg et al., 2025), whether it is "grounded" in the paper (Guo et al., 2023; Sadallah et al., 2025; Ou et al., 2025), or whether individual sentences are factual (Du et al., 2024). Each of these prior approaches treats the review utterance as the unit of evaluation.

The paper's diagnostic is that this unit choice is the root cause of poor reliability—both for human annotators and for LLM judges. The evidence is direct: during the initial group discussion (Appendix B), human annotators discovered that they "sometimes struggle which parts in a review point they should weigh more to evaluate the review point." A weakness that mixes a true premise with a false premise produces divergent holistic scores because different annotators implicitly weight the premises differently. This is not a problem that better guidelines or more careful annotators can solve—it is structural. A holistic score on a multi-premise argument is always an underspecified function of the annotator's implicit weighting scheme, which varies across annotators, across time, and across arguments.

What makes this insight intellectually distinctive is that it reframes review evaluation from a judgment task to a decomposition-then-verification task. Rather than asking "is this criticism valid?", the framework asks "what are the atomic factual claims this criticism depends on, and which of them are true?" This decomposition makes the evaluation objective in a specific sense: two evaluators may still disagree on whether a particular premise is true, but they cannot disagree on what they are disagreeing about. The premise is an explicit, discrete claim whose truth conditions are defined relative to a specific knowledge base. This transparency is what enables the substantial improvement in human-model agreement for ADVANCED REVIEWSCORE over BASE REVIEWSCORE (3.40× higher F1 for ArgScore vs. Base with Gemini 2.5 Flash, Table 3). The model is not necessarily better at evaluating factuality—it is being asked an easier, more constrained question.

This reframing has implications beyond the specific task. It suggests that many "subjective" evaluation problems in NLP—annotating argument quality, evaluating explanation faithfulness, scoring summarization accuracy—might be productively decomposed into atomic factuality checks against source documents, with aggregation rules making the evaluation logic explicit rather than implicit.

Relation to prior work: Sentence-level factuality (Du et al., 2024) already moved toward finer granularity, but the paper argues this is insufficient because "a sentence-level factuality cannot fully capture correctness of an underlying logic." An argument can contain individually true sentences that collectively support a false conclusion through flawed logical structure, or an argument can contain a mix of true and false statements within a single sentence. The premise-level approach addresses both problems: reconstruction makes the logical structure explicit (so logical errors are visible) and atomic premises are individually verifiable (so mixed-content sentences are decomposed).

Evidence anchor: Table 3 shows that even without aggregation, ADVANCED REVIEWSCORE (premise-level scoring) substantially outperforms BASE REVIEWSCORE (weakness-level scoring). For Gemini 2.5 Flash, premise-level F1 is 0.395 vs. 0.137 for BASE (2.88× improvement). With logical conjunction aggregation, F1 rises further to 0.466 (3.40× vs. BASE). This confirms that the decomposition itself—not just the aggregation rule—provides most of the benefit.

Limitation: The approach requires that weaknesses have reconstructable logical structure. For bare claims (assertions without supporting reasons), no decomposition is possible, and the paper's results show these remain problematic—ClaimScore shows the lowest human-model agreement across all review point types (F1 scores as low as 0.0 for GPT-4o and DeepSeek-V3, Table 2). The premise-level approach thus has a domain of applicability: it works for argumentative content but not for value judgments or underspecified criticisms.

Innovation 2: The Argument Reconstruction Engine as a Bridge Between Natural Language Criticism and Formal Verification

The paper's second conceptual contribution is the dual-loop argument reconstruction architecture that combines LLM generation with SAT solver verification to produce logically valid and semantically faithful premise-conclusion structures from unpolished, domain-specific review text. This is not merely an engineering solution to a preprocessing problem—it represents a principled approach to a challenge that the argument mining and computational argumentation communities have grappled with: how do you extract the logical structure of real-world arguments when the arguments are informal, implicit, and require domain context to interpret?

Prior computational work on argument reconstruction (Betz and Richardson, 2021) trained sequence-to-sequence models on synthetic or polished argument pairs, where the input argument was already structured and the reconstruction did not require external context. The paper's setting is fundamentally harder: review arguments are written by real reviewers under time pressure, they contain domain-specific technical content that cannot be understood without the full paper, they often omit premises that the reviewer considers obvious, and their logical structure is entangled with hedging language, asides, and evaluative commentary.

The insight that makes the proposed architecture work is the separation of validity from faithfulness. An LLM, left to its own devices, will produce reconstructions that are internally plausible but may be logically invalid (the premises do not actually entail the conclusion) or semantically unfaithful (the reconstruction changes the meaning of the original argument). By routing validity through a SAT solver—a tool that is perfectly reliable for logical entailment but cannot handle natural language—and faithfulness through an LLM judge—a tool that understands semantics but is unreliable for logic—the architecture exploits the complementary strengths of formal methods and neural models. The LLM's job is to propose candidate reconstructions in natural language and to translate between natural language and formal logic; the SAT solver's job is to verify logical entailment without error.

This architecture is conceptually significant because it points toward a general pattern for AI systems that need to perform rigorous reasoning over unstructured text: use neural models for generation and interpretation (where flexibility and semantic understanding are needed), and use formal methods for verification (where correctness guarantees are required). The paper does not frame it this way, but the reconstruction engine is essentially a generate-and-verify loop with formal verification—a pattern that has become prominent in code generation (AlphaCode, CodeRL) and mathematical reasoning (Lean-based systems) but had not been applied to argument analysis from peer review text.

Relation to prior work: The Betz and Richardson (2021) DeepA2 framework is the closest prior work, training a T5 model for argument reconstruction and formalization. However, DeepA2 used synthetic and polished training data where arguments were already structured, and the reconstruction task did not require access to external documents (the arguments were self-contained). The paper's reconstruction engine is the first to handle real-world, unpolished arguments that require full paper context for interpretation, and the first to use iterative feedback loops with SAT solver verification to ensure validity.

Evidence anchor: Table 5 (Appendix C) quantifies the contribution of the feedback loops. Without feedback, direct LLM reconstruction achieves only 0.895 validity (SAT solver verification) and 2.91/5 faithfulness (human evaluation), with only 36.9% of attempts producing a valid and faithful reconstruction within a single generation. With feedback, validity reaches 1.00 (perfect), faithfulness reaches 4.47/5, and pass rate reaches 93.5%, using an average of 3.09 iterations. The feedback loops are not a minor refinement—they are essential for the engine to work at all on this data.

Limitation: The reconstruction quality depends on the underlying LLM's capability. The paper notes that upgrading from Claude Sonnet 3.7 to Claude Sonnet 4 produces "considerable output quality improvement," indicating the engine is not a complete solution but rather an architecture that amplifies the base model's strengths. Furthermore, the reconstruction engine does not handle arguments that are inherently non-deductive (e.g., inductive arguments, analogical reasoning), which limits its applicability to review points whose logical structure can be formalized as deductive entailment.

Innovation 3: The "Misinformed Review Point" as a Unifying Concept — Defining a New Category of Review Quality That Is Simultaneously Specific and Broadly Applicable

The paper identifies and formalizes a category of review error—the misinformed review point—that fills a specific gap in prior work on review quality evaluation. Before this paper, the landscape of automatic review evaluation (summarized in Table 1) was split between approaches that were broad but subjective (Goldberg et al., 2025; Du et al., 2024: criteria like "substantiation" and "understanding" that apply widely but are hard to operationalize reliably) and approaches that were specific but narrow (Guo et al., 2023; Sadallah et al., 2025; Ou et al., 2025; Purkayastha et al., 2025: criteria like groundedness and claim-matching that are checkable but capture only a subset of review errors).

The misinformed review point concept navigates this trade-off by combining two individually checkable properties—answerability of questions and factuality of premises—under a single framework that covers the main content types in real reviews (questions, simple claims, and complex arguments). The key insight is that both types of error are failures of the reviewer to correctly process the paper's content, just in different directions: an unanswerable question is a failure to find information that is present in the paper; an incorrect premise is a failure to correctly interpret information that is present in the paper. Both are "misinformed" in the sense that the reviewer's review point would change if they had correctly processed the paper.

What makes this concept intellectually distinctive is that it connects review quality evaluation to the specific cognitive failures that reviewer guidelines already warn against. The paper explicitly cites the ACL 2023 guideline: "before writing a negative review, check whether your questions are already answered," and the NeurIPS 2025 guideline: "reviewers should minimize the chance of misunderstandings." These guidelines reflect a community consensus that answerable questions and factual misunderstandings are quality problems, but prior to this work there was no systematic way to detect them. REVIEWSCORE operationalizes existing community norms rather than inventing new ones, which makes it more likely to be adopted and less likely to be contested on normative grounds.

The empirical finding that 26.4% of questions and 15.2% of weaknesses are misinformed is itself a conceptual contribution: it quantifies a problem that the community has discussed anecdotally but never measured systematically. These numbers provide a baseline against which future improvements (in reviewer training, in review screening systems, in LLM-assisted review writing) can be measured.

Relation to prior work: The groundedness-focused approaches (Guo et al., 2023; Sadallah et al., 2025; Ou et al., 2025) detect whether a reviewer's claims reference the paper, but not whether the referenced content is correctly interpreted. A reviewer could write "The paper claims to achieve 95% accuracy on benchmark X" (grounded in the paper) when the paper actually reports 59% accuracy (misinformed—the critic misread the number). Groundedness checks would pass this review point; REVIEWSCORE would flag it as misinformed. The paper's concept thus captures a different and complementary dimension of quality.

Evidence anchor: The human annotation results (Section 3.3, Appendix D) establish the prevalence of misinformed review points: 15.2% of weaknesses and 26.4% of questions across ICLR 2021–2023 reviews on 40 papers. The inter-annotator agreement analysis (Table 9) shows that QScore (unanswerability of questions) achieves the highest agreement (median Krippendorff's Alpha 0.554), while ClaimScore and ArgScore show moderate agreement (0.392 and 0.457 respectively), confirming that the criteria are sufficiently objective for reliable annotation, particularly for questions.

Caveat: The scope is limited to ICLR reviews in AI/ML, and the annotators are AI graduate students, not the original paper authors (who would be the ultimate authority on whether a reviewer misunderstanding is genuinely misinformed). The paper acknowledges this limitation implicitly by emphasizing that the task is difficult and that annotator expertise varies. The concept of a "misinformed review point" is well-defined, but its operationalization requires expert judgment that is not perfectly replicable.

Innovation 4: A Negative Result with Positive Implications — Establishing That Current SOTA LLMs Are Not Reliable Enough for Fully Automated Review Screening

The paper's fourth contribution is a carefully measured negative result: eight current state-of-the-art LLMs, including GPT-5 and Claude Sonnet 4, achieve only moderate agreement with human experts on REVIEWSCORE evaluation, with F1 scores of 0.4–0.5 and kappa scores of 0.3–0.4 (Table 2). In an era where LLM-as-a-judge papers often report high human-model correlations and claim near-human or superhuman performance, this paper's willingness to report and analyze a moderate-agreement result is itself an intellectual contribution.

What makes this result significant is not just the numbers but the disagreement analysis that explains why models fail (Figure 3, Section 4.4). The paper categorizes human-model disagreements into specific error types:

Misunderstanding or incorrect reasoning: 53% of argument disagreements, 40% of question disagreements, and 55% of claim disagreements fall into this category. Models literally fail to understand what reviewers mean or make logical errors when evaluating whether a premise is supported by the paper.
Minority score prediction: 37% of argument disagreements and 45% of question disagreements occur because the model predicts a score that corresponds to a minority human annotator's judgment. This is not necessarily an error—the model may be picking up on a legitimate interpretation that only one human annotator noticed—but it highlights that the task has irreducible ambiguity that models cannot resolve.
Value judgments and lack of supporting reasons: 36% of claim disagreements come from claims being "value-laden" or lacking supporting reasoning, making them inherently subjective.

This analysis transforms a "moderate agreement" finding from a disappointing performance number into a diagnostic map of where and why LLMs fall short on review evaluation. It tells future researchers: improving models' ability to understand nuanced technical criticism (not just retrieve facts from papers) is the bottleneck, and claims (bare assertions without supporting reasoning) may require fundamentally different approaches than factuality checking.

The paper's positioning of this result is also notable. Rather than concluding "LLMs cannot do this task," the paper concludes "fully automating the evaluation remains challenging" and uses the disagreement analysis to identify specific failure modes that future work should target. This is a model of how to report LLM evaluation results responsibly on hard judgment tasks where human agreement is itself imperfect.

Evidence anchor: Table 2 provides the aggregate results. The best-performing model (GPT-5) achieves 0.482 F1 and 0.341 Kappa for overall REVIEWSCORE. Figure 3 (disagreement type breakdown) provides the diagnostic detail. The QScore analysis (Appendix F, Figures 13–14) provides qualitative examples of specific model failures, such as Claude Sonnet 3.7 scoring a question as answerable (score 4) when the human consensus was unanswerable (final label 2), due to the model misunderstanding which aspects of the question the paper addresses.

Significance beyond performance: This result establishes REVIEWSCORE evaluation as a benchmark task that is not yet solved, providing a clear target for model improvement. If state-of-the-art LLMs had achieved 0.8+ F1, the paper would have been a successful engineering demonstration but would not have defined a research challenge. Instead, the moderate agreement numbers and the detailed failure analysis create a research agenda: building models that can (a) correctly interpret nuanced technical criticism, (b) distinguish between different possible interpretations of ambiguous review language, and (c) handle bare claims where factuality judgments are inherently subjective.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary dataset is the REVIEWSCORE dataset constructed by the authors, consisting of 657 annotated review points (143 questions, 92 claims, 422 arguments containing 1,748 premises) drawn from reviews of 40 papers submitted to ICLR 2021–2023, with an additional 5-paper pilot subset. The reviews are sourced from OpenReview. For the automatic evaluation experiments, the LLMs are provided only the main paper text and tables (parsed from PDF), excluding figures and appendices, as the knowledge base against which factuality and unanswerability are judged (Section 4.1, Limitations).
Base model(s). Eight state-of-the-art LLMs are evaluated as judges: five proprietary models—Claude Sonnet 3.7 (Anthropic, 2025a), Claude Sonnet 4 (Anthropic, 2025b), GPT-4o (OpenAI, 2024), GPT-5 (OpenAI, 2025), and Gemini 2.5 Flash (Gemini Team, Google, 2025)—and three open-sourced models—Qwen3-235B-A22B (Yang et al., 2025) with 235B parameters, DeepSeek-V3 (DeepSeek-AI et al., 2024) with 671B parameters, and Llama 3.3 (AI@Meta, 2024) with 70B parameters. Reasoning models are excluded because "REVIEWSCORE mostly depends on grounding and evidence, not longer or smarter chains of thought" (Section 4.1). Anthropic Claude models are used without thinking modes, and GPT-5 is used with minimal reasoning effort. All models are called with temperature 0 to ensure deterministic and reproducible outputs, following the LLM-as-a-judge best practices cited by the authors (Liang et al., 2022; Liu et al., 2023; Gu et al., 2024).
Metrics. The paper evaluates alignment between human-annotated and model-estimated REVIEWSCORE using two complementary metric sets corresponding to two problem formulations. For binary classification (misinformed = scores 1–2, not misinformed = scores 3–5): Precision, Recall, and F1 score, with F1 emphasized as the primary metric because it is robust to class imbalance (the majority of human-annotated labels are "Not misinformed"). For 5-point scale scoring: Quadratic Weighted Kappa (Warrens, 2012)—referred to as "Kappa" throughout—as the primary metric because it is robust to skewed data distributions where scores concentrate at 4 and 5; additionally, Pearson rank correlation and Gwet's AC2 (Gwet, 2001) are reported for comprehensiveness in Appendix F (Tables 10–13). For inter-annotator agreement among humans, Krippendorff's Alpha (Hughes, 2021) is used (Section 3.3, Appendix E.2).
Baselines. The paper's evaluation is primarily an absolute measurement of human-model agreement rather than a comparative benchmark against alternative methods. The key comparison is internal: BASE REVIEWSCORE (weakness-level factuality, Definition 3) versus ADVANCED REVIEWSCORE (premise-level factuality with aggregation, Definition 4) for arguments. There are no external baseline systems from prior work evaluated on this dataset, since the paper introduces both the task and the dataset. Within the analysis, the "w/o aggregation" condition (premise-level scores without aggregation) serves as an intermediate ablation point between BASE and ADVANCED REVIEWSCORE (Table 3).
Generation budget / compute accounting. No generation budget or compute accounting is reported in the traditional sense, since LLMs are used as zero-shot evaluators rather than for sampling-based search. The computational cost is measured implicitly through the number and size of LLM API calls: each review point evaluation requires one LLM call for scoring (with the full paper as context), and for arguments, the argument reconstruction engine requires additional calls (average 3.09 iterations of the reconstruction loop, plus extraction, streamlining, and faithfulness checking calls, estimated at roughly 10–15 LLM calls per argument in Section 3.2's analysis). The paper acknowledges this cost only in the Limitations section, noting that API calling costs motivated the exclusion of figures and appendices from the model input. No FLOPs, token counts, or dollar costs are provided.
Cross-validation / statistical protocol. The paper does not employ cross-validation for the LLM evaluation, since evaluation is zero-shot on a fixed dataset. For the human annotation protocol, the paper employs a multi-stage procedure to ensure label quality (Section 3.3, Appendix E): three annotators independently annotate each instance → instances with score range ≥ 2 are flagged for disagreement-aware recheck (each annotator independently revises if clear mistakes are identified, without seeing others' labels) → remaining disagreements are resolved through group discussion, with final labels typically taken as the median of three annotations (exceptionally adopting post-discussion labels). No statistical significance tests or confidence intervals are reported for the human-model agreement metrics, which is a notable omission given the modest test set size (657 total review points, with per-type counts as low as 92 claims).

Main Quantitative Results

The paper's experimental results are organized around three axes: (1) overall human-model agreement on REVIEWSCORE evaluation across review point types, (2) the comparison between BASE and ADVANCED REVIEWSCORE for arguments, and (3) diagnostic analyses of disagreement patterns and the effect of providing authors' responses.

Overall Human-Model Agreement on REVIEWSCORE Evaluation

Headline result (Table 2): Across all eight LLMs and all review point types combined, F1 scores for binary misinformed detection range from 0.383 (DeepSeek-V3) to 0.482 (GPT-5), and Quadratic Weighted Kappa scores for 5-point scale agreement range from 0.262 (Qwen3-235B-A22B) to 0.378 (Claude Sonnet 4 and Gemini 2.5 Flash). The paper characterizes this as "moderate agreement" and concludes that "fully automating the evaluation remains challenging."

Performance by review point type. A consistent pattern emerges across all models (Table 2):

QScore (unanswerability of questions) shows the highest human-model agreement. The best-performing model, Claude Sonnet 4, achieves F1 = 0.579 and Kappa = 0.425. The worst-performing model on QScore, GPT-4o, achieves F1 = 0.476 and Kappa = 0.291. This relatively higher agreement aligns with the inter-annotator agreement pattern (Table 9, Appendix E.2), where QScore showed the highest median Krippendorff's Alpha (0.554)—answerability by the paper is the most objectively checkable criterion.
ArgScore (premise factuality for arguments) shows intermediate agreement. GPT-5 achieves the highest F1 = 0.481 and Gemini 2.5 Flash achieves the highest Kappa = 0.402. The range across models is substantial: F1 varies from 0.298 (DeepSeek-V3) to 0.481 (GPT-5), and Kappa from 0.108 (Llama 3.3) to 0.402 (Gemini 2.5 Flash). The open-sourced models (Llama 3.3, DeepSeek-V3, Qwen3-235B-A22B) consistently underperform proprietary models on Kappa for ArgScore, though the gap is smaller for F1.
ClaimScore (factuality of bare claims without supporting reasons) shows the lowest agreement across all models. GPT-4o and DeepSeek-V3 achieve F1 = 0.000 (zero true positives for misinformed claims). The highest ClaimScore F1 is 0.240 (Gemini 2.5 Flash) and the highest Kappa is 0.180 (DeepSeek-V3). Several models show Kappa near zero or negative for ClaimScore: GPT-5 achieves Kappa = 0.024, indicating effectively random agreement. This is consistent with the inter-annotator agreement pattern (median Krippendorff's Alpha 0.392 for ClaimScore, the lowest across types), confirming that claims are inherently more subjective and difficult to evaluate reliably for both humans and models.

The proprietary vs. open-source gap. While the paper does not emphasize this comparison, Table 2 shows a notable pattern: the open-sourced models consistently achieve the lowest Kappa scores across all review point types. For overall REVIEWSCORE Kappa, the three open-sourced models occupy the bottom three positions (0.262 Qwen3, 0.301 DeepSeek-V3, 0.311 Llama 3.3), while the five proprietary models occupy the top five (0.336–0.378). This gap is narrower for F1 scores (DeepSeek-V3 at 0.383 vs. GPT-5 at 0.482), suggesting that the open-source models' weakness is primarily in the fine-grained 5-point scale calibration rather than coarse binary detection.

Detailed metric results (Appendix F, Tables 10–13). The supplementary metric tables reveal additional patterns:

For ClaimScore (Table 10): Precision is extremely low across all models—the highest is 0.167 (Claude Sonnet 4). Since most claims are labeled "Not misinformed" and models rarely predict "Misinformed," Recall is highly variable (0.000–0.667) and F1 is penalized heavily by low precision. The Pearson correlation for the 5-point scale ranges from 0.020 (GPT-5) to 0.226 (Gemini 2.5 Flash), confirming that models struggle to rank claims correctly even on the continuous scale.
For ArgScore (Table 11): GPT-4o shows unusually high Precision (0.550) but very low Recall (0.239), indicating it is conservative in flagging arguments as misinformed. Gemini 2.5 Flash achieves the best balance across all metrics: F1 = 0.466, Pearson = 0.469, AC2 = 0.387, Kappa = 0.402.
For QScore (Table 12): The open-sourced models achieve high Recall (0.733–0.848) but low Precision (0.380–0.415) compared to proprietary models, suggesting they over-predict misinformed questions relative to human annotators. GPT-5 achieves the best balance: Precision = 0.512, Recall = 0.618, F1 = 0.560.
For overall REVIEWSCORE (Table 13): Gemini 2.5 Flash achieves the highest Recall (0.699) for detecting misinformed review points, but its Precision (0.347) is among the lowest, indicating a tendency to over-flag. GPT-5 achieves the best Precision (0.421) with solid Recall (0.565), yielding the highest F1 (0.482).

BASE vs. ADVANCED REVIEWSCORE Comparison

Headline result (Table 3): ADVANCED REVIEWSCORE (premise-level factuality with aggregation) substantially outperforms BASE REVIEWSCORE (weakness-level factuality) for argument evaluation across all four models tested. With Gemini 2.5 Flash, the improvement is 3.40× in F1 (0.137 → 0.466) and 3.24× in Kappa (0.124 → 0.402). The gains are consistent across all models, though the magnitude varies.

Model-by-model results (Table 3):

Gemini 2.5 Flash: Base F1 = 0.137, Base Kappa = 0.124 → Advanced w/o Agg F1 = 0.395, Kappa = 0.386 → Advanced F1 = 0.466, Kappa = 0.402. The largest relative improvement of all models tested. Even without aggregation, premise-level scoring improves F1 by 2.88× over BASE.
Claude Sonnet 3.7: Base F1 = 0.157, Base Kappa = 0.120 → Advanced w/o Agg F1 = 0.333, Kappa = 0.308 → Advanced F1 = 0.462, Kappa = 0.261. Notably, Kappa slightly decreases from the no-aggregation condition (0.308) to the with-aggregation condition (0.261), while F1 substantially increases (0.333 → 0.462). This suggests that logical conjunction aggregation boosts binary classification accuracy at the cost of 5-point scale calibration for this model.
GPT-5: Base F1 = 0.218, Base Kappa = 0.182 → Advanced w/o Agg F1 = 0.370, Kappa = 0.366 → Advanced F1 = 0.481, Kappa = 0.353. Similar pattern to Claude Sonnet 3.7: aggregation improves F1 (0.370 → 0.481) but slightly reduces Kappa (0.366 → 0.353).
DeepSeek-V3: Base F1 = 0.146, Base Kappa = 0.091 → Advanced w/o Agg F1 = 0.167, Kappa = 0.193 → Advanced F1 = 0.298, Kappa = 0.176. The gains are more modest than for proprietary models: the no-aggregation condition shows only marginal improvement over BASE, and aggregation provides most of the benefit. This suggests that DeepSeek-V3's premise-level factuality judgments are less reliable than those of proprietary models, limiting the benefit of decomposition alone.

The contribution of aggregation. Comparing "Advanced w/o Agg" to "Advanced" in Table 3 reveals that logical conjunction aggregation consistently improves F1 (e.g., GPT-5: 0.370 → 0.481, a 30% relative gain), but has mixed effects on Kappa (improves for Gemini 2.5 Flash, worsens for Claude Sonnet 3.7 and GPT-5). This pattern is explained by the nature of logical conjunction: it converts premise-level scores into a binary argument-level verdict by checking whether any premise is misinformed (score 1–2). This sharpens the binary classification (improving F1) but discards the gradations of the 5-point scale that Kappa measures, potentially reducing Kappa if the model's premise-level scores were well-calibrated on the continuous scale but the binary thresholding introduces new errors. The weighted average aggregation is reported to show "marginal improvements or slight degradations" (Section 4.3), which the paper attributes to human-model misalignment on untrivialness criteria.

The takeaway. These results validate the paper's central design hypothesis: decomposing argumentative weaknesses into premises and evaluating factuality at the premise level substantially improves alignment with human judgments compared to holistic weakness-level evaluation. The effect is largest for the strongest models (Gemini 2.5 Flash, GPT-5) and more modest for weaker models (DeepSeek-V3), suggesting that premise-level decomposition amplifies existing model capabilities rather than compensating for fundamental weaknesses.

Disagreement Analysis and Authors' Response Effect

Human-model disagreement types (Figure 3). The paper categorizes disagreements between human annotators and LLM evaluators into six types, with the distribution varying by review point type:

For arguments (422 instances): "Misunderstanding / incorrect reasoning" accounts for 53% of disagreements, and "Minority opinion" (model predicts a score matching a minority human annotator's judgment) accounts for 37%. Together, these two categories explain 90% of argument disagreements. The remaining 10% is distributed across "Lack of appendix" (models not having access to appendix material), "Incorrect pdf parsing," and a negligible amount of value-judgment-related issues.
For questions (143 instances): "Misunderstanding / incorrect reasoning" accounts for 40%, and "Minority opinion" accounts for 45%. The higher minority-opinion rate for questions compared to arguments suggests that question unanswerability has more genuine ambiguity—different human annotators sometimes draw different reasonable conclusions about whether a question is answered by the paper, and the model may agree with the minority interpretation. Practical issues (appendix, pdf parsing) account for 15% combined.
For claims (92 instances): "Misunderstanding / incorrect reasoning" dominates at 55%. However, a distinctive additional category appears: "Value judgment" (20%) and "Lack of supporting reasons" (16%) together account for 36% of claim disagreements. These are unique to claims—bare assertions like "The experimental findings are somewhat trivial" or "The improvement is quite limited compared with other methods" (Figures 9–10) inherently involve subjective judgment that neither humans nor models can resolve objectively. This explains why ClaimScore shows the lowest human-model agreement across all metrics.

What "minority opinion" disagreements mean. The paper does not treat these as clear model errors. When the model predicts a score that matches one out of three human annotators but not the median-based final label, this could indicate that the model picked up on a legitimate interpretation that the majority of annotators missed, or that the model and the minority annotator share a common misunderstanding. Without a gold-standard ground truth (there is no "correct" answer for many of these judgments), these cases are inherently ambiguous. The paper's framing of this as a disagreement type rather than an error rate is appropriate for the task.

Effect of providing authors' responses (Table 4). Using Claude Sonnet 4, the paper tests whether providing the authors' rebuttal (from OpenReview) to the model improves REVIEWSCORE evaluation:

Overall REVIEWSCORE: F1 improves from 0.448 to 0.498 (+11.2% relative), Kappa improves from 0.378 to 0.416 (+10.1% relative). Authors' responses provide useful additional context.
ArgScore: F1 improves from 0.403 to 0.493 (+22.3% relative). The authors' response is most helpful for evaluating argument premises, likely because rebuttals often directly address whether reviewer premises are correct or not, providing explicit signals for factuality judgment.
ArgScore w/o Agg: F1 improves from 0.315 to 0.385 (+22.2% relative), similar magnitude. The benefit applies to individual premise evaluation, not just the aggregation step.
WScore (BASE REVIEWSCORE for weaknesses): F1 improves from 0.194 to 0.243 (+25.3% relative), a substantial relative gain from a low base.
QScore: F1 is essentially unchanged (0.579 without AR vs. 0.578 with AR), and Kappa slightly decreases (0.425 → 0.410). The paper attributes this to "injecting authors bias"—authors' responses naturally defend the paper and may claim questions are answered even when the answer is incomplete or implicit, which could mislead the model into judging questions as more answerable than human annotators (who are neutral third parties) would.
ClaimScore: F1 decreases from 0.222 to 0.125, a notable degradation. Like QScore, claims may be subject to authors' bias in rebuttals (authors disputing criticisms), but claims lack the supporting reasoning that would allow the model to independently verify the author's counter-argument.

The practical implication. Providing authors' responses helps for argument-level and premise-level factuality evaluation but may harm question unanswerability and claim factuality evaluation. Any deployment of automatic REVIEWSCORE evaluation should consider whether to include authors' responses on a per-review-point-type basis, or should present the response to the model but instruct it to maintain neutrality.

Ablation Studies and Robustness Checks

Faithfulness threshold for premise annotation (Section 3.3): The dataset construction protocol only evaluates premise factuality when the argument reconstruction faithfulness score is ≥ 4 (on a 5-point scale). This means that for the ~6.5% of arguments where the reconstruction engine's pass rate fails to produce a reconstruction with faithfulness ≥ 4 (Table 5 shows 93.5% pass rate), premises are not annotated. The paper does not report a sensitivity analysis varying this threshold, but the threshold choice is conservative: it ensures that premise-level human labels are not contaminated by reconstruction errors, at the cost of excluding the hardest-to-reconstruct arguments from ADVANCED REVIEWSCORE evaluation. Whether these excluded arguments are systematically different (e.g., more complex, more ambiguous) is not analyzed.

Logical conjunction vs. weighted average aggregation (Section 3.1, Table 3 discussion): The weighted average aggregation (using untrivialness as weights) "shows marginal improvements or slight degradations" compared to logical conjunction. The paper's manual investigation of disagreements reveals that "humans and models have slightly misaligned criteria on whether a given premise is trivially true or not based on the common knowledge of CS/AI-majoring undergrad students" (Section 4.3). This is a meaningful negative result: untrivialness, while conceptually motivated as a way to filter out logical scaffolding premises, proves difficult to operationalize reliably because what counts as "common knowledge" varies between human annotators, between humans and models, and potentially across subfields. The paper does not report the inter-annotator agreement specifically for untrivialness judgments, which would help quantify this issue.

Oracle vs. predicted argument type classification (implicit): The paper uses LLM-based classification to categorize review points as claims, arguments, or questions, with human verification (Section 3.3). There is no ablation testing whether human-only classification (without LLM preprocessing) would change the results, nor is there an analysis of LLM classification errors that survived human verification. The human verification step makes this a minor concern, but any systematic bias in the LLM's classification proposals (e.g., misclassifying weak arguments as claims) could influence which instances enter which evaluation category.

Reconstruction engine: with vs. without feedback (Appendix C, Table 5): The ablation comparing direct LLM reconstruction (w/o feedback) to the full feedback-loop engine (w/ feedback) demonstrates the necessity of the validity and faithfulness checks. Without feedback: validity = 0.895, faithfulness = 2.91/5, pass rate = 0.369. With feedback: validity = 1.00, faithfulness = 4.47/5, pass rate = 0.935. The average of 3.09 ± 2.73 loop iterations confirms active use of the feedback signals. The faithfulness improvement (2.91 → 4.47) is larger in absolute terms than the validity improvement (0.895 → 1.00), suggesting that faithfulness is the harder criterion to satisfy and that the LLM judge's faithfulness feedback is effective. The paper does not ablate the individual contributions of validity feedback vs. faithfulness feedback (e.g., running with only validity feedback or only faithfulness feedback), which would help disentangle which loop drives the improvement.

Reconstruction engine: base model dependence (Limitations): The paper reports that upgrading the base model from Claude Sonnet 3.7 to Claude Sonnet 4 produces "considerable output quality improvement," but does not provide quantitative results for Claude Sonnet 4 in Table 5. This is a missed opportunity to quantify how reconstruction quality scales with base model capability, which would inform whether the engine's performance bottleneck is the architecture or the underlying LLM.

Effect of excluding figures and appendices (Figure 3, Limitations): The paper excludes figures and appendices from the model input to save API costs. The disagreement analysis (Figure 3) shows that "Lack of appendix" accounts for 10% of question disagreements, 8% of argument disagreements, and 6% of claim disagreements. "Incorrect pdf parsing" accounts for an additional 6%, 0%, and 8% respectively. Combined, these practical input limitations cause 16% of question disagreements and 14% of claim disagreements, representing a non-trivial floor on achievable human-model agreement that could be reduced with better input preprocessing and inclusion of appendix material. The paper does not provide an upper-bound estimate of what agreement could be achieved with perfect input (full paper, figures, appendices, correct parsing), which limits the interpretability of the moderate-agreement results—some unknown fraction of the disagreement is attributable to input quality rather than model capability.

Multiple human annotators protocol (Section 3.3, Appendix E): The paper's use of three annotators per instance with disagreement resolution through recheck and discussion is a strong robustness measure, but the final labels are primarily medians. The paper reports that occasional instances use post-discussion labels instead of medians ("we exceptionally take post-discussion labels as final ones"), but does not specify how many instances fall into this exception or provide an ablation showing whether using medians-only would change the results. The disagreement-aware recheck protocol is sensible but is not itself ablated (e.g., comparing recheck vs. no-recheck agreement).

Critical Assessment

Claim: "15.2% of weaknesses and 26.4% of questions are misinformed." This finding is grounded in the human annotation of 40 ICLR papers and 155 reviews. The evidence is the dataset statistics reported in Section 1 and Appendix D.3. The claim is supported within the scope of the annotated data, but its generalizability is untested. The 40 papers were selected by annotator groups based on shared research interests (non-random sampling), and all are from ICLR 2021–2023 (a specific conference with specific review norms). Whether these percentages generalize to other conferences (NeurIPS, ICML, ACL), other years, or other fields is not established. The inter-group variance in inter-annotator agreement (Krippendorff's Alpha ranging from 0.254 to 0.663, Table 9) suggests that paper selection and annotator expertise substantially influence the measured misinformed rate, so the 15.2% and 26.4% figures should be treated as estimates with unknown confidence intervals on a non-random sample.

Claim: "ADVANCED REVIEWSCORE clearly outperforms BASE REVIEWSCORE, which proves the effectiveness of premise-level factuality scoring." Table 3 provides strong evidence for this claim on arguments. With Gemini 2.5 Flash, the improvement is 3.40× F1. The claim is well-supported for the argument review point type across all four tested models. However, the paper's phrasing "proves the effectiveness" overstates what the experiment demonstrates. The experiment shows that premise-level evaluation improves human-model agreement, which is a measure of alignment with human judgments, not necessarily detection accuracy. If human annotators themselves make systematic errors in holistic weakness evaluation (which the paper argues they do, due to implicit premise weighting), then higher alignment with their premise-level judgments does not necessarily mean the premise-level approach is more accurate—it means it is more consistent with human annotators when they are forced to be explicit about their reasoning. This is still a desirable property, but it is a property of the evaluation protocol, not a direct proof that premise-level evaluation better identifies truly misinformed review points.

The improvement is also heterogeneous across models: DeepSeek-V3 shows more modest gains (F1: 0.146 → 0.298, ~2× improvement) than Gemini 2.5 Flash (F1: 0.137 → 0.466, ~3.4×), suggesting that premise-level evaluation amplifies the capabilities of stronger models more than weaker ones. The approach is effective but its magnitude of benefit is model-dependent.

Claim: "Current state-of-the-art LLMs only show moderate agreement with human experts (F1 0.4–0.5, Kappa 0.3–0.4)." Table 2 provides the evidence. The claim is well-supported. The paper is transparent about the moderate performance and does not overclaim. However, the interpretation of "moderate" depends on context. The paper does not benchmark against simpler baselines that would contextualize these numbers: How well would a random baseline perform? How well would a simple keyword-matching or embedding-similarity baseline perform? How well would the human annotators agree with each other in a held-out test (the inter-annotator agreement in Table 9 is measured on the training/annotation data, not on held-out instances)? Without these reference points, a reader cannot assess whether 0.4–0.5 F1 is surprisingly low (models are barely better than random) or surprisingly high (models nearly match human-level agreement on a very difficult task). The median inter-annotator Krippendorff's Alpha of 0.489 for overall REVIEWSCORE provides some context—models achieve Kappa of 0.262–0.378, which is below but not dramatically below human agreement levels. The gap between best model Kappa (0.378) and median human Alpha (0.489) is meaningful but not enormous given the task's difficulty.

What the experiments do NOT demonstrate. Several important questions are left open:

Does REVIEWSCORE correlate with other measures of review quality? The paper does not compare REVIEWSCORE against existing review quality metrics (e.g., the "substantiation" and "understanding" criteria from Goldberg et al., 2025, or the groundedness metrics from Guo et al., 2023). If REVIEWSCORE identifies different reviews as low-quality than existing metrics, that would suggest it captures a complementary dimension. If it largely overlaps, its added value is less clear. This comparison is absent.
Would flagging misinformed review points improve downstream outcomes? The paper does not test whether providing REVIEWSCORE to authors (to help write rebuttals), reviewers (to self-check), or meta-reviewers (to calibrate decisions) improves any measurable outcome. The practical utility of the metric is asserted in Appendix A but not demonstrated. A user study with area chairs or authors would substantially strengthen the paper's impact claim.
How does model performance vary with paper topic, review length, or argument complexity? The paper's difficulty analysis groups results by review point type but does not stratify by paper area (the five annotator groups cover different AI subfields: image generation, time series ML, LLM reasoning/compression, LLM/VLM, LLM prompting), review length, number of premises in the argument, or any other difficulty proxy. The inter-group variance in human agreement (Table 9) suggests there may be systematic differences in task difficulty across subfields that the current analysis does not explore.
What would a "solved" REVIEWSCORE evaluation look like? The paper establishes that current models achieve 0.4–0.5 F1, but provides no estimate of the ceiling—what F1 would a perfect model achieve given the inherent ambiguity in the task? The inter-annotator agreement provides an approximate ceiling (Krippendorff's Alpha 0.489 median), but agreement among annotators is not the same as agreement between a model and the final consensus label. Computing human-human F1 on held-out instances would establish an empirical upper bound for model performance and contextualize the 0.4–0.5 F1 numbers.

Missing ablations and experiments. Several experiments that would strengthen the paper are absent:

No confidence calibration analysis. The paper reports point estimates of agreement (F1, Kappa) but does not evaluate whether models are calibrated—does a model's confidence in its score predict the likelihood that it agrees with humans? This is important for deployment: if models are well-calibrated, high-confidence predictions could be trusted even if average agreement is moderate; low-confidence instances could be flagged for human review.
No ensemble or multi-model analysis. The paper evaluates eight models independently but does not test whether ensembling multiple models (e.g., majority voting across models) improves agreement, which is a standard technique for improving reliability in LLM-as-a-judge settings.
No prompt ablation. The prompts used for REVIEWSCORE evaluation (Appendix H, Figures 18–20) are detailed and include specific instructions (e.g., handling of conditional premises, knowledge source selection rules). The paper does not ablate whether simpler prompts would perform comparably, whether the detailed instructions help or confuse the model, or whether few-shot examples would improve performance. Given the sensitivity of LLM-as-a-judge to prompt design, this is a significant gap.
No analysis of the 6.5% of arguments that fail reconstruction. The reconstruction engine's pass rate is 93.5% (Table 5)—6.5% of arguments do not produce a valid and faithful reconstruction within 10 iterations. These failed cases are not analyzed. If they are systematically harder or more ambiguous arguments, excluding them from ADVANCED REVIEWSCORE evaluation biases the comparison with BASE REVIEWSCORE (which does not require reconstruction and thus includes all arguments). The paper should report whether BASE REVIEWSCORE performance differs between the reconstructable and non-reconstructable subsets.
The supplementary metrics in Tables 10–13 are not discussed in the main text. The Pearson correlation and Gwet's AC2 metrics provide additional information about model performance on the 5-point scale, but the main text only discusses F1 and Kappa from Table 2. The Pearson correlation for ArgScore (Table 11) ranges from −0.002 (Llama 3.3, effectively no linear correlation) to 0.469 (Gemini 2.5 Flash), showing that some models have essentially no ability to rank arguments on the 5-point scale even when their binary F1 is reasonable. This important detail is buried in the appendix.

Conditions on the paper's claims. The central ADVANCED REVIEWSCORE improvement claim holds when: (1) the weakness is an argument with reconstructable logical structure (not a bare claim), (2) the argument reconstruction is sufficiently faithful (faithfulness ≥ 4), and (3) the evaluation model is sufficiently capable (gains are larger for stronger models). The claim that REVIEWSCORE identifies misinformed review points holds when: (1) the definition of "misinformed" aligns with what the community considers problematic (the paper argues this via reviewer guidelines, but community consensus is not empirically measured), and (2) the human annotators' judgments are treated as ground truth (the inter-annotator agreement of 0.489 Krippendorff's Alpha indicates moderate but imperfect reliability). The claim that automatic evaluation "remains challenging" holds unconditionally—the 0.4–0.5 F1 numbers make this clear.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Unaccounted for in Deployment

The assumption or constraint. The entire compute-optimal framework that the paper proposes—adaptively allocating inference budget based on prompt difficulty—depends on knowing the difficulty of each prompt before deciding how to spend the compute budget. The method the paper uses to estimate difficulty is generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted). The authors explicitly acknowledge this cost in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. In a realistic deployment, the total cost would be the difficulty estimation cost plus the strategy execution cost. Generating 2048 samples per prompt is extraordinarily expensive—it exceeds the largest test-time compute budgets studied in the paper (256–512 generations). This means that for any single-prompt deployment (e.g., answering one user question), the difficulty estimation overhead would make the approach strictly more expensive than simply running best-of-N with the full budget, completely negating the reported 4× efficiency gains. The gains only materialize when the difficulty estimation cost is amortized across many prompts of the same difficulty level—a scenario the paper does not evaluate.

The paper also does not analyze the trade-off between difficulty estimation accuracy and the number of samples used. Could 16 samples provide a sufficiently good difficulty estimate to recover most of the gains? What about 64? Without this analysis, a practitioner cannot determine whether the approach is practical at any scale.

What evidence exists in the paper. The paper provides clear evidence that difficulty estimation works in principle: Figures 4 and 8 show that predicted difficulty bins (using PRM scores from 2048 samples) track the oracle bins (using ground-truth correctness) closely, with the curves "largely overlapping." However, there is no experiment measuring performance as a function of the number of samples used for difficulty estimation, no ablation showing that cheaper difficulty estimates (e.g., from 8 or 16 samples) preserve the gains, and no accounting of difficulty estimation cost in any of the efficiency calculations.

Mitigation status. The paper acknowledges this as a gap and frames it as "a key avenue for future work" (Section 3.2), specifically suggesting "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). No such model is developed or evaluated. The paper also does not explore adaptive difficulty estimation schemes—generating a few initial samples, assessing difficulty from those, and then allocating the remaining budget—even though this is a natural extension of the framework. Until cheap difficulty estimation is demonstrated, the reported 4× efficiency gains are an upper bound on what a deployment system could achieve, not a realized practical benefit.

Hard Problems Remain Essentially Unsolved—Test-Time Compute Cannot Create Capability

The assumption or constraint. The compute-optimal framework is built on the premise that test-time compute can substitute for pretraining under certain conditions. However, the paper's results reveal a hard boundary: test-time compute amplification only works when the base model already has some non-trivial probability of producing a correct answer. On problems where the base model's pass@1 is near zero, additional inference compute provides essentially no benefit regardless of how it is allocated.

The paper explicitly acknowledges this in Section 7:

"On the hardest questions (bin 5), test-time compute provides essentially zero benefit regardless of budget, meaning that some capabilities can only be acquired through pretraining, not recovered at inference time."

The consequence. This is a fundamental capability ceiling, not a mere inefficiency. For any problem domain or difficulty tier where the base model cannot produce correct solutions even occasionally, REVIEWSCORE-based allocation (or any test-time compute strategy) offers zero improvement. In the paper's taxonomy, this applies to difficulty bin 5, where accuracy hovers at 1–3% across all methods, all budgets, and all allocation strategies (Figure 3, right panel; Figure 7, right panel; Figure 9 for bin 5). The FLOPs-matched comparison (Figure 9) shows that scaling pretraining to a ~14× larger model does improve bin 5 performance, confirming that these problems are solvable with more training compute but not with more inference compute on the smaller model.

For practitioners, this means test-time compute is not a general-purpose substitute for model capability. If a deployment involves a non-trivial fraction of genuinely hard problems (where the model's raw pass@1 is near zero), the system must either route those problems to a larger model (hybrid deployment), accept near-zero accuracy on them, or invest in pretraining. The compute-optimal framework provides no guidance for this regime.

What evidence exists in the paper. The evidence is stark and consistent across every experiment that breaks results out by difficulty. In Figure 3 (right), bin 5 shows near-flat lines at 1–3% accuracy for all search methods across all budgets. In Figure 7 (right), bin 5 shows 2–3% accuracy regardless of the sequential-to-parallel ratio. In Figure 9, the bin 5 scaling curve (blue line) sits at 0–5% accuracy and is consistently below the ~14× larger model's performance (star markers) across all values of R. The paper's FLOPs-matched comparison results (Section 7, bar chart in Figure 1) show negative relative performance for hard questions at R ≫ 1: −37.2% for revisions and −52.9% for PRM search. These are not marginal degradations—test-time compute is substantially worse than simply using the larger model.

Mitigation status. The paper is transparent about this limitation, stating it explicitly in the Section 7 takeaway and discussing it in the Executive Summary. However, no mitigation is proposed beyond acknowledging that pretraining remains necessary for hard problems. The paper does not explore whether combining test-time compute on the larger model (rather than only on the smaller one) could push the capability boundary further—this would require a three-way comparison (small model + test-time, large model greedy, large model + test-time) that is absent. A practitioner reading the paper is left with a clear warning about the limitation but no strategy for addressing it beyond "use a bigger model."

The Revision Model Has a Structural Correct-to-Incorrect Reversion Problem

The assumption or constraint. The revision model is trained exclusively on trajectories where all in-context answers are incorrect, followed by a correct final answer (Section 6.1). The model never sees training examples where a correct answer appears in context and should be preserved. This is an intentional design choice to teach the model to improve incorrect answers, but it creates a structural blind spot: the model has no training signal for what to do when its current answer is already correct.

The consequence. At test time, when the revision model generates a chain of sequential revisions, approximately 38% of correct answers produced during the chain get "revised" into incorrect answers in the subsequent step (Section 6.1). This means the revision chain is not monotonically improving—it can degrade previously correct outputs. The paper states:

"since the model was trained only on sequences where all in-context answers are incorrect (followed by a correct target), at test time the model may encounter correct answers in its context (produced during earlier revisions) and incorrectly 'revise' them into wrong answers."

This fundamentally limits the effectiveness of long revision chains. Even if step 5 produces a correct answer, step 6 has a 38% chance of breaking it. The paper mitigates this by not taking the final revision output but instead selecting the best answer from any point in the chain using majority voting or verifier-based selection (Section 6.1). However, this means the model is generating many revisions that are not only wasted but actively harmful to the pool of candidates. The compute spent on revisions after a correct answer is reached is, in expectation, counterproductive.

What evidence exists in the paper. The paper cites the 38% correct-to-incorrect reversion rate without providing a detailed breakdown (e.g., does the reversion rate increase with chain length? Is it higher for certain difficulty levels?). Figure 6 (left) shows that pass@1 across revision steps improves from ~18.2% at step 1 to ~24–25% by steps 15–20, and remains in the 23–25% range out to 64 steps. The plateau (rather than continued improvement) is consistent with the reversion problem acting as a brake on chain-length scaling. The fact that within-chain selection (majority or verifier) substantially outperforms taking the final revision (implied by the sequential vs. parallel results in Figure 6, right) is indirect evidence of the reversion problem's impact. The ReST^EM experiment (Appendix K, Figure 16) provides additional evidence: attempting to further optimize the revision model with RL-style training caused sequential revisions to substantially hurt performance, with fully sequential dropping to ~33.5% compared to ~38.5% at the optimal ratio, suggesting the revision training procedure is fragile and the reversion tendency may worsen with on-policy data collection.

Mitigation status. The paper mitigates the problem through within-chain answer selection (majority voting or verifier-based best-of-N weighted selection across all steps in the chain), but this is a post-hoc fix, not a solution to the underlying training deficiency. A principled solution—such as training the model on mixed sequences that include correct answers in context, teaching it to recognize when no revision is needed—is not explored. The paper does not propose modifications to the training data construction (e.g., including some trajectories where the correct answer appears mid-sequence and is preserved) or the inference procedure (e.g., an explicit "stop revising" mechanism) to address the root cause. This limitation is significant for practitioners because it means that simply generating longer revision chains does not reliably improve output quality—there is a built-in degradation mechanism that requires external selection to manage.

Revisions and PRM Search Are Studied Independently, Not Combined

The assumption or constraint. The paper treats revisions (modifying the proposal distribution) and PRM-guided search (modifying the selection mechanism) as independent axes, evaluating each separately. Section 8 acknowledges:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The experiments on revisions use an ORM trained on revision model outputs for answer selection (Section 6, Appendix J), not the PRM used for search. Conversely, the search experiments use the few-shot prompted base model, not the revision model, as the proposal distribution.

The consequence. The paper's central framework decomposes test-time compute into two complementary axes—proposal distribution modification and verifier optimization (Section 2)—with the implication that combining them should yield gains beyond either alone. The empirical results confirm that these axes have complementary difficulty-dependent strengths: revisions help most on easy problems (local refinement), while search helps most on medium problems (global exploration). Yet the paper never tests whether these complementary strengths can be combined within a single inference procedure—for example, using the revision model as the proposal distribution within beam search (so each beam step conditions on previous steps and revisions of the same problem), or using the PRM to guide which revisions to pursue (pruning revision chains that the PRM scores poorly).

This means the reported results are a lower bound on what a fully integrated system could achieve. The 4× efficiency gains over best-of-N (Figures 4, 8) are achieved by selecting either a search strategy or a revision strategy per difficulty bin, not by combining them. A system that used revision-model outputs within beam search, or that adaptively switched between search and revisions mid-computation, could potentially exceed the performance of either strategy alone, particularly on medium-difficulty problems where both mechanisms show benefits. The absence of this experiment means the paper's headline result—while impressive in isolation—may substantially understate the potential of the framework.

What evidence exists in the paper. The paper provides results for search in isolation (Section 5, Figure 3–4) and revisions in isolation (Section 6, Figure 6–8), with separate verifiers for each (PRM for search, ORM with revision context for revisions). The difficulty-dependent behavior differs between the two mechanisms: search shows the strongest relative advantage on bins 3–4 (Figure 3, right), while revisions show an optimal intermediate sequential-to-parallel ratio on bins 3–4 (Figure 7, right). This suggests complementarity—search excels at exploration, revisions at refinement—but the interaction is untested. Appendix J (Figure 15a) shows that the base-LM PRM underperforms the revision-specific ORM when scoring revision model outputs, confirming that the verifiers are not interchangeable and that combining search with revisions would require either a PRM trained on revision outputs or an adapted verifier—neither of which is developed.

Mitigation status. The paper explicitly identifies this as future work in Section 8 but provides no preliminary results, no analysis of what a combined system would look like architecturally, and no discussion of the practical challenges (e.g., the need for a PRM that generalizes to revision model outputs). A practitioner seeking to implement the full framework would need to resolve the verifier–proposal mismatch independently, with no guidance from the paper on whether the combination is likely to be synergistic or redundant. Given that the paper's framework explicitly identifies these as complementary axes, the absence of even a small-scale combined experiment (e.g., using revision model outputs with the existing PRM, even if suboptimal) is a notable gap.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained, and the FLOPs-Matched Comparison Gives the Larger Model No Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares the base model (PaLM 2-S*) with compute-optimal test-time scaling against a model with approximately 14× more parameters but the same amount of training data, using greedy decoding with no test-time compute augmentation. The paper acknowledges that this departs from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. The FLOPs-matched comparison systematically favors test-time compute in two ways. First, a Chinchilla-optimal model (Hoffmann et al., 2022) trained with 14× more total FLOPs would scale both parameters and training data equally, likely outperforming a parameter-only-scaled model at the same total compute budget. The paper's larger model is effectively undertrained relative to compute-optimal scaling laws, making it a weaker baseline than it should be. This inflates the reported advantages of test-time compute over pretraining—the +27.8% relative improvement on easy questions at R ≪ 1 (Figure 1, Section 7) would likely be smaller against a compute-optimally trained larger model.

Second, and perhaps more consequentially, the larger model is given no test-time compute budget of its own. It uses greedy decoding—a single generation with temperature 0. The framework the paper champions (adaptive, difficulty-aware test-time compute allocation) is specifically designed to amplify model capabilities, and the paper demonstrates that a small model with this framework can outperform a much larger model without it. But the natural comparison for a practitioner is: given a fixed total FLOPs budget, should I spend it all on making a larger model and running it with greedy decoding, or should I spend some on making a moderately larger model and also give it some test-time compute? The paper's setup prevents answering this question because the larger model is prohibited from using the very techniques the paper advocates.

In the extreme, a Chinchilla-optimal larger model with its own compute-optimal test-time scaling (even a modest budget of 4–8 generations) would be a substantially stronger baseline that could shift or reverse the reported pretraining-vs-inference tradeoff conclusions, particularly in regimes where the current baseline already performs competitively (R ≫ 1, medium-to-hard problems).

What evidence exists in the paper. Figure 9 provides the raw comparison: the ~14× larger model's greedy performance (star markers) is compared against PaLM 2-S* with compute-optimal scaling (line plots) at three values of R. The paper is transparent about the parameter-only scaling assumption in the Section 7 discussion. However, the paper does not provide an ablation where the larger model is given even a small test-time compute budget (e.g., best-of-4 or best-of-8), which would reveal whether the pretraining-vs-inference tradeoff conclusions are robust to the larger model also benefiting from inference-time optimization. The paper also does not discuss what fraction of the reported gains could be attributed to the larger model's suboptimal training recipe versus the test-time compute allocation strategy.

Mitigation status. The authors acknowledge the parameter-only scaling assumption as a limitation and frame the Chinchilla-optimal comparison as future work (Section 7, Section 8). They do not acknowledge the asymmetry in test-time compute allocation between the two models as a separate limitation, nor do they discuss how the comparison would change if the larger model received a proportional inference budget. The "representative" framing—that parameter-only scaling follows the LLaMA paradigm—is a reasonable argument, but it does not address the fundamental question a practitioner would ask: "If I have X FLOPs total, should I spend them on a bigger model or on test-time compute for a smaller one?" The current setup answers an asymmetric version of this question (bigger model with greedy decoding vs. smaller model with optimized inference), which is not the decision most practitioners face.

Single Benchmark, Single Model Family, Small Test Set—Generalization Is Unproven

The assumption or constraint. All experiments in the paper use the MATH benchmark (500 test questions from Hendrycks et al., 2021, using the Lightman et al., 2022 split) with PaLM 2-S* as the base model. The paper states in Section 4 that the authors "believe this model is representative of the capabilities of many contemporary LLMs," but provides no evidence from other model families or benchmarks. The test set is split into five difficulty quintiles of ~100 questions each, and two-fold cross-validation within each bin further halves the data used for strategy selection to ~50 questions per fold per bin.

The consequence. The core findings—that test-time compute strategies have qualitatively different scaling behavior depending on difficulty, that compute-optimal allocation yields 4× efficiency gains, and that test-time compute can substitute for pretraining on easy-to-medium problems—may not generalize beyond PaLM 2 models on competition-level math problems. Several aspects could be model-specific or domain-specific:

PRM quality and over-optimization behavior. The over-optimization curve (beam search degrading easy-problem performance at high budgets, Figure 3 right) depends on the specific calibration properties of the PRM trained on PaLM 2-S* outputs. A model with different output distributions (e.g., fewer false positives that the PRM identifies as correct) might exhibit different or weaker over-optimization, changing the optimal strategy allocation.
Revision model learning dynamics. The revision model's ability to improve through sequential revisions depends on the base model's in-context learning and self-correction capabilities. Models with different architectural properties or training procedures might learn different revision behaviors—or fail to learn revision at all, as the ReST^EM experiment (Appendix K) demonstrates even within the PaLM 2 family.
MATH as a task. MATH consists of formal math problems with unique correct answers and multi-step symbolic reasoning. The difficulty-dependent patterns (sequential for easy, parallel for hard) might not transfer to tasks requiring factual recall (where the "correct answer" is memorized rather than reasoned), code generation (where correctness is validated by unit tests, not string matching), or open-ended generation (where there is no ground-truth answer at all). The paper's framework assumes access to clean correctness signals for PRM training (via Monte Carlo rollouts) and difficulty estimation (via pass@1)—assumptions that break for tasks without verifiable answers.
Small test set and strategy selection variance. With ~50 questions per fold per difficulty bin used for strategy selection, the chosen optimal strategies may not be robust. A different random split of the 500 questions could select different optimal strategies, and the paper does not report confidence intervals on the compute-optimal scaling curves. The observed 4× efficiency gains could be partially attributable to overfitting the strategy selection to the specific test set.

What evidence exists in the paper. The paper provides no cross-model or cross-benchmark results. The difficulty-dependent patterns are consistent within the MATH benchmark—both search (Figure 3 right) and revisions (Figure 7 right) show clear difficulty stratification—but this only demonstrates that the patterns exist for PaLM 2 on MATH, not that they are universal. The paper's inter-annotator agreement analysis (Table 9, Appendix E) for the human annotation task shows substantial inter-group variance (Krippendorff's Alpha ranging from 0.254 to 0.663), which is a different task but hints at the sensitivity of evaluation to specific paper selections and annotator expertise—a parallel caution about the representativeness of a single benchmark. The paper does not discuss how the choice of MATH (with its specific difficulty distribution, answer formats, and problem types) might influence the optimal allocation strategies.

Mitigation status. The authors acknowledge the single-model, single-benchmark scope only indirectly, stating the model is "representative" without evidence (Section 4). The Limitations section (Section 5, "Limitations" paragraph at the end of the paper) does not identify generalization as a limitation, focusing instead on practical issues (API costs, PDF parsing, annotator noise). This is the most significant unaddressed limitation for practitioners: before adopting the framework for a new model family or task domain, they would need to replicate the full experimental pipeline—PRM training, revision model training, difficulty bin estimation, strategy selection—to verify that the difficulty-dependent patterns and efficiency gains transfer. The paper provides no guidance on which findings are likely to be universal (e.g., difficulty-dependence of optimal strategies) versus model-specific (e.g., the specific budget levels where beam search outperforms best-of-N).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around LLM-based peer review evaluation from asking "can LLMs judge review quality?" to asking "what kind of review judgment is reliable enough for LLMs to make, and under what conditions?" It introduces a specific, principled decomposition—evaluating misinformed review points by first reconstructing arguments into premise-conclusion structures, then checking factuality at the premise level rather than the weakness level—that directly addresses the subjectivity problems plaguing holistic review quality scoring. The empirical demonstration that this decomposition substantially improves human-model agreement (3.40× higher F1 for ArgScore vs. Base ReviewScore with Gemini 2.5 Flash, Table 3) establishes a new standard: argument reconstruction is not optional preprocessing but the mechanism that makes review evaluation objective enough to be operationalized.

This is not a paradigm shift—the paper does not claim LLMs can reliably detect all review quality problems, and indeed demonstrates the opposite with moderate 0.4–0.5 F1 scores. Rather, it is a diagnostic reframing that clarifies why holistic review evaluation is unreliable (human and model evaluators apply different implicit weights to mixed factual and non-factual content within a single criticism) and provides a concrete architecture to fix it (decompose, evaluate atomic premises, aggregate with explicit rules). The significance is that this reframing converts review quality evaluation from a subjective judgment task—where disagreement between evaluators is inherent and irreducible—to a structured fact-checking task where disagreements can be attributed to specific, addressable causes: reconstruction errors, knowledge base mismatches, or genuine factual ambiguity in individual premises.

The paper also reconciles conflicting intuitions in the community about LLM-as-reviewer capabilities. On one hand, the fact that 26.4% of real ICLR review questions are already answerable by the submitted paper (Section 1) suggests there is substantial low-hanging fruit for automation—reviewers clearly miss information that is present in papers. On the other hand, the moderate LLM agreement scores (F1 0.4–0.5, Table 2) demonstrate that current models are not reliable enough for unsupervised screening. The paper's contribution is to show that both intuitions are correct simultaneously: there is a real problem to solve (quantified at 15–26% misinformed rates), and current LLMs can make progress on it (premise-level factuality shows meaningful agreement with humans), but they are not yet deployed-ready. This framing is productive because it defines a clear research trajectory rather than a binary "can LLMs review?" debate.

The paper also changes the attractiveness of different research directions in peer review NLP:

More attractive: Training dedicated argument reconstruction and premise-level fact-checking models for scientific text. The paper's reconstruction engine (Section 3.2) demonstrates that off-the-shelf LLMs with feedback loops can already produce high-quality reconstructions (4.47/5 faithfulness, Table 5), and the ADVANCED REVIEWSCORE results (Table 3) show that premise-level evaluation significantly improves agreement. This suggests that fine-tuning specialized models for these subtasks—particularly premise factuality classification against specific knowledge bases—could yield large gains, since the decomposition step already works well.
Less attractive: End-to-end review scoring without explicit structure. The paper's comparison between BASE and ADVANCED REVIEWSCORE (Table 3) provides direct evidence that holistic weakness-level evaluation produces substantially lower human-model agreement than premise-level evaluation. This suggests that approaches which treat review quality as a single regression target or classification label (without argument structure) are fundamentally limited by the same ambiguity that caused human annotator divergence in the paper's initial group discussion (Appendix B). The paper does not disprove the value of holistic scoring—it may capture dimensions orthogonal to misinformed detection—but it establishes that for the specific goal of detecting factually incorrect criticisms, structure-free approaches are suboptimal.
Newly attractive: The intersection of argument mining and scientific fact-checking. The paper effectively demonstrates that peer review is a natural domain where argument structure (premises, conclusions, logical entailment) and factual verification (checking claims against a knowledge base) must be handled jointly. This creates a bridge between two largely separate NLP communities and provides a benchmark (the REVIEWSCORE dataset) where both skills are required.

The identification of claim evaluation as a hard failure mode is also landscape-shifting. The paper shows that bare claims—weaknesses without supporting reasons—show the lowest human-model agreement (F1 as low as 0.0, Tables 2 and 10), with 36% of disagreements caused by claims being "value-laden" or "lacking supporting reasons" (Figure 3). This is not a model weakness that better training will fix; it is a fundamental property of underspecified criticism. The implication for future work is that review evaluation systems should either (a) flag underspecified claims as requiring reviewer clarification rather than attempting to evaluate them, or (b) prompt reviewers to provide supporting reasoning for all criticisms at submission time, reducing the claim-to-argument ratio in real reviews. The paper does not make this recommendation explicitly, but it follows directly from the results.

Follow-Up Research This Work Enables

Training a dedicated premise factuality classifier fine-tuned on the REVIEWSCORE dataset. The paper uses general-purpose LLMs in a zero-shot setting for premise-level factuality evaluation, achieving moderate agreement (ArgScore F1 0.298–0.481 across models, Table 2). The REVIEWSCORE dataset provides 1,748 annotated premises with factuality labels (1–5 scale), knowledge base selections, and untrivialness scores. This is sufficient training data to fine-tune a smaller, specialized model (e.g., Llama 3.3 70B or a T5 variant) specifically for the task of binary or 5-point factuality classification given a premise and a knowledge base (paper text). A strong follow-up would: (a) fine-tune on the premise annotations, (b) evaluate on a held-out set of premises from unseen papers, (c) compare the fine-tuned model's agreement with human annotators against the zero-shot LLM baselines from Table 3, and (d) measure whether the fine-tuned model generalizes to reviews from conferences or years not in the training data (ICLR 2024–2025, NeurIPS, ACL). The hypothesis is that task-specific fine-tuning should substantially outperform zero-shot prompting, especially for open-source models (DeepSeek-V3, Llama 3.3) that underperform proprietary models on ArgScore Kappa (0.108–0.176 vs. 0.244–0.402, Table 11).

Evaluating whether argument reconstruction quality improvement translates to downstream evaluation gains. The paper reports that upgrading the base LLM from Claude Sonnet 3.7 to Claude Sonnet 4 produces "considerable output quality improvement" in the reconstruction engine (Limitations), but does not measure this quantitatively or test whether better reconstructions lead to better REVIEWSCORE evaluation. A follow-up experiment would: (a) run the reconstruction engine with 3–5 different base models spanning a quality range, (b) measure reconstruction faithfulness for each, (c) have human annotators evaluate premise factuality on the same premises across all reconstruction outputs (to isolate the effect of reconstruction quality from premise evaluation capability), and (d) correlate reconstruction faithfulness with downstream human-model agreement on ArgScore. The paper's current faithfulness of 4.47/5 (Table 5) suggests there is limited headroom for improvement, but the correlation analysis would establish whether the remaining ~0.5-point faithfulness gap is a meaningful bottleneck. If the correlation is weak, future work should focus on premise evaluation accuracy rather than reconstruction. If strong, further investment in reconstruction (e.g., domain-specific fine-tuning, better formalization of non-deductive arguments) is warranted.

Stress-testing the claim that claims are inherently subjective and should be flagged rather than evaluated. The paper shows that ClaimScore has the lowest human-model agreement (F1 as low as 0.0) and attributes this to value judgments and lack of supporting reasons (Figure 3). This finding could be strengthened or qualified by: (a) having human annotators re-annotate a subset of claims while explicitly providing the justification they would require to make the claim evaluable (e.g., "this claim would need a comparison baseline to be checkable"), (b) correlating the "evaluability" of a claim (as judged by annotators) with human-human and human-model agreement, and (c) testing whether an LLM can be prompted to convert claims into arguments by generating plausible supporting reasons, then evaluating the resulting argument premise-by-premise. If claims with generated premises show higher agreement than bare claims, this would suggest that the subjectivity problem is not inherent to the criticism itself but to its underspecification—and that automated "argument completion" could extend the ADVANCED REVIEWSCORE approach to claims, addressing the paper's most significant failure mode. If agreement does not improve, it would confirm that certain review content (value judgments about novelty, significance, triviality) is fundamentally outside the scope of factuality-based evaluation.

Cross-conference and cross-domain generalization of the 15–26% misinformed rate. The paper's finding that 15.2% of weaknesses and 26.4% of questions are misinformed is based on 40 ICLR 2021–2023 papers selected by annotator groups with shared research interests (non-random sampling). A replication study across venues and fields would: (a) sample reviews from NeurIPS, ICML, ACL, and EMNLP (both AI and NLP, varying review norms), (b) sample papers randomly rather than by annotator interest to reduce selection bias, (c) apply the same REVIEWSCORE annotation protocol (with new annotators matched to paper topics), and (d) compare misinformed rates across venues and correlate them with known conference characteristics (acceptance rate, reviewer pool size, review length requirements). This would establish whether the 15–26% figures are ICLR-specific, AI-specific, or broadly representative of ML/NLP peer review. The paper's inter-group variance in inter-annotator agreement (Krippendorff's Alpha 0.254–0.663, Table 9) already suggests that subfield and paper selection matter. A cross-conference study would also test whether certain review norms (e.g., structured review forms with separate "questions" and "weaknesses" fields) naturally reduce misinformed rates by forcing reviewers to separate their content.

Combining the argument reconstruction engine with retrieval-augmented generation for referred paper verification. A significant fraction of premises in the REVIEWSCORE dataset involve claims about other papers (the "referred papers" knowledge base option in Definition 4). The current evaluation setup assumes the model has "a general understanding of referred papers" (Section 4.1) without providing the actual referred paper text. A follow-up could: (a) identify all premises in the dataset where the knowledge base is "referred papers," (b) retrieve the actual referred papers (they are cited in the review or the submitted paper), (c) provide the referred paper text to the model alongside the submitted paper, and (d) measure whether having the ground-truth referred paper improves premise-level factuality agreement. This tests whether the moderate agreement on referred-paper premises is due to model knowledge limitations (which retrieval would fix) or genuine ambiguity in comparing claims across papers (which retrieval would not fix). If agreement improves substantially, it suggests that retrieval-augmented REVIEWSCORE evaluation is the practical path forward. If not, it suggests that cross-paper factual claims are inherently harder to verify than within-paper claims, an important boundary condition on the approach.

Developing an adaptive, difficulty-aware review evaluation pipeline. The paper's results show that review point type (claim vs. argument vs. question) is the primary axis of evaluation difficulty, with claims being hardest and questions easiest (Table 2). This suggests an adaptive pipeline: (a) classify the review point type, (b) for questions, apply LLM-based unanswerability scoring (highest agreement, most deployable), (c) for arguments, run the reconstruction engine and premise-level evaluation (moderate agreement, requires reconstruction cost), (d) for claims, flag as "requires human review" rather than attempting automated evaluation (lowest agreement, least reliable). A follow-up would implement this pipeline, measure the coverage (fraction of review points that can be automatically evaluated with sufficient confidence), and estimate the cost savings over all-manual review. The key metric would be: what fraction of misinformed review points can be detected automatically with high precision, and what fraction of review points must be escalated to human meta-reviewers? The paper's current results (Table 2 precision: 0.365–0.421 for overall REVIEWSCORE) suggest that a high-precision configuration—accepting only high-confidence model predictions—could be practical even with moderate average agreement, but this requires confidence calibration analysis that the paper does not provide.

Practical Applications and Downstream Use Cases

Automated review quality screening for meta-reviewers and area chairs. The most direct application: before area chairs read reviews to make acceptance decisions, an automated REVIEWSCORE pipeline could flag review points that are likely misinformed—questions already answered by the paper and weaknesses resting on factually incorrect premises. The paper's results suggest that for questions (QScore), this is the most deployable component: QScore shows the highest human-model agreement (F1 0.476–0.579, Table 2) and the highest inter-annotator agreement (Krippendorff's Alpha 0.554 median, Table 9). An area chair reviewing a paper with three reviewer reports could see, for each review, a flag on specific questions that the system judges as already answerable by the paper, with citations to the relevant paper sections. This would directly operationalize the ACL 2023 guideline ("before writing a negative review, check whether your questions are already answered") and help area chairs identify when reviewers may not have read the paper carefully. The 26.4% misinformed question rate (Section 1) suggests this would flag roughly one in four reviewer questions, a substantial fraction. The main risk is false positives (flagging genuinely unanswerable questions as answerable), which the moderate F1 scores indicate is a real concern—hence the tool should be assistive, presenting flags with evidence for human verification rather than making automatic decisions.

Author rebuttal assistance: surfacing the logical structure of reviewer criticisms. The argument reconstruction engine (Section 3.2, Figure 2b) produces explicit premise-conclusion structures from reviewer arguments. Beyond evaluation, these reconstructions are directly useful to authors writing rebuttals: they show which specific factual claims the reviewer's criticism depends on, making it clear where to focus the rebuttal. For example, from the ViT review example (Figure 2b), the reconstructed premises (P1–P5) reveal that the reviewer's criticism hinges on the premise that "if the paper claims avoiding inductive biases is an advantage, then avoiding inductive biases is an advantage" (P2). An author can see this and directly address whether they intended to claim that all inductive biases should be avoided, or whether the claim was more nuanced (e.g., avoiding certain inductive biases for flexibility). This use case does not require perfect automated factuality evaluation—it only requires that the reconstruction is faithful to the original argument (4.47/5 faithfulness, Table 5), which the paper demonstrates is achievable. The benefit is reduced author time spent interpreting vague reviewer criticisms and more targeted, productive rebuttals.

Reviewer self-check during review writing. REVIEWSCORE could be integrated into the review writing workflow (e.g., OpenReview's review form) to provide real-time feedback to reviewers before submission. When a reviewer writes a question, the system could check whether the answer is already in the paper and prompt: "This question appears to be addressed in Section 3.2 of the submission. Consider whether the existing discussion answers your concern, or rephrase the question to target what is still unclear." When a reviewer writes an argumentative weakness, the system could reconstruct the premises and flag any that appear factually inconsistent with the paper text. The value here is preventative: catching misinformed review points before they reach authors and area chairs, reducing the burden on the rebuttal and decision processes. The feasibility depends on latency—the reconstruction engine takes an average of 3.09 iterations (Table 5), each requiring multiple LLM calls, which may be too slow for real-time feedback. But for asynchronous review writing (where reviewers compose over days), even seconds-to-minutes latency is acceptable. The main risk is that reviewers ignore or over-rely on the feedback, but as an optional assistive tool, the downside is minimal.

Calibrating reviewer quality in reviewer assignment and incentivization systems. Conferences increasingly track reviewer quality for assignment (matching good reviewers to borderline papers) and incentivization (recognizing high-quality reviewers). REVIEWSCORE provides a specific, measurable dimension of review quality: the fraction of a reviewer's review points that are misinformed. If a particular reviewer consistently has 40% of their questions flagged as answerable by the paper (compared to a conference average of ~26%), this suggests they tend to skim papers or miss details. If a reviewer has 30% of their weaknesses flagged as resting on incorrect premises, this suggests they tend to misunderstand technical content. These metrics could inform reviewer training (targeted feedback on specific error patterns), reviewer assignment (pairing detail-oriented reviewers with papers that need careful reading), or reviewer recognition (highlighting reviewers with consistently low misinformed rates). The current moderate LLM agreement (F1 0.4–0.5) means this application is not yet reliable for high-stakes decisions (e.g., removing reviewers from the pool), but it could be used for aggregate statistics across many reviews per reviewer and many reviewers per conference, where individual errors average out. A conference with 5,000 reviews and 3,000 reviewers could compute average misinformed rates per reviewer and identify outliers (reviewers >2 standard deviations above the mean) for human review of their review quality—a lower-stakes, higher-volume use case where imperfect automation still adds value.