AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context

ArXiv: 2601.19494

🎯 Pitch

Simply giving LLMs more context for code review can slash their precision by over half—unless they use an Agent architecture, in which case precision surges but recall collapses. This benchmark, built from 200 PRs across 10 languages with 285%-expanded defect annotations, reveals that the optimal strategy flips entirely depending on the model, language, and retrieval method.

1. Executive Summary

This paper introduces AACR-Bench, the first multilingual, repository-level context-aware benchmark for evaluating Large Language Models in Automated Code Review, constructed from 200 PRs across 50 repositories spanning 10 programming languages with 1,505 expert-annotated review comments. The benchmark employs an "AI-assisted, Expert-verified" annotation pipeline that augments raw GitHub PR comments with LLM-generated defect candidates—verified by over 80 senior engineers—yielding a 285% increase in defect coverage over traditional datasets. Through comprehensive evaluation of mainstream LLMs (including GPT-5.2, Claude-4.5-Sonnet, DeepSeek-V3.2, and Qwen3-Coder-480B) under different context retrieval methods (BM25, embedding-based retrieval, and Agent-based frameworks like Claude Code), the paper reveals that the granularity of context and choice of retrieval method significantly impact ACR performance, with Agent-based approaches achieving dramatically higher precision (e.g., Claude-4.5-Sonnet reaching 39.90% in Agent mode versus 8.70% without context) but at the cost of substantially lower recall, establishing that context-aware code review is not universally beneficial—different models exhibit distinct optimal retrieval strategies, and the effectiveness of providing repository-level context varies entirely depending on the programming language and whether an Agent architecture is employed.

2. Context and Motivation

The Core Problem: ACR Benchmarks Don't Test What Matters in Production

This paper targets a fundamental disconnect between how automated code review systems are evaluated and how they must operate in real-world software development. The central problem is not that ACR benchmarks don't exist—several do—but that existing benchmarks systematically fail to capture two dimensions of difficulty that define actual code review: the need to reason across file boundaries and the reality that review comments in historical data are incomplete and noisy.

The gap the paper identifies can be understood through a concrete scenario. Imagine an LLM reviewing a pull request that modifies a function signature. The change itself appears in a single diff hunk—perhaps swapping two parameter types. A naïve reviewer (or an LLM restricted to diff-only context) might check that the new types are internally consistent within the function body and approve. But a competent human reviewer would ask: where is this function called? Are callers updated? Does this change violate invariants in code that imports this module? Answering those questions requires traversing cross-file dependencies—reading code that never appears in the diff but controls whether the change is safe.

Existing benchmarks, the paper argues, would not penalize an LLM that missed those cross-file issues because those benchmarks either (a) provide only the diff itself, with no way to verify cross-file reasoning, or (b) use ground truth derived from raw PR comments, where human reviewers themselves may have overlooked the cross-file implications. The result is that evaluations systematically overestimate LLM capability by testing on a simplified version of the task.

This is not a minor methodological quibble. It means that ACR systems deployed on the basis of optimistic benchmark results may silently miss the very defects that make code review valuable in practice—the subtle, context-dependent bugs that arise from interactions between components.

Why This Matters: The Stakes of Software Quality and Developer Productivity

The practical stakes are high and operate at two levels.

At the individual pull request level, missed defects carry compounding costs. A security vulnerability that passes review enters the codebase, potentially persisting through multiple release cycles before discovery. The later a defect is caught, the more expensive it is to fix—a well-established principle in software engineering that applies with full force to LLM-assisted review. If an ACR system misses cross-file security issues because its benchmark never tested cross-file reasoning, the downstream cost is born by users and maintainers who trust the tool's assessment.

At the organizational level, the efficiency argument is equally important. Code review is among the most time-consuming activities in modern software development, with studies consistently showing it consumes 10–15% of engineering time. If LLMs can reliably automate even a fraction of that burden, the productivity gains are substantial. But reliability is the operative word: a tool that generates false positives erodes trust and waste reviewer time (a concern the paper explicitly echoes in its Agent prompt design in Figure 18), while false negatives—missed defects—defeat the purpose. The benchmark's ability to measure both precision and recall under realistic conditions is therefore directly tied to the economic case for deploying ACR systems at scale.

The paper also touches on a deeper theoretical concern: language-specific bias in evaluation. If all benchmarks are Python-only (as the paper notes for SWR-Bench and CodeFuse-CR-Bench), then a model that happens to be strong on Python—because of abundant training data in that language—may appear generally competent, when in fact its performance collapses on C, Rust, or PHP. Organizations working in polyglot codebases (common in large enterprises) need benchmarks that surface these disparities, not conceal them behind single-language test sets.

Where Prior Work Falls Short

The paper organizes prior benchmarks along two axes, each with a critical deficiency.

Single-language, repository-level benchmarks (SWR-Bench, CodeFuse-CR-Bench). SWR-Bench, derived from SWE-Bench's 12 Python projects, and CodeFuse-CR-Bench, built from 70 Python projects, both provide full repository context—PR metadata and complete repository code. This is the right idea: they recognize that code review requires understanding the broader codebase. But their exclusive focus on Python introduces two problems. First, generalizability: findings on Python may not transfer to languages with different semantics, type systems, or dependency structures. Second, structural bias: Python's characteristics (dynamic typing, import semantics, relatively simple compilation model) may systematically advantage certain review strategies or model architectures, making the benchmark an unreliable predictor of performance on, say, Rust (with its borrow checker and strict lifetime semantics) or C (with manual memory management and preprocessor macros). The paper's finding that C and C++ performance patterns differ sharply from Python—with context often degrading performance on C# and Python while improving it on Go and Java—would be invisible in a Python-only benchmark.

Multi-language, file-level benchmarks (ContextCRBench). ContextCRBench addresses the language diversity problem, covering 9 languages across 90 repositories. But it restricts context to the file level: the LLM sees the diff and the full file being modified, but not cross-file dependencies. This means ContextCRBench can evaluate whether an LLM catches issues that are self-contained within a single file (e.g., a logic error visible in the diff), but cannot evaluate whether it catches issues that require understanding of callers, callees, or type definitions in other files. These are precisely the issues that benefit most from automated tooling—they are cognitively demanding for humans to track—so excluding them from evaluation systematically inflates apparent performance.

Diff-only benchmarks (CodeReviewer). CodeReviewer, the most widely adopted ACR evaluation dataset, provides only diff-level code fragments. It is the simplest to construct and has been used in numerous subsequent studies, but its limitation is the most severe: it tests the LLM's ability to spot issues within isolated code changes, which is a fundamentally easier task than production code review. A model that achieves high scores on CodeReviewer may in practice fail to detect that a seemingly clean function signature change breaks three call sites in separate modules.

The deeper data quality problem: incomplete and noisy ground truth. Beyond context scope, the paper identifies a subtler but equally consequential issue: existing benchmarks treat raw PR comments as ground truth without addressing two well-documented problems in code review data. First, label incompleteness: as Bacchelli and Bird (2013) established, human code reviews in practice are "often insufficient"—reviewers miss defects due to cognitive load, time pressure, or lack of domain expertise. Using these incomplete reviews as ground truth means the benchmark cannot penalize models for missing defects that human reviewers also missed, creating a ceiling effect where all models (and humans) appear equally effective at low recall. Second, comment noise: raw PR comments contain conversational elements (discussions, information sharing), non-substantive approvals ("LGTM", "Merge"), and multi-turn dialogues where the actual defect is only clarified over several exchanges—none of which map cleanly to an evaluation metric. The paper cites Rong et al. (2024) on this point, building on established findings in the mining software repositories community.

How This Paper Positions Itself

The paper explicitly frames itself as synthesizing the strengths of prior benchmarks while addressing their weaknesses through three design decisions.

First, multilingual + repository-level context. AACR-Bench spans 10 programming languages (JavaScript, Python, TypeScript, Java, C#, C++, C, PHP, Go, Rust) drawn from the StackOverflow Developer Survey 2025 top-10 list, with 5 repositories per language selected by a recency and activity criterion (top 2,000 repositories by new stars and closed PRs from December 2024 to December 2025). This directly addresses the generalizability gap in SWR-Bench and CodeFuse-CR-Bench. The paper does not claim that 10 languages is exhaustive, but rather that it is sufficient to surface language-specific performance patterns that single-language benchmarks mask—a claim the experimental results in Section 4.2.3 substantiate.

Second, AI-assisted expert annotation to address label incompleteness. Rather than simply curating raw PR comments, the paper augments them with defect candidates generated by two heterogeneous review systems (an internal system and the open-source Claude Code agent) powered by 6 different LLMs, then subjects all candidates—both human-origin and model-origin—to expert verification by over 80 senior engineers. Importantly, the annotation process is not a rubber stamp: it involves three rounds, including double-blind independent review in the first two rounds and adjudication by a core expert team in the third. The result is a 285% increase in defect coverage relative to the original PR comments (391 augmented from human reviews vs. 1,114 generated by LLMs), meaning the benchmark can penalize models for missing defects that raw-PR baselines wouldn't capture. The paper explicitly notes that validation accuracy on the augmentation pipeline was 95% (at a 95% confidence level with 4.74% margin of error), giving statistical backing to the annotation quality claim.

Third, hierarchical context annotation. A distinctive feature is the labeling of each review comment with the scope of context required to detect the issue—Diff level, File level, or Repository level (Table 1). This is a novel metadata dimension that enables a type of analysis not possible with prior benchmarks: measuring not just overall accuracy, but accuracy as a function of required context depth. This directly tests the paper's central hypothesis that context retrieval methods matter differently across context levels, and it enables the finding in Section 4.2.2 that non-Agent methods show monotonically declining recall as required context scope increases (Diff > File > Repo), while Agent-based methods invert this trend.

The paper's positioning is therefore one of filling a measurement gap, not proposing a new ACR method. It does not argue for a particular model architecture or retrieval strategy. Instead, it provides the measurement infrastructure to determine which strategies work under which conditions, and its empirical contributions are the discoveries that emerge from that measurement: that Agent-based and traditional methods occupy different points on the precision-recall frontier, that context retrieval can harm performance as much as help it, and that language-specific effects dominate any universal claims about what works. These findings, the paper implies, are only visible with a benchmark that spans languages and context levels simultaneously—precisely what AACR-Bench provides.

3. Technical Approach

3.1 Reader Orientation

AACR-Bench is not a new code review model or method—it is a measurement instrument: a carefully constructed dataset and evaluation protocol designed to test how well LLMs perform automated code review under conditions that mirror real-world software development. The core problem it solves is that existing benchmarks evaluate ACR systems on a simplified version of the task—either by restricting the programming language to Python alone, by limiting context to single files rather than entire repositories, or by using incomplete ground truth derived from raw PR comments that miss many defects human reviewers overlooked. AACR-Bench's solution takes the form of a multilingual, repository-level, expert-augmented benchmark that makes all three dimensions of difficulty explicit and measurable, enabling researchers to quantify not just how accurate a model is overall, but how its accuracy varies with context scope, programming language, and retrieval strategy.

3.2 Big-Picture Architecture

AACR-Bench is a benchmark dataset + evaluation framework with five major components. The diagram flows left to right through data collection, then annotation, then evaluation execution:

Repository and PR Selection Pipeline — Selects 50 GitHub repositories (5 per language × 10 languages) based on recency and activity metrics, extracts 200 PRs meeting quality criteria from an initial pool of 12,715, and applies stratified sampling to ensure diversity across problem domains and change sizes.
Review Comment Augmentation Pipeline — Takes raw multi-turn PR review comments and uses an LLM to distill confirmed code defects into clean "Augmented Review Comments," filtering out conversational noise and non-substantive remarks while preserving technical substance.
LLM-Based Defect Generation + Expert Annotation Pipeline — Generates additional defect candidates using two heterogeneous review frameworks (an internal system plus the open-source Claude Code agent) powered by 6 different LLMs, then subjects all candidates to three-round expert verification by over 80 senior software engineers, producing 1,505 verified ground-truth review comments across four issue categories and three context-dependency levels.
Evaluation Execution Framework — For each PR in the benchmark, the ACR system under test iterates through all diff hunks generating review comments. The framework supports multiple context retrieval modes: no context (baseline), BM25 retrieval, embedding-based retrieval (via Qwen3-Embedding-8B), and Agent-based operation (via Claude Code). All modes receive the PR title and description; retrieval modes additionally receive top-3 relevant code snippets from the repository.
Metric Computation Module — Compares generated review comments against the 1,505 ground-truth items using Precision, Recall, and F1-score, computed both globally and disaggregated by context level, programming language, and retrieval method.

Information flows as follows: the pipeline first constructs the benchmark (steps 1–3: collecting PRs, augmenting comments, expert-annotating defects), then the evaluation framework executes (step 4: model generates comments on each diff hunk under a specified context mode), then metrics are computed (step 5: matching generated comments to ground truth). The benchmark construction is a one-time process; evaluation is repeated for each model–method combination.

3.3 Roadmap for the Deep Dive

First, the repository and PR selection criteria, because the statistical properties of the resulting dataset (language distribution, problem domain coverage, PR size distribution) determine what claims the benchmark can support.
Second, the review comment augmentation procedure, since it transforms noisy raw PR data into the clean "Augmented Review Comments" that form one source of ground truth (391 of 1,505 items).
Third, the LLM-based defect generation and expert annotation pipeline, which is the paper's core methodological contribution—this produces the remaining 1,114 ground-truth items and establishes the benchmark's claim of 285% increased defect coverage.
Fourth, the evaluation execution protocol for non-Agent and Agent-based methods, including the specific prompts, context retrieval configurations, and output parsing rules, since the paper's empirical findings depend on precisely how models are queried.
Fifth, the metric computation methodology, including how generated comments are matched to ground truth and what the paper counts as a hit, miss, or false positive.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a benchmark construction and evaluation methodology paper whose core idea is that measuring ACR capability requires simultaneously varying context scope and programming language while using ground truth that captures defects missed by human reviewers—and that the resulting measurement surface reveals context-dependence patterns invisible to single-language, diff-only, or noisy-ground-truth benchmarks.

Repository and PR Selection: The Statistical Foundation

Objective. The benchmark must represent real-world code review across diverse languages without being so large that expert annotation becomes infeasible. The selection pipeline therefore applies a series of filtering and sampling steps to reduce an initial pool of thousands of PRs to a manageable 200 while preserving diversity along multiple dimensions.

Programming language selection. The 10 languages are chosen based on the StackOverflow Developer Survey 2025 top-10 ranking: JavaScript, Python, TypeScript, Java, C#, C++, C, PHP, Go, and Rust. This is an explicit recency criterion—the paper uses the 2025 survey, not historical rankings—to ensure the benchmark reflects contemporary development practices rather than legacy language distributions. The choice is not claimed to be exhaustive but rather sufficient to surface language-specific performance patterns, which the experimental results in Section 4.2.3 validate by showing substantial variance across languages.

Repository selection. For each language, the paper defines a candidate set of active GitHub repositories meeting two simultaneous criteria: (1) ranked within the top 2,000 repositories by newly acquired stars between December 1, 2024, and December 1, 2025, and (2) ranked within the top 2,000 repositories by closed pull requests in the same period. The intersection of these two rankings ensures that selected repositories are both popular (stars as a proxy for community interest) and actively developed (closed PRs as a proxy for ongoing engineering work). From this candidate pool, the top 5 repositories by new star count are selected per language, yielding a final set of 50 repositories (5 × 10 languages).

The one-year window (December 2024 to December 2025) is deliberately chosen: it ensures the benchmark contains recent code and review practices, avoiding the staleness problem that affects benchmarks constructed from older repositories where coding conventions, language features, and review norms may have shifted. The paper explicitly states that this recency criterion is applied to produce "evaluation data that reflects contemporary development."

Initial PR extraction. Using the GitHub API, the paper collects all PRs created within the December 2024–December 2025 window from the 50 selected repositories, yielding 12,715 records. For each PR, the following metadata is extracted:

Title and description (natural language fields providing author intent and change rationale)
Number of lines of code changed (for size filtering)
The base commit (for later repository checkout and diff computation)
Comprehensive review comment data: target revision, file paths, line ranges, and comment content

Preprocessing: domain classification, language identification, revision selection, size categorization. Before filtering, the pipeline enriches each PR with structured metadata:

Problem domain classification: Using the Qwen3-235B-A22B-Thinking-2507 model, each PR is classified into one of nine categories adapted from SWE-Bench: Bug Fix, New Feature Addition, Code Refactoring/Architectural Improvement, Documentation Update, Test Suite/CI Enhancements, Performance Optimization, Security Patch/Vulnerability Fix, Dependency Update/Environment Compatibility, and Code Style/Linting/Formatting Fix. The prompt used (Figure 5) provides few-shot examples and requires output as a category code. Manual verification on 350 randomly sampled instances showed 92.36% accuracy with a 95% confidence interval margin of error of ±5.06%.
Natural language identification: The language of the PR title and description is classified.
Revision selection: For PRs with multiple revisions, the pipeline identifies and extracts the revision containing the highest number of inline review comments, along with all corresponding review data for that revision.
Size categorization: Each PR is classified by the number of changed lines using a T-Shirt size scheme (XS through XXL), as detailed in Table 6.
Reviewed language identification: The primary programming language of the files targeted by review comments is identified from file extensions.

Five filtering criteria. From the preprocessed pool, PRs are retained only if they satisfy all of:

The PR title and description must be in English (language standard to enable uniform LLM prompting).
Changed lines of code must be ≤ 1,000 (justified by Google's code review practice that changes exceeding this threshold are too large for effective review—this is a direct citation of established industry practice, not an arbitrary cutoff).
The primary programming language of the modified files must match the repository's main language (preventing polyglot PRs from contaminating per-language analysis).
The PR must contain > 2 inline comments, including at least one constructive comment that was adopted and led to code modification (ensuring the PR has substantive review activity, not just superficial feedback).
Semantic validity: after automated filtering, a second round of manual verification excludes "trivial changes that were detached from the project's business context or lacked actual semantic meaning."

After automated filtering, 3,328 PRs pass criteria 1–4. After the manual semantic validity check (criterion 5), 573 PRs remain.

Stratified sampling to 200 PRs. The final reduction from 573 to 200 is performed via stratified sampling along three dimensions: source repository (to prevent any single repository from dominating), problem domain (to ensure representation across bug fixes, features, refactoring, etc.), and PR size (to ensure representation across change magnitude). The paper reports the resulting distributions in Figures 6 and 7: the 200 PRs span all nine problem domain categories (with Bug Fix being the modal category) and the full T-Shirt size range, with Small and Medium being the most common sizes.

Why this sampling strategy over alternatives. The paper's design choices reflect tradeoffs between representativeness and annotation cost. An alternative would be random sampling from all open-source PRs—but this would yield a dataset dominated by trivial changes, single-language repositories, and PRs with minimal review activity. The filtering criteria (especially the >2 inline comments with adoption requirement) bias the sample toward PRs where review actually occurred, which is necessary for the benchmark to have ground truth at all. The stratified sampling preserves diversity along known axes of variation (domain, size) while keeping the total count manageable for expert annotation. The 200-PR target is not theoretically derived—it reflects a practical budget for the 80+ engineer annotation effort—but the paper argues that the diversity constraints (10 languages, 5 repos each, stratified domains) make the sample sufficiently representative for comparative evaluation even if it cannot claim statistical representativeness of all GitHub activity.

Review Comment Augmentation: From Noisy Threads to Clean Defect Statements

The problem with raw PR comments. Code review on GitHub typically involves multi-turn threaded discussions: a reviewer posts an initial comment, the author responds, the reviewer clarifies, agreement is reached (or not), and code may be modified in subsequent revisions. The actual technical defect being discussed—"this null check is missing" or "this loop has an off-by-one error"—is often distributed across several messages, embedded in conversational context, and intermixed with non-technical remarks. Using such threads directly as ground truth creates two evaluation problems: (1) the defect itself may be ambiguous, making it unclear what constitutes a correct detection by an ACR system, and (2) non-defect comments (approvals, discussions, style preferences) add noise that inflates false positive rates if counted as ground truth or false negative rates if counted as issues an ACR system should detect.

The augmentation procedure. For each selected PR revision, the paper identifies all review comment threads associated with code diff hunks. An LLM (the specific model is not named in the main text but is described as "an LLM to perform deep semantic analysis") processes each thread with the prompt shown in Figure 8. The prompt instructs the model to:

Analyze both the conversation text and the associated diff hunk
Extract "the most concise and actionable review feedback"
Output 1–2 sentences capturing the core technical issue with minimal context
Handle edge cases: if the thread concludes no issue exists, output "No code issues identified"; if multiple issues are discussed, output bullet points

The model is explicitly instructed to "Focus on technical substance, not conversational details" and to produce output that is "direct and specific about what needs attention."

Augmentation quality validation. The augmentation process was applied to 1,119 conversation threads. The paper evaluated correctness on a random sample of 300 augmented results, where "correct" was defined as either accurately identifying confirmed code issues from the thread or correctly determining that the thread contained no code issues. The validation showed 95% accuracy at a 95% confidence level with a 4.74% margin of error. This is a critical quality gate: it establishes that the augmentation step does not introduce substantial errors that would contaminate the ground truth.

Output format. The augmented comments are standalone defect descriptions—for example, "The function processData does not handle the case where input is null, which will cause a NullPointerException at line 47 when input.length is accessed." They are stripped of conversational context, reviewer identity, and temporal markers, making them directly comparable to the output an ACR system would generate (a single, focused defect statement).

Design rationale. Why not just use the raw threads directly and ask annotators to extract defects? The paper's implicit argument is that the augmentation step front-loads the extraction work onto an LLM (which is fast and cheap at scale) and reserves expert human effort for verification (which is expensive but essential for quality). The 95% validation accuracy suggests this division of labor is effective: the LLM performs the tedious extraction task with high reliability, and humans verify correctness rather than performing extraction from scratch. An alternative—having humans read full threads and write defect descriptions—would be slower, more expensive, and potentially less consistent across annotators.

Relationship to the final benchmark. The augmented comments constitute 391 of the 1,505 total ground-truth items (approximately 26%). They are labeled "Aug" in Figure 2's distribution chart. These represent defects that were actually identified by human reviewers during real PR review but have been cleaned into a standardized format.

LLM-Based Defect Generation and Expert Annotation: The Core Methodological Innovation

This pipeline is the paper's most distinctive contribution and the mechanism by which AACR-Bench achieves its claimed 285% increase in defect coverage. The logic is as follows: human reviewers, constrained by time, cognitive load, and domain knowledge, miss many defects in PRs (a finding the paper cites from Bacchelli and Bird, 2013). A benchmark whose ground truth consists only of defects that humans did catch cannot measure whether an ACR system catches defects that humans missed. To enable that measurement, the benchmark must include defects that are genuinely present in the code but were not identified in the original PR review. The paper uses LLMs to propose such defects and then uses expert human engineers to verify them, creating a ground truth set that includes both human-identified and human-verified model-identified defects.

Generation Matrix: Six Models × Two Frameworks

Model selection. The paper constructs a generation matrix of six mainstream models:

Three open-source models: Qwen3-Coder-480B-A35B-Instruct, GLM-4.7, DeepSeek-V3.2
Three commercial models: Claude-4.5-Sonnet, GPT-5.2, Gemini-3-Pro

The explicit rationale is to "mitigate single-model bias and ensure output diversity." If only one model generated defect candidates, the resulting ground truth might reflect that model's idiosyncratic strengths and weaknesses—for example, a model that is particularly strong at catching null-pointer issues but weak at concurrency bugs would produce a benchmark that over-rewards null-pointer detection and under-rewards concurrency detection. Using six diverse models (open-source and commercial, with different architectures and training distributions) increases the likelihood that the generated candidate pool spans the space of detectable defects broadly.

Two heterogeneous generation frameworks. The six models generate review comments through two distinct frameworks operating in parallel:

An Internal Review System — a custom-built ACR pipeline (details are not specified in the paper, but it is described as a system that processes diffs and generates review comments, presumably with some form of context retrieval or structured prompting).
Claude Code — the open-source agent framework that supports autonomous code review with tool-use capabilities (repository navigation, git operations, multi-step reasoning).

The two frameworks are described as "heterogeneous" to suggest they approach the review task differently: the internal system likely follows a traditional scan-and-comment paradigm (iterate through diffs, retrieve context, prompt an LLM to comment), while Claude Code operates as an agent (autonomously exploring the repository, making tool calls, reasoning across multiple steps). Using both frameworks further diversifies the generation process and reduces the risk that the benchmark reflects a single architectural approach to ACR.

Semantic de-duplication. After all models generate comments on all PRs, there will be substantial overlap—multiple models detecting the same defect. To avoid presenting annotators with redundant candidates, the pipeline performs semantic de-duplication using Qwen3-235B-A22B-Thinking-2507. The procedure:

Comments are grouped by repository, PR, file path, and the specific diff hunk they address.
Within each group, comments are compared pairwise by the LLM using the prompt in Figure 9.
The LLM determines whether two comments "express the same concern or suggestion regarding the provided diff hunk," with the explicit instruction to "Ignore differences in wording, tone, or formatting—focus solely on semantic equivalence of the underlying issue."
To increase robustness, the paper uses an election of results from 5 repeated judgments—meaning a pair is considered duplicate only if a majority of the 5 judgments agree.

This de-duplication step is important for annotation efficiency (fewer items to verify) and for preventing the ground truth from over-weighting defects that happen to be detected by many models (which would bias the benchmark toward model-consensus defects).

Merging with augmented human comments. The de-duplicated LLM-generated comments are merged with the 391 augmented human review comments to form a "candidate set for verification." This candidate set therefore contains both refined versions of defects originally caught by humans and novel defects proposed by models that may or may not be genuine.

Why this generation approach? The alternative—having human experts review all 200 PRs from scratch and write down all defects they find—would produce ground truth but at immense cost (200 PRs × average 7.5 defects = 1,500 items to identify, but experts would need to read full codebases to identify cross-file defects). The paper's hybrid approach leverages LLMs to do the expensive search (scanning code for potential issues) and reserves humans for the higher-value judgment task (verifying whether a proposed issue is genuine). The distribution of detection frequency (Table 8) validates the diversity rationale: only 11 comments achieved consensus among 3 models, while a "significant portion was identified by only a single model." This low overlap confirms that using multiple models captures defects that any single model would miss.

Expert Annotation: Three-Round Verification

Annotation workforce. The paper recruited over 80 senior software engineers, each with more than two years of company experience. The workforce was organized into a Core Expert Team of 6 members and a General Annotation Pool comprising the remaining participants. Task allocation was strictly matched to annotators' programming language expertise—annotators reviewed code only in languages they were professionally competent in, which is essential for reliable defect verification.

Three annotation dimensions. For each candidate review comment in the verification set, annotators assessed:

Correctness: Is the identified issue a genuine defect? This is a binary judgment: the comment either correctly identifies a real problem in the code or it does not (false positive, hallucination, or misunderstanding of the code).
Issue type categorization: Each verified correct comment is classified into one of four categories (defined in Table 7):
- Security Vulnerability: a point in the code open to attack
- Code Defect: a coding mistake that can lead to an error or unexpected behavior at runtime
- Maintainability and Readability: an issue making code confusing and difficult to maintain
- Performance: inefficient algorithms, improper resource usage, poor concurrency handling, or architectural flaws causing excessive response times, insufficient throughput, high resource consumption, or limited scalability
Context dependency scope: The level of context required to formulate the comment, defined in Table 1:
- Diff level: the issue is entirely contained within the changed code shown in the diff—no additional context beyond the diff hunk is needed to understand and verify the defect.
- File level: the issue requires understanding the full file being modified—for example, a change that is inconsistent with other functions in the same file, or that violates invariants established elsewhere in the file.
- Repository level: the issue requires cross-file reasoning—for example, a changed function signature whose callers in other files are not updated, or a new dependency that introduces a security vulnerability through transitive includes.

This third dimension is AACR-Bench's most innovative annotation feature. It transforms the benchmark from a flat accuracy measurement into a stratified capability measurement: researchers can measure not just whether a model is accurate, but whether its accuracy degrades (or improves) as the required context scope expands. This directly operationalizes the paper's central research question about context-dependence in ACR.

Three-round annotation protocol. The process is designed to produce high-reliability labels through structured disagreement resolution:

Rounds 1 and 2 (double-blind independent annotation): Each comment is independently annotated by two different annotators from the General Annotation Pool, with neither annotator seeing the other's judgments. Task allocation respects language expertise. This double-blind design prevents anchoring bias and enables measurement of inter-annotator agreement.
Round 3 (adjudication by Core Expert Team): The Core Expert Team of 6 members reviews all cases where the two independent annotations disagree. The experts discuss the conflicting results and determine the final annotation. This is not a simple majority vote—it involves discussion and consensus, which allows resolving ambiguities that mechanical voting would miss (e.g., one annotator interpreting a comment as a Code Defect while the other interprets it as a Performance issue—the expert discussion can determine the appropriate category based on deeper analysis).

Final benchmark composition. After annotation, only comments verified as correct are retained as ground truth. The final AACR-Bench contains 1,505 review comments:

391 augmented from original human reviews (the "Aug" category in Figure 2)
1,114 generated by LLMs and verified by experts (the "Gen" category in Figure 2)

This represents a 285% increase relative to the 391 comments that would constitute the ground truth if only raw human reviews were used. Figure 2 shows the language distribution: Python, TypeScript, and JavaScript are the most represented languages (consistent with their popularity on GitHub), while PHP, C, and Rust have smaller counts (consistent with smaller representation in the selected repositories).

The complementarity evidence (Table 8). The paper analyzes detection overlap to validate the multi-model generation strategy. The distribution of the 1,114 LLM-generated comments by how many models detected them:

Only 11 comments (approximately 1%) were detected by 3 models—essentially no consensus items
The overwhelming majority were detected by a single model

This low overlap is the key evidence for the paper's claim that "the necessity to include multi-model results in review comment generation process for better issue coverage." If models largely agreed on which defects exist, a single model would suffice. The empirical finding that they do not agree justifies the paper's choice to use six models.

Why expert annotation rather than automated verification? Automated verification of code review comments—determining whether a stated defect genuinely exists—is itself an unsolved AI problem (it is essentially the same code understanding task that ACR systems are trying to perform). Using an LLM to verify LLM-generated comments would create a circular dependency where the verifier's errors are confounded with the generator's errors. Human expert verification breaks this circularity: the ground truth is established by a process (expert human judgment) that is independent of the systems being evaluated, even if the proposals were LLM-generated. The paper's three-round protocol with double-blind design and expert adjudication provides explicit quality guarantees that automated verification cannot.

Evaluation Execution Protocol: How Models Are Tested

The evaluation protocol differs fundamentally between non-Agent and Agent-based methods because the two paradigms handle context retrieval differently.

Non-Agent Methods: Standardized Prompt-and-Scan

Setup. For each PR in the benchmark, the evaluation framework:

Clones the repository locally and ensures both the Base version (pre-PR) and Target version (post-PR) states are available.
Extracts all diff hunks between the two versions using GitPython (a Python library wrapping git operations).
For methods incorporating context retrieval, builds a code index on the Target version to enable similarity-based retrieval.

Context retrieval configurations. The paper evaluates four context modes:

No context: The model receives only the PR title, PR description, and the current diff hunk. This serves as the comparative baseline.
BM25: A classic sparse retrieval method (bag-of-words with TF-IDF weighting, as formalized by Robertson et al., 2009). The model receives the top-3 most similar code snippets from the repository, retrieved by BM25 scoring against the diff hunk content. The choice of top-3 is consistent across BM25 and embedding methods and is not theoretically justified—it represents a practical default that balances context informativeness against prompt length.
Embedding-based: Uses Qwen3-Embedding-8B—described as "one of the current state-of-the-art models"—to encode code snippets and the diff hunk into dense vector representations, then retrieves the top-3 snippets by cosine similarity. This captures semantic similarity (e.g., variable names, code structure) rather than just lexical overlap, potentially retrieving conceptually related code even if it uses different naming.
Agent-based: Uses Claude Code in autonomous mode (see below).

In all cases, even the "No context" mode, the model always receives the PR title and PR description as repository-level context. This is important: the benchmark never tests models in a completely context-free setting, because PR metadata (title and description) is always available in real review scenarios. The "No context" designation refers specifically to the absence of additional repository code context.

The generation prompt (Figure 16, 17). The model under test is given a structured prompt with four components:

System role: "You are an expert code reviewer. Your task is to review the following code changes and provide constructive feedback."
PR information: Title and Description, provided verbatim.
Code changes: The diff hunk to review, wrapped in a markdown code block.
Relevant code context (when applicable): Up to 3 retrieved code snippets, each presented with file path and content.
Output format instructions: A precise specification requiring each review comment to contain:
- <diff>...</diff> tags wrapping the minimal problematic code snippet (not the entire diff hunk—the model must localize to the specific lines containing the issue)
- <side>left</side> or <side>right</side> indicating whether the comment applies to deleted code (left/old version) or added code (right/new version)
- <note>...</note> tags wrapping the review comment text
- <notesplit/> to separate multiple comments
- <end/> to terminate the response (or immediately if no issues found)
Focus areas: "Code defects, Performance issues, Security vulnerability, Maintainability and readability"
Example: A complete formatted example showing how to wrap a comment about missing documentation.
Edge case handling: "If you believe the code changes are acceptable and have no issues to report, simply output <end/>"

The output format is designed to be machine-parseable while still allowing natural language comment content. The requirement to extract "only the minimal code snippet" (rather than the full diff hunk) is significant: it forces the model to demonstrate precise defect localization, not just general awareness that something is wrong. This enables the evaluation to distinguish between a model that vaguely identifies a problem area and one that precisely pinpoints the defective lines.

Inference hyperparameters. All models are run with Temperature = 0.7, Top_p = 0.95, Top_k = 20. These are standard generation parameters that introduce some stochasticity (temperature > 0) to enable diverse outputs while keeping the distribution reasonably peaked (Top_p = 0.95, Top_k = 20 prevent the tail of the distribution from being sampled). The paper does not justify these specific values against alternatives or report sensitivity to hyperparameter variation—this is a methodological limitation that affects reproducibility if other researchers use different sampling parameters.

The prompt for classification of model-generated comments. To measure performance across the four issue categories (Security, Defect, Maintainability, Performance), the pipeline classifies each generated review comment using Qwen3-235B-A22B-Instruct-2507 with the prompt in Figure 20. The prompt provides category definitions and requires single-token output. Validation against the benchmark's ground-truth category labels (which are expert-annotated, not model-generated) achieved 97% accuracy. This high validation accuracy justifies using automated classification rather than manual categorization of every model output, which would be prohibitively expensive given the number of model–method combinations evaluated.

Agent-Based Methods: Autonomous Review via Claude Code

Framework selection. The paper selects Claude Code as the representative Agent framework. The justification is pragmatic: Claude Code is "a widely-used agent framework that supports code review." The paper does not claim Claude Code is optimal or representative of all Agent architectures—it is one concrete instantiation of the Agent paradigm.

Agent definition (Figure 18). The Claude Code agent is defined with:

Name: "code-reviewer"
Description: "Code review and post inline comments if issues are found"
Tools: Read, Write, Bash (enabling file reading, output writing, and shell command execution)
Model: Inherited from the parent Claude Code configuration (Claude-4.5-Sonnet in the main experiments, though the framework supports plugging in other models)

The agent's internal instructions (the system prompt embedded in the agent definition) specify a multi-step review process:

Identify possible issues: The agent is instructed to flag only issues it is certain about—"If you are not certain an issue is real, do not flag it. False positives erode trust and waste reviewer time." It must provide subagents with the PR title and description for context about author intent.
Validate issues via parallel subagents: For each candidate issue, launch a parallel subagent to verify with high confidence that the issue is genuinely present in the code. The example given: if an issue like "variable is not defined" is flagged, the subagent's job is to validate that this is actually true by examining the codebase.
Post inline comments: For verified issues, output in a structured format with <path>, <side>, <from>, <to>, and <note> tags, separated by <notesplit/>.

The agent is explicitly instructed to avoid flagging: subjective concerns, style preferences, potential issues that "might" be problems, pre-existing issues, pedantic nitpicks, linter-catchable issues, and issues mentioned in CLAUDE.md but explicitly silenced in code. This is a deliberately conservative stance—it trades recall for precision by instructing the agent to comment only when confident.

Triggering the review (Figure 19). The evaluation framework issues a prompt to Claude Code specifying:

The commit range to review: {base_commit}...{target_commit}
Output destination: write comments to comments.txt, clearing any existing content
Output format: the same structured format used in the agent definition
An escape hatch: "If you think there are no issues with the code, there's no need to create the file or write comments."
A recommendation: "If possible, use code-reviewer first" (referring to the defined subagent)

Key difference from non-Agent methods. The Agent approach does not pre-build a code index or pre-retrieve contexts. The agent autonomously decides what code to explore, what context to retrieve, and how many retrieval steps to take. The paper notes that "the Agent method allows the Claude Code framework to autonomously decide the number of contexts to retrieve." This is a fundamentally different paradigm: non-Agent methods follow a fixed retrieval-then-review pipeline, while Agent methods interleave exploration and review, potentially revisiting earlier judgments in light of later discoveries.

Why Claude Code was chosen over alternative Agent frameworks. The paper does not explicitly justify this choice beyond describing Claude Code as "widely-used." Implicitly, the selection seems driven by practical considerations: Claude Code is an open-source framework with documented tool-use capabilities, making it reproducible and inspectable. Alternative frameworks might include proprietary internal tools or research prototypes that are not publicly available. The limitation is that results on "Agent-based" methods are results on one specific Agent implementation, and the paper's claims about Agent behavior (e.g., the inverse context-dependence trend) may not generalize to Agents with different architectures, planning strategies, or tool capabilities.

Metric Computation: How Performance Is Quantified

The matching problem. The core evaluation challenge is determining whether a generated review comment "matches" a ground-truth comment. This is non-trivial because:

A ground-truth comment and a generated comment may describe the same defect using different natural language phrasing.
A generated comment may cover multiple ground-truth issues (overly broad comment) or split a single ground-truth issue into multiple comments (overly granular).
A generated comment may be correct but address an issue not in the ground truth (a genuine defect that the annotation pipeline missed).

Matching approach. The paper uses the standard information retrieval metrics—Precision, Recall, F1-score—but does not describe the exact matching algorithm in the main text (Appendix C may contain details, but the provided paper content describes the evaluation process at a high level). The implicit approach, given the output format requirements (structured tags for file path, line range, and comment side), is that matching considers both:

Localization: Does the generated comment target the same file and line range (or overlapping range) as a ground-truth comment?
Semantic content: Does the comment text address the same issue? This likely involves some form of semantic similarity threshold or, given the paper's use of LLMs for other classification tasks, possibly an LLM-based matching judge.

The paper does not report inter-annotator agreement for the matching process itself, which is a methodological gap—different matching algorithms or thresholds could produce different Precision/Recall values, and without reporting this sensitivity, the absolute metric values are less interpretable than the relative comparisons (which would be robust if the same matching algorithm is applied consistently across all models and methods).

What counts as a true positive, false positive, and false negative.

True Positive (TP): A generated comment that matches a ground-truth comment (both localization and semantic content align).
False Positive (FP): A generated comment that does not match any ground-truth comment. This includes both genuinely incorrect comments (hallucinations, misunderstandings) and comments that identify real issues not captured in the ground truth. The latter case means that the benchmark's Precision metric is a lower bound on true precision—some FPs may actually be valid defects that the annotation pipeline missed.
False Negative (FN): A ground-truth comment that no generated comment matches. This represents defects that the ACR system failed to detect.
True Negative (TN): Not applicable in this setting, as the benchmark does not define negative examples (code changes with no issues).

Metric formulas. The standard definitions apply:

$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$

$\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$

$\text{F1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$

where TP is the count of generated comments matching ground-truth comments, FP is the count of generated comments not matching any ground-truth, and FN is the count of ground-truth comments with no matching generated comment.

What Precision and Recall measure operationally. Precision measures trustworthiness: what fraction of the model's comments are correct? A high-precision model says little but what it says is reliable. Recall measures coverage: what fraction of known defects does the model find? A high-recall model misses few issues. F1 is the harmonic mean, penalizing extreme imbalance between the two. The paper's finding that Agent-based methods achieve dramatically higher Precision but lower Recall than non-Agent methods (Table 3) reflects a fundamental tradeoff: conservative commenting (only flagging high-confidence issues) increases Precision but reduces Recall, while aggressive commenting (flagging everything that might be an issue) increases Recall at the cost of Precision.

Why this metric framework over alternatives? Alternative evaluation frameworks could include:

Exact match accuracy: Count a comment as correct only if its text exactly matches a ground-truth comment. This would be trivially low for all methods due to natural language variation and is clearly inappropriate.
Human evaluation of generated comments: Have experts rate the quality of each generated comment on a Likert scale. This would provide richer quality signals but at dramatically higher cost and with lower reproducibility.
Developer feedback simulation: Deploy models in live PRs and measure whether developers accept, modify, or reject comments. This is the most ecologically valid metric but requires live deployment infrastructure and is confounded by developer behavior, project norms, and latency constraints.

The paper's Precision-Recall framework represents a pragmatic middle ground: it is reproducible, scalable, and captures the two dimensions that matter most for deployment decisions (how much noise vs. how much coverage). The limitation, which the paper acknowledges implicitly through its construction methodology, is that Recall is bounded by the completeness of the ground truth—FN counts only measure missed defects that are in the ground truth, not all missed defects.

Per-level and per-language disaggregation. The paper computes metrics not just globally but also:

By context level (Table 4): Recall computed separately for Diff-level, File-level, and Repo-level ground-truth comments. This enables the finding that non-Agent Recall decays as context scope increases.
By programming language (Figure 3): F1 computed separately for each of the 10 languages. This enables the finding of language-specific bias.
By issue category (Table 10): Performance metrics broken out by Security Vulnerability, Code Defect, Maintainability/Readability, and Performance categories.

These disaggregations are where AACR-Bench's architectural choices (multi-language, hierarchical context annotation) pay off: they enable analyses that would be impossible with single-language, context-agnostic benchmarks.

Output statistics tracked. Beyond the core metrics, the paper tracks auxiliary statistics that illuminate model behavior:

Average comments per patch: This reveals how aggressively a model comments. Non-Agent methods generate many comments (e.g., GPT-5.2 averages 2.47 per patch in no-context mode, per Table 3), while Agent methods generate very few (Claude-4.5-Sonnet in Agent mode averages 0.08–0.15 per patch). This statistic contextualizes the Precision-Recall tradeoff: models with high Precision but low comments-per-patch are conservative reviewers; models with low Precision but high comments-per-patch are verbose but noisy.
Issue category distribution: After classifying generated comments using the LLM-based classifier (Figure 20), the paper reports how each model–method combination distributes its output across the four issue categories (Table 10). This enables analysis of whether certain methods are biased toward detecting, for example, maintainability issues while missing security vulnerabilities.

Why this output tracking matters for interpreting results. Consider a model that achieves Precision = 0.40 and Recall = 0.10 (roughly Claude-4.5-Sonnet in Agent mode). Without auxiliary statistics, this looks like poor overall performance. But with the additional data—comments per patch = 0.08, meaning the model comments on only 8% of patches—the interpretation shifts: the model is extremely conservative, but when it does comment, it is correct 40% of the time. For a deployment scenario where false positives are costly (e.g., automated review that blocks merges), this might be preferable to a model with Precision = 0.08, Recall = 0.28, and comments per patch = 2.47 (GPT-5.2 in no-context mode), which finds more defects but also generates many false alarms. The auxiliary statistics make this nuanced comparison possible, and the paper's decision to report them alongside the core metrics reflects a deliberate choice to support deployment-relevant evaluation rather than just leaderboard ranking.

Summary of Design Choices and Their Justifications

10 languages from StackOverflow 2025 survey rather than all languages: balances representativeness of contemporary development against annotation cost; the recency criterion (2025 survey, 2024–2025 PR window) ensures the benchmark reflects current practices.
Stratified sampling (200 PRs from 573 candidates) rather than random sampling: preserves diversity across problem domains, PR sizes, and source repositories while keeping annotation feasible.
LLM-based comment augmentation rather than using raw PR threads directly: extracts clean, standalone defect descriptions from noisy multi-turn discussions, enabling direct comparison with model-generated comments.
Six-model generation matrix with semantic de-duplication rather than single-model generation: increases defect coverage diversity (validated by low inter-model overlap in Table 8) while preventing redundant annotation through de-duplication.
Three-round expert annotation with double-blind design rather than single-pass annotation: provides reliability through independent verification and expert adjudication of disagreements; expert verification breaks the circularity of using LLMs to verify LLM-generated content.
Context dependency annotation (Diff/File/Repo levels) rather than flat defect labeling: enables the stratified analysis that reveals context-dependence effects (Section 4.2.2), which is the paper's central empirical contribution.
Structured output format with explicit localization tags rather than free-text output: enables automated matching to ground truth with file path and line range, making the evaluation reproducible and scalable.
Agent framework (Claude Code) with autonomous tool use rather than a custom Agent implementation: uses a publicly available, documented framework with known capabilities, enabling reproducibility and comparison across studies.
Temperature = 0.7, Top_p = 0.95, Top_k = 20 rather than greedy decoding (temperature = 0): introduces controlled stochasticity to prevent degenerate outputs while maintaining reasonable determinism; these are common defaults rather than rigorously tuned values, representing a limitation in reproducibility.
Top-3 retrieval for BM25 and embedding methods rather than variable-k retrieval: provides a consistent baseline for comparing retrieval methods without introducing the confound of varying context volume; the choice of 3 is conventional rather than theoretically optimal.
Precision, Recall, F1 as primary metrics rather than single-score quality ratings: captures the precision–recall tradeoff that the paper identifies as fundamental to ACR system design; enables the finding that Agent and non-Agent methods occupy different operating points on this frontier.

4. Key Insights and Innovations

Innovation 1: Context Is Not a Monotonic Resource—The "Contextual Backwardness" Principle

The paper's most conceptually disruptive finding is that providing repository-level context to LLMs during code review does not universally improve performance—and in many cases, it actively degrades it. This challenges a deeply held assumption in the ACR and broader LLM literature: that more context is always better, and that the primary challenge is retrieving the right context rather than knowing when context becomes noise.

Prior work on ACR (SWR-Bench, CodeFuse-CR-Bench) implicitly endorsed this assumption by building benchmarks that provide full repository context, treating context availability as an unalloyed good. The broader RAG literature similarly frames the problem as retrieving relevant documents and injecting them into prompts, with the understanding that relevance improves performance. AACR-Bench's experimental design—systematically varying context retrieval methods (None, BM25, Embedding) while holding the evaluation framework constant—reveals that this assumption is false in code review, and that the relationship between context and accuracy is model-dependent, language-dependent, and method-dependent.

The evidence in Table 3 is stark. Claude-4.5-Sonnet achieves its highest F1 score of 14.46 in "No context" mode; adding top-3 BM25 context drops F1 to 9.98 (a 31% decrease), and embedding-based retrieval similarly degrades performance. This is not a marginal effect—it is a qualitatively different operating regime where the supposedly helpful context is actively harmful. The paper names this phenomenon implicitly through its framing of "contextual tunnel vision" (Section 4.2.1) and formally through the finding that "introducing contextual information does not always yield positive gains" (Section 4.2).

What makes this an innovation rather than just an empirical result is that it reframes the ACR problem from context retrieval (how to find relevant code) to context filtering (how to determine whether retrieved code helps or hurts). This is a fundamental shift in the optimization target. The dominant R&D focus in repository-level code tasks has been on building better retrievers—denser embeddings, hybrid BM25+dense pipelines, iterative retrieval strategies. AACR-Bench's results suggest that for code review specifically, the bottleneck may not be retrieval quality but robustness to irrelevant context. A retriever with perfect recall that retrieves 3 highly relevant snippets may still hurt performance if the LLM cannot distinguish between genuinely informative context and superficially similar but task-irrelevant code.

The paper does not solve this problem—it diagnoses it. But the diagnostic move is itself the contribution because it redirects research attention. Before AACR-Bench, a researcher studying ACR context might ask "How can I retrieve better context?" After AACR-Bench, the more productive question becomes "Under what conditions does context help versus hurt, and how can I build models that are robust to context-induced noise?" This is conceptually analogous to the shift in NLP from "bigger models are better" to "bigger models have failure modes that need targeted mitigation"—the benchmark doesn't propose the mitigation, but it makes the failure mode visible and measurable for the first time.

The paper's language-wise analysis (Section 4.2.3, Figure 3) deepens this insight by showing that context affects languages differently. On C#, GPT-5.2 drops from F1 = 0.309 (no context) to 0.095 (Agent), while on Java, Claude-4.5-Sonnet improves from 0.142 to 0.241 using the same Agent framework. This means the question "does context help?" has no single answer—it depends on a three-way interaction between model, language, and retrieval method. The paper does not fully characterize this interaction (that would require a much larger experimental matrix), but establishing its existence is a prerequisite for any principled ACR system design.

Innovation 2: Ground Truth Augmentation as a Measurement Strategy—The "Label Incompleteness" Problem Operationalized

The paper's core methodological innovation is not the expert annotation pipeline per se (expert-verified datasets exist in many domains), but rather the explicit recognition that ground truth incompleteness is not a data quality issue to be minimized—it is a measurement bias that affects which types of ACR errors can be detected, and that augmenting ground truth changes the kind of evaluation the benchmark performs.

Prior ACR benchmarks (CodeReviewer, SWR-Bench, CodeFuse-CR-Bench) use raw PR comments as ground truth. This means their evaluation can only answer the question: "Does the ACR system catch the defects that human reviewers caught?" But the more practically important question—especially for deployment decisions—is: "Does the ACR system catch defects that human reviewers missed?" The two questions have different answers, and a benchmark built on raw PR comments cannot distinguish between them because it has no information about missed defects. The result is a systematic bias: all models appear to have lower recall than they actually do (because the ground truth is incomplete), and differences between models in detecting human-missed defects are invisible.

The paper makes this bias explicit and quantifies it through the 285% coverage increase figure—391 augmented human comments versus 1,505 total after LLM generation and expert verification. This number is not just a data size statistic; it is a measure of how much the evaluation landscape shifts when the benchmark tests for defects that original reviewers overlooked. A model evaluated on the 391-comment subset might appear to achieve high recall because it catches the same easy-to-spot issues humans flagged, while another model that excelled at detecting subtle cross-file issues invisible in raw PR comments would appear to perform identically. The augmented benchmark disentangles these capabilities.

What distinguishes this from standard "make the dataset bigger" approaches is the verification methodology that breaks the LLM circularity. Many recent benchmarks use LLMs to generate candidate test cases or annotations, but they then use automated metrics or another LLM to verify correctness—a circular process where the verifier's errors are confounded with the generator's errors. AACR-Bench avoids this by using LLMs only for proposal generation (six diverse models producing candidate defects) and reserving verification for human experts through a three-round double-blind protocol. The proposal step leverages LLM scalability; the verification step leverages human judgment reliability. The two steps are independent in a way that single-LLM-generation-followed-by-LLM-verification pipelines are not.

The evidence that this matters comes from Table 8: the low overlap between different models' detected defects (only 11 comments detected by 3 models) confirms that any single model's proposals would miss many genuine defects. If the benchmark had used a single LLM for both generation and verification, those single-model-missed defects would never enter the ground truth, and the benchmark would inherit that model's blind spots. The multi-model generation plus human verification design specifically targets this failure mode.

This innovation is conceptual rather than algorithmic. It does not propose a new way to do ACR—it proposes a new way to measure ACR that reveals capabilities invisible to existing benchmarks. The paper's empirical findings about context-dependence and language-specific bias are downstream consequences of having a measurement instrument that can detect these patterns. In this sense, the ground truth augmentation methodology is the foundational innovation that enables all the other insights in the paper.

Innovation 3: The Precision–Recall Operating Curve as a System Design Spectrum, Not a Metric Tradeoff

The paper's third conceptual contribution is reframing the ACR evaluation problem from "which model is best?" to "which operating point on the precision–recall frontier is appropriate for which deployment context?" This shifts the role of the benchmark from a leaderboard ranker (producing a single F1 number that sorts models into a best-to-worst order) to a system design tool (characterizing the space of possible precision–recall tradeoffs and the architectural choices that produce different points in that space).

The evidence for this reframing comes from the stark divergence between Agent-based and traditional methods in Table 3. Agent-based methods occupy one extreme of the precision–recall spectrum: Claude-4.5-Sonnet in Agent mode achieves Precision = 39.90% but Recall = 10.10%, with an average of 0.08–0.15 comments per patch. Traditional methods occupy the opposite extreme: GPT-5.2 in no-context mode achieves Precision = 8.70% but Recall = 28.37%, averaging 2.47 comments per patch. Neither operating point is "better" in absolute terms—each is optimal for a different deployment scenario.

The paper does not explicitly make this argument as a formal "innovation," but the experimental design and Result presentation structurally embed it. Table 3 is organized to make the precision–recall divergence immediately visible; the average comments per patch statistic contextualizes the tradeoff; and the discussion in Section 4.2.1 explicitly notes that "Agents excel in precision-sensitive settings" while "traditional methods often generate excessive comments... which can bury valuable insights in noise." This is not a claim that one method is superior—it is a claim that method evaluation is meaningless without specifying the deployment context.

Prior ACR benchmarks (CodeReviewer, etc.) report accuracy metrics as if there is a single correct answer to "how good is this model?" By contrast, AACR-Bench's multi-method evaluation reveals that the question is ill-posed—there are at least two distinct definitions of "good" (high precision vs. high recall) that correspond to different use cases (automated merge-blocking vs. assistive suggestion). The benchmark's contribution is not resolving this tension but making it explicit and measurable.

A subtle but important implication: this reframing suggests that hybrid architectures that combine Agent and traditional methods—for example, an Agent that identifies high-confidence issues with near-certainty and a traditional scanner that suggests lower-confidence candidates for human review—may outperform either approach alone on both precision and recall, because they can occupy different points on the frontier for different issue types. The paper does not test such hybrids, but the conceptual framework it establishes makes the design space visible in a way that single-method benchmarks cannot.

Innovation 4: Context-Dependency Scope as a Stratification Dimension—The Inverse Scaling Pattern

The paper's annotation of each ground-truth comment with its required context scope (Diff, File, or Repository level, defined in Table 1) is the most elegant methodological move in the benchmark design, because it transforms a single accuracy number into a capability profile that reveals how a model fails as task complexity increases, not just that it fails.

The key finding enabled by this stratification (Table 4) is that non-Agent methods show monotonically declining Recall as required context scope widens (Diff > File > Repo), while Agent-based methods show the inverse pattern—they perform better at Repo-level than Diff-level. This is a fundamentally new empirical observation that would be invisible without the context-scope annotation. In a benchmark that reports only aggregate accuracy, both method classes might appear to have similar overall performance, masking the qualitative difference in where their errors concentrate.

The significance goes beyond the specific finding. The context-scope annotation is a diagnostic tool for ACR system development: a researcher building a new method can measure not just whether it improves overall F1, but whether its improvements come from better Diff-level detection, better File-level detection, or better Repo-level detection. These improvements have different practical value—a gain in Repo-level detection is more impactful for production code review than a gain in Diff-level detection, because Repo-level issues are cognitively harder for humans to catch and therefore benefit more from automation. The stratification enables developers to target improvements where they matter most.

Why this is an innovation rather than just an annotation choice: the insight is that context dependence is not a continuous variable where "more context = harder" for all methods. The paper could have defined a single difficulty score based on how many files need to be read to verify a defect. Instead, it defined a categorical hierarchy that captures qualitatively different types of reasoning—local syntactic analysis (Diff), single-file semantic understanding (File), and cross-file dependency tracking (Repo)—and discovered that different ACR paradigms have strengths in different qualitative regimes. The inverse scaling pattern (non-Agent decline vs. Agent improvement across context levels) suggests that these qualitative categories correspond to fundamentally different cognitive demands that existing ACR architectures handle differently, and that no current architecture handles all three well.

This is a diagnostic reframing: the benchmark doesn't just rank methods; it characterizes their failure modes in a structured way that guides future development. A researcher who sees that their method's Repo-level Recall is near-zero while its Diff-level Recall is competitive can focus development on cross-file reasoning specifically, rather than trying to improve an opaque aggregate metric.

Innovation 5: Language as a First-Class Experimental Variable in ACR Evaluation

The paper's decision to make AACR-Bench multilingual (10 languages) might appear to be a feature-list item ("supports more languages") rather than an intellectual contribution. But the results in Section 4.2.3 and Figure 3 transform this design choice into a genuine insight: language is not just a coverage dimension—it is an experimental variable that reveals model biases, context sensitivity differences, and structural limitations that single-language benchmarks systematically conceal.

The evidence for this as an insight rather than a feature is the pattern of language-specific effects, not just their existence. The paper identifies three distinct types of language-driven performance variation:

Training data bias: Claude-4.5-Sonnet shows a 3× F1 gap between Python (0.247) and TypeScript (0.081) in Agent mode. This is consistent with training corpus imbalance—Python's extensive open-source ecosystem provides abundant training data—but the key point is that this bias would be invisible in Python-only benchmarks. Evaluations on SWR-Bench or CodeFuse-CR-Bench cannot reveal whether a model's apparently strong ACR performance is general or Python-specific.

Structural language effects: The paper observes that C# consistently achieves high performance across models and methods (GPT-5.2 reaches F1 = 0.309 in no-context mode on C#, the highest single-language score), while C consistently performs worst (0.085 for the same model–method combination). The authors hypothesize that this reflects "programming language intrinsic characteristics"—C#'s explicit type system, namespace management, and structured dependency model make code reasoning easier, while C's pointer operations, macros, and implicit dependencies (header files, linking logic) present fundamentally harder reasoning challenges. This is not a training data artifact (C has abundant open-source code); it is a capability ceiling imposed by language semantics. A benchmark that tests only Python cannot surface this distinction because Python occupies a single point on the language-difficulty spectrum.

Context sensitivity by language: The most nuanced finding is that introducing context affects languages differently. For C#, C++, JavaScript, PHP, Python, Rust, and TypeScript, context retrieval degrades performance relative to no-context baselines. For C, Go, and Java, context (particularly via Agent frameworks) improves performance. This means the question "should I provide repository context to my ACR system?" has different answers for different languages—a finding that would be literally unanswerable using any single-language benchmark.

The innovation here is methodological: by making language a variable rather than a constant, AACR-Bench transforms ACR evaluation from a point estimate (how good is this model on Python?) to a performance surface (how does this model's accuracy vary across languages, and what does that variation reveal about its capabilities?). This is analogous to how multi-task benchmarks in NLP revealed that "average performance" masked large per-task variance, leading to more nuanced evaluation practices. AACR-Bench does the same for code review, establishing that language diversity is not an optional benchmark feature but a necessary condition for meaningful evaluation.

The paper does not claim to have explained the language-specific patterns—it identifies them and offers plausible hypotheses (training data, language semantics, context interference) but does not conduct controlled experiments to test these hypotheses. The contribution is establishing that the patterns exist and that they are large enough (3× gaps, qualitative reversals of context effects) to invalidate any evaluation that does not account for language variance.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use AACR-Bench, the benchmark constructed and described in Section 3: 200 PRs drawn from 50 repositories across 10 programming languages, with 1,505 expert-annotated ground-truth review comments. The benchmark is used in its entirety—there is no train/validation/test split because AACR-Bench is an evaluation-only benchmark, not a training dataset. The 200 PRs serve as the evaluation instances; the 1,505 comments serve as the ground truth against which model-generated comments are matched.
Base model(s). The paper evaluates five models: three open-source (Qwen3-Coder-480B-A35B-Instruct, referred to as Qwen-480B-Coder; DeepSeek-V3.2; GLM-4.7) and two commercial (GPT-5.2; Claude-4.5-Sonnet). The selection criterion is "the latest models from mainstream open-source providers and the most recent versions of major commercial large models"—a recency-based choice that ensures the evaluation reflects current capability levels rather than historical baselines. For the Agent-based evaluations, Claude-4.5-Sonnet is the default model powering the Claude Code framework, though the framework supports model substitution. The paper does not evaluate smaller model variants or ablate model scale, meaning all results characterize the performance ceiling at contemporary model sizes rather than scaling trends.
Metrics. The primary metrics are Precision, Recall, and F1-score, computed by matching generated review comments against ground-truth comments. Precision = TP / (TP + FP), Recall = TP / (TP + FN), F1 = 2 × (Precision × Recall) / (Precision + Recall), where TP is a generated comment matching a ground-truth comment, FP is a generated comment with no ground-truth match, and FN is a ground-truth comment with no matching generated comment. The matching process considers both localization (file path and line range) and semantic content, though the exact matching algorithm threshold is not detailed in the provided paper content. An auxiliary statistic—average number of comments generated per patch—is tracked to contextualize the precision–recall tradeoff by revealing how aggressively each model comments.
Baselines. The paper uses a "No context" condition as the primary comparative baseline, where models receive only the PR title, PR description, and the current diff hunk without any additional repository code context. This is not a trivial baseline—it represents a meaningful lower bound on context-dependent performance and is used to measure whether providing additional context (via BM25, Embedding, or Agent frameworks) yields improvement or degradation relative to the model's intrinsic code-review capability operating on diff-level information alone. There are no prior-method baselines (e.g., CodeReviewer, Llama-Reviewer) because the paper evaluates models and context retrieval strategies, not specific ACR systems from the literature. The evaluation is a benchmark characterization, not a method comparison.
Generation budget / compute accounting. The paper does not use generation budget or FLOP accounting as an experimental axis. There is no systematic variation of the number of samples per prompt, no comparison of different decoding budgets, and no cost-efficiency analysis. All models are run with fixed inference hyperparameters: Temperature = 0.7, Top_p = 0.95, Top_k = 20. The paper's comparisons are therefore across models and context retrieval methods at a fixed quality–cost operating point, not across different compute allocations. For context retrieval, BM25 and Embedding methods uniformly retrieve the top-3 most relevant code snippets; the Agent method (Claude Code) autonomously decides how many retrieval steps to take, making its compute usage variable and not directly controlled.
Cross-validation / statistical protocol. There is no cross-validation, statistical significance testing, or confidence interval reporting for the main experimental results. The paper reports point estimates (Precision, Recall, F1) for each model–method combination on the full 200-PR benchmark without estimating variance. The only statistical validation reported is for pipeline components: domain classification accuracy (92.36% on 350 samples, ±5.06% margin at 95% confidence), review comment augmentation accuracy (95% on 300 samples, ±4.74% margin at 95% confidence), and generated comment classification accuracy (97% on the ground-truth dataset). The main experimental metric values in Tables 3, 4, and Figure 3 are reported as point estimates without error bars or significance tests, meaning the reliability of observed differences (e.g., Claude-4.5-Sonnet's F1 = 14.46 in no-context vs. 9.98 in BM25) cannot be assessed for statistical significance from the provided data.

Main Quantitative Results

Agent-Based vs. Traditional Methods: The Precision–Recall Divergence (Table 3)

The headline finding from Table 3 is a fundamental structural difference in how Agent-based and traditional (non-Agent) methods distribute their predictions. Agent-based methods produce far fewer review comments but with substantially higher precision; traditional methods produce many more comments but with lower precision and higher recall.

Comment volume divergence. Agent methods generate 0.08–0.15 comments per patch on average, while traditional methods generate 1.22–2.47 comments per patch—roughly a 10× to 20× difference in output volume. This is not an implementation artifact; it reflects a deliberate design choice in the Claude Code agent's instructions (Figure 18), which explicitly direct the agent to flag only issues it is certain about: "If you are not certain an issue is real, do not flag it. False positives erode trust and waste reviewer time." Traditional methods, by contrast, use a general "expert code reviewer" prompt (Figures 16, 17) that instructs the model to review code changes and provide feedback without an explicit certainty threshold, resulting in more liberal commenting behavior.

Precision comparison. Claude-4.5-Sonnet in Agent mode achieves Precision = 39.90%, compared to 8.70% in its own "No context" traditional mode—a 4.6× improvement. GPT-5.2 shows the opposite pattern: Precision = 8.70% in no-context traditional mode but drops to 9.90% in Agent mode (Table 3). The key observation is that Agent mode does not uniformly improve precision; its effectiveness is model-dependent. Claude-4.5-Sonnet benefits dramatically from the Agent architecture, while GPT-5.2 does not, despite GPT-5.2 showing the second-highest no-context F1 (14.90) among all traditional configurations.

Recall comparison. The tradeoff is stark. Claude-4.5-Sonnet in Agent mode achieves Recall = 10.10%, versus 41.52% for DeepSeek-V3.2 in no-context mode (the highest Recall among all configurations). No Agent configuration achieves Recall above 10.10%, while no traditional configuration achieves Precision above 10.90% (DeepSeek-V3.2 with BM25, the highest traditional Precision). The two method classes occupy non-overlapping regions of the precision–recall space: Agent methods are precision-dominant (high precision, low recall), traditional methods are recall-dominant (moderate recall, low precision).

F1 comparison. The best overall F1 is DeepSeek-V3.2 with BM25 at 15.59, followed by DeepSeek-V3.2 with no context at 14.97, and GPT-5.2 with no context at 14.90. The best Agent F1 is Claude-4.5-Sonnet at 12.33—meaning that despite the dramatically higher precision, the very low recall pulls the harmonic mean below the best traditional configurations. This reinforces the paper's emphasis that no single method dominates: Agent methods are optimal if precision is the primary concern, traditional methods are optimal if recall (coverage of known defects) is the priority.

Model-specificity of Agent performance. Table 3 reveals that Agent effectiveness is not a general property of the Agent paradigm but depends on the underlying model. Claude-4.5-Sonnet achieves Precision = 39.90% in Agent mode, while GPT-5.2 achieves only 9.90% in the same framework—worse than its own no-context Precision of 8.70% but not dramatically better. DeepSeek-V3.2 and Qwen-480B-Coder show intermediate Agent Precision (14.90% and 18.40%, respectively) but with Recall values so low (3.32% and 3.19%) that their F1 scores are the worst among all configurations (5.32 and 5.44). The paper hypothesizes that "general-purpose capabilities may not straightforwardly translate into competence for Agent-based ACR scenarios" (Section 4.2.1), suggesting that the Claude Code agent's internal reasoning structure and tool-use patterns are better aligned with Claude-4.5-Sonnet's training than with the other models' capabilities.

Impact of Context Retrieval Methods (Table 3)

The second major finding from Table 3 is that context retrieval methods affect different models in qualitatively different ways—there is no universally optimal retrieval strategy.

Claude-4.5-Sonnet: Context degrades performance. Claude-4.5-Sonnet achieves its highest F1 in "No context" mode (14.46). Adding BM25 retrieval drops F1 to 9.98 (a 31% relative decrease), and Embedding retrieval produces F1 = 12.30, still below the no-context baseline. The degradation comes primarily from reduced precision: Precision falls from 8.70% (no context) to 5.80% (BM25) and 6.50% (Embedding), while recall drops from 41.79% (no context—the model generates many comments, 2.31 per patch on average) to 36.15% (BM25) and 32.96% (Embedding), with a corresponding reduction in comments per patch (1.98 and 1.51). This suggests that for Claude-4.5-Sonnet, retrieved context acts as noise that suppresses both the quantity and quality of generated comments.

DeepSeek-V3.2: BM25 is optimal. DeepSeek-V3.2 shows the opposite pattern. BM25 achieves the highest F1 (15.59) among all model–method combinations in the study, with Precision = 10.90% (the highest among all non-Agent configurations) and an average of only 0.89 comments per patch, down from 2.29 in no-context mode. Embedding retrieval yields F1 = 13.11, and no-context yields F1 = 14.97. The key mechanism: BM25 retrieval dramatically reduces comment volume (from 2.29 to 0.89 per patch) while increasing precision (from 5.10% to 10.90%), suggesting that the retrieved context helps DeepSeek-V3.2 filter out spurious issues rather than identify new ones. BM25's lexical matching may retrieve code that is syntactically related to the diff (same function names, variable names) and therefore provides relevant constraint information, while Embedding's semantic matching may retrieve conceptually related but syntactically distinct code that is less informative for the specific review task.

Qwen-480B-Coder: Embedding is optimal. Qwen-480B-Coder achieves its best F1 with Embedding retrieval (14.36), outperforming both BM25 (11.69) and no-context (13.22). The pattern differs from DeepSeek-V3.2: comment volume remains relatively high across all modes (1.32–1.95 per patch), and the gain from Embedding comes from higher precision (7.80% vs. 6.20% for BM25) while maintaining moderate recall (27.58%). This suggests Qwen-480B-Coder benefits from the semantic similarity captured by dense embeddings in a way that DeepSeek-V3.2 does not, and vice versa for BM25's lexical matching.

GPT-5.2 and GLM-4.7: Context provides marginal or negative benefit. GPT-5.2's best F1 is in no-context mode (14.90), with BM25 (14.03) and Embedding (13.20) providing slightly lower performance. GLM-4.7 shows a similar pattern: no-context F1 = 10.54, BM25 = 10.02, Embedding = 9.07. For both models, the addition of retrieved context either maintains or slightly degrades performance, with no clear benefit from either retrieval strategy.

Key observation from Section 4.2.1. The paper summarizes: "Different models respond differently to context retrieval methods, e.g., pairing Claude-4.5-Sonnet with Agent frameworks, DeepSeek with BM25, and Qwen with Embedding consistently present optimal performance." This is not a claim that one method is best—it is a claim that the optimal method is model-specific, and that blanket recommendations about context retrieval are misleading.

Context Level-Wise Impact on ACR Performance (Table 4)

Table 4 presents Recall disaggregated by the context level required to detect each ground-truth defect (Diff, File, or Repo, as defined in Table 1). Because it is impossible to know what context level a model implicitly used during inference, only Recall is reported—Precision cannot be meaningfully decomposed by required context level of the ground truth.

Non-Agent methods: Monotonically declining Recall with context scope. For all non-Agent configurations (No context, BM25, Embedding), Recall follows a consistent hierarchy: Diff > File > Repo. Taking Qwen-480B-Coder with no context as a representative example: Recall drops from 33.82% (Diff-level issues) to 22.59% (File-level) to 17.60% (Repo-level)—a nearly 2× gap between the easiest and hardest context tiers. GPT-5.2 with no context shows the same pattern: 42.05% → 24.63% → 14.45%. Even the best-performing non-Agent configuration for Repo-level issues (DeepSeek-V3.2 with BM25, achieving 20.80% Repo Recall) still shows a substantial decline from its File-level Recall (30.90%). This monotonic decline is universal across non-Agent methods: as the required context scope expands, detection capability systematically degrades. Context retrieval (BM25 or Embedding) does not reverse this trend—the hierarchy Diff > File > Repo is preserved under all non-Agent configurations, though the absolute values shift.

Agent methods: Inverse trend. Agent-based methods (implemented via Claude Code) show the opposite pattern. DeepSeek-V3.2 in Agent mode improves from 4.28% (Diff) to 5.66% (File) to 8.00% (Repo). Qwen-480B-Coder in Agent mode improves from 4.49% (Diff) to 4.83% (File) to 5.94% (Repo). GLM-4.7 in Agent mode improves from 2.86% (Diff) to 4.44% (File) to 5.48% (Repo). However, the absolute Recall values are extremely low compared to non-Agent methods—the Agent's Repo-level Recall of 8.00% (DeepSeek-V3.2) is far below Qwen-480B-Coder's no-context Repo-level Recall of 17.60%. The inverse trend is qualitatively interesting but quantitatively small in absolute terms.

Interpretation. The paper interprets this pattern as reflecting a fundamental difference in how the two paradigms allocate attention. Non-Agent methods focus primarily on the diff itself and lose signal as the required reasoning scope expands beyond the immediately visible code. Agent methods, by contrast, may become "preoccupied with external dependencies, thereby overlooking conspicuous local issues inherent within the diff itself" (Section 4.2.2). The extremely low Diff-level Recall for Agent methods (2.86–4.49%) compared to non-Agent Diff-level Recall (25.98–42.05%) supports this interpretation: Agents sacrifice local defect detection for cross-file reasoning capability. The paper frames this as evidence that "Agents may become preoccupied with external dependencies, thereby overlooking conspicuous local issues inherent within the diff itself."

Language-Wise Impact on ACR Performance (Figure 3, Section 4.2.3)

Figure 3 presents F1 scores disaggregated by programming language across the four method types (No context, BM25, Embedding, Agent) for the five evaluated models. The paper identifies three distinct patterns.

Model-specific language hierarchies. Claude-4.5-Sonnet—the best-performing model in Agent tasks—shows a sharp performance gradient across languages in Agent mode: F1 = 0.247 (Python), 0.241 (Java), 0.218 (Go), 0.189 (C) forming a top tier, versus 0.091 (Rust), 0.082 (PHP), 0.081 (TypeScript) forming a bottom tier. The gap between the best and worst language is approximately 3×. The paper attributes this primarily to training data imbalance: "Languages such as Python and Java, with their extensive ecosystems and abundance of high-quality open-source repositories, likely provide sufficient and well-curated training corpora. In contrast, for languages like Rust, the relatively scarce corpus often hinders LLMs/agents from acquiring the necessary knowledge."

Structural language effects independent of training data. A cross-comparison across all methods reveals that C# consistently achieves high performance while C consistently ranks at the bottom. In "No context" mode, GPT-5.2 achieves F1 = 0.309 on C# (the highest single-language F1 in the entire study) versus only 0.085 on C—a 3.6× gap. Qwen-480B-Coder shows a similar ratio: 0.268 on C# versus 0.104 on C. The paper argues this reflects programming language semantics rather than training data volume: "Given the application status of both languages, we can reasonably assume that C# and C provide comparable abundance for training corpora. In this sense, this phenomenon deeply reflects the impact of programming language intrinsic characteristics on the performance of LLMs." The hypothesis is that C#'s strong typing, namespace management, and explicit type definitions make code reasoning easier, while C's pointer operations, macro definitions, and implicit memory management create fundamentally harder reasoning challenges.

Context sensitivity varies by language. Comparing "No context" mode with Agent/Retrieval modes reveals two behavioral clusters:

Languages where context degrades performance: C#, C++, JavaScript, PHP, Python, Rust, and TypeScript. For GPT-5.2, C# drops from F1 = 0.309 (no context) to 0.095 (Agent); Python drops from 0.165 to 0.071. The paper interprets this as evidence that "externally retrieved context or redundant planning steps generated by the Agent severely interfered with the model's intrinsic judgment."
Languages where context maintains or improves performance: C, Go, and Java. Claude-4.5-Sonnet shows substantial improvements via Agent on Go (0.120 → 0.218), Java (0.142 → 0.241), and C (0.106 → 0.189). These are the only languages where Agent methods outperform traditional approaches, and the effect is concentrated in Claude-4.5-Sonnet specifically.

The paper does not provide a causal explanation for why these particular languages benefit from context while others do not, but the finding itself has a clear practical implication: "When considering whether to provide contextual information to models to improve their performance in ACR, the answer varies entirely depending on the programming language" (Section 4.2.3).

Ablation Studies and Robustness Checks

The paper does not contain a dedicated ablation study section. There is no systematic variation of benchmark construction parameters (e.g., number of PRs, number of retrieved contexts, choice of k for top-k retrieval), no sensitivity analysis of the matching algorithm's thresholds, and no evaluation of how results change under different inference hyperparameters (the paper uses fixed Temperature = 0.7, Top_p = 0.95, Top_k = 20 throughout without testing alternatives).

What the paper does provide are several pipeline validation studies that establish the reliability of the benchmark construction process rather than ablating experimental conditions:

Domain classification accuracy (Section 3.2, Appendix B.1). The Qwen3-235B-A22B-Thinking-2507 model achieves 92.36% accuracy on 350 manually verified PR domain classifications, with a 95% confidence interval of ±5.06%. This validates that the PR problem domain labels used for stratified sampling are reliable.
Review comment augmentation accuracy (Section 3.2, Appendix B.1). The LLM-based augmentation of raw PR comments into clean defect statements achieves 95% accuracy on 300 sampled results, with a 95% confidence interval of ±4.74%. This validates that the augmentation step does not introduce substantial errors into the 391 augmented human-origin comments.
Generated comment classification accuracy (Appendix C.1). The Qwen3-235B-A22B-Instruct-2507 model achieves 97% accuracy when classifying review comments into the four issue categories (Security Vulnerability, Code Defect, Maintainability/Readability, Performance), validated against the expert-annotated category labels in the ground truth. This justifies using automated classification rather than manual annotation for model outputs.
Semantic de-duplication robustness (Appendix B.1). The de-duplication procedure for LLM-generated defect candidates uses an election from 5 repeated LLM judgments, with duplicate status determined by majority vote. This increases robustness over a single judgment, but the paper does not report agreement rates across the 5 judgments or analyze how the duplication threshold affects the final benchmark composition.

What is not ablated:

Number of retrieved contexts (k). All BM25 and Embedding methods use top-3 retrieval. The paper does not test k = 1, 5, 10, or any other value, so the sensitivity of results to context volume is unknown. If top-3 retrieval introduces noise for some models (e.g., Claude-4.5-Sonnet), a smaller k might reduce noise; if top-3 provides insufficient context, a larger k might help. The absence of this ablation means the paper cannot distinguish between "context retrieval is harmful" and "retrieving 3 snippets is the wrong amount of context."
Retrieval quality vs. retrieval method. BM25 and Embedding are different retrieval algorithms, but they also retrieve different documents. The paper cannot determine whether performance differences between BM25 and Embedding are caused by the retrieval algorithm itself or by the specific documents retrieved. An ablation that used BM25 to retrieve documents but Embedding to rank them (or vice versa) would disentangle these factors.
Inference hyperparameters. All results use Temperature = 0.7, Top_p = 0.95, Top_k = 20. No results are reported for greedy decoding (Temperature = 0), for different temperature values, or for different sampling strategies, so the sensitivity of the main findings to stochastic decoding is unknown.
Agent framework choice. All Agent results use Claude Code. There is no comparison with alternative Agent frameworks (e.g., SWE-Agent, AutoCodeRover, or custom ReAct-style agents), so the paper cannot distinguish between properties of Agent-based ACR in general and properties specific to Claude Code's implementation.
PR count and benchmark size. The benchmark contains 200 PRs. There is no subsampling analysis showing whether results stabilize at smaller sample sizes or whether the 200-PR scale is necessary to detect the reported effects.
Annotation quality metrics. While pipeline component accuracies are reported, the paper does not report inter-annotator agreement (e.g., Cohen's kappa) for the core annotation task of verifying review comment correctness. The three-round protocol with double-blind design implies that disagreement occurs (otherwise adjudication would be unnecessary), but the rate and nature of disagreements are not quantified. This makes it difficult to assess the reliability of the ground truth itself—if two expert annotators disagree on 15% of comments, the benchmark's ceiling performance is at most 85%, and all reported Recall values should be interpreted relative to that ceiling.

Critical Assessment

Does the benchmark support the claim that AACR-Bench is "the first multilingual, repository-level context-aware benchmark"?

Yes, with qualifications about what "repository-level" means in practice. The benchmark provides full repository code for all 200 PRs across 10 languages, and the evaluation framework supports retrieving context from the entire repository. This is a genuine advance over CodeReviewer (diff-only) and ContextCRBench (file-level). However, the claim that the benchmark itself is "repository-level context-aware" conflates what the benchmark provides (full repository access) with what the evaluation quantifies (context-dependent performance). The main results in Table 3 show that non-Agent methods use only top-3 retrieved snippets—a tiny fraction of the repository—while Agent methods autonomously decide what to retrieve but are not evaluated on whether they retrieved the correct context for a given defect. The benchmark demonstrates that context matters, but it does not measure whether models are actually using repository-level context effectively. A model could achieve high Repo-level Recall by retrieving the right file by chance, not by systematic cross-file reasoning. The benchmark reveals context-dependence of performance but does not validate that performance gains come from genuine repository-level understanding.

Does the claim of "285% increase in defect coverage" hold up under scrutiny?

The coverage increase is real in the sense that the benchmark contains 1,505 ground-truth comments versus 391 that would exist from raw PR comments alone, yielding (1505 - 391) / 391 = 285% increase. However, the claim conflates two distinct concepts: data volume increase (there are more annotated comments) and defect coverage increase (the benchmark covers defects that a raw-PR benchmark would miss). The paper provides evidence for the former but only indirect evidence for the latter.

The coverage increase claim depends on two unverified assumptions: (1) that the LLM-generated defects verified by experts represent defects that human reviewers genuinely missed (rather than defects that human reviewers noticed but did not comment on, or defects that were added in later revisions), and (2) that the expert verification process is sufficiently reliable that the added comments are not contaminated by false positives that survived the annotation pipeline. The first assumption is untestable without having the original human reviewers re-review the code with the benefit of the LLM-generated suggestions—which is not done. The second assumption depends on the inter-annotator agreement metrics that the paper does not report.

The low overlap in Table 8 (only 11 comments detected by 3 models) is presented as evidence of complementarity, but it could alternatively indicate that many of the single-model-detected comments are model-specific hallucinations that survived de-duplication because they were semantically unique while still being false. The expert verification process should catch such hallucinations, but without inter-annotator agreement data, it is impossible to estimate the false positive rate in the final ground truth. If 5% of the 1,114 LLM-generated comments are false positives that slipped through verification, that introduces approximately 56 erroneous ground-truth items—which would artificially depress reported Precision for all evaluated models.

The coverage increase should therefore be understood as "285% more annotated comments, verified by expert process with reported 95% accuracy on augmentation and comparable quality on annotation" rather than "285% more genuine defects"—the paper provides evidence for the former interpretation, while the latter requires stronger assumptions than the reported validation data can fully support.

Do the experiments support the claim that "the granularity/level of context and the choice of retrieval methods significantly impact ACR performance"?

Yes, but the nature of the impact is more nuanced than a blanket statement conveys. Table 3 provides clear evidence that retrieval method choice matters quantitatively—DeepSeek-V3.2's F1 varies from 14.97 (no context) to 15.59 (BM25) to 13.11 (Embedding), a range of approximately 2.5 F1 points, and Claude-4.5-Sonnet varies from 14.46 (no context) to 9.98 (BM25), a range of approximately 4.5 F1 points. These are meaningful differences. Table 4 provides evidence that context level matters qualitatively—the Diff > File > Repo decline for non-Agent methods versus the inverse pattern for Agent methods represents fundamentally different behavior, not just numeric variation.

However, the paper does not establish statistical significance for any of these differences. With 200 PRs and no reported variance estimates, a difference of 2.5 F1 points could be within sampling error or could be robust—the reader cannot determine which from the provided data. The claim would be stronger with confidence intervals or significance tests, even non-parametric ones (e.g., bootstrap confidence intervals for the F1 difference between methods).

More importantly, the experiments establish that context and retrieval method matter but do not establish why. The paper offers plausible post-hoc hypotheses (training data imbalance for language effects, "contextual tunnel vision" for Agent behavior, noise introduction for context degradation) but does not conduct experiments that would test these hypotheses. For example, if context degrades Claude-4.5-Sonnet's performance because irrelevant snippets introduce noise, an experiment that retrieved ground-truth-relevant context (from the context-dependency annotations) versus BM25-retrieved context would show whether the degradation is due to retrieval quality or context processing capability. This experiment is not performed.

Do the experiments support the claim that "this influence varies depending on the LLM, programming language, and the LLM usage paradigm"?

Yes, this is the most robustly supported claim in the paper. The model-dependence is directly visible in Table 3: optimal retrieval method differs by model (BM25 for DeepSeek-V3.2, Embedding for Qwen-480B-Coder, No context for Claude-4.5-Sonnet and GPT-5.2). The language-dependence is visible in Figure 3: the 3× gap between best and worst language, and the qualitative reversal of context effects across language clusters. The paradigm-dependence is visible in the Agent vs. traditional divergence in Table 3 and the inverse context-scope pattern in Table 4.

The claim does not depend on statistical significance testing because the effects are large and structured—they form patterns (model-specific optimal strategies, language clusters, method-class behavioral differences) rather than isolated data points. A critic could argue about whether a specific pairwise difference is significant, but the overall pattern of three-way interaction between model, language, and method is well-supported by the experimental design's systematic variation across all three dimensions.

The limitation is that the claim is descriptive, not explanatory. The paper identifies that these factors interact but cannot say how or why—whether language effects are driven by training data, language semantics, or benchmark sampling; whether Agent effectiveness depends on model size, training methodology, or instruction-tuning quality; whether retrieval method preferences reflect model architecture, pretraining data composition, or prompt sensitivity. Answering these questions would require controlled experiments that isolate individual factors, which the current benchmark-based evaluation design does not permit.

What experiments would have strengthened the paper?

Ablation of context volume (k). Testing k = 1, 3, 5, 10 for BM25 and Embedding retrieval would characterize the context volume–performance relationship and determine whether the optimal k is model-specific (as the method-specific results suggest). This is a low-cost experiment that would substantially increase the actionable insight from the evaluation.

Retrieval quality experiment. Using the context-dependency annotations (which identify the specific files needed to detect each defect) to construct an "oracle" context condition—where the model receives exactly the files known to be necessary for each ground-truth comment—would establish an upper bound on context-aided performance and reveal whether suboptimal performance stems from retrieval failures or reasoning failures. If oracle context does not improve performance, the bottleneck is the model's ability to use context, not the retriever's ability to find it. This is a critical diagnostic that the current design cannot provide.

Inter-annotator agreement reporting. Reporting Cohen's kappa or percentage agreement for the core annotation task (verifying comment correctness) would establish the reliability ceiling for the benchmark. If inter-annotator agreement is, say, 85%, then benchmark accuracy above 85% is impossible, and all reported metrics should be interpreted with that ceiling in mind.

Multiple Agent framework comparison. Including at least one additional Agent framework (e.g., a simple ReAct loop without Claude Code's specific tool-use patterns) would help distinguish between "Agent-based ACR" as a paradigm and specific properties of Claude Code's implementation. The current results conflate these.

Deterministic decoding comparison. Reporting results with Temperature = 0 (greedy decoding) would establish whether the observed performance patterns are robust to sampling variation or are artifacts of the stochastic decoding at Temperature = 0.7. Given the high temperature, some of the variance between model–method configurations could reflect sampling noise rather than systematic differences.

Statistical significance testing. At minimum, bootstrap confidence intervals for the main F1 comparisons would allow readers to assess whether the reported differences (e.g., Claude-4.5-Sonnet F1 = 14.46 vs. 9.98) are reliable or could arise from sampling variation across the 200 PRs.

These missing experiments do not invalidate the paper's contributions—the benchmark itself is the primary contribution, and the empirical results serve to demonstrate the benchmark's value by revealing patterns invisible to prior benchmarks. But they do mean that the empirical findings should be treated as existence proofs (showing that context-dependence, language-specificity, and method-dependence exist and are measurable) rather than precise characterizations (quantifying exactly how large these effects are, under what conditions they generalize, and what causal mechanisms produce them). The paper's strongest scientific contribution is establishing that these dimensions of variation matter; precisely characterizing them is left to future work.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Not Accounted For in the Benchmark's Value Proposition

The assumption or constraint. AACR-Bench's core value proposition—that it enables measurement of context-dependent and language-specific ACR performance—depends on the benchmark having been constructed with expert-augmented ground truth. The augmentation pipeline (LLM-based defect generation followed by three-round expert annotation) is the mechanism that produces the 285% coverage increase. However, the paper acknowledges the cost of this pipeline only in describing the annotation process, not in evaluating whether the benchmark's construction methodology is sustainable or replicable. The paper explicitly states the annotation involved "over 80 senior software engineers with more than two years of experience" performing three rounds of review (Appendix B.1), but does not estimate the total person-hours, the monetary cost, or the feasibility of other research groups constructing similar benchmarks at comparable scale.

The consequence. The practical implication is that AACR-Bench is not an easily reproducible or extensible benchmark. A research group that wanted to add an 11th programming language, update the benchmark with more recent PRs, or construct a domain-specific variant (e.g., for embedded systems code review) would face a prohibitive annotation cost. The benchmark therefore functions as a one-time measurement instrument rather than a living benchmark that can evolve with the field. This matters because language models and code review practices evolve rapidly—the 2025 PRs in AACR-Bench will become stale, and the benchmark's value will decay if it cannot be refreshed without re-incurring the full annotation cost.

Furthermore, the cost asymmetry affects the interpretation of the benchmark's coverage claim. The 285% increase over raw-PR ground truth is achieved by spending substantial expert annotation resources that single-model-generation approaches (e.g., using one LLM to propose defects and another to verify) would not require. The paper does not compare its multi-model, expert-verified ground truth against a cheaper alternative (e.g., single-model generation with automated deduplication, no expert verification) to establish how much of the coverage benefit comes from the multi-model strategy versus the expensive expert verification. It is possible that a simpler pipeline would capture most of the coverage gain at a fraction of the cost, making the benchmark's construction methodology inefficient rather than uniquely necessary.

What evidence exists in the paper. Section 3.2 and Appendix B.1 describe the annotation process in detail (80+ engineers, three rounds, double-blind design) but provide no cost estimates. Figure 2 and the 285% coverage figure quantify the output of the pipeline but not the input cost. The paper does not report the number of annotation hours, the compensation model, the time elapsed during annotation, or any cost-efficiency metric (e.g., verified comments per annotator-hour). Table 8 shows that most LLM-generated defects were detected by only a single model, which justifies the multi-model strategy for coverage, but does not address whether expert verification of all candidate defects (rather than, say, verification of a stratified sample) was necessary to achieve comparable ground-truth quality.

Mitigation status. The paper does not address the benchmark construction cost as a limitation or propose lower-cost alternatives for future extensions. The annotation cost is treated as an implementation detail of the construction process rather than a property of the benchmark that affects its sustainability. The paper suggests future work on "more advanced semi-automated methods to refine the Ground Truth quality" (Section 6), which implicitly acknowledges the scalability challenge, but this is framed as improving quality rather than reducing cost.

Limitation 2: Single Representative Agent Framework—Claude Code as "Agent-Based ACR"

The assumption or constraint. The paper evaluates Agent-based ACR exclusively through Claude Code, stating in Section 4.1 that it "selected Claude Code, a widely-used agent framework that supports code review." The experimental design draws conclusions about "Agent-based methods" as a category (e.g., the inverse context-scope pattern in Table 4, the precision-recall divergence in Table 3, and the discussion in Section 4.2.1 of "Agent frameworks" generally), but all Agent results come from a single framework implementation.

The consequence. The paper cannot distinguish between properties of Agent-based ACR as a paradigm and properties of Claude Code's specific implementation. Claude Code's agent definition (Figure 18) embeds very specific design decisions: a multi-step review-then-validate workflow with parallel subagents, an explicit instruction to avoid flagging low-certainty issues ("If you are not certain an issue is real, do not flag it"), a deliberately conservative commenting posture that suppresses subjective concerns and style preferences, and a specific tool set (Read, Write, Bash). These design choices directly produce the observed behavior—the very low comment volume (0.08–0.15 per patch), the high precision (39.90% for Claude-4.5-Sonnet), and the low recall (10.10%). An alternative Agent framework with different instructions might produce entirely different results.

For example, an Agent instructed to maximize recall (flag all potential issues even at low certainty) might show the same Diff > File > Repo decline as traditional methods. An Agent without the explicit validation subagent step might achieve higher comment volume but lower precision. An Agent with different tool capabilities (e.g., static analysis integration, test execution) might detect different issue categories. By evaluating only one Agent implementation, the paper's claims about Agent behavior are claims about Claude Code's behavior, not about Agent-based ACR behavior in general.

The model-specificity of Agent performance reinforces this concern. Table 3 shows that Claude-4.5-Sonnet achieves Precision = 39.90% in Agent mode, while GPT-5.2 achieves only 9.90% in the same framework. The paper interprets this as evidence that "general-purpose capabilities may not straightforwardly translate into competence for Agent-based ACR scenarios" (Section 4.2.1). But an alternative interpretation is that Claude Code's agent definition was designed—either explicitly through prompt engineering or implicitly through the framework's development history—to work well with Claude-4.5-Sonnet specifically, and the poor performance of other models reflects a mismatch between the agent's instruction style and those models' training distributions rather than a general property of Agent-based ACR. The paper provides no evidence to distinguish these explanations.

What evidence exists in the paper. The limitation is visible in the experimental design: Section 4.1 names only Claude Code as the Agent framework, and all Agent results in Table 3, Table 4, Figure 3, and the case studies in Appendix D are Claude Code results. Section 4.2.1 discusses "Agent frameworks" and "Agent-based methods" in general terms (e.g., "Agent methods (e.g., Claude Code) produce far fewer review comments," "the effectiveness of Agent-based methods exhibits a pronounced dependency on model-specific characteristics"), but the "e.g." is the only acknowledgment that Claude Code is an instance rather than the definition. Appendix C.1 provides the Claude Code agent definition and trigger prompt (Figures 18, 19) but does not discuss alternative agent architectures or justify the choice of Claude Code over alternatives.

Mitigation status. The paper does not acknowledge this as a limitation, does not evaluate any alternative Agent framework, and does not qualify its claims about Agent behavior as being specific to Claude Code. The Discussion in Section 6 draws general conclusions about Agent architectures (e.g., "Enabling models to actively explore context, verify hypotheses through multi-turn interactions, and filter noise—mimicking human expert behavior—represents the most promising direction") without noting that this conclusion is based on a single implementation. Future work would need to evaluate multiple Agent frameworks (ReAct-style loops, SWE-Agent-style architectures, custom multi-agent systems) to establish whether the observed patterns generalize.

Limitation 3: No Statistical Significance Testing or Variance Estimation for Core Metrics

The assumption or constraint. All main experimental results in Tables 3, 4, and Figure 3 are reported as point estimates—single Precision, Recall, and F1 values for each model–method combination on the 200-PR benchmark—without confidence intervals, standard errors, or significance tests. The paper reports validation statistics for pipeline components (domain classification accuracy with 95% confidence interval, comment augmentation accuracy with 95% confidence interval, generated comment classification accuracy) but does not extend this statistical rigor to the core evaluation metrics that support the paper's main claims.

The consequence. Readers cannot determine whether observed differences between model–method configurations are reliable or could arise from sampling variation across the 200 PRs. Consider the paper's central finding about context degradation: Claude-4.5-Sonnet's F1 drops from 14.46 (no context) to 9.98 (BM25), a 4.48-point decrease. Is this a robust effect, or could re-running the experiment on a different sample of 200 PRs (or with different random seeds given Temperature = 0.7) produce a smaller or reversed difference? Without variance estimates, the 4.48-point drop is a single observation, not a statistically grounded finding.

This matters particularly for the model-specific optimal retrieval strategy claim. The paper concludes that "pairing Claude-4.5-Sonnet with Agent frameworks, DeepSeek with BM25, and Qwen with Embedding consistently present optimal performance" (Section 4.2.1). But the F1 differences between methods for a given model are sometimes small: DeepSeek-V3.2 achieves F1 = 15.59 (BM25) vs. 14.97 (no context)—a gap of 0.62 F1 points. Without variance estimates, it is impossible to know whether this difference is meaningful or whether DeepSeek-V3.2's performance is effectively identical across BM25 and no-context conditions, which would invalidate the claim that BM25 is specifically optimal for this model.

The contextual backwardness findings in Section 4.2.3 face the same issue. The paper reports that GPT-5.2's F1 on C# drops from 0.309 (no context) to 0.095 (Agent). This is a large absolute difference, but with an unknown variance, the precision of this estimate is unknown—the true population difference could be substantially larger or smaller. The qualitative conclusions about language-specific context sensitivity are probably robust given the large effect sizes and consistent patterns, but the precise magnitudes and rankings that the paper reports are not statistically validated.

What evidence exists in the paper. The limitation is visible by absence: no confidence intervals, error bars, significance stars, or p-values appear in Tables 3, 4, or Figure 3. The paper reports the benchmark size (200 PRs, 1,505 ground-truth comments) and the number of models evaluated (5) but provides no power analysis or minimum detectable effect size. The fixed inference hyperparameters (Temperature = 0.7, Top_p = 0.95, Top_k = 20) introduce stochastic variation that is not quantified—the paper does not report whether results are averaged over multiple runs or represent single evaluations. The paper does not describe any procedure for estimating metric variance (e.g., bootstrap resampling over PRs, cross-validation, or repeated runs with different seeds).

Mitigation status. The paper does not acknowledge the absence of statistical inference as a limitation. The pipeline validation statistics (92.36% ± 5.06%, 95% ± 4.74%, 97%) demonstrate that the authors are capable of computing and reporting confidence intervals when they consider validation important, making the omission from the core experimental results more notable. Future work should report at minimum bootstrap confidence intervals for the main F1 comparisons, which would be straightforward to compute by resampling the 200 PRs with replacement and recomputing metrics, requiring no additional experiments.

Limitation 4: Top-3 Retrieval Is an Untested Default—No Ablation of Context Volume

The assumption or constraint. All BM25 and Embedding retrieval experiments use exactly three retrieved code snippets as context ("the number of retrieved code contexts was uniformly set to 3," Section 4.1). The paper provides no theoretical or empirical justification for this choice. It is described as a fixed parameter of the experimental design, not as a tuned optimum or a systematically varied variable.

The consequence. The paper's central finding—that context retrieval can degrade performance—is confounded with the specific context volume of k = 3. It is possible that the observed performance degradation is caused by context volume (too much context overwhelming the model) rather than context relevance (irrelevant context introducing noise), or vice versa. If Claude-4.5-Sonnet's performance degrades with top-3 BM25 because 3 snippets introduce too much text for the model to process effectively, then top-1 retrieval might maintain or improve performance. Conversely, if performance degrades because the 3rd retrieved snippet is often irrelevant noise, then a higher-quality retriever that ensures all 3 snippets are genuinely relevant might eliminate the degradation. The current design cannot distinguish these mechanisms.

This matters for the paper's prescriptive implications. The conclusion that "introducing context retrieval methods may even degrade performance" (Section 4.2.3 key observation) might lead practitioners to avoid context retrieval entirely for certain model–language combinations. But if the degradation is specific to k = 3, then smaller or larger context windows might be beneficial. The paper's findings about model-specific optimal retrieval strategies (BM25 for DeepSeek-V3.2, Embedding for Qwen-480B-Coder, No context for Claude-4.5-Sonnet) might change if k were optimized per model. DeepSeek-V3.2 with top-1 BM25 might outperform top-3 BM25; Qwen-480B-Coder with top-5 Embeddings might outperform top-3. The paper cannot rule out these possibilities.

The choice of k = 3 also affects the comparison between retrieval methods and Agent methods. Agent methods (via Claude Code) autonomously decide how many retrieval steps to take and how much context to gather. If the Agent retrieves, on average, 5–10 relevant code snippets while traditional methods are capped at 3, the observed differences might reflect different effective context budgets rather than different retrieval paradigms. The paper notes that "the Agent method allows the Claude Code framework to autonomously decide the number of contexts to retrieve" (Section 4.1) but does not report the average or distribution of retrieved context volume for Agent runs, making it impossible to control for this confound.

What evidence exists in the paper. Section 4.1 states the k = 3 setting as a fixed parameter without justification. The paper does not report experiments with k = 1, 5, 10, or any other value. There is no analysis of how retrieval rank (1st, 2nd, 3rd retrieved snippet) correlates with comment quality or how performance would change if only the top-1 or top-5 snippets were provided. The paper does not discuss the choice of k as a limitation or suggest that context volume optimization is important future work.

Mitigation status. The paper does not address this limitation. A minimal ablation—testing k = 1, 3, 5, 10 for one representative model (e.g., DeepSeek-V3.2 with BM25) on a subset of the benchmark—would provide evidence for whether the observed effects are robust to context volume or are artifacts of the specific k = 3 choice. The paper's silence on this issue means that practitioners using AACR-Bench to guide context retrieval decisions must accept the k = 3 default without knowing whether it is near-optimal or substantially suboptimal for their model of interest.

Limitation 5: Benchmark Ceiling Effects from Incomplete Ground Truth Are Unquantified

The assumption or constraint. The paper's core methodological contribution is addressing "Label Incompleteness" through LLM-generated defect augmentation and expert verification. However, the augmentation process is itself incomplete—it adds defects that the six LLMs detected, but cannot add defects that no LLM detected and no human reviewer originally commented on. The 1,505 ground-truth comments represent a lower bound on the true number of defects in the 200 PRs, and the gap between 1,505 and the true total is unknown.

The consequence. All Recall values reported in the paper are upper bounds on true Recall, because the denominator (1,505) is smaller than the true number of defects. If the 200 PRs actually contain 2,000 genuine defects (with 495 missed by both humans and the six LLMs), then a model's true Recall is lower than reported. More importantly, the relative Recall comparisons between models may be biased if the missed defects are systematically different from the detected defects. If the 495 hypothetical missed defects are predominantly subtle cross-file issues that require deep repository understanding, then a model that specializes in cross-file reasoning (potentially an Agent-based method) would be penalized more severely by the incomplete ground truth than a model that excels at local diff-level issues, because the Agent model's strengths lie in a defect category that the ground truth undersamples.

This creates a potential circular evaluation bias: the ground truth is constructed from LLM-generated defects verified by humans, which means defects that LLMs are bad at detecting are underrepresented in the ground truth, which means the benchmark cannot measure whether a new model is better at detecting exactly those underrepresented defect types. The paper's multi-model strategy (six diverse LLMs) mitigates but does not eliminate this circularity—the six models collectively have blind spots (e.g., all may struggle with certain types of concurrency bugs, domain-specific logic errors, or language-specific pitfalls), and those blind spots become the benchmark's blind spots.

The paper's finding that hard problems (Repo-level) show near-zero Recall for Agent methods (4.28–8.00% across models, Table 4) must be interpreted with this limitation in mind. If the ground truth undersamples the types of Repo-level defects that Agents are good at detecting (e.g., because the six generation LLMs share the Agents' blind spots), then Agent Repo-level Recall may be artificially depressed. Conversely, if the generation LLMs are particularly good at detecting a certain class of Diff-level defects that traditional methods also catch easily, then traditional Diff-level Recall may be artificially inflated. The paper cannot quantify these biases because it has no access to the true defect distribution.

What evidence exists in the paper. The limitation is structural—it follows logically from the construction methodology and is not something the paper can empirically measure without an oracle defect list. The paper implicitly acknowledges the issue by noting that "constructing a fully comprehensive Ground Truth remains a formidable challenge due to the inherent complexity and subjectivity of real-world software systems" (Section 6). The distribution in Table 8 shows that most LLM-generated defects were detected by only a single model, which demonstrates that the multi-model strategy captures more defects than a single-model strategy would—but does not demonstrate that it captures all or most defects. The paper does not estimate the completeness of the ground truth or discuss how incomplete ground truth might systematically bias Recall comparisons.

Mitigation status. The paper acknowledges the limitation in its Conclusion (Section 6: "constructing a fully comprehensive Ground Truth remains a formidable challenge") and suggests future work on "more advanced semi-automated methods to refine the Ground Truth quality," but does not propose specific methods for estimating ground-truth completeness (e.g., capture-recapture models from ecology, which estimate population size from overlap patterns in multiple sampling methods) or for bounding the bias in Recall comparisons. The limitation is inherent to the benchmark construction paradigm—there is no known way to guarantee complete defect coverage without exhaustive manual review, which is infeasible at scale—but the paper does not discuss how researchers using AACR-Bench should account for ceiling effects when interpreting Recall values.

Limitation 6: Temperature = 0.7 Introduces Uncontrolled Stochastic Variance

The assumption or constraint. All experiments use fixed inference hyperparameters: Temperature = 0.7, Top_p = 0.95, Top_k = 20 (Appendix C.1). These are standard generation parameters that balance output diversity against coherence, but Temperature = 0.7 is substantially above zero and introduces non-trivial stochastic variation in model outputs. The paper does not report whether results are averaged over multiple runs, whether the reported metrics come from single evaluations, or how much variance exists across different random seeds.

The consequence. The reported Precision, Recall, and F1 values are noisy estimates of the model's expected performance at these hyperparameter settings. A model–method combination that achieves F1 = 15.59 in one run might achieve F1 = 14.8 or 16.3 in another run with a different random seed, purely due to sampling variation in token generation. This stochastic noise is confounded with the systematic effects the paper aims to measure (differences between retrieval methods, differences between models, language-specific patterns). Without quantifying the noise level, it is impossible to determine whether a 0.62 F1-point difference between BM25 and no-context for DeepSeek-V3.2 (15.59 vs. 14.97) reflects a real benefit of BM25 or is within the range of run-to-run variation.

The problem is exacerbated because different model–method configurations generate different numbers of comments, making some configurations more sensitive to stochastic noise than others. A configuration that generates very few comments (e.g., Claude-4.5-Sonnet in Agent mode, averaging 0.08–0.15 per patch) has high variance in its metric estimates because each individual comment has a large influence on the aggregate Precision and Recall. A single hallucinated comment in a low-volume configuration can substantially change Precision, while a single missed defect can substantially change Recall. Conversely, high-volume configurations (e.g., GPT-5.2 in no-context mode, averaging 2.47 per patch) have more stable estimates because individual comments have proportionally less influence. This means that comparisons between low-volume and high-volume configurations are particularly susceptible to stochastic noise—the very comparison the paper emphasizes (Agent vs. traditional) is also the comparison most affected by unreported run-to-run variance.

The fixed hyperparameters also mean that the paper cannot distinguish between model capability and decoding strategy. A model that performs poorly at Temperature = 0.7 might perform substantially better with greedy decoding (Temperature = 0), which eliminates stochastic variation and produces the most likely output at each step. The paper cannot determine whether the observed performance reflects the model's fundamental code review capability or a suboptimal decoding strategy for the task. For code review specifically—where precision and factual accuracy are paramount—greedy decoding might be more appropriate than the relatively high-temperature sampling used in the experiments.

What evidence exists in the paper. Appendix C.1 reports the hyperparameters (Temperature = 0.7, Top_p = 0.95, Top_k = 20) but does not mention multiple runs, seed variation, or metric variance. The paper does not report whether results are from single evaluations or averaged over multiple runs. There is no discussion of why Temperature = 0.7 was chosen over alternatives, no comparison with Temperature = 0 (greedy) or lower temperature values, and no sensitivity analysis showing how metrics change with different random seeds.

Mitigation status. The paper does not address stochastic variance as a limitation. The optimal mitigation would be to report metrics averaged over multiple runs (e.g., 5 independent evaluations with different random seeds) with standard deviations or confidence intervals, which would directly quantify the noise level and enable readers to assess whether observed differences exceed run-to-run variation. At minimum, reporting results for Temperature = 0 (greedy decoding) would establish a deterministic baseline against which the Temperature = 0.7 results could be compared, revealing how much of the performance arises from stochastic exploration versus the model's modal behavior. Neither mitigation is implemented.

7. Implications and Future Directions

How This Work Changes the Landscape

AACR-Bench does not propose a new code review model or a novel retrieval algorithm—it proposes a new measurement philosophy for ACR evaluation, and that philosophical shift is the paper's most consequential contribution. Prior benchmarks treated ACR evaluation as a single-dimensional problem: choose a model, run it on a dataset of PRs with known review comments, and report accuracy. AACR-Bench reframes evaluation as a multi-dimensional capability characterization problem where the answer to "how good is this model?" depends on the programming language, the context retrieval method, the required reasoning scope, and the deployment paradigm. This is not an incremental refinement of existing benchmarks—it is a structural change in what counts as a valid evaluation, analogous to how multi-task benchmarks in NLP transformed evaluation from "average GLUE score" to "performance profile across qualitatively different task types."

The shift has three concrete consequences for how the ACR field operates.

First, the paper dismantles the universal-context assumption. The finding that Claude-4.5-Sonnet's F1 drops from 14.46 to 9.98 (a 31% relative decrease) when given top-3 BM25 context—and that this degradation is language-specific, with C, Go, and Java bucking the trend—establishes that context is not a monotonic resource. The dominant narrative in repository-level code intelligence has been "build better retrievers to provide better context." AACR-Bench's results redirect attention toward a harder question: when does context become noise, and how do we build models robust to irrelevant retrieval? This reframing is consequential because it changes the optimization target. If context can hurt as much as help, then improving retrieval precision (finding the right files) is insufficient—the system must also be robust to retrieval errors, which is a fundamentally different technical challenge. Research programs focused exclusively on retriever quality (denser embeddings, hybrid retrieval, iterative retrieval) become less attractive on their own; programs focused on context filtering and robustness gain urgency.

Second, the paper reconciles the apparent contradiction between optimistic and pessimistic views of ACR context. Prior work could be read as either confirming that context helps (SWR-Bench and CodeFuse-CR-Bench provide repository-level context and report reasonable performance) or confirming that context is unnecessary (CodeReviewer's diff-only benchmark has been widely adopted and yields seemingly useful results). AACR-Bench shows that both perspectives are partially correct—and the correct answer is that context effects are model-dependent, language-dependent, and method-dependent. The apparent contradiction in prior literature dissolves once language and method are treated as experimental variables rather than held constant. This is not a trivial "it depends" conclusion; it is a specific mapping of where and how context matters, backed by the three-way interaction visible in Table 3, Table 4, and Figure 3. Future papers that claim a new ACR method improves over baselines must now demonstrate that the improvement holds across languages and context retrieval strategies, not just on a single language with a single retrieval method. This raises the methodological bar for ACR research.

Third, the paper operationalizes the precision–recall frontier as a system design spectrum. The finding that Agent-based methods achieve Precision = 39.90% but Recall = 10.10%, while traditional methods achieve Recall = 41.52% but Precision = 5.10% (Table 3) is not presented as one method being "better"—it is presented as two operating points on a frontier that corresponds to different deployment constraints. This shifts the role of benchmarks from ranking (producing leaderboards that sort models into a best-to-worst order) to profiling (characterizing the performance surface of a model–method combination across multiple dimensions so practitioners can select the appropriate operating point for their use case). The implication is that future ACR systems should be evaluated not by a single F1 number but by their entire precision–recall curve across context levels and languages, and that hybrid architectures occupying multiple points on the frontier simultaneously (e.g., high-precision Agent for blocking issues, high-recall traditional scanner for suggestions) may outperform any single-method approach.

The paper also identifies verifier over-optimization as a bottleneck in a different guise. While the reference example paper (on test-time compute scaling) documents PRM over-optimization from aggressive search, AACR-Bench documents a parallel phenomenon: the over-optimization of retrieval methods toward precision at the expense of robustness to retrieval noise. Just as the prior work showed that better search algorithms can paradoxically reduce performance by exploiting verifier weaknesses, AACR-Bench shows that better context retrieval (in terms of relevance) can paradoxically reduce ACR performance if the model cannot filter irrelevant information. Both papers converge on the same meta-insight: optimizing a component of a pipeline (search, retrieval) without considering the downstream model's robustness to that component's failure modes can be counterproductive.

Follow-Up Research This Work Enables

Characterizing the context volume–performance relationship for code review. The paper's most actionable open question is whether the observed context degradation is caused by the amount of context (k = 3 snippets may be too much or too little) or by the quality of context (BM25 retrieves irrelevant noise). A strong follow-up would systematically vary k ∈ {1, 3, 5, 10} for both BM25 and Embedding retrieval across 3 representative models (Claude-4.5-Sonnet, DeepSeek-V3.2, GPT-5.2) and 3 representative languages (Python, C#, C) on AACR-Bench. The hypothesis: if context degradation is volume-driven, all models should show a ∩-shaped curve (improvement from k = 1 to some optimum, then decline at higher k); if it is quality-driven, models with better inherent reasoning (Claude-4.5-Sonnet) should be harmed less by noisy retrieval than models with weaker reasoning. A flat or monotonically declining curve would suggest that k = 3 is already on the right side of the optimum for most models, validating the paper's default choice. This experiment requires no new benchmark construction—only re-running existing evaluation pipelines with different k values—and would transform the paper's descriptive finding ("context can hurt") into a prescriptive guideline ("use k = X for model Y on language Z").

Oracle-context experiment to separate retrieval failure from reasoning failure. The paper demonstrates that Repo-level Recall is universally low (4.28–20.80% across all configurations, Table 4), but cannot determine whether this is because retrievers fail to find the necessary cross-file dependencies or because models fail to reason about them even when provided. A critical diagnostic experiment would use the benchmark's context-dependency annotations (Table 1) to construct an "oracle" context condition: for each ground-truth Repo-level comment, provide the model with exactly the specific files known to contain the relevant cross-file dependencies (identified from the expert annotation process). If oracle context substantially improves Repo-level Recall (e.g., from 17.60% to 40%+ for Qwen-480B-Coder), the bottleneck is retrieval—and research should focus on better cross-file dependency retrieval. If oracle context produces minimal improvement, the bottleneck is reasoning—and the model fundamentally cannot perform cross-file inference even when given the right files, suggesting that architectural advances (not retrieval advances) are needed. This experiment would be labor-intensive (requiring manual identification of dependency files for each Repo-level comment from the annotation records) but would provide the single most informative diagnostic for the ACR field about where to invest research effort.

Multi-Agent-framework comparison to establish whether Agent behavior generalizes beyond Claude Code. The paper's claims about Agent-based ACR rest entirely on Claude Code's implementation, which embeds very specific design choices (conservative commenting posture, validation subagents, Read/Write/Bash tools). A necessary stress-test is to replicate the Agent experiments using at least two alternative frameworks: a minimal ReAct-style agent (basic thought–action–observation loop with file reading and code search tools, no explicit validation step) and a multi-agent debate framework (e.g., two agents independently review the same diff, then reconcile disagreements). The prediction: the ReAct agent should produce intermediate results between traditional methods and Claude Code (higher recall than Claude Code, higher precision than traditional methods), while the debate agent might achieve the highest precision by requiring consensus before flagging issues. If different Agent architectures produce qualitatively different precision–recall operating points, then "Agent-based ACR" is not a coherent category—it is a design space, and the paper's findings about inverse context-dependence (Table 4) may be Claude Code-specific rather than Agent-general. This experiment requires implementing alternative agents within the AACR-Bench evaluation framework (the benchmark's modular design supports this) and would calibrate the generality of the paper's Agent-related claims.

Training data provenance analysis for language-specific performance gaps. The paper identifies large language-specific performance disparities (GPT-5.2 F1 = 0.309 on C# vs. 0.085 on C in no-context mode) and hypothesizes that they reflect both training data imbalance and language semantics, but cannot distinguish these causes. A concrete follow-up would correlate per-language AACR-Bench performance with independent estimates of training data volume per language for each model (where available—some open-source models release training data composition statistics). If F1 and training data volume are strongly correlated (r > 0.7), the gap is primarily a data problem, solvable by targeted pretraining on low-resource languages. If the correlation is weak (r < 0.3), language semantics dominate, and the gap reflects fundamental reasoning difficulty—meaning that even perfect training data parity might not close the C#–C gap. For commercial models where training data composition is unknown, a weaker but still informative approach would measure cross-language transfer: fine-tune an open-source model (e.g., DeepSeek-V3.2) on ACR data in one language (Python) and measure whether AACR-Bench performance improves in other languages, testing whether ACR skill transfers across languages or is language-specific.

Capture-recapture estimation of ground-truth completeness. The paper's most fundamental unquantified assumption is that the 1,505 ground-truth comments represent a sufficiently complete sample of true defects that Recall comparisons are unbiased. A rigorous follow-up would apply capture-recapture methodology (from ecology) to estimate the total number of defects in the 200 PRs. The idea: treat the six LLMs used in generation as six independent "samplers" of the defect population. The overlap pattern—how many defects were detected by 1, 2, 3, 4, 5, or 6 models (the paper already reports the distribution up to "3 models" in Table 8; the full distribution would be needed)—can be fitted to a capture-recapture model to estimate the number of defects detected by zero models, i.e., the ground-truth gap. If the estimated gap is small (e.g., 100–200 additional defects), the 1,505-comment benchmark is substantially complete, and Recall values are close to true Recall. If the estimated gap is large (e.g., 1,000+ additional defects), the benchmark captures only a fraction of true defects, and Recall comparisons—particularly between methods targeting different defect types—may be systematically biased. This analysis requires no new data collection; it uses the existing overlap pattern from the generation pipeline, making it a high-leverage follow-up with minimal marginal cost.

Hybrid precision–recall architecture combining Agent and traditional methods. The paper identifies that Agent methods excel at precision (39.90%) while traditional methods excel at recall (41.52%), but never tests whether combining them yields a better operating point. A natural follow-up architecture: run both an Agent review (Claude Code, producing a small set of high-confidence comments) and a traditional review (GPT-5.2 with no context, producing a large set of lower-confidence comments) on the same PR, then use a confidence-weighted aggregation where Agent comments are treated as near-certain issues and traditional comments are presented as suggestions requiring human verification. The prediction: this hybrid would achieve precision comparable to the Agent on high-confidence issues while maintaining recall comparable to the traditional method on lower-confidence issues, yielding an overall F1 that exceeds both individual methods. If successful, this would establish that the precision–recall frontier is not a tradeoff to be accepted but a design space to be spanned by multi-method systems—a finding with immediate practical implications for ACR deployment. AACR-Bench's evaluation framework already supports running multiple methods on the same PR and comparing outputs, making this experiment feasible without benchmark modification.

Practical Applications and Downstream Use Cases

Model selection for polyglot codebases using per-language performance profiles. Organizations maintaining codebases in multiple programming languages face a model selection problem that single-language benchmarks cannot inform. AACR-Bench's language-disaggregated F1 scores (Figure 3) directly support this decision. A team working primarily in Python and Go, for instance, would observe that Claude-4.5-Sonnet in Agent mode achieves F1 = 0.247 on Python and 0.218 on Go—the best Agent scores for both languages—making it the natural choice if precision-sensitive automated review is the goal. A team working in C# and JavaScript would observe that GPT-5.2 in no-context mode achieves F1 = 0.309 on C# (the highest single-language score in the study) but only 0.121 on JavaScript in the same configuration, and might instead choose DeepSeek-V3.2 with BM25, which achieves more balanced performance (0.236 on C#, 0.098 on JavaScript) at lower cost. These decisions are directly actionable today—they require no new research, only looking up the relevant language–model–method cells in the paper's results—and the performance differences are large enough (3× gaps between best and worst configurations for a given language) to materially affect review quality.

Deployment architecture design with tiered review pipelines. The paper's precision–recall divergence between Agent and traditional methods supports a two-tier deployment architecture that is more cost-effective than any single-method approach. Tier 1 runs a high-precision Agent review (Claude-4.5-Sonnet in Agent mode) that flags only issues it is near-certain about, achieving Precision = 39.90% with approximately 0.08 comments per patch. Issues flagged by Tier 1 are treated as blocking—the PR cannot merge until they are addressed—with minimal human review overhead because the low comment volume (approximately 16 comments per 200 patches) is manageable. Tier 2 runs a high-recall traditional scanner (GPT-5.2 with no context or DeepSeek-V3.2 with BM25) that generates suggestions at higher volume (2.47 or 0.89 comments per patch, respectively) with lower precision (8.70% or 10.90%), presented as non-blocking recommendations for the human reviewer to accept or dismiss. The economics: Tier 1's low comment volume means it imposes minimal reviewer burden despite lower recall; Tier 2's higher volume is acceptable because its comments are suggestions, not requirements. This architecture directly exploits the Agent–traditional precision–recall divergence documented in Table 3 and requires no new technical development—it is an integration decision guided by the benchmark's profiling capability.

Context retrieval strategy selection as a per-model, per-language configuration parameter. The paper's finding that optimal retrieval method is model-specific (BM25 for DeepSeek-V3.2, Embedding for Qwen-480B-Coder, No context for Claude-4.5-Sonnet and GPT-5.2) has an immediate practical implication: ACR deployment pipelines should not use a single retrieval strategy for all models. A platform that supports multiple backend LLMs for ACR should configure the retrieval strategy per model based on AACR-Bench results. Concretely, if a platform routes Python PRs to DeepSeek-V3.2, it should enable BM25 retrieval with top-3 snippets; if it routes to Qwen-480B-Coder, it should enable Embedding retrieval; if it routes to Claude-4.5-Sonnet or GPT-5.2, it should disable additional context retrieval entirely and rely on the model's intrinsic reasoning. This configuration is trivial to implement (it requires selecting a retrieval flag per model backend) and the F1 improvements are moderate but meaningful—approximately 0.6 F1 points for DeepSeek-V3.2 (15.59 vs. 14.97) and 1.1 F1 points for Qwen-480B-Coder (14.36 vs. 13.22). For high-volume ACR deployments processing thousands of PRs, these per-model optimizations compound to a non-trivial total quality improvement.