Consistency Amplifies: How Behavioral Variance Shapes Agent Accuracy

🎯 Pitch

Running the same SWE-bench task 5 times reveals that more capable models aren't just more accurate—they're dramatically more consistent, with Claude's variance (CV: 15.2%) being 3× lower than Llama's. Yet the real danger isn't inconsistency but its opposite: 71% of Claude's failures come from confidently repeating the exact same wrong interpretation across all runs, making errors harder to detect than sporadic mistakes.

1. Executive Summary

This paper studies how behavioral consistency—whether LLM-based agents produce similar action sequences when given identical tasks—relates to accuracy in complex, multi-step software engineering. Running Claude 4.5 Sonnet, GPT-5, and Llama-3.1-70B across 50 trials each (10 SWE-bench tasks × 5 runs), the authors find that across models, higher consistency aligns with higher accuracy: Claude achieves the lowest coefficient of variation (CV: 15.2%) and highest accuracy (58%), GPT-5 is intermediate (CV: 32.2%, accuracy: 32%), and Llama shows the highest variance (CV: 47.0%) with lowest accuracy (4%). The paper's central conceptual contribution is the amplification insight—consistency amplifies outcomes rather than guaranteeing correctness, with 71% of Claude's failures arising from "consistent wrong interpretation" (making the same incorrect assumption across all runs)—while also revealing that divergence timing alone does not determine consistency, as Claude and GPT-5 diverge at nearly identical steps (3.2 vs. 3.4) yet Claude achieves 2.1× better consistency. The findings establish a speed-accuracy-consistency tradeoff—GPT-5 is 4.7× faster than Claude (9.9 vs. 46.1 steps) but achieves 1.8× lower accuracy—and demonstrate that interpretation quality, not execution consistency, is the primary bottleneck for reliable agent deployment, with the rank ordering of consistency and accuracy holding across models only when tasks fall within a model's capability range.

2. Context and Motivation

The Core Problem: We Don't Know How Reliable LLM Agents Really Are

The fundamental question this paper tackles is deceptively simple: if you give an LLM-based agent the exact same software engineering task five times, will it behave the same way each time? This question matters because the dominant paradigm of evaluating these agents—running them once per task and reporting a single accuracy score—implicitly assumes that a single run is representative of how the agent would perform on that task. The paper challenges this assumption directly.

The specific gap this paper addresses is the near-total absence of empirical data on behavioral consistency in complex, multi-step agent trajectories. While previous work (Mehta, 2026) has established that LLMs exhibit variance even on simple reasoning tasks like factual questions or multi-hop QA, no one had systematically measured whether that variance persists, amplifies, or changes character when the action space expands dramatically—from selecting a single answer token to navigating large codebases, reading files, implementing multi-line edits, and running test suites across dozens of interdependent steps. The conceptual leap from "do LLMs give different answers to the same question?" to "do LLM agents trace entirely different paths through complex software engineering tasks?" is substantial, and was essentially unexplored territory before this paper.

This gap is significant for several reasons that the paper develops across its introduction and related work:

Why Consistency Matters: Practical and Theoretical Stakes

Unpredictable agents cannot be trusted in production. The paper frames this directly in Section 1: "inconsistent agents are unpredictable, making them difficult to trust in high-stakes applications." This is not merely an academic concern. If an agent tasked with fixing a production bug sometimes finds the correct root cause in 8 steps and sometimes wanders through 50 steps of exploration before failing, the deploying organization faces genuine operational risk. Worse, as the paper's "consistent wrong interpretation" finding demonstrates (71% of Claude's failures), an agent can be both consistent and wrong—meaning it will confidently and repeatedly make the same mistake, never self-correcting. An agent that fails identically on all five runs of a task presents a fundamentally different reliability profile from one that fails on three of five but succeeds on two. The former suggests the model's understanding is systematically flawed; the latter suggests stochastic noise that might be mitigated through techniques like majority voting or self-consistency methods (Wang et al., 2023).

Variance complicates debugging and improvement. The paper argues in Section 1 that "if an agent sometimes succeeds and sometimes fails on identical inputs, isolating the cause becomes challenging." This is a practical engineering concern: developers trying to improve agent performance need to understand why failures occur. If failures are stochastic across runs, they might be attributed to sampling noise or unlucky exploration choices using tools. If failures are systematic—the same error repeated identically—they point toward fundamental issues in task interpretation or model capability. Without measuring and characterizing consistency, developers lack the diagnostic signal to distinguish between these two failure modes and thus cannot allocate improvement efforts effectively.

Single-run benchmarking may be misleading. Perhaps the most consequential implication is for evaluation methodology. The paper explicitly states in Section 1 that "consistency is a prerequisite for meaningful benchmarking: if results vary substantially across runs, single-run evaluations may be misleading." This critique lands with force because SWE-bench—the benchmark the paper studies—is the de facto standard for evaluating software engineering agents, and its standard protocol involves a single run per task (Jimenez et al., 2023, 2024). If, as the paper demonstrates, every single one of 150 runs across three models produces a completely unique action sequence (Section 4.1: "100% of runs produce unique action sequences for all three models"), then the standard single-run evaluation is sampling from a distribution where no two samples are ever identical. A model's reported 58% accuracy might mean it solves 58% of tasks on every try, or it solves 100% of tasks 58% of the time—patterns with radically different implications for deployment reliability. The paper's key insight is that these two patterns are observationally equivalent under single-run evaluation but lead to fundamentally different failure modes, recovery strategies, and trustworthiness guarantees in production.

Prior Approaches and Where They Fall Short

The paper identifies several lines of prior work and explains why each fails to address the core question of consistency in complex agent trajectories.

LLM consistency research has been limited to simple tasks. Section 2 acknowledges that prior work has examined LLM consistency in domains like mathematical reasoning (Wang et al., 2023), factual questions (Elazar et al., 2021), and multi-hop QA (Mehta, 2026). These studies consistently find substantial variance even in straightforward settings. However, these tasks are fundamentally single-step or short-horizon: the model makes one or a few reasoning steps and produces a final answer. The actions are independent in the sense meant by Wang et al. (2023): self-consistency methods assume that sampling multiple reasoning chains and taking a majority vote improves accuracy because errors are uncorrelated across samples. But as the paper notes in Section 2, "this assumption may not hold for complex agent trajectories" where "actions are sequentially dependent." A software engineering agent's decision at step 15—which file to edit—is path-dependent on what it explored in steps 1–14. Two runs that diverge at step 3 may end up in entirely different regions of the codebase, reading different files, forming different mental models of the bug, and producing different fixes. The independence assumption that underlies self-consistency techniques is almost certainly violated, but no one had measured whether this matters in practice.

Agent benchmarks report accuracy but not consistency. The SWE-bench ecosystem (Jimenez et al., 2023, 2024) and other agent evaluation frameworks—WebArena for web navigation (Zhou et al., 2023), OSWorld for computer use (Xie et al., 2024), and coding benchmarks like HumanEval and MBPP (Chen et al., 2021; Austin et al., 2021)—universally report single-run accuracy. The paper highlights this as a critical methodological gap: "Most report single-run accuracy, leaving consistency unexplored." This is not a minor oversight. SWE-bench Verified, introduced by Jimenez et al. (2024) to provide human-validated solutions and enable more reliable evaluation, addresses ground-truth reliability (are the benchmark's correct answers actually correct?) but not agent reliability (does the agent reliably produce the same answer?). The paper positions itself as extending the evaluation methodology conversation from ground-truth reliability to behavioral reliability.

Failure analysis research focuses on categorization, not consistency. Section 2 references work on agent failure analysis, particularly Liu et al. (2023) on error categorization and Shinn et al. (2023) on Reflexion (verbal self-reflection for agent improvement). These approaches assume that failures are detectable and that agents can learn from them. The paper identifies a blind spot in this assumption: "many failures are 'consistent wrong'; the agent confidently repeats the same mistake, making reflection-based recovery unlikely." If an agent consistently interprets a task's bug as "this needs a deprecation warning" when the correct fix is "remove the code entirely," then even if it reflects on its failure and tries again, it will likely reach the same conclusion. The consistency measurement reveals that reflection-based improvement methods face a fundamental limitation: they cannot correct errors that the model cannot recognize as errors. This insight is made possible only by running multiple trials and observing that the same interpretation error recurs identically.

No framework for connecting consistency to accuracy in complex domains. Perhaps most importantly, prior work had no empirical data or theoretical framework for understanding how consistency relates to accuracy when the task complexity increases dramatically. The paper identifies an unresolved question in Section 1: "Do consistency challenges grow with action space size? Does the variance in simple tasks predict variance in complex ones?" Without a study spanning the gap from simple QA to multi-step software engineering, these questions were unanswerable. The paper fills this gap by providing the first systematic measurements of consistency on a complex agent benchmark, establishing baseline values that future work can build on.

How This Paper Positions Itself

The paper positions itself at the intersection of two conversations that have not previously been connected: the LLM consistency literature (which has studied simple tasks) and the agent evaluation literature (which has ignored consistency entirely). This framing is made explicit through the research questions organizing Section 4:

RQ1 asks whether models differ in consistency—establishing the basic empirical fact of the matter, which had never been measured for complex agent trajectories.

RQ2 asks whether consistency relates to accuracy—directly interrogating whether the common assumption that "more consistent = more reliable" actually holds. The paper's finding that consistency correlates with accuracy across models but not within models is a significant complication of that simple narrative.

RQ3 asks where variance originates—decomposing agent trajectories into phases (explore, understand, edit, verify) to understand whether consistency challenges are driven by exploration noise, editing decisions, or verification loops. This goes beyond prior work that has treated agent trajectories as opaque sequences.

RQ4 asks how failure modes differ—characterizing whether failures are "consistent wrong" (systematic misinterpretation) or "inconsistent wrong" (execution noise), a distinction that has direct implications for which improvement strategies (better interpretation vs. better execution) will be most effective.

RQ5 asks when trajectories diverge—pinpointing the exact step at which runs of the same task take different paths. This temporal analysis of divergence is novel and reveals the paper's surprising finding that divergence timing alone does not determine consistency (Claude and GPT-5 diverge at nearly identical steps yet achieve very different overall variance).

The paper thus positions itself not as proposing a new method for improving agents, but as providing the empirical foundation and conceptual vocabulary for reasoning about agent reliability. Terms like "the amplification insight," "the consistent wrong pattern," "the speed-accuracy-consistency tradeoff," and "the interpretation bottleneck" are introduced as organizing concepts that help practitioners and researchers think clearly about what consistency means, when it helps, when it hurts, and what to do about it. The paper's contribution is primarily descriptive and diagnostic—mapping the landscape of agent consistency so that future work on improving agents can target the right problems (interpretation quality, not execution consistency) with appropriate methods (multi-run evaluation standards, task difficulty estimation, adaptive strategy selection).

3. Technical Approach

3.1 Reader Orientation

This paper is primarily an empirical measurement study that constructs an experimental apparatus for repeatedly executing LLM-based software engineering agents on identical tasks and quantitatively characterizing how much their behavior varies from run to run. The system being built is not a new agent or training method, but rather a reproducible measurement pipeline that isolates the sources of behavioral variance—whether it originates in exploration, understanding, editing, or verification phases—and connects that variance to task outcomes. The core problem it solves is that standard single-run benchmarking cannot distinguish between an agent that reliably solves a task every time and one that succeeds on one run out of five, and the solution takes the shape of a multi-run experimental protocol with phase-decomposed trajectory analysis that produces a set of consistency metrics (coefficient of variation, divergence timing, sequence uniqueness) alongside traditional accuracy scores.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental system has six major components, arranged in a pipeline from benchmark selection through statistical analysis:

Task Sampler — selects a diverse subset of 10 tasks from SWE-bench Verified, all from the astropy repository, controlling for codebase familiarity and enabling cross-task comparison within a single software ecosystem. This component produces the fixed task slate that all models face.
Agent Scaffold (mini-SWE-agent) — a minimal agent framework that provides each model with an identical interface: a bash shell for file system navigation, code editing, and test execution. Critically, it uses subprocess.run in isolated Docker containers rather than a higher-level tool-calling API, which means the agent's action space is raw shell commands rather than structured function calls. The scaffold includes a format-error handling patch (developed during the study) that blocks patch submission during error recovery to prevent premature or corrupted submissions.
Model Interface Layer — wraps three different LLMs (Claude 4.5 Sonnet, GPT-5, Llama-3.1-70B-Instruct) behind an identical interface: same system prompt, same tool definitions, same temperature setting (0.5), same maximum step budget (250). This component ensures that any observed differences in behavior are attributable to model properties, not configuration asymmetries.
Trial Executor — runs each model-task pair through 5 independent trials in fresh Docker containers with fresh repository states, producing 150 complete agent trajectories (10 tasks × 5 runs × 3 models). Each trial generates a time-ordered sequence of actions (bash commands) with metadata about action type, success/failure, and intermediate outputs.
Trajectory Analyzer — processes the 150 raw trajectories into structured measurements. It decomposes each action step into one of five phases (EXPLORE, UNDERSTAND, EDIT, VERIFY, OTHER) based on command type, computes per-task step count variance using the coefficient of variation, identifies the first step where runs of the same task diverge in action category, and classifies each failed run's failure mode as either "interpretation error" or "execution error" through manual annotation of the final patch against the ground-truth fix.
Statistical Engine — computes cross-model comparisons using paired t-tests (for CV and divergence metrics, since the same 10 tasks are used across models) and Fisher's exact tests (for accuracy comparisons, since accuracy is binary per run and not naturally paired). Produces the aggregate statistics (CV values, accuracy rates, divergence timings, phase distributions) reported in tables and figures.

Information flows sequentially: the Task Sampler selects tasks → each Model Interface Layer configures an agent → the Trial Executor runs 5 independent trials per model-task pair in isolated containers → the Trajectory Analyzer processes the resulting action sequences into phase labels, variance metrics, and failure classifications → the Statistical Engine aggregates across tasks and runs to produce the reported findings.

3.3 Roadmap for the Deep Dive

The detailed breakdown follows the natural temporal order of an experiment, explaining what happens at each stage and why specific design choices were made:

First, the benchmark and task selection procedure—what SWE-bench Verified is, why astropy was chosen, what the 10 tasks look like, and how the task diversity constraints were set. Understanding the task space is essential because consistency might manifest differently on simple import errors versus complex logic bugs.
Second, the agent scaffold and model interface—how mini-SWE-agent works, what the bash-only interface implies for the action space, the format-error bug that was discovered and fixed during development, and the precise prompt and tool configurations that ensure fair comparison across models. This is the machinery that translates a task description and a model into an action sequence.
Third, the multi-run execution protocol—the temperature setting (0.5), the 5-run design, the Docker isolation strategy, the 250-step budget, the $10 cost cap, and why these numbers were chosen. This is where the paper's central experimental innovation lives: running identical tasks multiple times under controlled stochasticity.
Fourth, the trajectory decomposition and phase labeling system—the five phase categories (EXPLORE, UNDERSTAND, EDIT, VERIFY, OTHER), the command-type-to-phase mapping rules, and how this decomposition enables the paper to answer RQ3 ("where does variance originate?"). Without this phase analysis, the paper could only report that models vary, not where in the trajectory the variance emerges.
Fifth, the consistency metric design—the coefficient of variation formula, why CV rather than raw standard deviation or variance, how sequence uniqueness is measured (the 100% unique finding), and the divergence step identification algorithm. These metrics operationalize "consistency" into computable quantities.
Sixth, the failure mode annotation protocol—how "consistent wrong interpretation" versus "execution error" is defined, the manual annotation procedure, who performed it, the semantic criteria distinguishing interpretation from execution failures, and the acknowledged limitation (unblinded annotation). This taxonomy is the mechanism behind the paper's central insight that interpretation, not execution, is the bottleneck.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical measurement study whose core idea is that agent reliability cannot be assessed from single-run evaluations because behavioral consistency—the degree to which an agent produces similar action sequences on identical tasks—is both measurable and predictive of different failure modes, and that consistency amplifies outcomes (correct or incorrect) rather than guaranteeing correctness.

Benchmark and Task Selection

The paper uses SWE-bench Verified (Jimenez et al., 2024), a curated subset of the original SWE-bench (Jimenez et al., 2023) where human annotators have validated that each task's ground-truth solution is correct and that the test suite properly distinguishes correct from incorrect fixes. This matters because if the benchmark's ground truth were noisy (as the original SWE-bench had some issues with), consistency measurements would be contaminated: an agent might be "consistently right" according to the benchmark but actually wrong, or vice versa. SWE-bench Verified provides a trustworthy correctness signal against which to evaluate the relationship between consistency and accuracy.

From SWE-bench Verified, the authors select 10 tasks, all from the astropy repository. This single-repository constraint is a deliberate design choice that the paper does not explicitly justify in depth, but the reasoning is clear in context: by holding the codebase constant, the paper controls for codebase-specific factors that might affect consistency—differences in documentation quality, codebase size, test suite coverage, or domain-specific complexity. If the paper had mixed tasks from astropy, django, sympy, and other repositories, it would be unclear whether consistency differences across tasks were driven by task difficulty or by codebase characteristics. The single-repository design isolates task-level factors (bug type, fix complexity) as the drivers of any observed cross-task consistency variation.

The 10 selected tasks span a deliberate diversity of bug types and fix complexities, summarized in Table 1. The bug types include logic errors (task 12907), missing checks (13033), silent type conversions (13236), edge cases (13398), type handling issues (13453), format string bugs (13579), deprecation problems (13977), boundary checks (14096), default value issues (14182), and import errors (14309). The fix sizes range from 2 lines (13579, a format string fix) to 12 lines (13977, a deprecation fix), with a median of 4.5 lines. The number of files modified is 1 for most tasks and 2 for two tasks (13398 and 13977). This diversity serves an important experimental purpose: if consistency patterns are the same across bug types and fix complexities, the findings generalize within the astropy ecosystem; if they differ, the diversity reveals which task characteristics drive consistency. The paper explicitly notes that "finding the right location and understanding the bug context requires substantial exploration" even though the median fix is small—this is the crux of why SWE-bench is a complex agent benchmark: the hard part is not the edit itself but the diagnosis.

Agent Scaffold (mini-SWE-agent)

The paper uses mini-SWE-agent (Jimenez et al., 2023), which is explicitly described as "a minimal agent scaffold that provides only a bash interface (no tool-calling API)." This design choice has deep implications for the action space and, consequently, for what consistency means in this study.

Bash-only interface. Unlike newer agent frameworks that expose structured function-calling APIs (e.g., a read_file function, an edit_file function), mini-SWE-agent gives the model a raw bash shell. Every action the agent takes is a bash command string submitted to subprocess.run inside the Docker container. This means the agent must compose standard Unix commands—ls for directory listing, find for file search, grep for content search, cat for file reading, sed for stream editing—to accomplish the higher-level actions of exploring, understanding, and editing. This design significantly expands the action space compared to structured APIs: there are many syntactically different but semantically equivalent ways to, for example, "find all Python files containing the string 'NdarrayMixin'." One run might use grep -r "NdarrayMixin" *.py, another might use find . -name "*.py" -exec grep "NdarrayMixin" {} \;, and a third might use grep -rn "NdarrayMixin" --include="*.py" .. These are semantically identical actions but syntactically different trajectories. This matters for the paper's consistency measurements: "consistency" in a bash-only interface is about strategic coherence (do runs follow similar high-level plans?) rather than literal sequence identity (do runs type exactly the same commands?). The paper's finding that 100% of runs produce unique sequences is partially an artifact of this interface design—it would likely be lower (but probably not zero) under a structured API where equivalent actions collapse to identical function calls.

Docker isolation. Each trial executes in "an isolated Docker container" with a "fresh repository state for each run." This is critical for the validity of the 5-run comparison: if runs shared filesystem state, the second run could benefit from files already explored or edits already made by the first run, contaminating the independence assumption. Fresh containers ensure that each of the 5 runs starts from an identical initial state—same files, same directory structure, same git history, same test suite state. Any divergence in behavior is thus attributable to the model's stochasticity (driven by temperature 0.5) and not to environmental differences.

Format-error handling bug fix. The paper mentions discovering and fixing "a format-error bug where agents could inadvertently submit during error recovery; patch submission is now blocked during format-error handling." This is worth examining because it reveals a subtle interaction between the scaffold and agent behavior. The bug was presumably that when the agent produced a malformed command (e.g., a sed command with incorrect syntax), the scaffold would enter an error recovery loop where it would ask the agent to reformulate the command, but the agent might misinterpret the error message and submit a patch instead of fixing the command. This could cause corrupted or premature submissions that would artificially inflate failure rates and potentially make consistency look different from what it would be with proper error handling. By fixing this bug, the paper ensures that all observed failures are genuine agent errors rather than scaffold-induced artifacts. This also explains the "valid patches" row in Table 2: Claude submitted valid patches on all 50 runs (meaning every run produced a syntactically well-formed patch, though not necessarily a correct one), GPT-5 on 48/50 (two runs hit the format error before the fix? or encountered other issues), and Llama on only 40/50 (10 runs failed to produce a valid patch at all). The 21% "empty patch" failure rate for Llama (Table 6) is thus partially about the model's inability to produce a well-formed submission, not just about incorrect fixes.

Model Configuration and Fair Comparison

The paper configures three models with identical external conditions to isolate model-level differences:

Temperature 0.5. All models use temperature 0.5, described as providing "moderate stochasticity." This is the only source of behavioral variation across the 5 runs of each model-task pair—there is no prompt variation, no tool access variation, no environment variation. Temperature 0.5 is a deliberate middle ground: temperature 0 (deterministic) would produce identical runs (trivializing the consistency study), while temperature 1.0 (maximum stochasticity) might amplify noise to the point where no meaningful consistency patterns could be observed. The choice of 0.5 is conventional in many agent studies and aims to capture "typical" deployment stochasticity without extreme randomness.

Maximum steps: 250. Each run is capped at 250 steps, where a step is one action-reply cycle (the model issues a command, the scaffold executes it, the output is returned to the model). This budget is generous relative to the observed step counts: Claude averages 46 steps, GPT-5 averages 10, Llama averages 17. No model comes close to exhausting the budget, which means the step cap does not constrain agent behavior—the models stop when they decide to submit a patch or give up, not when they run out of steps. This is important because an active step constraint could artificially induce consistency (if all runs hit the cap, they might appear similarly "long") or inconsistency (if the cap truncates exploration at different points in different runs).

Identical system prompt and tool definitions. The paper states that "all models have access to identical tools: bash commands for file system navigation, code editing, and test execution" and "we use the same system prompt and tool definitions for all." The specific prompt text is not reproduced in the paper, but the key principle is that any observed behavioral differences are attributable to the models themselves, not to different instructions or different tool capabilities. This is a basic experimental control but one that is easy to get wrong when working across model providers who may have different prompt formatting conventions. The paper's approach of using mini-SWE-agent's bash-only interface simplifies this: there is no tool-calling API to configure differently across providers; the interface is just "here is a shell, type commands."

**Cost limit: $10 per run.** The paper sets "a per-run cost limit of$ 10 (estimated from API token pricing)" and reports that "no run was truncated by this budget—the maximum observed cost was $2.91 (Claude)." This is less an active constraint and more a sanity check: it ensures that the cost structure of the experiments is known and that the results are not distorted by budget-induced early termination. The cost figures in Table 2 ($ 1.50/run for Claude, $0.53 for GPT-5,$ 0.10 for Llama) reflect standard API pricing at the time of the study and provide a practical dimension to the speed comparison: Claude is not just slower in steps, it is also more expensive per run.

The three models represent a capability spectrum. Claude 4.5 Sonnet (described as "a frontier model known for strong coding capabilities, with extended context and tool use"), GPT-5 ("OpenAI's frontier model with strong reasoning capabilities"), and Llama-3.1-70B-Instruct ("an open-weights model, smaller but widely deployed in production systems") span from highest to lowest expected capability on software engineering tasks. This three-tier selection enables the paper's cross-model analysis (RQ1: do models differ in consistency? RQ2: does consistency relate to accuracy?) to operate across a wide performance range—from 58% accuracy (Claude) to 4% (Llama). A study that only compared two similar frontier models might find no consistency differences; a study that only compared frontier to small models might find trivial differences driven entirely by capability gaps. The three-tier design enables the paper to observe whether the consistency-accuracy relationship is monotonic (it appears to be, at least ordinally) and whether it breaks down within capability tiers (as the within-model correlation analysis in Section 4.2 suggests).

Multi-Run Execution Protocol

The core experimental design is 5 independent trials per model-task pair, producing 50 runs per model (10 tasks × 5 runs) and 150 total agent trajectories. This design choice—5 runs rather than 3, 10, or 100—reflects a practical tradeoff between statistical power and computational cost. With 150 agent trajectories at $0.10–$ 1.50 per run, the total experiment cost is modest (on the order of $100–$ 200 in API fees plus compute for the Docker evaluation harness), making the study reproducible by other researchers. But the statistical power is limited: with n=5 per task, the paper can detect large consistency differences (e.g., CV of 15% vs. 47%) but cannot reliably distinguish more subtle differences (e.g., CV of 15% vs. 20%). The paper acknowledges this limitation in Section 7 ("With n = 3 models and 10 tasks, we can identify trends but cannot establish strong statistical relationships").

The "fresh repository state for each run" ensures that the 5 trials are independent in the statistical sense: the outcome of run 2 is not influenced by the outcome of run 1. This is trivial for Docker-based execution but is worth stating explicitly because it would be easy to overlook in a less carefully designed study. Without fresh state, an agent's exploration in run 1 (e.g., discovering which files are relevant) could be cached or learned from in run 2, making run 2 artificially faster or more accurate—a contamination that would inflate consistency if the later runs converge on the same correct solution that the earlier runs found.

All runs completed without infrastructure failures or budget truncation. This is an important operational detail: it means the 150-trajectory dataset is complete with no missing data. Every model-task pair produced exactly 5 valid trajectories. This eliminates the need for imputation or partial-data analysis and ensures that the per-task CV calculations are based on identical sample sizes.

Trajectory Decomposition and Phase Labeling

The paper introduces a five-phase decomposition of agent actions to answer RQ3 ("where does variance originate?"). Each action in a trajectory is classified into one of five categories based on the command type:

EXPLORE: Directory listing and file search commands (ls, find, grep). These actions expand the agent's knowledge of the codebase structure and locate candidate files relevant to the bug. They represent the "breadth-first" component of information gathering.
UNDERSTAND: File reading commands (cat, head, less). These actions deepen the agent's knowledge of specific file contents. They represent the "depth-first" component of information gathering. The distinction between EXPLORE and UNDERSTAND is conceptually important: exploration identifies which files might be relevant; understanding determines what those files contain and how they relate to the bug.
EDIT: Code modification commands (sed, echo >>, file writes). These actions change the repository state. They are the "production" phase where the agent implements its fix hypothesis.
VERIFY: Testing commands (python, pytest). These actions check whether the agent's edits have resolved the bug and haven't broken existing functionality. They provide feedback that may trigger further exploration, understanding, or editing.
OTHER: Catch-all for commands that don't fit the above categories. The paper does not define what falls into OTHER, but it likely includes navigation commands (cd), environment setup, git operations, and any scaffolding interactions that aren't pure exploration, understanding, editing, or verification.

Table 5 reports the distribution of actions across these phases for each model. Claude spends 17.8% on EXPLORE, 41.2% on UNDERSTAND, 14.5% on EDIT, 19.3% on VERIFY, and 7.2% on OTHER. GPT-5 shows a markedly different profile: only 13.0% on EXPLORE but 32.3% on VERIFY—a heavy emphasis on testing relative to exploration. Llama spends the most on EXPLORE (28.1%), suggesting it has more difficulty efficiently locating relevant files.

The paper also computes pre-edit time (the fraction of steps before the first edit): 59.0% for Claude, 44.4% for GPT-5, and 58.6% for Llama. Claude and Llama invest heavily in diagnosis before acting; GPT-5 begins editing sooner. This metric operationalizes the "speed-accuracy-consistency" intuition: GPT-5's speed comes partly from spending less time on pre-edit diagnosis.

How the phase labeling works in practice. The paper does not provide the exact classification rules, but the mapping is presumably based on command name pattern matching: ls* → EXPLORE, cat/head/less → UNDERSTAND, sed/echo/write → EDIT, python/pytest → VERIFY. This rule-based classification is deterministic and reproducible, which is important for a measurement study: another researcher processing the same trajectories would assign the same phase labels. However, it also means the classification may be coarse—a grep command that also reads file contents could be classified as EXPLORE even though it provides UNDERSTAND-level information, and a python -c "..." one-liner that edits a file would be classified as VERIFY even though it performs EDIT functionality. The paper treats this coarse classification as sufficient for its purposes: identifying broad phase-level patterns in where models spend their time and where variance concentrates.

Within-phase variance. The paper reports that "Claude shows the lowest within-phase variance across all phases, while Llama shows erratic behavior especially in EXPLORE (CV: 123% vs Claude's 42%)." The EXPLORE CV of 123% for Llama means that the number of EXPLORE steps varies by more than the mean itself—the standard deviation exceeds the mean—indicating extreme inconsistency in how Llama navigates the codebase. This is the phase where Llama's high overall CV is concentrated, consistent with the interpretation that Llama is inefficient at locating relevant files and sometimes finds them quickly (by luck) and sometimes wanders extensively.

Consistency Metric Design

The paper uses two primary metrics to operationalize "consistency": the coefficient of variation (CV) of step counts and sequence uniqueness.

Coefficient of Variation. The CV is defined as:

$\text{CV} = \frac{\sigma_{\text{steps}}}{\mu_{\text{steps}}} \times 100\%$

where $\sigma_{\text{steps}}$ is the standard deviation of step counts across the 5 runs of a given model-task pair, and $\mu_{\text{steps}}$ is the mean step count across those 5 runs.

What it computes: for each task and model, the CV expresses the spread of step counts as a percentage of the mean. A CV of 15.2% (Claude's aggregate) means that, on average, the step count for a given task varies by about 15% relative to the task's mean step count. A CV of 47.0% (Llama) means the variation is nearly half the mean.

Why CV rather than raw variance or standard deviation: the CV is scale-invariant, which matters because the mean step count varies dramatically across models (Claude: 46.1, GPT-5: 9.9, Llama: 17.0). A raw standard deviation of, say, 3 steps means something very different for GPT-5 (where it's 30% of the mean) than for Claude (where it's 6.5% of the mean). The CV normalizes for this, enabling fair comparison across models with different characteristic step budgets. Without normalization, GPT-5 might appear more consistent simply because it takes fewer steps—a spurious result driven by scale differences rather than genuine behavioral consistency.

How CV is aggregated: the paper computes CV per task per model (there are 10 CV values per model, one per task), then reports the mean CV across tasks. Table 9 in the appendix shows the per-task CVs: for Claude, they range from 8.8% (task 13398) to 27.0% (task 13236); for GPT-5, from 0.0% (task 13977) to 66.7% (task 13033); for Llama, from 28.7% (task 13453) to 67.2% (task 14309). The cross-model aggregate CVs (15.2%, 32.2%, 47.0%) are the means of these per-task values. The paper uses paired t-tests for CV comparisons because "models are evaluated on the same 10 tasks"—the pairing controls for task-level difficulty effects that might otherwise confound the comparison.

Sequence uniqueness. The paper reports that "100% of runs produce unique action sequences for all three models." This is measured by comparing the literal sequence of commands across the 5 runs of each task: if any two runs produce exactly the same sequence of commands in the same order, they would count as non-unique. The finding that no two runs are identical across all 150 trajectories is striking. It means that even at temperature 0.5, the 5-run sample is drawing from a distribution where the probability of any specific sequence is extremely low. This has implications for self-consistency methods (Wang et al., 2023): if no two trajectories are ever identical, then majority voting over complete action sequences is impossible—there is never a majority sequence to select.

The paper interprets this finding carefully: "behavioral consistency (low CV) does not mean deterministic behavior; models can be consistent in strategy while varying in execution details." In other words, Claude's low CV means its 5 runs take roughly the same number of steps, not that they take the same steps. Strategic consistency (following a similar high-level plan) can coexist with execution diversity (different specific commands, different file exploration orders).

Divergence step. The paper introduces a third consistency-related metric: the step at which not all 5 runs of a task execute the same action category. Specifically, "we define the divergence step as the first step where not all 5 runs of a task execute the same action category." This metric captures how long the model maintains perfect agreement before stochasticity drives the trajectories apart.

The calculation procedure is: for each task, align the 5 trajectories by step index (step 1 across all runs, step 2 across all runs, etc.), label each step with its phase category (EXPLORE, UNDERSTAND, EDIT, VERIFY, OTHER), and find the earliest step index where the 5 runs do not all share the same phase label. If at step 1, all 5 runs execute EXPLORE actions (even if different EXPLORE commands), the divergence step is at least 2. If at step 2, 4 runs execute EXPLORE and 1 executes UNDERSTAND, the divergence step is 2.

Table 7 reports the results: Claude's mean divergence is 3.2 steps (median 3.0), GPT-5's is 3.4 (median 3.5), and Llama's is 1.4 (median 1.0). The paper emphasizes the surprising finding: "Claude and GPT-5 diverge at similar steps, but Claude achieves 2.1× better consistency" in terms of overall CV. Divergence timing alone does not determine consistency—what happens after divergence (whether the model maintains strategic coherence or scatters widely) matters more.

Failure Mode Annotation Protocol

The paper introduces a critical distinction between two types of failures: interpretation errors (the agent's patch addresses a different semantic goal than the ground-truth fix) and execution errors (the patch targets the correct goal but contains implementation bugs). This distinction is the mechanism behind the paper's central "amplification insight"—that consistency amplifies interpretation quality, so when interpretation is wrong, consistency makes it reliably wrong rather than occasionally right by chance.

Annotation procedure. The paper states: "the first author examined each failed run's trajectory and final patch. A failure was labeled 'interpretation error' if the agent's patch addressed a different semantic goal than the ground-truth fix (e.g., adding a deprecation warning instead of removing code). 'Execution error' was assigned when the patch targeted the correct goal but contained implementation bugs." The annotation was performed on the 21 Claude failures, yielding the finding that 71% (15/21) are consistent wrong interpretation—the same incorrect approach across all 5 runs of those 3 tasks.

Why "semantic goal" rather than "exact match": two patches can be syntactically different but semantically equivalent (e.g., using a try/except block versus an if check to handle the same edge case). If the paper classified all non-identical patches as interpretation errors, it would overcount interpretation failures by treating implementation diversity as misunderstanding. The "semantic goal" criterion distinguishes between "the agent understood the bug correctly but implemented the fix imperfectly" (execution error) and "the agent fundamentally misunderstood what the bug was and thus solved a different problem" (interpretation error). The example given—"adding a deprecation warning instead of removing code"—clarifies the distinction: the agent correctly identified the relevant code location but wrongly concluded that the desired behavior was to warn about deprecation rather than to change the behavior entirely. This is an interpretation failure, not an execution failure, because no amount of better implementation of the deprecation warning would have passed the tests.

Limitation acknowledged. The paper explicitly notes: "This labeling was not blinded; we note it as a limitation and release all trajectories for independent verification." Unblinded annotation means the annotator (the first author) knew the model identity and the research hypotheses while labeling, which could introduce bias—for example, a tendency to label Claude's failures as "interpretation errors" because the paper's thesis emphasizes interpretation bottlenecks. The paper mitigates this by releasing the raw trajectories, enabling other researchers to verify or challenge the labels. This transparency is important for a study introducing a new failure taxonomy.

The consistent wrong quantification. The 71% figure (15/21) is computed as: out of Claude's 21 total failed runs, 15 belong to tasks where all 5 runs made the same interpretation error. On three tasks (astropy-13236, astropy-13398, astropy-13977), Claude makes "the identical mistake every time." This per-task pattern—all 5 runs failing identically on 3 tasks, contributing 15 failures—is what the paper calls "consistent wrong interpretation." The remaining 6 failures (21 - 15 = 6) come from the two "mixed results" tasks (Table 3) where Claude sometimes succeeded and sometimes failed, suggesting execution noise rather than systematic misinterpretation.

This annotation scheme enables the paper's key conceptual move: transforming the observation that "Claude fails on 42% of tasks" into the more informative diagnosis that "most of Claude's failures are systematic misinterpretations, not stochastic execution errors." The practical implication follows directly: improving Claude's execution consistency (e.g., better testing, more thorough editing) would not help on these 3 tasks, because the executions are already consistent—they're just consistently wrong. Only improving interpretation quality (better initial bug understanding, more flexible hypothesis generation, mechanisms for recognizing and escaping incorrect interpretations) would convert these failures to successes.

Statistical Analysis Design

The paper uses two families of statistical tests, chosen based on the structure of the comparison:

Paired t-tests for CV and divergence metrics. Because the same 10 tasks are used across all three models, CV values and divergence steps are naturally paired: each task contributes one CV value for Claude, one for GPT-5, and one for Llama. The paired t-test ( $df = 9$ ) tests whether the mean difference between two models' per-task CVs is significantly different from zero. This controls for task-level difficulty as a confound: if some tasks are inherently more variable than others (regardless of model), the pairing removes this variance component from the comparison, increasing power. The paper reports specific values: Claude vs. GPT-5 CV comparison yields $t(9) = 2.59, p = 0.029$ , which is significant at the conventional 0.05 threshold. GPT-5 vs. Llama yields $t(9) = -1.92, p = 0.087$ , which is not significant at 0.05 but would be at 0.10—the paper correctly reports the exact p-value and lets readers interpret it rather than dichotomizing.

Fisher's exact tests for accuracy comparisons. Accuracy is binary per run (pass/fail) but the paper aggregates to per-task accuracy (number of successes out of 5 runs per task), producing counts rather than means. The paper uses Fisher's exact test rather than a chi-squared test because the sample sizes are small (10 tasks per model) and expected cell counts may be low. The reported odds ratios (OR = 0.34 for GPT-5 vs. Claude, OR = 11.3 for GPT-5 vs. Llama, OR = 33.1 for Claude vs. Llama) express how much more or less likely one model is to succeed compared to another. An OR of 0.34 means GPT-5 is about one-third as likely to produce a correct run as Claude; an OR of 33.1 means Claude is about 33 times more likely to succeed than Llama—a massive gap that is unequivocally significant ( $p < 0.001$ ). The paper notes that "accuracy is binary per run, not naturally paired," which is why Fisher's test rather than a paired test is appropriate.

Effect sizes reported. The paper reports Cohen's d alongside the t-tests for CV comparisons: $d = 1.16$ for Claude vs. GPT-5, $d = -0.86$ for GPT-5 vs. Llama, $d = -3.02$ for Claude vs. Llama. These are large (d > 0.8) to very large (d > 2.0) effect sizes by conventional benchmarks, suggesting that the consistency differences observed are not just statistically detectable but substantively meaningful. However, with only 10 tasks and high variance in per-task CVs (as visible in Figure 1's point cloud), these effect sizes may be unstable estimates.

Within-model correlations. For RQ2 ("does consistency relate to accuracy?"), the paper computes Pearson correlation coefficients between per-task CV and per-task accuracy for each model separately. The reported values ( $r = -0.10$ for Claude, $r = -0.15$ for GPT-5, $r = 0.30$ for Llama) are all non-significant ( $p > 0.4$ for all). The negative signs for Claude and GPT-5 (higher CV associated with lower accuracy, which would be the expected direction if consistency helps) are weak, and Llama's positive sign is the opposite direction (higher CV associated with higher accuracy). With only 10 data points per model, the power to detect moderate correlations is very low—a correlation of $r = 0.3$ requires n ≈ 85 to detect at 80% power. The null finding should thus be interpreted cautiously: the paper cannot rule out a real but moderate within-model relationship that 10 tasks lack the power to detect.

Summary of Design Choices and Their Justifications

Single repository (astropy): controls for codebase-specific factors (documentation quality, codebase size, domain complexity) that could otherwise confound cross-task consistency comparisons. The tradeoff is reduced external validity—findings may not generalize to other codebases.
Bash-only interface via mini-SWE-agent: provides a uniform, model-agnostic action space that avoids differential tool-calling API implementations across model providers. The tradeoff is that the expanded action space inflates sequence uniqueness, making literal consistency unachievable and forcing the analysis to focus on strategic consistency.
Temperature 0.5: balances stochasticity (needed to observe variance) against excessive randomness (which would swamp any consistency signal). The paper does not sweep temperature, so the choice is a fixed assumption rather than an empirically optimized value.
5 runs per task: provides enough samples to compute per-task CVs (which require at minimum 2 runs, and are unreliable with fewer than 4–5) while keeping total experiment cost manageable (150 trajectories). The tradeoff is low statistical power for within-task analyses.
Phase decomposition (EXPLORE/UNDERSTAND/EDIT/VERIFY/OTHER): a deterministic, reproducible, command-based classification that enables localization of variance to specific activity types. The tradeoff is that classification is coarse—commands can serve multiple purposes—but the paper judges this acceptable for its macroscopic phase-level analysis.
CV as primary consistency metric: scale-invariant normalization enables fair comparison across models with very different mean step counts. The alternative—raw variance or standard deviation—would confound consistency with step budget magnitude.
Manual, unblinded failure annotation: provides the semantic distinction between interpretation and execution errors that drives the paper's central insight, but at the cost of potential annotator bias. The release of raw trajectories provides a partial mitigation by enabling independent verification.
Paired statistical tests where possible: leverages the shared-task design to control for task-level confounds, increasing power given the small sample of 10 tasks. Fisher's exact test is used where pairing is not applicable (binary accuracy outcomes).

4. Key Insights and Innovations

Innovation 1: The Amplification Insight — Consistency as an Outcome Multiplier, Not a Correctness Guarantee

The paper's most intellectually distinctive contribution is what it calls the amplification insight: consistency does not independently predict correctness; it amplifies whatever outcome the agent's interpretation quality produces. When the agent correctly understands a task, consistency ensures reliable success across all runs (Claude's 5 tasks with correct interpretation achieved 100% accuracy—25/25 runs, Table 3). When the agent incorrectly understands a task, consistency ensures reliable failure across all runs (the 3 tasks with wrong interpretation achieved 0% accuracy—0/15 runs). This is not merely an empirical observation about Claude; it is a conceptual reframing of what consistency means for agent reliability.

Prior to this work, the dominant intuition in the field—implicit in how consistency was discussed and measured—was that higher consistency is uniformly desirable. Self-consistency methods (Wang et al., 2023) treat variance as noise to be averaged out, operating on the premise that sampling multiple reasoning paths and taking a majority vote improves accuracy because the model is more often right than wrong. Consistency research in simpler domains (Elazar et al., 2021; Mehta, 2026) frames inconsistency as a problem to be solved. The underlying assumption is that if an agent is consistent, it will be consistently right, or at least more often right than wrong. This paper demolishes that assumption by demonstrating—with concrete evidence from complex, multi-step trajectories—that consistency is valence-neutral: it amplifies correct interpretations into reliable successes and incorrect interpretations into reliable failures with equal fidelity.

What makes this a fundamental insight rather than an incremental observation is that it changes the diagnostic framework for agent improvement. Under the old mental model, if an agent fails on some runs of a task, the natural response is to improve consistency—better execution, more thorough testing, longer trajectories. The amplification insight reveals that this response is only appropriate when interpretation is already correct. If interpretation is wrong—as it was for 71% of Claude's failures—improving consistency would only make the agent fail more reliably, not succeed more often. The correct improvement strategy depends on the failure mode: interpretation errors require fundamentally different interventions (better initial understanding, hypothesis diversity, mechanisms for detecting and escaping wrong interpretations) than execution errors (better verification, more careful editing, error recovery). This diagnostic distinction was invisible under single-run evaluation and becomes actionable only when multi-run consistency data is available and correctly interpreted through the amplification lens.

The finding that consistency correlates with accuracy across models (Claude > GPT-5 > Llama on both metrics) but not within any single model (Section 4.2, Figure 3: within-model Pearson r values of −0.10, −0.15, and 0.30, all non-significant) is the empirical signature of the amplification mechanism. If consistency directly caused accuracy, the relationship should appear within models as well as across them—tasks where a model is more consistent should also be tasks where it is more accurate. The absence of this within-model correlation is evidence that consistency is not an independent causal factor but rather a property that interacts multiplicatively with interpretation quality. A model with better interpretation (Claude) benefits from its consistency; a model with worse interpretation would only have its errors amplified. The cross-model correlation exists because interpretation quality and consistency happen to co-vary across these three models, but the mechanism is amplification, not direct causation.

This insight also explains why the paper's title is "Consistency Amplifies" rather than "Consistency Improves" or "Consistency Predicts." The verb choice is precise: amplification is value-neutral, making good outcomes better and bad outcomes worse. This is a conceptual contribution that will likely travel beyond software engineering agents to any domain where multi-step autonomous decision-making is evaluated—robotics, scientific reasoning, medical diagnosis—because the diagnostic question "are failures interpretation errors or execution errors?" is universally applicable, and the answer determines whether the appropriate response is better understanding or better execution.

Innovation 2: Divergence Timing as a Necessary but Insufficient Condition for Consistency

The paper's second conceptual contribution is the finding that when trajectories first diverge does not determine how much they ultimately vary. Claude and GPT-5 diverge at nearly identical steps (mean divergence: 3.2 vs. 3.4 steps; Table 7), yet Claude achieves 2.1× better overall consistency (CV: 15.2% vs. 32.2%). This is a non-obvious result with significant implications for how we think about and analyze agent trajectories.

The intuitive mental model—one that the paper itself may have initially entertained—is that consistency is primarily about early alignment: if a model makes the same initial decisions across runs, those runs will follow similar paths and end up with similar outcomes. Under this model, divergence timing would be the primary predictor of overall consistency: later divergence means more consistent behavior. The paper's data falsifies this model. Llama diverges immediately (60% of tasks at step 1, mean divergence 1.4 steps) and is indeed highly inconsistent (CV: 47.0%). But the critical comparison is Claude vs. GPT-5: they share early agreement yet diverge sharply in overall consistency. Something happens after step 3 that causes GPT-5's trajectories to scatter while Claude's remain coherent.

What that "something" is, the paper characterizes qualitatively: "Claude maintains coherent strategies across runs even after initial divergence, while GPT-5's trajectories scatter more widely" (Section 6.4). This suggests a two-phase model of agent consistency: an initial phase of strategic agreement (do runs agree on the high-level plan?) followed by a sustained phase of strategic coherence (do runs execute variations of the same plan, or do they switch to different plans?). Divergence timing captures only the first phase. The second phase—post-divergence coherence—is what distinguishes Claude from GPT-5, and it is a dimension of consistency that prior work had not identified because prior work had not measured divergence timing at all.

This is a conceptual advance because it decomposes consistency into two mechanistically distinct components. Early strategic agreement is about the model's initial interpretation: given the task description, does it converge on the same first few actions (typically exploration commands to locate relevant files)? Post-divergence coherence is about the model's stability under path-dependence: once stochasticity causes two runs to take different actions at step 4, do they both continue pursuing the same underlying strategy (e.g., "find all files related to X, then read them, then edit the most relevant one") or do they fork into qualitatively different strategies (e.g., one run pivots to a different hypothesis about the bug while the other deepens its initial approach)? GPT-5's high variance despite late divergence suggests that small early differences cascade into large strategic divergences—a butterfly effect in agent trajectories—while Claude's low variance suggests that small early differences remain constrained within the same strategic basin of attraction.

For practitioners, this decomposition suggests that evaluating agents solely on whether they "start correctly" is insufficient. Two agents that both pass the first-3-steps agreement test may have radically different reliability profiles depending on what happens after divergence. This has implications for agent monitoring in production: measuring post-divergence trajectory dispersion could serve as an early warning signal that an agent is about to enter an unreliable regime, even if its initial actions look correct. It also suggests that training interventions to improve consistency should target post-divergence coherence—teaching models to maintain strategic focus even when execution details vary—rather than just improving initial agreement.

Innovation 3: The Speed-Accuracy-Consistency Tradeoff as a Trilemma

The paper identifies and empirically characterizes what it calls the speed-accuracy-consistency tradeoff, revealed most starkly by GPT-5: it is 4.7× faster than Claude (9.9 vs. 46.1 steps per run, Table 2), but achieves 1.8× lower accuracy (32% vs. 58%) and 2.1× worse consistency (CV: 32.2% vs. 15.2%). This is not simply a restatement of the well-known speed-accuracy tradeoff from cognitive psychology and machine learning; it adds consistency as a third, partially independent dimension that complicates the standard two-axis tradeoff.

The traditional speed-accuracy tradeoff is familiar from many domains: faster decisions are less accurate; slower, more deliberate decisions are more accurate. What the paper shows is that consistency is not simply a byproduct of accuracy—if it were, GPT-5's lower accuracy would fully explain its lower consistency, and the within-model correlation between consistency and accuracy would be strong (which it is not, as discussed in Innovation 1). Instead, consistency varies independently enough to constitute a third dimension of the tradeoff space. A model can be fast and moderately accurate but inconsistent (GPT-5), or slow and highly accurate and highly consistent (Claude), or intermediate on all three dimensions (Llama, though its accuracy is so low that the tradeoff is barely relevant).

The trilemma framing is significant because it changes the deployment decision from a binary choice to a three-way optimization. Before this work, a practitioner choosing between Claude and GPT-5 for a software engineering task might have considered only speed and accuracy: "GPT-5 is faster but less accurate—which matters more for my use case?" The consistency dimension adds a new consideration: GPT-5's lower consistency means its accuracy is less predictable per-run. A 32% accuracy rate from an inconsistent agent might mean it solves some tasks reliably and others never, or it might mean it solves many tasks occasionally—patterns with different operational implications. The paper's Figure 8 further illustrates this by plotting the three models in a step-count vs. CV space, showing that Claude clusters in a high-step, low-CV region, GPT-5 in a low-step, moderate-CV region, and Llama in a high-CV region regardless of step count.

This trilemma is not merely descriptive; it has prescriptive force. The paper argues (Section 6.3) that "practitioners must choose their priority based on deployment context: rapid prototyping may favor GPT-5's speed, while production systems may require Claude's reliability." But the trilemma also implies that there is no single model that dominates on all three dimensions for the tasks studied. A model that was simultaneously fast, accurate, and consistent would be strictly superior, but none of the three models achieves this. This suggests that the speed-accuracy-consistency tradeoff may be a fundamental constraint—perhaps arising from architectural properties like the depth of reasoning chains or the calibration of uncertainty—rather than an accidental property of these particular models. The paper does not prove this, but it establishes the empirical pattern that future work can attempt to explain or overcome.

The phase decomposition in Table 5 provides mechanistic texture to the tradeoff. GPT-5's speed comes partly from spending less time on pre-edit diagnosis (44.4% of steps before first edit, vs. 59.0% for Claude) and more on verification (32.3% vs. 19.3%). Claude invests heavily in understanding before acting; GPT-5 acts sooner and verifies more. This suggests that the speed-accuracy-consistency tradeoff is partly about where in the trajectory the model allocates its step budget—early investment in understanding buys accuracy and consistency at the cost of speed. An optimal agent might adapt this allocation based on task complexity, a direction the paper flags in Section 5.2 ("An optimal agent might adapt its strategy based on estimated task complexity") and in the conclusion ("Future work should explore adaptive strategies that balance thoroughness with efficiency based on task complexity"). The trilemma thus opens a concrete research direction: can we build agents that dynamically modulate their speed-accuracy-consistency operating point based on real-time signals of task difficulty or confidence?

Innovation 4: The Interpretation Bottleneck Diagnosis — Why Better Execution Cannot Fix Most Failures

The paper's most actionable diagnostic finding is that interpretation quality, not execution consistency, is the primary bottleneck for reliable agent deployment on complex software engineering tasks. This is not an assumption or a theoretical claim—it is an empirical diagnosis supported by the failure mode taxonomy (Table 6) and the within-model consistency analysis (Table 3).

The diagnosis works as follows. Claude's 21 failed runs come from 5 tasks (Table 4). On 3 of those 5 tasks—accounting for 15 of the 21 failures, or 71%—every single run makes the same incorrect interpretation of the bug. The remaining 6 failures come from 2 tasks where Claude sometimes succeeds and sometimes fails, suggesting execution noise rather than systematic misunderstanding. This distribution tells a clear story: Claude's remaining errors are overwhelmingly systematic rather than stochastic. It is not that Claude occasionally messes up an otherwise correct understanding; it is that on some tasks, Claude's understanding is consistently wrong, and no amount of additional runs, better testing, or more careful editing would fix it.

This diagnosis has immediate practical implications that invert the conventional wisdom about improving agents. The dominant approaches to agent improvement focus on execution quality: better tool use, more thorough testing, longer trajectories, error recovery mechanisms (Shinn et al., 2023), and self-consistency through majority voting (Wang et al., 2023). All of these assume that the agent's interpretation is basically correct and that failures arise from execution shortcomings. The paper's diagnosis reveals that for a frontier model like Claude on software engineering tasks, this assumption is violated on a substantial fraction of failures. No amount of execution improvement can convert a deprecation warning into a code removal (the astropy-13236 case study in Section 5.1) because the agent never considers the correct hypothesis to execute.

What makes this a conceptual innovation rather than just an empirical observation is that it identifies the bottleneck through a novel diagnostic method—multi-run consistency measurement with failure mode annotation—that was not previously available. Single-run evaluation cannot distinguish between "the agent happened to fail this time" and "the agent would always fail on this task." It is only by running 5 trials and observing that all 5 produce the same wrong fix that the interpretation bottleneck becomes visible. The paper thus provides both the diagnosis and the diagnostic tool that enables it, making the contribution methodologically self-contained.

The interpretation bottleneck also explains why the "consistent wrong" failure mode is particularly dangerous for reflection-based improvement methods like Reflexion (Shinn et al., 2023). Reflexion assumes that agents can detect their own failures through testing and then revise their approach. But if the agent's interpretation is wrong at a level that test failures cannot localize—"the tests fail because my deprecation warning has a bug" rather than "the tests fail because a deprecation warning is the wrong approach entirely"—reflection will lead to iterative refinement of an incorrect solution rather than a strategic pivot. The paper's case study of astropy-13236 (Section 5.1) provides a concrete example: Claude spent 30–50 steps per run "implementing and debugging a FutureWarning," never questioning whether a warning was the right intervention. Llama, by contrast, succeeded on one run not because it was more consistent or more thorough, but because its variance caused it to stumble onto the correct interpretation—"remove the conversion code"—which it executed in 13 steps. This is a stark demonstration that interpretation quality can matter more than execution quality, and that consistency without interpretation quality is counterproductive.

For the research agenda going forward, this diagnosis redirects attention from execution improvement to interpretation improvement. The paper suggests (Section 6.1) that "improving initial task interpretation may yield larger gains" than "better tool use, more thorough testing, longer trajectories." Concrete directions include: building agents that generate and maintain multiple competing hypotheses about the bug rather than committing early to a single interpretation; developing verifiers that can detect not just whether a patch passes tests but whether the patch's semantic goal matches the task description; and designing scaffold mechanisms that periodically prompt the agent to reconsider its fundamental assumptions rather than iteratively refining its current approach. These are qualitatively different interventions from the execution-focused improvements that dominate current agent research, and they are made salient specifically by the consistency-based diagnostic framework this paper introduces.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses SWE-bench Verified (Jimenez et al., 2024), a curated subset of the original SWE-bench (Jimenez et al., 2023) where human annotators have validated that each task's ground-truth solution is correct and that the test suite properly distinguishes correct from incorrect fixes. From this benchmark, the authors select 10 tasks, all from the astropy repository, chosen for diversity in bug types and fix complexity (Table 1). The tasks span logic errors, missing checks, silent type conversions, edge cases, format string bugs, deprecation issues, boundary checks, default value problems, and import errors, with fix sizes ranging from 2 to 12 lines (median 4.5 lines) across 1–2 files. The single-repository constraint controls for codebase-specific factors (documentation quality, codebase size, domain complexity) that could otherwise confound cross-task consistency comparisons.
Base models. The paper compares three models representing different capability tiers: Claude 4.5 Sonnet (claude-sonnet-4-5-20250929, described as "a frontier model known for strong coding capabilities, with extended context and tool use"), GPT-5 (openai-gpt-5, February 2026, described as "OpenAI's frontier model with strong reasoning capabilities"), and Llama-3.1-70B-Instruct (described as "an open-weights model, smaller but widely deployed in production systems"). All three are configured with identical external conditions: temperature 0.5, maximum 250 steps, identical system prompt and tool definitions, and access only to bash commands for file system navigation, code editing, and test execution via the mini-SWE-agent scaffold.
Metrics. The paper uses several complementary metrics to operationalize "consistency." The primary metric is the coefficient of variation (CV) of step counts: CV = σ_steps / μ_steps × 100%, computed per task per model across the 5 runs, then averaged across the 10 tasks to produce the aggregate CV values (15.2%, 32.2%, 47.0%). Accuracy is measured using the official SWE-bench evaluation harness, which applies each submitted patch to an isolated Docker container and runs the repository's test suite; a patch is marked "resolved" only if all previously-failing tests pass. The paper also reports sequence uniqueness (the fraction of runs producing unique action sequences) and divergence step (the first step where not all 5 runs of a task execute the same action category). Phase decomposition categorizes each action as EXPLORE (ls, find, grep), UNDERSTAND (cat, head, less), EDIT (sed, echo, file writes), VERIFY (python, pytest), or OTHER, enabling localization of variance to specific activity types. Failure mode classification distinguishes "interpretation errors" (the agent's patch addresses a different semantic goal than the ground-truth fix) from "execution errors" (correct goal, implementation bugs).
Baselines. The paper does not compare against prior methods—it is a measurement study, not a method-proposing paper. The three models serve as each other's baselines in the cross-model comparisons. The implicit baseline being challenged is the standard SWE-bench evaluation protocol of single-run accuracy reporting.
Generation budget / compute accounting. Each run is capped at 250 steps (one action-reply cycle per step), though no model approaches this limit (Claude averages 46.1 steps, GPT-5 9.9, Llama 17.0). A per-run **cost limit of $10** is set based on API token pricing, but no run was truncated by this budget (maximum observed cost was$ 2.91 for Claude). The cost figures in Table 2 ( $1.50/run for Claude,$ 0.53 for GPT-5, $0.10 for Llama) reflect inference cost only and exclude the SWE-bench evaluation harness compute (approximately 110 seconds per task for test suite execution).
Cross-validation / statistical protocol. The paper does not use cross-validation (there is no model training or hyperparameter selection). For statistical comparisons, paired t-tests (df = 9) are used for CV and divergence metrics since the same 10 tasks are used across all three models, controlling for task-level difficulty as a confound. Fisher's exact tests are used for accuracy comparisons since accuracy is binary per run and not naturally paired. Pearson correlation coefficients are computed for within-model per-task CV vs. accuracy relationships. Effect sizes (Cohen's d) and odds ratios are reported alongside significance tests. The complete statistical details appear in Appendix C.

Main Quantitative Results

RQ1: Cross-Model Consistency Hierarchy

The headline finding is a clear three-tier hierarchy in both consistency and accuracy across models (Table 2). Claude achieves the lowest CV (15.2%), indicating its step counts vary by approximately 7 steps around a mean of 46. GPT-5 is intermediate (CV: 32.2%), with variation of approximately 3 steps around a mean of 10. Llama shows the highest variance (CV: 47.0%). The rank ordering—Claude > GPT-5 > Llama—holds identically for accuracy: 58%, 32%, and 4% respectively.

These differences are statistically significant for the key comparisons. Claude is significantly more consistent than GPT-5 (t(9) = 2.59, p = 0.029, Cohen's d = 1.16), and Claude is dramatically more consistent than Llama (t(9) = −6.75, p < 0.001, d = −3.02). The GPT-5 vs. Llama consistency comparison is not significant at the conventional 0.05 threshold (t(9) = −1.92, p = 0.087, d = −0.86). For accuracy, all pairwise comparisons are significant by Fisher's exact test: GPT-5 vs. Claude (OR = 0.34, p = 0.015), GPT-5 vs. Llama (OR = 11.3, p < 0.001), Claude vs. Llama (OR = 33.1, p < 0.001).

A striking finding reported in Section 4.1 is that 100% of runs produce unique action sequences for all three models. Despite this, behavioral consistency (low CV) is achievable—Claude's low CV means its runs take roughly the same number of steps and follow similar high-level strategies, even though no two runs produce literally identical command sequences. Figure 1 visualizes the consistency gap: individual task CVs are shown as points, with Claude's clustering tightly at low values and Llama's scattering widely. Figure 2 provides a heatmap representation across all 150 runs, where Claude's uniform coloring indicates similar step counts across runs while Llama's patchy appearance reflects high variance.

The paper also establishes what it calls the speed-accuracy-consistency tradeoff (Section 4.1): GPT-5 is 4.7× faster than Claude (9.9 vs. 46.1 steps) but achieves 1.8× lower accuracy and 2.1× worse consistency. This tradeoff is visualized in Figure 8, which plots the three models in a step-count vs. CV space, showing Claude clustering in a high-step, low-CV region, GPT-5 in a low-step, moderate-CV region, and Llama showing high variance regardless of step count.

RQ2: The Consistency-Accuracy Relationship

The paper finds a perfect rank correlation across models (Claude: most consistent and most accurate; GPT-5: middle on both; Llama: least consistent and least accurate). However, the within-model analysis reveals the crucial nuance: no significant correlation between per-task CV and accuracy within any model (Section 4.2, Figure 3). The Pearson r values are Claude: r = −0.10 (p = 0.78), GPT-5: r = −0.15 (p = 0.68), Llama: r = 0.30 (p = 0.40). This null within-model finding, combined with the strong cross-model pattern, is the empirical foundation for the paper's central amplification insight.

The mechanism is revealed in Table 3, which decomposes Claude's performance by interpretation quality. On the 5 tasks where Claude correctly understood the bug, it achieved 100% accuracy (25/25 runs). On the 3 tasks where it incorrectly interpreted the bug, it achieved 0% accuracy (0/15 runs). On the 2 tasks with mixed results, accuracy was 40% (4/10). This tri-modal distribution—100%, 0%, or intermediate—is precisely what the amplification mechanism predicts: when interpretation is correct, consistency ensures reliable success; when interpretation is wrong, consistency ensures reliable failure; only on tasks where interpretation is ambiguous or partial does execution noise determine outcomes on a run-by-run basis.

Cross-model per-task performance is detailed in Table 4 and Figure 4. Claude solves 7/10 tasks at least once, GPT-5 solves 6/10, and Llama solves 2/10. GPT-5 achieves perfect 5/5 accuracy on astropy-14309, matching Claude. Notably, on astropy-13236, Llama (20% accuracy, 1/5) outperforms both Claude and GPT-5 (both 0%), a result that becomes a case study in Section 5.1 demonstrating how variance can occasionally stumble onto correct interpretations that consistency-driven models miss.

RQ3: Phase Decomposition and Variance Origins

Table 5 presents the phase distribution findings. Claude invests heavily in UNDERSTAND (41.2% of actions) and spends 59.0% of steps before making the first edit. GPT-5 shows a distinctive profile: it spends the most time on VERIFY (32.3%), heavily uses the nl (numbered line output) command (19.4% of actions, a command rarely used by the other models), and begins editing sooner (only 44.4% pre-edit steps). Llama spends the most on EXPLORE (28.1%), suggesting difficulty efficiently locating relevant files. These distributions are visualized in Figure 5.

Within-phase variance analysis (Section 4.3) reveals that "Claude shows the lowest within-phase variance across all phases, while Llama shows erratic behavior especially in EXPLORE (CV: 123% vs Claude's 42%)." The EXPLORE CV of 123% for Llama means the standard deviation of exploration steps exceeds the mean—indicating extreme inconsistency in codebase navigation, where Llama sometimes finds relevant files quickly (by luck) and sometimes wanders extensively. GPT-5 falls between the two, "with notably low variance in VERIFY due to its consistent testing strategy."

RQ4: Failure Mode Taxonomy

Table 6 presents the failure mode breakdown across all models. The dominant failure mode across all three models is submitting incorrect fixes (WRONG FIX): 100% of Claude's 21 failures, 94% (32/34) of GPT-5's failures, and 79% (38/48) of Llama's failures. The secondary failure mode—submitting empty patches (EMPTY PATCH), effectively "giving up"—is largely absent from the more capable models: 0% for Claude, 6% (2/34) for GPT-5, but 21% (10/48) for Llama. No model experienced LOOP DEATH (infinite action loops). Figure 6 visualizes these distributions.

The paper's critical diagnostic finding is the consistent wrong pattern (Section 4.4): "71% (15/21) of Claude's failures are 'consistent wrong interpretation': the same incorrect approach across all 5 runs of a task. On three tasks (astropy-13236, astropy-13398, astropy-13977), Claude makes the identical mistake every time." This is the evidence that interpretation failures dominate execution failures for the most capable model studied.

Annotation protocol (described in Section 4.4): The first author examined each failed run's trajectory and final patch. A failure was labeled "interpretation error" if the agent's patch addressed a different semantic goal than the ground-truth fix (the example given: "adding a deprecation warning instead of removing code"). "Execution error" was assigned when the patch targeted the correct goal but contained implementation bugs. The paper explicitly notes that "this labeling was not blinded; we note it as a limitation and release all trajectories for independent verification."

RQ5: Divergence Timing Analysis

Table 7 presents a surprising finding: Claude and GPT-5 diverge at nearly identical steps despite achieving very different consistency. Claude's mean divergence is 3.2 steps (median 3.0), GPT-5's is 3.4 (median 3.5), a difference that is not statistically significant (t(9) = 0.25, p = 0.808). Yet Claude's CV of 15.2% is 2.1× better than GPT-5's 32.2%. Both models diverge much later than Llama (mean 1.4 steps, median 1.0, with 60% of tasks diverging at step 1), and the Claude vs. Llama and GPT-5 vs. Llama divergence comparisons are both significant (p = 0.030 and p < 0.001 respectively).

Figure 7 visualizes the divergence step distribution. The key pattern: 0% of Claude's tasks and 0% of GPT-5's tasks diverge at step 1, while 60% of Llama's tasks do. By step 5, 90% of Claude's tasks and 100% of GPT-5's tasks have diverged.

First action analysis (Table 8) reveals that the opening move strongly predicts model but not success. GPT-5 always starts with ls (100% of runs), making it the most predictable opening of any model, yet it achieves only 32% accuracy. Claude uses find 68% of the time, ls 26%, and achieves 85% success on its ls-first runs. Llama shows the most diverse openings (ls 54%, grep 24%, find 20%). The paper concludes that "strategic coherence, not starting action, predicts success."

Ablation Studies and Robustness Checks

This paper, as a measurement study rather than a method-proposing paper, does not contain traditional ablation studies in the sense of removing components from a proposed system and measuring performance degradation. However, several analyses serve the same diagnostic function by isolating specific factors and examining their relationship to consistency.

Phase-level variance decomposition: By computing within-phase CV values and comparing them across models (Section 4.3), the paper implicitly ablates the contribution of different activity types to overall variance. The finding that Llama's EXPLORE CV is 123% versus Claude's 42% isolates codebase navigation as the primary driver of Llama's inconsistency, while Claude's low within-phase variance across all categories demonstrates that its consistency advantage is not confined to a single phase but is pervasive.

Within-model per-task correlation analysis (Figure 3): This serves as an ablation of the hypothesis that consistency and accuracy are monotonically related. By computing per-task CV-vs-accuracy correlations within each model separately, the paper tests whether the cross-model consistency-accuracy relationship survives when holding model capability constant. The null results (r = −0.10, −0.15, 0.30, all non-significant) demonstrate that consistency does not predict accuracy at the task level—a finding that could not have been obtained from the cross-model comparison alone and that directly motivates the amplification insight as an alternative to a simple causal model where consistency improves accuracy.

Failure mode annotation (Table 6): By manually classifying failures as interpretation vs. execution errors, the paper ablates the contribution of these two qualitatively different error sources to overall failure rates. The 71% figure for Claude's interpretation failures quantifies the upper bound on improvement achievable through better execution alone—even if execution errors were completely eliminated (the remaining 29%, or 6/21 failures), Claude's accuracy would only improve from 58% to at most 70% (solving 7/10 tasks instead of 5/10 plus the mixed-results tasks). The interpretation errors on 3 tasks would remain intractable to execution-focused improvements.

First-action vs. success analysis (Table 8): This ablates the hypothesis that the initial action choice predicts task success. GPT-5's 100% ls first action yields only 32% accuracy, while Claude's ls-first runs (26% of its total) succeed 85% of the time. The identical starting action produces radically different outcomes depending on the model employing it, demonstrating that the opening move is not independently predictive.

Negative results. The paper's most informative negative result is the statistical non-significance of the GPT-5 vs. Llama consistency comparison (p = 0.087). Despite a 14.8 percentage point difference in aggregate CV (32.2% vs. 47.0%), the high per-task variance and small sample (n = 10 tasks) mean this difference cannot be confidently distinguished from noise. This is an important caution against over-interpreting the intermediate position of GPT-5 in the consistency hierarchy. The paper also notes that "we cannot establish causality from n = 3 models" (Section 4.2), explicitly identifying the cross-model rank correlation as suggestive rather than conclusive.

Critical Assessment

The paper makes several substantive claims, each requiring careful scrutiny against the evidence presented.

Claim 1: Across models, higher consistency aligns with higher accuracy (Claude > GPT-5 > Llama on both metrics). This claim is empirically supported by the data in Table 2 and the statistical tests in Appendix C. The rank ordering is unambiguous, and the pairwise accuracy comparisons are all significant. However, the claim is based on n = 3 models, which is trivially small for establishing a general relationship. The paper acknowledges this limitation explicitly (Section 7: "With n = 3 models and 10 tasks, we can identify trends but cannot establish strong statistical relationships"). A study with 5–10 models spanning the full capability range would be needed to determine whether the consistency-accuracy correlation is monotonic, logarithmic, or follows some other functional form. The current evidence demonstrates that for these three specific models, consistency and accuracy co-vary in the same rank order, but the paper's implicit suggestion that this pattern generalizes to "across models" writ large is an extrapolation beyond the data. A counterexample—a model with high consistency but low accuracy (imagine a model that consistently outputs empty patches) or low consistency but high accuracy (a model that succeeds through diverse strategies)—would falsify the generalization, and such models likely exist in the broader space of possible agents.

Claim 2: Consistency amplifies outcomes rather than guaranteeing correctness, with 71% of Claude's failures being "consistent wrong interpretation." This is the paper's strongest and most distinctive claim, and the evidence for it is substantial but comes with methodological caveats. Table 3 and the per-task breakdown in Table 4 clearly show that Claude's accuracy is bimodal at the task level: tasks where it achieves either 100% or 0% accuracy dominate (8 of 10 tasks fall into these extremes, with only 2 "mixed" tasks). The 71% figure (15 of 21 failed runs belonging to tasks where all 5 runs fail identically) is a direct computation from these tables. The qualitative evidence from the case study (Section 5.1, astropy-13236) provides a concrete illustration of the mechanism.

However, the failure mode classification is unblinded and performed by a single annotator (the first author). The paper acknowledges this limitation and releases trajectories for independent verification, but until such verification occurs, the 71% figure should be treated as an initial estimate subject to revision. The distinction between "interpretation error" and "execution error" is semantic and may have borderline cases—imagine an agent that correctly identifies the buggy code location and the general type of fix needed, but makes a subtle wrong assumption about the desired behavioral change (e.g., raising a specific exception type vs. handling the edge case silently). Is this an interpretation error (wrong semantic goal) or an execution error (correct goal, wrong implementation detail)? The paper's example of "adding a deprecation warning instead of removing code" is a clear-cut case, but not all failures will be so cleanly categorized. The reliability of the 71% figure depends on how these borderline cases were resolved.

Additionally, the paper's claim that "more testing cannot help if the fundamental understanding is wrong" (Section 4.4) is a logical inference from the data, not something directly tested. An experiment that explicitly varied the amount of testing or verification budget for the "consistent wrong" tasks—to confirm that additional execution effort does not convert them to successes—would strengthen this claim considerably. As it stands, the paper observes that Claude did not succeed on these tasks across 5 runs each, but does not test whether, say, 50 runs or a modified scaffold that forces hypothesis reconsideration would change the outcome.

Claim 3: Divergence timing alone does not determine consistency, as Claude and GPT-5 diverge at nearly identical steps (3.2 vs. 3.4) yet Claude achieves 2.1× better consistency. This claim is statistically rigorous and clearly supported. The non-significant t-test (p = 0.808) for the Claude vs. GPT-5 divergence comparison, paired with the significant CV difference (p = 0.029), cleanly demonstrates that divergence timing and overall consistency are dissociable. The finding is further strengthened by the contrast with Llama, which diverges significantly earlier (mean 1.4 steps, p < 0.001 vs. both other models) and has significantly worse consistency—showing that divergence timing is not irrelevant, just insufficient.

However, a limitation is that divergence is measured at the phase-category level, not the specific-action level. Claude and GPT-5 both have all 5 runs executing the same action category through step ~3, but the specific commands within that category may already differ. It is possible that measuring divergence at the specific-command level (e.g., "at what step do the exact command strings first differ?") would reveal differences between Claude and GPT-5 that the coarser phase-level analysis misses. The paper's choice of phase-level divergence is defensible—exact-command divergence would likely happen at step 1 or 2 for all models due to the expanded action space of the bash interface—but it means the claim is specifically about strategic divergence timing, not execution-level divergence.

Claim 4: GPT-5 reveals a speed-accuracy-consistency tradeoff: 4.7× faster, 1.8× lower accuracy, 2.1× worse consistency than Claude. The numerical ratios are directly computed from Table 2 and are factually correct for these two models on these tasks. However, calling this a "tradeoff" implies a causal relationship—that GPT-5's speed is the reason for its lower accuracy and consistency, or that there is a fundamental tension between these dimensions. The paper's evidence for this is primarily the phase decomposition in Table 5: GPT-5 spends less time on pre-edit diagnosis (44.4% vs. Claude's 59.0%) and more on verification (32.3% vs. 19.3%), suggesting that speed comes from earlier commitment to editing. The case study of astropy-14309 (Section 5.2) provides a concrete instance where GPT-5's fast approach matched Claude's thorough approach, showing that speed is not always detrimental—for simple tasks, the tradeoff may not exist or may reverse.

But the paper only studies one temperature setting (0.5) and does not experimentally manipulate speed (e.g., by prompting models to "be quick" or "be thorough," or by varying the step budget). The observed differences in step counts, accuracy, and consistency are correlations between model properties, not experimentally demonstrated causal tradeoffs. It could be, for example, that Claude's slower pace and GPT-5's faster pace are both near-optimal for their respective model architectures, and that forcing Claude to go faster would harm its accuracy while forcing GPT-5 to go slower would not help—which would mean the "tradeoff" is model-specific rather than universal. Without experimental manipulation of speed, the trilemma framing remains an empirical observation about these three models rather than an established constraint.

What experiments would strengthen the paper:

Temperature sweep. Running a subset of tasks at temperatures 0.0, 0.25, 0.5, 0.75, and 1.0 for each model would reveal how the consistency-accuracy relationship changes with stochasticity. The paper's choice of 0.5 is arbitrary, and the key findings (especially the amplification insight) might be temperature-dependent. At temperature 0, all runs would be deterministic, producing identical trajectories and making consistency trivially perfect—but would accuracy change? At higher temperatures, would the "consistent wrong" pattern break down as variance introduces occasional correct interpretations? This experiment would test the robustness of the central claims to the stochasticity parameter.
Model scaling within a family. Testing 3–4 sizes of the same model family (e.g., Llama-3.1-8B, 70B, 405B if available) would isolate the effect of model scale on consistency while holding architecture and training data constant. The current three-model comparison confounds model family (Anthropic vs. OpenAI vs. Meta), architecture, training data, and scale. A within-family scaling analysis would determine whether consistency is primarily a function of capability (larger models are more consistent regardless of family) or whether architectural and training differences dominate.
Multiple codebases. All 10 tasks are from astropy. Reproducing the study on tasks from django, sympy, flask, or other repositories would test whether the consistency patterns—particularly the "consistent wrong interpretation" rate—are codebase-dependent. The astropy codebase may have characteristics (well-documented, mature, scientifically-oriented) that make interpretation errors more or less likely than other codebases.
Forced hypothesis reconsideration. An intervention experiment where, after a run fails, the agent is explicitly prompted to reconsider its interpretation ("Your fix did not pass the tests. Is it possible that your understanding of the bug is incorrect? Consider alternative interpretations.") would directly test the paper's claim that interpretation errors are resistant to execution-focused improvement. If such prompting converts some "consistent wrong" tasks to successes, it would both support the interpretation-bottleneck diagnosis (by showing that interpretation-focused interventions help) and provide a practical mitigation strategy.
More runs per task. With only 5 runs per task, the paper cannot reliably distinguish between "truly consistent wrong" (would fail 100/100 times) and "mostly wrong" (fails 80% of the time, but some runs succeed through stochastic exploration). Running 20–50 trials on the 3 "consistent wrong" tasks would provide a more precise estimate of the true failure probability and determine whether the amplification is truly complete or just very strong. If even 1 out of 50 runs succeeded on these tasks, it would complicate the narrative of "consistency amplifies to 0% or 100%."

Conditional nature of the claims. The paper's central findings are most robust for the specific configuration studied: frontier and near-frontier models (Claude 4.5 Sonnet, GPT-5) plus one smaller open model (Llama-3.1-70B), running at temperature 0.5 on SWE-bench Verified astropy tasks using the mini-SWE-agent scaffold. The following boundary conditions should be noted:

The finding that "consistency amplifies outcomes" is demonstrated for Claude (100% accuracy on correctly-interpreted tasks, 0% on incorrectly-interpreted tasks) but is only partially visible for GPT-5, whose per-task accuracy ranges more continuously across values (Table 4 shows patterns like 2/5, 3/5, 5/5, 0/5). GPT-5's consistency may be insufficient to drive the bimodal outcome distribution that makes amplification so stark in Claude. The amplification mechanism may thus be most relevant for models that have crossed a consistency threshold—it may not apply to less consistent models where execution noise dominates.
The 71% "consistent wrong interpretation" figure is specific to Claude on these 10 astropy tasks. It should not be interpreted as a general property of frontier models or of SWE-bench more broadly. A different set of 10 tasks, or tasks from a different repository, could yield a very different ratio of interpretation to execution failures.
The speed-accuracy-consistency trilemma is demonstrated with n = 3 models and may not generalize. A model that is simultaneously faster than Claude, more accurate than Claude, and more consistent than Claude is not ruled out by any theoretical argument in the paper—it simply wasn't among the three models tested.
The divergence timing analysis (Table 7) depends on the phase-level measurement granularity. If measured at the specific-command level, the Claude vs. GPT-5 comparison might look different. The paper's claim that "divergence timing alone does not determine consistency" should be understood as applying to strategic (phase-level) divergence, not execution-level divergence.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Gains

The assumption or constraint. The entire compute-optimal framework rests on the ability to estimate prompt difficulty before deciding how to allocate the inference budget. The paper's method for doing so—generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted)—is extraordinarily expensive. At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. The 4× figure should therefore be understood as an upper bound on achievable efficiency rather than a realized deployment gain. For batch inference with many questions, the amortized cost per question might be acceptable, but for single-question inference (the typical production setting), the overhead is prohibitive.

What evidence exists in the paper. Section 3.2 describes the difficulty estimation procedure explicitly but provides no cost analysis or amortization calculation. The paper's Figure 4 and Figure 8 both show compute-optimal scaling curves that use predicted difficulty bins, but these curves are computed without including the cost of generating those predictions. The gap between oracle and predicted bins is small but present (Figure 8: roughly 41% vs. 44% at 256 generations for revisions), suggesting that even with predicted difficulty, there is a performance cost that compounds with estimation cost.

Mitigation status. The paper partially acknowledges the issue (Section 3.2, Section 8) and flags it as "a key avenue for future work," suggesting that future work could explore "pretraining or finetuning models to directly predict difficulty of a question." No such model is developed or evaluated in the paper. The difficulty estimation cost is explicitly excluded from all budget calculations. This is the single most consequential gap between the paper's reported gains and what a practitioner could achieve in deployment.

Single Benchmark, Single Model Family

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. The paper studies only one model family (PaLM 2), one benchmark (MATH), and one task domain (competition-level mathematical reasoning).

The consequence. Several aspects of the findings could be model-specific or domain-specific. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution—a model with different calibration properties might exhibit different difficulty-dependent scaling curves. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The MATH benchmark consists exclusively of problems with symbolic reasoning and exact-match evaluation; it is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) generalize to code generation, logical reasoning, scientific QA, or tasks requiring factual knowledge rather than inference. Perhaps most critically, the PRM training pipeline's reliance on Monte Carlo rollouts requires a model that can generate diverse solutions at adequate pass@1 for the training questions—a capability that weaker models may lack, limiting the applicability of the entire verifier-based search approach.

What evidence exists in the paper. The paper provides no evidence of replication across benchmarks or model families. Section 4 explicitly limits the study to MATH with PaLM 2-S*. Section 8 acknowledges this implicitly when suggesting future work on "extension to other domains and modalities." The paper does not report results with alternative base models, alternative PRM training data sources (beyond PRM800k, which was tried and found ineffective due to distribution shift), or alternative benchmarks.

Mitigation status. Not addressed. The authors are transparent about the single-benchmark, single-model scope (Section 4), but make no attempt to mitigate it experimentally. This is the most significant generalization gap in the paper. A practitioner considering adopting compute-optimal test-time scaling for a non-MATH domain or a non-PaLM model has no empirical evidence that the approach transfers.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The authors acknowledge that this departs from compute-optimal pretraining (Hoffmann et al., 2022), where both data and parameters are scaled equally:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the 14× larger model uses only greedy decoding—no majority voting, no best-of-N, no search—while the smaller model is augmented with up to 512 generations of sophisticated test-time computation.

The consequence. The pretraining baseline is systematically weaker than it could be. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, making the pretraining baseline stronger. Giving the larger model even a modest test-time compute budget (say, best-of-8 or best-of-16) would create a much more realistic comparison—the question should be "small model + heavy test-time compute vs. large model + light test-time compute," not "small model + heavy test-time compute vs. large model + zero test-time compute." The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at R << 1 in Figure 1) may shrink or reverse against a properly compute-optimal larger model with a fair test-time compute allocation.

What evidence exists in the paper. Section 7 describes the FLOPs accounting framework and the parameter-only scaling approach. The Figures 1 and 9 bar charts and scaling curves show the comparison against a greedy-decoding 14× larger model. No ablation varies the larger model's test-time compute budget or compares against a Chinchilla-optimal pretraining allocation. The paper does not report what the larger model's accuracy would be with best-of-8 or best-of-16, which would be a natural and inexpensive baseline.

Mitigation status. The authors explicitly acknowledge this limitation in Section 7 and frame it as a design choice rather than an oversight, noting that "this setting is representative of a canonical approach" (the LLaMA paradigm). However, this framing doesn't change the fact that the comparison is asymmetric: the smaller model gets the full benefit of the paper's methods while the larger model gets none. This limits the strength of the claim that test-time compute can substitute for pretraining—the substitution is demonstrated against a specific, potentially suboptimal pretraining approach.

Verifier Over-Optimization Is a Hard Ceiling, Not a Solved Problem

The assumption or constraint. The paper's compute-optimal policy selects the best strategy per difficulty bin, but this selection operates within the constraints of the available verifier quality. When the verifier signal becomes unreliable under aggressive optimization, the policy can only route away from aggressive search—it cannot improve the verifier itself.

The consequence. On easy problems (difficulty bins 1–2), beam search degrades performance with increasing budget (Figure 3, right)—a hallmark of verifier over-optimization. The compute-optimal policy mitigates this by routing easy problems to best-of-N rather than beam search, but this means the policy is avoiding the verifier's weakness rather than solving it. On medium problems where beam search is deployed, over-optimization still limits the scaling ceiling: the beam search curves in Figure 3 flatten and may decline well before the budget is exhausted. This means the compute-optimal approach is fundamentally bounded by verifier quality, and the specific difficulty thresholds and strategy selections are tied to the particular PRM trained via Monte Carlo rollouts with specific hyperparameters. A different PRM (better calibrated, more robust, or trained on a different model's outputs) would shift the optimal policy, potentially significantly. The paper provides no guidance on how to predict or estimate verifier quality limits in advance.

What evidence exists in the paper. Figure 3 (right) shows beam search performance declining on easy problems at high budgets—the clearest quantitative evidence. Lookahead search, the strongest optimizer, paradoxically performs worst overall (Figure 3, left), further evidence that optimization strength and verifier reliability are inversely related. Appendix M provides qualitative examples of degenerate outputs (repetitive low-information steps, overly short 1–2 step solutions) that score highly under the PRM. Figure 4 and Figure 8 show compute-optimal policies saturating or plateauing at high budgets—evidence that verifier quality, not strategy selection, is the binding constraint at scale.

Mitigation status. The paper identifies verifier over-optimization as a central phenomenon (Section 5.3, Section 8) and the compute-optimal policy partially mitigates it through difficulty-aware routing. However, the underlying problem is not solved: the policy can only choose among available strategies, all of which are constrained by the same verifier. The paper suggests future work on "robust verifiers resistant to over-optimization" (Section 8), including adversarial training, ensemble verification, and KL-penalty approaches, but none of these are implemented. A practitioner using this framework would need to train their own PRM and would have no tools for predicting or testing when over-optimization will occur.

Hard Problems Remain Essentially Unsolved Regardless of Budget

The assumption or constraint. The paper's framework assumes that test-time compute can improve performance if the base model has some non-trivial probability of generating a correct solution. On difficulty bin 5 (the hardest quintile), this assumption fails: the base model's pass@1 is near zero.

The consequence. Across all methods—search, revisions, and their compute-optimal combinations—the hardest questions show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%. Test-time compute can amplify existing capability but cannot create it from nothing. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help—there are no correct solutions in the proposal distribution to find or refine. This means the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, pretraining remains the only viable path.

What evidence exists in the paper. The difficulty bin analysis consistently shows bin 5 as a flat line near zero across all figures (Figure 3 right, Figure 7 right, Figure 9). Section 7 explicitly states this in the takeaway box: "hard questions (bins 4–5): Pretraining is almost always more effective. Test-time compute provides minimal gains on problems that are fundamentally outside the base model's capability range." The FLOPs-matched comparison quantifies this: at R >> 1 with PRM search, hard questions show a −52.9% relative disadvantage from using test-time compute instead of the larger model (Figure 1 bar chart).

Mitigation status. The authors are transparent about this limitation (Section 7, Section 8) and do not claim to solve it. The finding is presented as a boundary condition: test-time compute is powerful when problems are within the base model's reach but cannot compensate for fundamental capability gaps. This is a clean, well-supported negative result that properly scopes the applicability of the approach. The paper does not suggest mitigations for the hardest problems beyond "pretraining remains the only viable path."

Revisions and Search Are Studied Independently, Not Combined

The assumption or constraint. The paper studies two complementary axes of test-time compute—PRM search and iterative revisions—but never combines them. Section 8 explicitly acknowledges this:

"we did not experiment with PRM tree-search techniques in combination with revisions"

Each mechanism is analyzed separately: Section 5 studies search against a PRM using the base LLM as the proposal distribution; Section 6 studies revisions using a fine-tuned model with its own ORM verifier. The studies use different proposal distributions (base model vs. revision model), different verifiers (PRM trained on base model outputs vs. ORM trained on revision model outputs), and different selection mechanisms.

The consequence. The paper's results represent a lower bound on what a fully integrated system could achieve. The two mechanisms have complementary, difficulty-dependent strengths: revisions improve the proposal distribution (generating better candidates through iterative refinement), while PRM search improves candidate selection (finding the best among generated candidates through step-level scoring). These strengths are demonstrated to be complementary in the difficulty-bin analysis: revisions dominate on easy problems, search dominates on medium problems. A combined system that uses the revision model as the proposal distribution within beam search—or uses the PRM to guide which revision chains to pursue—could potentially outperform either method alone, particularly on medium-difficulty problems where both mechanisms show partial effectiveness. The current compute-optimal policy can only choose between search and revisions, not combine them.

What evidence exists in the paper. Sections 5 and 6 present search and revisions as independent studies with separate experimental setups, separate verifiers, and separate compute-optimal policies. Figure 4 and Figure 8 show separate compute-optimal scaling curves that are never combined. Appendix J (Figure 15a) shows that the base-LM PRM underperforms when scoring revision model outputs, confirming distribution shift as a barrier to naïve combination. Section 8 flags this as an explicit avenue for future work but provides no combined results.

Mitigation status. The paper acknowledges this gap directly (Section 8) and frames it as future work: "A natural next step is to use the revision model as the proposal distribution within beam search... Alternatively, the PRM could guide which revisions to pursue." No combined experiments are reported, and no analysis is provided of how distribution shift between base model and revision model outputs would need to be addressed (the PRM trained on base model outputs does not transfer well). This is a significant architectural limitation: the paper demonstrates that both mechanisms work independently, but provides no evidence about whether they would compose synergistically, interfere, or require additional engineering to combine.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around LLM agent evaluation from a world where reliability is assumed to follow from accuracy to one where consistency is recognized as an independent, measurable dimension of agent quality that interacts with accuracy in non-obvious ways. Before this work, the standard mental model was straightforward: a model that achieves 58% accuracy on SWE-bench is, roughly speaking, a 58%-reliable agent—it solves about 6 out of 10 tasks, and you can count on it accordingly. This paper demonstrates that this mental model is incomplete and potentially misleading. Claude's 58% accuracy could, in principle, mean it solves every task 58% of the time (highly inconsistent, unreliable) or it solves 58% of tasks every time (perfectly consistent, highly reliable on a subset of tasks)—and the paper shows empirically that the reality is closer to the latter: Claude achieves either 100% or 0% accuracy on 8 of 10 tasks (Table 3). This transforms the interpretation of a single accuracy number from a summary of expected per-task performance into a summary of how many tasks the agent can handle at all, with consistency guaranteeing that the ones it can handle, it handles reliably.

This is not a paradigm shift on the scale of, say, the introduction of attention mechanisms or reinforcement learning from human feedback. But it is a methodological reframing of how the field should evaluate agents—one that adds a new required measurement (multi-run consistency) to the standard evaluation toolkit. The paper's most durable contribution may be less the specific numbers (15.2% CV for Claude, 71% consistent-wrong failures) and more the demonstration that single-run evaluation is insufficient for characterizing agent reliability. The finding that 100% of runs produce unique action sequences for all three models (Section 4.1) means that every single-run evaluation is sampling from a vast behavioral distribution and treating that single sample as representative. The paper provides the first systematic evidence that this practice can obscure qualitatively different reliability profiles—an agent that always fails on certain tasks versus one that sometimes succeeds—that have radically different implications for deployment.

The paper also resolves, or at least clarifies, a latent tension in the self-consistency and agent-improvement literatures. Self-consistency methods (Wang et al., 2023) operate on the premise that sampling multiple reasoning paths and taking a majority vote improves accuracy because the model's errors are uncorrelated across samples. Reflexion (Shinn et al., 2023) assumes that agents can detect their own failures and improve through iterative reflection. Both approaches implicitly assume that failures are stochastic—that a model that fails once might succeed on a second try. This paper provides evidence that, for complex agent trajectories on tasks within a frontier model's capability range, the dominant failure mode is not stochastic execution noise but systematic misinterpretation: Claude fails on the same 3 tasks all 5 times, making the identical mistake each time (Section 4.4). If this pattern generalizes, it means that self-consistency and reflection-based methods may be fundamentally limited in their ability to convert failures to successes—they can mitigate execution noise but cannot correct interpretation errors that the model cannot recognize as errors. This redirects research attention from "how do we help agents recover from mistakes?" to "how do we help agents avoid forming wrong interpretations in the first place?"—a qualitatively different question.

The specific research directions that become more attractive after this paper include: multi-hypothesis agent architectures that maintain competing interpretations rather than committing early (since early commitment to wrong interpretations is the dominant failure mode); verifiers or scaffold mechanisms that can detect not just whether a patch works but whether it addresses the right semantic goal; and adaptive agents that modulate their thoroughness based on task complexity (motivated by the speed-accuracy-consistency tradeoff and the case study of astropy-14309 where GPT-5's fast approach matched Claude's for a simple task, Section 5.2). Directions that become less attractive include: methods that focus purely on execution quality without addressing interpretation (better tools, longer trajectories, more thorough testing—these would not have helped Claude on its 3 consistent-wrong tasks); and majority-voting approaches that assume independent errors across runs (the paper's sequence-uniqueness finding and sequential dependency in agent trajectories call this assumption into question).

A secondary contribution that may have lasting methodological impact is the phase decomposition approach for analyzing agent trajectories. By classifying each action as EXPLORE, UNDERSTAND, EDIT, VERIFY, or OTHER, the paper provides a reproducible, low-dimensional representation of agent behavior that enables localization of variance to specific activity types. The finding that Llama's inconsistency is concentrated in EXPLORE (CV: 123%, Section 4.3) while Claude's consistency is pervasive across all phases suggests that phase-level analysis can serve as a diagnostic tool for identifying where an agent's reliability breaks down—is it exploration inefficiency, shallow understanding, editing errors, or verification failures? This decomposition is simple enough to adopt in other agent evaluation studies and provides more actionable diagnostic information than aggregate step-count CVs alone.

Follow-Up Research This Work Enables

Multi-hypothesis agent architectures. The paper's central diagnosis—that 71% of Claude's failures stem from incorrect initial interpretation that is never reconsidered—directly motivates agents that maintain multiple competing hypotheses about the bug rather than committing to a single interpretation early. A concrete experiment: modify the agent scaffold to require the model to generate three distinct hypotheses about the bug's root cause (with explicit rationales) before beginning any editing, then explore each hypothesis in parallel, using test failures to eliminate wrong interpretations rather than iteratively refining a single one. The paper's data on which tasks produce "consistent wrong" failures (Table 4: astropy-13236, 13398, 13977) provides a ready-made test suite for such an intervention. A strong follow-up would measure whether multi-hypothesis agents convert any of the 0/5 consistent-wrong tasks to >0/5 successes, and whether the overhead of maintaining multiple hypotheses (increased step counts, more exploration) is justified by improved accuracy on interpretation-bottlenecked tasks.

Temperature-dependent consistency characterization. The paper uses a single temperature setting (0.5) for all experiments, leaving open the question of how the consistency-accuracy relationship changes with stochasticity. A natural follow-up would run the same 10 SWE-bench tasks at temperatures spanning 0.0 (deterministic, guaranteed perfect consistency but potentially different accuracy), 0.25, 0.5, 0.75, and 1.0 for each model, measuring CV, accuracy, and the fraction of consistent-wrong vs. inconsistent-wrong failures at each temperature. Specific hypotheses to test: (1) At temperature 0, does accuracy increase (because noise is eliminated) or decrease (because the model cannot escape incorrect deterministic paths)? (2) At high temperatures, does the "consistent wrong" pattern break down as increased variance occasionally produces correct interpretations on tasks where the low-temperature model was consistently wrong? (3) Is there an optimal temperature that balances the amplification benefit (consistency on correctly-interpreted tasks) against the diversification benefit (occasional correct interpretations on hard tasks)? The paper's current data cannot answer these questions, and they are practically important for anyone deploying agents with configurable temperature.

Cross-codebase replication on SWE-bench Verified. All 10 tasks in this study are from the astropy repository—a well-documented, mature scientific Python codebase with a particular structure and domain (astronomy). The generalizability of the findings to other codebases is unknown. A direct replication study would select 10 tasks each from 3–4 other repositories in SWE-bench Verified (e.g., django for web frameworks, sympy for symbolic mathematics, flask for microframeworks, requests for HTTP libraries) and run the same 5-trial protocol with the same three models. Key questions: (1) Is the "consistent wrong" fraction (71% for Claude on astropy) similar across codebases, or does it vary with codebase characteristics like documentation quality, codebase size, or domain complexity? (2) Does the speed-accuracy-consistency tradeoff hold across codebases, or are there codebases where GPT-5's fast approach matches or exceeds Claude's thorough approach across the board? (3) Are there codebases where Llama's variance-driven occasional successes (as on astropy-13236) become a significant factor, or is astropy-13236 an outlier? This replication would establish the boundary conditions for the paper's findings and determine whether codebase-specific factors are first-order or second-order in determining agent consistency.

Within-model-family scaling analysis. The paper compares three models from three different families (Anthropic, OpenAI, Meta), confounding model scale, architecture, training data, and post-training with the consistency measurements. A clean scaling analysis would test 3–4 sizes of the same model family (e.g., Llama-3.1-8B, 70B, 405B if available) on the same 10-task protocol, measuring how consistency scales with model size within a family. Does CV decrease smoothly with parameter count? Does the "consistent wrong" fraction change, or do larger models simply succeed on more tasks while remaining equally consistent-wrong on the ones they fail? Is there a threshold model size below which behavior becomes too noisy to exhibit the amplification pattern (i.e., models are too inconsistent for the "consistent wrong" mode to exist)? The current study cannot distinguish between "consistency is a property of model capability" (larger models are more consistent regardless of family) and "consistency is a property of model family" (Anthropic's training produces more consistent models at a given capability level). A within-family scaling analysis would isolate the capability-consistency relationship.

Forced reconsideration interventions. The paper claims that interpretation errors are the bottleneck and that "more testing cannot help if the fundamental understanding is wrong" (Section 4.4). This claim is inferential—the paper observes that Claude fails identically on 3 tasks across 5 runs but does not experimentally test whether interventions can change this. A direct test would run the same 3 consistent-wrong tasks with a modified scaffold that, after the agent submits a failing patch, explicitly prompts: "Your fix did not pass the tests. Is it possible that your fundamental understanding of the bug is incorrect? Consider at least two alternative interpretations of what the bug might be, then choose the best one and implement a new fix." If such prompting converts any of the 0/5 tasks to non-zero successes, it would simultaneously confirm the interpretation-bottleneck diagnosis (by showing that interpretation-focused interventions help where execution-focused ones don't) and provide a practical mitigation strategy. If it doesn't help, it would suggest that the interpretation errors are too deep to be corrected by simple prompting and require more fundamental architectural changes. The paper's released trajectories provide the baseline data against which such an intervention study would be compared.

Verifier-guided hypothesis evaluation for software engineering agents. The paper studies behavioral consistency but does not introduce any mechanism for the agent itself to evaluate the quality of its own interpretation. A natural extension—motivated by the finding that interpretation quality is the bottleneck—is to develop a lightweight "interpretation verifier" that, given the task description and the agent's current hypothesis about the bug, predicts whether that hypothesis is semantically aligned with the ground-truth fix. Training such a verifier would require labeled data (bug descriptions paired with ✅/❌ interpretations), which could potentially be generated at scale by having frontier models produce interpretations and then testing whether they lead to correct fixes using the SWE-bench evaluation harness. If such a verifier could be made reliable, it could be used to: (1) warn the agent early that its current interpretation appears misaligned, triggering reconsideration before editing begins; (2) filter candidate hypotheses in a multi-hypothesis architecture; (3) provide a confidence score that predicts whether the agent will succeed on a given task, enabling deployment-time routing decisions (send high-confidence tasks to the agent, escalate low-confidence tasks to human review). This direction builds directly on the paper's finding that interpretation, not execution, is the bottleneck.

Practical Applications and Downstream Use Cases

Multi-run benchmark reporting as standard evaluation practice. The paper's most immediately actionable implication is that agent evaluation benchmarks should report multi-run consistency metrics alongside single-run accuracy. The finding that 100% of runs produce unique action sequences means that any single-run accuracy score is a noisy estimate of an underlying distribution whose variance is unknown unless explicitly measured. A concrete recommendation: SWE-bench leaderboards should require 5-run evaluations per task and report mean accuracy, 95% confidence intervals, and a per-model CV score. This would enable consumers of benchmark results to distinguish between a model that achieves 32% accuracy because it solves 32% of tasks reliably every time (high consistency, predictable deployment) versus one that solves 100% of tasks 32% of the time (low consistency, unpredictable deployment). The cost of 5-run evaluation—5× the inference compute—is modest relative to the information gained, especially given that the paper's total experiment cost was on the order of $100–$ 200 in API fees for 150 trajectories. For model developers, reporting consistency would provide a competitive signal about reliability that is currently absent from leaderboards and could drive investment in interpretation quality, since consistency amplifies whatever accuracy a model achieves on its interpretable subset of tasks.

Production deployment with consistency-informed routing. For organizations deploying software engineering agents in production—for example, automated bug-fixing pipelines that triage GitHub issues and propose patches—the paper's findings support a deployment architecture that routes tasks based on estimated difficulty or interpretation confidence. The pattern in Table 3 is informative: Claude's tasks fall into three categories—those it always solves (5 tasks, 100% accuracy), those it never solves (3 tasks, 0% accuracy), and those where it sometimes succeeds (2 tasks, 40% accuracy). If a deployment system could identify which category a new task falls into before committing significant compute, it could route trivially: always-solved tasks go to the agent; never-solved tasks go directly to human review without wasting agent compute; sometimes-solved tasks might be run with multiple attempts or escalated based on cost constraints. The paper does not provide a difficulty predictor, but the consistency measurements establish the business case for building one: Claude's 100% success rate on 5 tasks and 0% on 3 tasks means that a perfect classifier of task solvability could eliminate 30% of agent runs (the 3 never-solved tasks) while maintaining the same overall accuracy. With Claude costing $1.50/run (Table 2), this represents direct cost savings in a production pipeline, and the consistency measurement is what makes such a classifier's value quantifiable.

Cost-aware model selection based on the speed-accuracy-consistency tradeoff. The paper's characterization of the speed-accuracy-consistency tradeoff (GPT-5: 4.7× faster, 1.8× lower accuracy, 2.1× worse consistency than Claude; Table 2 and Section 4.1) provides a quantitative framework for model selection in different deployment contexts. A startup building a rapid-prototyping tool for developers might prefer GPT-5: at $0.53/run and 9.9 steps average, it provides fast, cheap feedback even if it's less reliable—developers will review the output anyway and can catch the 68% of tasks where GPT-5 fails. A financial institution building an automated patch-generation pipeline for critical infrastructure might prefer Claude: at$ 1.50/run and 46.1 steps, it's slower and more expensive, but its 58% accuracy and 15.2% CV (meaning its successes are reliable and its failures are predictable) provide the reliability guarantees needed for high-stakes deployment. The specific numbers are model- and task-specific, but the framework—making the speed-accuracy-consistency tradeoff explicit and measurable—transfers to any model selection decision where reliability matters.

When to Prefer This Method

This paper is a measurement study that characterizes consistency and its relationship to accuracy; it does not propose a specific method (no new agent architecture, training procedure, or inference algorithm) that competes against named alternatives. The paper's contribution is a diagnostic framework and evaluation methodology, not a technique for which there are competing approaches. Therefore, a "when to prefer this method over alternatives" decision matrix is not applicable. The relevant decision is not "when to use this method" but rather "when to adopt multi-run consistency evaluation and the associated diagnostic vocabulary (amplification insight, interpretation bottleneck, consistent-wrong pattern) in your own agent evaluation and deployment practices." The answer implicit in the paper is: whenever the reliability characterization of your agent matters for downstream trust, debugging, or cost optimization—which is to say, in any deployment where the agent operates without human-in-the-loop verification on every output, and where the distinction between "fails occasionally" and "fails always on certain tasks" has operational consequences.